Dig101 — Go slice header, slicing tricks, and append growth

Contents

  1. 1. 0x01 What is a slice?
  2. 2. 0x02 Can you compare slices?
  3. 3. 0x03 Slicing expressions
  4. 4. 0x04 What append does
  5. 5. 0x05 Compiler lowers for append
    1. 5.1. Growth policy
    2. 5.2. Why uint(n) > uint(cap(s))?
    3. 5.3. memclrNoHeapPointers vs typedmemclr?

Dig101: dig more, simplify more, know more.

Slices are everywhere in Go. Unlike fixed-size arrays, a slice is a small header over a variable-length backing array.

0x01 What is a slice?

Runtime definition:

1
2
3
4
5
6
// src/runtime/slice.go
type slice struct {
array unsafe.Pointer
len int
cap int
}

array points at the backing store; len is length; cap is capacity. Declare a, inspect the header, and print %p for &a, sh, a, and &a[0]:

1
2
3
4
5
6
7
8
a := make([]int, 1, 3)
// reflect.SliceHeader mirrors the runtime layout (unsafe for production code)
sh := (*reflect.SliceHeader)(unsafe.Pointer(&a))
fmt.Printf("slice header: %#v\naddress of a: %p &a[0]: %p | &a: %p sh:%p ",
sh, a, &a[0],&a, sh)

//slice header: &reflect.SliceHeader{Data:0xc000018260, Len:1, Cap:3}
//address of a: 0xc000018260 &a[0]: 0xc000018260 | &a: 0xc00000c080 sh:0xc00000c080

a and &a[0] match—the slice “value” for printing is the first element’s address. &a, sh, and the header’s storage address match—that is the slice variable on the stack, not the array blob. Keeping those two addresses straight helps with later puzzles.

Side note: what does fmt.Printf("%p", a) do? Chain: fmtPointer → Value.Pointer. For slices, Pointer returns the first element (or 0 if nil)—exactly the data pointer above.

Go passes slices by value (copying the header). For large arrays, passing a slice is far cheaper than copying the array. I covered for-range over big arrays in another Dig101 note (array copies vs slice headers).

Summary: a slice is a header (pointer, len, cap) plus slicing/append ergonomics.

0x02 Can you compare slices?

Only to nil.

1
2
3
4
s := make([]int, 5)
a := s
println(s == a)
//invalid operation: s == a (slice can only be compared to nil)

Comparing headers is rarely what you want; comparing backing arrays would also need len/cap rules. The language punts.

0x03 Slicing expressions

Form: x[from:high:max]—start at from, length up to high, capacity up to max (each bound must fit the source). Omitted bounds default to x[0:len(x):cap(x)]. Works on arrays too:

1
2
3
4
5
s:=make([]int,5)
x:=s[:]

arr:=[5]int{}
y:=arr[:]

With copy/append, see the wiki Slice Tricks. Example—copy without sharing:

b = append(a[:0:0], a...)

Cut/delete with nil slots to drop pointers and avoid leaks:

1
2
3
4
5
6
7
8
9
10
11
12
13
//Cut
copy(a[i:], a[j:])
for k, n := len(a)-j+i, len(a); k < n; k++ {
a[k] = nil // or the zero value of T
}
a = a[:len(a)-j+i]

//Delete
if i < len(a)-1 {
copy(a[i:], a[i+1:])
}
a[len(a)-1] = nil // or the zero value of T
a = a[:len(a)-1]

Worth typing these out until they feel automatic.

0x04 What append does

High level:

  • Room in cap: write after len in the same array; return new header with larger len.
  • No room: allocate, copy the live prefix, append, return header pointing at the new array.
1
2
3
4
5
6
7
s := make([]int, 5)
x := append(s, 1)
fmt.Printf("x dataPtr: %p len: %d cap: %d\ns dataPtr: %p len: %d cap: %d",
x, len(x), cap(x),
s, len(s), cap(s))
// x dataPtr: 0xc000094000 len: 6 cap: 10
// s dataPtr: 0xc000092030 len: 5 cap: 5

x and s no longer share storage after growth.

0x05 Compiler lowers for append

The compiler classifies append and rewrites. Besides growslice, three shapes matter:

  • x = append(y, make([]T, y)...) — memclr fast path for cleared tail
  • x = append(l1, l2...) or append(slice, string) — memmove from l2
  • x = append(src, a, b, c) — known argument count → direct stores

From walk.go append (comments abbreviated):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
switch {
case isAppendOfMake(r):
// x = append(y, make([]T, y)...) will rewrite to

// s := l1
// n := len(s) + l2
// if uint(n) > uint(cap(s)) {
// s = growslice(T, s, n)
// }
// s = s[:n]
// lptr := &l1[0]
// sptr := &s[0]
// if lptr == sptr || !hasPointers(T) {
// // growslice did not clear the whole underlying array
// (or did not get called)
// hp := &s[len(l1)]
// hn := l2 * sizeof(T)
// memclr(hp, hn)
// }

// memclr for bulk zeroing
r = extendslice(r, init)
case r.IsDDD(): // DDD is ... syntax
// x = append(l1, l2...) will rewrite to

// s := l1
// n := len(s) + len(l2)
// if uint(n) > uint(cap(s)) {
// s = growslice(s, n)
// }
// s = s[:n]
// memmove(&s[len(l1)], &l2[0], len(l2)*sizeof(T))

// copy l2 in one memmove
r = appendslice(r, init) // also works for append(slice, string).
default:
// x = append(src, a, b, c) will rewrite to

// s := src
// const argc = len(args) - 1
// if cap(s) - len(s) < argc {
// s = growslice(s, len(s)+argc)
// }
// n := len(s)
// s = s[:n+argc]
// s[n] = a
// s[n+1] = b
// ...

// fixed argc → direct assignments
r = walkappend(r, init, n)
}

Growth policy

At least the requested length n, often double while small.

  • len < 1024: double cap
  • larger: ~1.25× until overflow
  • overflow: grow to exact need
  • then round through mallocgc size classes (between n and 2n)

Extra headroom cuts realloc churn. Size-class rounding is allocator policy (alignment included). Rainy Huang’s Go notes and memory alignment in Go go deeper.

Why uint(n) > uint(cap(s))?

1
2
3
4
// n is total len after append; cap(s) is int
if uint(n) > uint(cap(s))
s = growslice(T, s, n)
}

Unsigned comparison catches int overflow: a negative n becomes a large uint, forces growslice, which panics on invalid n instead of corrupting memory.

memclrNoHeapPointers vs typedmemclr?

Depends on whether memory is already type-safe and whether T contains pointers.

memclrNoHeapPointers when clearing non-pointer storage that may hold garbage bits before first use.

Otherwise typedmemclr; pointerful types run write barriers for the GC.

See zero initialization vs zeroing, memclrNoHeapPointers, typedmemclr.

Sample code: NewbMiao/Dig101-Go/types/slice

Questions? Leave a comment below or email: newbvirgil@gmail.com