I understand it's handy in reusing a buffer rather than allocating every time hen using io.Copy. However, having printed its value several times, I get all zeros and the size of my buffer never changes. I tried to set the size to 8 and 1.
On a related note, to what value should I set my buffer size?
io.CopyBuffer() documents that:
func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error)
CopyBuffer is identical to Copy except that it stages through the provided buffer (if one is required) rather than allocating a temporary one. If buf is nil, one is allocated; otherwise if it has zero length, CopyBuffer panics.
If either src implements WriterTo or dst implements ReaderFrom, buf will not be used to perform the copy.
So io.CopyBuffer() copies data (bytes) from src to dst. The source is an io.Reader and the destination is an io.Writer. These interfaces allow you to read and write slices of bytes ([]byte).
In the general case to do the copying, we need a slice to read into from the source, which we can write into the destination. So io.CopyBuffer() needs a buffer. The buf param allows you to pass a byte slice if you already have one, and if you do so, that buffer will be used to do the job, so no new slice have to be allocated (which would be thrown away at the end of the operation).
What size should it be? The bigger the better, but no bigger is needed than the data you want to copy. Obviously bigger requires more memory, so there's a trade-off. Typically a few KB is a good compromise.
Note that as documented, if the source implements io.WriterTo or the destination implements io.ReaderFrom, those interfaces allow to read /
write without having to pass a slice, so in that case the buffer you pass will not be used. Like in this example:
srcData := []byte{1, 2, 3, 4, 5, 6, 7}
src := bytes.NewBuffer(srcData)
dst := &bytes.Buffer{}
buf := make([]byte, 10)
io.CopyBuffer(dst, src, buf)
fmt.Println(srcData)
fmt.Println(dst.Bytes())
fmt.Println(buf)
Which outputs (try it on the Go Playground):
[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7]
[0 0 0 0 0 0 0 0 0 0]
Since we used bytes.Buffer as the source and destination (and since it implements both io.ReaderFrom and io.WriterTo), the buffer is not used.
Let's construct a source and destination that does not implement these interfaces, so we can test if / how our passed buffer is used.
For this, I will embed *bytes.Buffer in a struct, but specify a WriteTo and ReadFrom fields, so those methods will not get promoted from the embedded bytes.Buffer:
srcData := []byte{1, 2, 3, 4, 5, 6, 7}
src := struct {
WriteTo int // "disable" WriteTo method
*bytes.Buffer
}{0, bytes.NewBuffer(srcData)}
dst := struct {
ReadFrom int // "disable" ReadFrom method
*bytes.Buffer
}{0, &bytes.Buffer{}}
buf := make([]byte, 10)
io.CopyBuffer(dst, src, buf)
fmt.Println(srcData)
fmt.Println(dst.Bytes())
fmt.Println(buf)
This will output (try it on the Go Playground):
[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7 0 0 0]
As you can see, the data from the source was read into the buffer, which then was written to the destination.
Note that you may pass a buffer smaller than the data to be copied, in which case reading / writing will be done in several iterations. In such cases, the data in the buffer may hold only the last iteration, and may only hold partial data (if the copied size is not an integer multiplication of the buffer size). It also depends on how the Read() method is implemented on the source, as Read() is not required to read the full slice passed to it.
Also note that io.CopyBuffer() does not document that the data written to the passed buffer is retained, it may get cleared / zeroed. Although this clearing is not implemented for performance reasons, but you should not count on it holding valid data after io.CopyBuffer() returns.
Related
I need to populate a struct that has a member of type [8]uint8. This needs be populated with a byte array of type []byte initialized to length 8. The simplistic approach does not work:
Data: [8]uint8(RequestFrame(0x180, r)),
gives
cannot convert .. (type []byte) to type [8]uint8
Since both arrays are structurally identical it would be nice if this could be done with casting/assignment rather than copying?
Background
The problem with your "simplistic approach" is that a slice
(of any type) is a struct-typed value consisting of a pointer
and two integers; the pointer contains the address of the
underlying (backing) data array, and the integers contain
what len() and cap() builtins return for that slice.
In other words, a slice is sort of a view into an array.
Then, in Go, there is no concept of a type cast; there are only
type conversions, and these conversions may only happen between
types with the same underlying representation¹.
Since a slice and an array may not have the same underlying
representation (array is literally a contiguous block of memory
of the size just enough to contain all the array's elements),
your alleged type conversion may not be legal.
Possible solutions
There are two possible solutions.
The simplest is to just copy the data from the slice's
backing array into a newly-allocated array:
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dst [8]uint8
)
copy(dst[:], src[:8])
Note that there exists an inherent disparity between slice an
array types: an array type encodes both the type of its elements
and its length (that is, the length is a part of the type),
while a slice type only encodes the type of its elements
(and may be of any length at runtime).
This means that you might need to have a check before such
copying that makes sure the source slice has exactly 8
elements, that is, len(src) == len(dst).
This invariant may be enforced by some other code, but I think
I'd warn you up front about this: if src has less than 8
elements, the src[:8] expression will panic at runtime,
and if it contains more, then there's the question of whether
copying just the first 8 of them is exactly what's needed.
The second approach (admittedly messier) is to just re-use
the underlying array of the slice:
import "unsafe"
var (
src = []byte{1, 2, 3, 4, 5, 6, 7, 8}
dstPtr *[8]uint8
)
if len(src) != len(*dstPtr) {
panic("boom")
}
dstPtr = (*[8]uint8)(unsafe.Pointer(&src[0]))
Here, we've just taken the address of the first element
contained in the slice's underlying array and peformed
a "dirty" two-phase type-conversion to make the obtained
pointer to be of type *[8]uint8—that is, "an address of
an array of 8 uint8s".
Note two caveats:
The resulting pointer now points to
the same memory block the original slice does.
It means it's now possible to mutate that memory both through the
slice and the pointer we obtained.
As soon as you'll decide to assign the array's data
to a variable of type [8]uint8 (and passing it as an argument
to a function's parameter of that type), you will dereference
that pointer (like with *dstPtr), and at that moment
the array's data will be copied.
I'm specifically mentioning this as often people resort
to hacks like this one to pull the backing array out of
a slice precisely in an attempt to not copy the memory.
TL;DR
Copy the data (after supposedly verifying the
len(src) == len(dst) invariant holds).
Copying 8 bytes is fast (on a typical 64-bit CPU this will be
a single MOV instruction, or two at most), and the code will
be straightforward.
Only resort to hacks from the second solution when you really
need to optimize on some critical hot path.
In that case, comment the solution extensively and watch for
not accidentally dereferencing your pointer.
¹ There are notable exceptions to this rule:
A []byte is type-convertible to string, and vice-versa.
A string is type-convertible to []rune, and vice-versa.
An int is type-convertible to string (but since Go 1.15 go vet gives a warning about it, and this feature may probably be prohibited in the future).
You can copy the contents of your byte slice into your uint8 array very simply by using copy, like this:
package main
import (
"fmt"
)
func main() {
slice := []byte{1, 2, 3, 4, 5, 6, 7, 8}
array := [8]uint8{}
copy(array[:], slice)
fmt.Println(array)
}
Outputs
[1 2 3 4 5 6 7 8]
Try it out on the playground.
But may I ask why you are using an array? It's usually better to just use slices, unless you have a really good reason.
Starting from Go 1.17 you are able to use type conversion directly, from a slice to an array pointer:
a := make([]byte, 8)
b := (*[8]uint8)(a) // b is pointer to [8]uint8
The you can just dereference to obtain a non-pointer [8]uint8 type.
a := make([]byte, 8)
b := *(*[8]uint8)(a) // b is [8]uint8
Notes:
unlike copy, the conversion approach does not incur in extra allocations (not yours, nor any possibly done by copy), because it simply yields a pointer to the existing backing array. Though dereferencing the array pointer will make a copy.
the conversion panics if the length of the array is greater than the slice's
a := make([]byte, 5)
b := (*[10]byte)(a) // panics
the pointer points to the slice's underlying array, therefore the same values will be visible by indexing either:
a := []byte{0xa1, 0xa2}
b := (*[2]uint8)(a)
fmt.Printf("%x\n", a[0]) // a1
b[0] = 0xff
fmt.Printf("%x\n", a[0]) // ff
you can convert from byte to uint8, including type literals derived from them, because byte is an alias (identical to) of uint8.
Related: How do you convert a slice into an array?
In Golang, we can use the builtin make() function to create a slice with a given initial length and capacity.
Consider the following lines, the slice's length is set to 1, and its capacity 3:
func main() {
var slice = make([]int, 1, 3)
slice[0] = 1
slice = append(slice, 6, 0, 2, 4, 3, 1)
fmt.Println(slice)
}
I was surprised to see that this program prints:
[1 6 0 2 4 3 1]
This got me wondering- what is the point of initially defining a slice's capacity if append() can simply blow past it? Are there performance gains for setting a sufficiently large capacity?
A slice is really just a fancy way to manage an underlying array. It automatically tracks size, and re-allocates new space as needed.
As you append to a slice, the runtime doubles its capacity every time it exceeds its current capacity. It has to copy all of the elements to do that. If you know how big it will be before you start, you can avoid a few copy operations and memory allocations by grabbing it all up front.
When you make a slice providing capacity, you set the initial capacity, not any kind of limit.
See this blog post on slices for some interesting internal details of slices.
A slice is a wonderful abstraction of a simple array. You get all sorts of nice features, but deep down at its core, lies an array. (I explain the following in reverse order for a reason). Therefore, if/when you specify a capacity of 3, deep down, an array of length 3 is allocated in memory, which you can append up to without having it need to reallocate memory. This attribute is optional in the make command, but note that a slice will always have a capacity whether or not you choose to specify one. If you specify a length (which always exists as well), the slice be indexable up to that length. The rest of the capacity is hidden away behind the scenes so it does not have to allocate an entirely new array when append is used.
Here is an example to better explain the mechanics.
s := make([]int, 1, 3)
The underlying array will be allocated with 3 of the zero value of int (which is 0):
[0,0,0]
However, the length is set to 1, so the slice itself will only print [0], and if you try to index the second or third value, it will panic, as the slice's mechanics do not allow it. If you s = append(s, 1) to it, you will find that it has actually been created to contain zero values up to the length, and you will end up with [0,1]. At this point, you can append once more before the entire underlying array is filled, and another append will force it to allocate a new one and copy all the values over with a doubled capacity. This is actually a rather expensive operation.
Therefore the short answer to your question is that preallocating the capacity can be used to vastly improve the efficiency of your code. Especially so if the slice is either going to end up very large, or contains complex structs (or both), as the zero value of a struct is effectively the zero values of every single one of its fields. This is not because it would avoid allocating those values, as it has to anyway, but because append would have to reallocate new arrays full of these zero values each time it would need to resize the underlying array.
Short playground example: https://play.golang.org/p/LGAYVlw-jr
As others have already said, using the cap parameter can avoid unnecessary allocations. To give a sense of the performance difference, imagine you have a []float64 of random values and want a new slice that filters out values that are not above, say, 0.5.
Naive approach - no len or cap param
func filter(input []float64) []float64 {
ret := make([]float64, 0)
for _, el := range input {
if el > .5 {
ret = append(ret, el)
}
}
return ret
}
Better approach - using cap param
func filterCap(input []float64) []float64 {
ret := make([]float64, 0, len(input))
for _, el := range input {
if el > .5 {
ret = append(ret, el)
}
}
return ret
}
Benchmarks (n=10)
filter 131 ns/op 56 B/op 3 allocs/op
filterCap 56 ns/op 80 B/op 1 allocs/op
Using cap made the program 2x+ faster and reduced the number of allocations from 3 to 1. Now what happens at scale?
Benchmarks (n=1,000,000)
filter 9630341 ns/op 23004421 B/op 37 allocs/op
filterCap 6906778 ns/op 8003584 B/op 1 allocs/op
The speed difference is still significant (~1.4x) thanks to 36 fewer calls to runtime.makeslice. However, the bigger difference is the memory allocation (~4x less).
Even better - calibrating the cap
You may have noticed in the first benchmark that cap makes the overall memory allocation worse (80B vs 56B). This is because you allocate 10 slots but only need, on average, 5 of them. This is why you don't want to set cap unnecessarily high. Given what you know about your program, you may be able to calibrate the capacity. In this case, we can estimate that our filtered slice will need 50% as many slots as the original slice.
func filterCalibratedCap(input []float64) []float64 {
ret := make([]float64, 0, len(input)/2)
for _, el := range input {
if el > .5 {
ret = append(ret, el)
}
}
return ret
}
Unsurprisingly, this calibrated cap allocates 50% as much memory as its predecessor, so that's ~8x improvement on the naive implementation at 1m elements.
Another option - using direct access instead of append
If you are looking to shave even more time off a program like this, initialize with the len parameter (and ignore the cap parameter), access the new slice directly instead of using append, then throw away all the slots you don't need.
func filterLen(input []float64) []float64 {
ret := make([]float64, len(input))
var counter int
for _, el := range input {
if el > .5 {
ret[counter] = el
counter++
}
}
return ret[:counter]
}
This is ~10% faster than filterCap at scale. However, in addition to being more complicated, this pattern does not provide the same safety as cap if you try and calibrate the memory requirement.
With cap calibration, if you underestimate the total capacity required, then the program will automatically allocate more when it needs it.
With this approach, if you underestimate the total len required, the program will fail. In this example, if you initialize as ret := make([]float64, len(input)/2), and it turns out that len(output) > len(input)/2, then at some point the program will try to access a non-existent slot and panic.
Each time you add an item to a slice that has len(mySlice) == cap(mySlice), the underlying data structure is replaced with a larger structure.
fmt.Printf("Original Capacity: %v", cap(mySlice)) // Output: 8
mySlice = append(mySlice, myNewItem)
fmt.Printf("New Capacity: %v", cap(mySlice)) // Output: 16
Here, mySlice is replaced (through the assignment operator) with a new slice containing all the elements of the original mySlice, plus myNewItem, plus some room (capacity) to grow without triggering this resize.
As you can imagine, this resizing operation is computationally non-trivial.
Quite often, all the resize operations can be avoided if you know how many items you will need to store in mySlice. If you have this foreknowledge, you can set the capacity of the original slice upfront and avoid all the resize operations.
(In practice, it's quite often possible to know how many items will be added to a collection; especially when transforming data from one format to another.)
The capacity parameter in making a slice in Go does not make much sense to me. For example,
aSlice := make([]int, 2, 2) //a new slice with length and cap both set to 2
aSlice = append(aSlice, 1, 2, 3, 4, 5) //append integers 1 through 5
fmt.Println("aSlice is: ", aSlice) //output [0, 0, 1, 2, 3, 4, 5]
If the slice allows inserting more elements than the capacity allows, why do we need to set it in the make() function?
The builtin append() function uses the specified slice to append elements to if it has a big enough capacity to accomodate the specified elements.
But if the passed slice is not big enough, it allocates a new, big enough slice, copies the elements from the passed slice to the new slice and append the elements to that new slice. And returns this new slice. Quoting from the append() documentation:
The append built-in function appends elements to the end of a slice. If it has sufficient capacity, the destination is resliced to accommodate the new elements. If it does not, a new underlying array will be allocated. Append returns the updated slice. It is therefore necessary to store the result of append, often in the variable holding the slice itself:
When making a slice with make if the length and capacity are the same, the capacity can be omitted, in which case it is defaulted to the specified length:
// These 2 declarations are equivalent:
s := make([]int, 2, 2)
s := make([]int, 2)
Also note that append() appends elements after the last element of the slice. And the above slices already have len(s) == 2 right after declaration so if you append even just 1 element to it, it will cause a reallocation as seen in this example:
s := make([]int, 2, 2)
fmt.Println(s, len(s), cap(s))
s = append(s, 1)
fmt.Println(s, len(s), cap(s))
Output:
[0 0] 2 2
[0 0 1] 3 4
So in your example what you should do is something like this:
s := make([]int, 0, 10) // Create a slice with length=0 and capacity=10
fmt.Println(s, len(s), cap(s))
s = append(s, 1)
fmt.Println(s, len(s), cap(s))
Output:
[] 0 10
[1] 1 10
I recommend the following blog articles if you want to understand slices in more details:
Go Slices: usage and internals
Arrays, slices (and strings): The mechanics of 'append'
It is mainly an optimization, and it is not unique to go, similar structures in other languages have this as well.
When you append more than the capacity, the runtime needs to allocate more memory for the new elements. This is costly and can also cause memory fragmentation.
By specifying the capacity, the runtime allocates what is needed in advance, and avoids reallocations. However if you do not know the estimated capacity in advance or it changes, you do not have to set it, and the runtime reallocates what is needed and grows the capacity itself.
I read go slice usage and internals and Slice and Effective go#slice but there is nothing about slicing a slice with 3 number like this : slice[a:b:c]
For example this code :
package main
import "fmt"
func main() {
var s = []string{"a", "b", "c", "d", "e", "f", "g"}
fmt.Println(s[1:2:6], len(s[1:2:6]), cap(s[1:2:6]))
fmt.Println(s[1:2:5], len(s[1:2:5]), cap(s[1:2:5]))
fmt.Println(s[1:2], len(s[1:2]), cap(s[1:2]))
}
go playground result is this :
[b] 1 5
[b] 1 4
[b] 1 6
I can understand that the third one is something about capacity, but what is the exact meaning of this?
Do I miss something in documents?
The syntax has been introduced in Go 1.2, as I mentioned in "Re-slicing slices in Golang".
It is documented in Full slice expressions:
a[low : high : max]
constructs a slice of the same type, and with the same length and elements as the simple slice expression a[low : high].
Additionally, it controls the resulting slice's capacity by setting it to max - low.
Only the first index may be omitted; it defaults to 0.
After slicing the array a:
a := [5]int{1, 2, 3, 4, 5}
t := a[1:3:5]
the slice t has type []int, length 2, capacity 4, and elements
t[0] == 2
t[1] == 3
The design document for that feature had the following justification:
It would occasionally be useful, for example in custom []byte allocation managers, to be able to hand a slice to a caller and know that the caller cannot edit values beyond a given subrange of the true array.
The addition of append to the language made this somewhat more important, because append lets programmers overwrite entries between len and cap without realizing it or even mentioning cap.
2022: svitanok adds for Go 1.19+:
while the capacity of a "derivative" slice doesn't exceed the one specified by the third index during its creation the slice is still "from" the same spot in the memory as its original ("true") slice, so the changes applied to it will affect the original slice.
And if then, for example, you append to this derivative slice the amount of elements that would cause its capacity to be increased, this new slice will occupy a different place in the memory, and so the changes made to it will not affect the slice it originated from.
In a slice expression slice[a:b:c] or aSlice[1:3:5]
a:b or 1:3 -> gives length
a:c or 1:5 -> gives capacity
We can extract both length and capacity from a slice expression with 3 numbers/indices, without looking at the source slice/array.
expression| aSlice[low:high:max] or aSlice[a:b:c] or aSlice[1:3:7]
------------------------------------------------------------------------
Length | len(aSlice[low:high]) or len(aSlice[a:b]) or len(aSlice[1:3])
Capacity | len(aSlice[low:max]) or len(aSlice[a:c]) or len(aSlice[1:7])
------------------------------------------------------------------------
Read more here at Slice Expressions
Playground
Actually Go slice have a pointer and pointing to the array and it holds length and capacity of the array and we can show it like will be
pointer:length:capacity
and append is used for adding same new length.
sl1 := make([]int, 6)
fmt.Println(sl1)
sl2 := append(sl1, 1)
fmt.Println(sl2)
[0 0 0 0 0 0]
[0 0 0 0 0 0 1]
The bytes.Buffer object has a Truncate(n int) method to discard all but the first n bytes.
I'd need the exact inverse of that - keeping the last n bytes.
I could do the following
b := buf.Bytes()
buf.Reset()
buf.Write(b[offset:])
but I'm not sure if this will re-use the slice efficiently.
Are there better options?
There are two alternatives:
The solution you give, which allows the first 'offset' bytes to be reused.
Create a bytes.NewBuffer(b[offset:]) and use that. This will not allow the first 'offset' bytes to be collected until you're done with the new buffer, but it avoids the cost of copying.
Let bytes.Buffer handle the buffer management. The internal grow method slides the data down. Use the Next method. For example,
package main
import (
"bytes"
"fmt"
)
func main() {
var buf bytes.Buffer
for i := 0; i < 8; i++ {
buf.WriteByte(byte(i))
}
fmt.Println(buf.Len(), buf.Bytes())
n := buf.Len() / 2
// Keep last n bytes.
if n > buf.Len() {
n = buf.Len()
}
buf.Next(buf.Len() - n)
fmt.Println(buf.Len(), buf.Bytes())
}
Output:
8 [0 1 2 3 4 5 6 7]
4 [4 5 6 7]
I reckon the problem with your idea is that "truncating the buffer from its start" is impossible simply because the memory allocator allocates memory in full chunks and there's no machinery in it to split an already allocated chunk into a set of "sub chunks" — essentially what you're asking for. So to support "trimming from the beginning" the implementation of bytes.Buffer would have to allocate a smaller buffer, move the "tail" there and then mark the original buffer for reuse.
This naturally leads us to another idea: use two (or more) buffers. They might either be allocated separately and treated as adjacent by your algorythms or you might use custom allocation: allocate one big slice and then reslice it twice or more times to produce several physically adjacent buffers, or slide one or more "window" slices over it. This means implementing a custom data structure of course…