What is the easiest method to create an empty buffer of size n in Go using bytes.NewBuffer()?
Adding some additional info here. The quick way to create a new buffer is briefly mentioned at the end of the doc string:
b := new(bytes.Buffer)
or
b := &bytes.Buffer{}
The Buffer struct define includes a 64 byte internal bootstrap field that is initially used for small allocations. Once the default size is exceeded, a byte slice Buffer.buf is created and internally maintained.
As #leafbebop suggested we can pre-initalize the buf field of the Buffer struct using a new slice.
b := bytes.NewBuffer(make([]byte,0,N))
I also found another option to use the Grow() method:
b := new(bytes.Buffer)
b.Grow(n)
Also it's interesting to point out that the internal buf slice will grow at a rate of cap(buf)*2 + n. This means that if you've written 1MB into a buffer and then add 1 byte, your cap() will increase to 2097153 bytes.
Related
I'm rather new with go and I'm trying to access a portion of a byte slice and copy to a another fixed length byte slice but doesn't find the proper solution.
My best bet was :
var extracted []byte
var newSlice [512]byte = extracted[0 : 511]
But this gives me a conversion error :
cannot use extracted[0:511] (value of type []byte) as [512]byte value in variable declarationcompilerIncompatibleAssign
Notes :
this will be in a loop to iterate over the entire size of extracted 512 bytes at a time;
extracted actually has a fixed size of 512*n bytes, but if I fix that length I have the same issue
I thought I could use a io.Reader but this approach failed miserably as well.
Any help welcome :)
Here are a couple of approaches:
Convert the slice to an array pointer and dereference that pointer:
var pixels [512]byte
pixels = *(*[512]byte)(extracted[:512])
This can be done in one statement using a short variable declaration:
pixels := *(*[512]byte)(extracted[:512])
Use the builtin copy function to copy elements from a slice to an array (this point was covered in the question comments):
var pixels [512]byte
copy(pixels[:], extracted[:512])
The documentation in the registry package for GetValue() says :
GetValue retrieves the type and data for the specified value associated with an open key k. It fills up buffer buf and returns the retrieved byte count n. If buf is too small to fit the stored value it returns ErrShortBuffer error along with the required buffer size n. If no buffer is provided, it returns true and actual buffer size n. If no buffer is provided, GetValue returns the value's type only. If the value does not exist, the error returned is ErrNotExist.
GetValue is a low level function. If value's type is known, use the appropriate Get*Value function instead."
In my case, I don't know the value type of the registry key. However, I only need to print the value as a string. GetValue() takes in the value name and a "buffer" but the buffer is of type []byte. It is not passed by reference so I can't just create var buf []byte, pass that in and read it. I can't pass it in with &buf (type *[]byte). I can't use byte.Buffer (also type mismatch). I feel like there is something really simple I'm missing.
Code:
var buf []byte //????
_, _, e := myKey.GetValue(valuename, buf)
if e != nil {
panic(e)
}
fmt.Printf("Value: %s\n", string(buf)) // Prints blank
I suppose the registry API you mention is the Windows registry. To use these kinds of APIs, you have to take your best guess on the size of output you expect from the call:
buf:=make([]byte,1024)
typ, n, e := myKey.GetValue(valuename, buf)
if e==ErrShortBuffer {
// Go back, try with a larger buffer size
buf=make([]byte,n)
typ, n, e = myKey.GetValue(valuename, buf)
}
I need to create a string in Go that is 1048577 characters (1MB + 1 byte). The content of the string is totally unimportant. Is there a way to allocate this directly without concatenating or using buffers?
Also, it's worth noting that the value of string will not change. It's for a unit test to verify that strings that are too long will return an error.
Use strings.Builder to allocate a string without using extra buffers.
var b strings.Builder
b.Grow(1048577)
for i := 0; i < 1048577; i++ {
b.WriteByte(0)
}
s := b.String()
The call to the Grow method allocates a slice with capacity 1048577. The WriteByte calls fill the slice to capacity. The String() method uses unsafe to convert that slice to a string.
The cost of the loop can be reduced by writing chunks of N bytes at a time and filling single bytes at the end.
If you are not opposed to using the unsafe package, then use this:
p := make([]byte, 1048577)
s := *(*string)(unsafe.Pointer(&p))
If you are asking about how to do this with the simplest code, then use the following:
s := string(make([]byte, 1048577)
This approach does not meet the requirements set forth in the question. It uses an extra buffer instead of allocating the string directly.
I ended up using this:
string(make([]byte, 1048577))
https://play.golang.org/p/afPukPc1Esr
func Encode(i interface{}) ([]byte, error) {
buffer := bytes.NewBuffer(make([]byte, 0, 1024))
// size := unsafe.Sizeof(i)
size := reflect.TypeOf(i).Size()
fmt.Println(size)
ptr := unsafe.Pointer(&i)
startAddr := uintptr(ptr)
endAddr := startAddr + size
for i := startAddr; i < endAddr; i++ {
bytePtr := unsafe.Pointer(i)
b := *(*byte)(bytePtr)
buffer.WriteByte(b)
}
return buffer.Bytes(), nil
}
func TestEncode(t *testing.T) {
test := Test{10, "hello world"}
b, _ := Encode(test)
ptr := unsafe.Pointer(&b)
newTest := *(*Test)(ptr)
fmt.Println(newTest.X)
}
I am learning how to use golang unsafe and wrote this function for encoding any object. I meet with two problems, first, dose unsafe.Sizeof(obj) always return obj's pointer size? Why it different from reflect.TypeOf(obj).Size()? Second, I want to iterate the underlying bytes of obj and convert it back to obj in TestEncode function by unsafe.Pointer(), but the object's values all corrupt, why?
First, unsafe.Sizeof returns the bytes that needs to store the type. It is a little bit tricky, but it does not mean bytes that needs to store the data.
For example, a slice, as it is well known, stores 3 4-byte ints on a 32bit machine. One uintptr for memory address of the underlying array, and two int32 for len and cap. So no matter how long a slice is or what type it is of, a slice takes always 12 bytes on a 32 bit machine. Likely, a string uses 8 bytes: 1 uintptr for address and 1 int32 for len.
As for difference between reflect.TypeOf().Size, it is about interface. reflect.TypeOf looks into the interface and gets an concrete type, and reports bytes needed about the concrete type, while unsafe.Sizeof just returns 8 for an interface type: 2 uintptr for a pointer to the data and a pointer to the method lists.
Second part is quite clear now. For one, unsafe.Pointer is taking the address of the interface, instead of the concrete type. Two, in TestEncode, unsafe.Pointer is taking address to the 12-byte slice "header". There might be other errors, but with the two mentioned, they are meaningless to spot.
Note: I avoid talking about orders of the uintptr and int32 not only because I don't know, but also becuase they are not documented, unsafe, and implentation depended.
Note 2: Conclusion: Don't try to dump memory of a Go data.
Note 3: I change everything to 32 bit becuase playground is using it, so it is easier to check.
Is it true that converting from string to []byte allocates new memory? Also, does converting from []byte to string allocates new memory?
s := "a very long string"
b := []byte(s) // does this doubled the memory requirement?
b := []byte{1,2,3,4,5, ...very long bytes..}
s := string(b) // does this doubled the memory requirement?
Yes in both cases.
String types are immutable. Therefore converting them to a mutable slice type will allocate a new slice. See also http://blog.golang.org/go-slices-usage-and-internals
The same with the inverse. Otherwise mutating the slice would change the string, which would contradict the spec.