Is there an implementation of io.ReadWriteSeeker to use in Golang?
Since, bytes.Buffer does not implement Seek method, I need to find such an implementation to use as a buffer written by zipwriter and to be read with seeking.
In addition I wont go with Reader(buff.Bytes()) to covert with memory copy, because I can not afford double memory size for buffered data.
In addition, when using os.File as the option, if I wont call f.Sync, it will never touch file system, right? Thanks.
My simplified codes:
func process() {
buff := new(bytes.Buffer)
zipWriter := zip.NewWriter(buff)
// here to add data into zipWriter in sequence
zipWriter.Close()
upload(buff) // upload(io.ReadSeeker)
}
For example, using the same underlying array for (uBuf and zBuf) buffers,
package main
import (
"archive/zip"
"bytes"
"io"
)
func upload(io.ReadSeeker) {}
func process() {
zBuf := new(bytes.Buffer)
zipWriter := zip.NewWriter(zBuf)
// add data into zipWriter in sequence
zipWriter.Close()
uBuf, zBuf := zBuf.Bytes(), nil
// upload(io.ReadSeeker)
upload(bytes.NewReader(uBuf))
}
func main() {}
Playground: https://play.golang.org/p/8TKmnL_vRY9
Package bytes
import "bytes"
func (*Buffer) Bytes
func (b *Buffer) Bytes() []byte
Bytes returns a slice of length b.Len() holding the unread portion of
the buffer. The slice is valid for use only until the next buffer
modification (that is, only until the next call to a method like Read,
Write, Reset, or Truncate). The slice aliases the buffer content at
least until the next buffer modification, so immediate changes to the
slice will affect the result of future reads.
The tuple assignment statement
uBuf, zBuf := zBuf.Bytes(), nil
gets the slice descriptor for the zipped bytes (zBuf.Bytes()) and assigns it to the slice descriptor uBuf. A slice descriptor is a struct with a pointer to the underlying array, the slice length, and the slice capacity. For example,
type slice struct {
array unsafe.Pointer
len int
cap int
}
Then, for safety, we assign nil to zBuf to ensure that no further changes can be made to its underlying array, which is now used by uBuf.
Related
I'm working on a project where I frequently convert []int32 to []byte. I created a function intsToBytes to perform an inplace conversion to minimize copying. I noticed that Go's escape analysis doesn't realize that ints and bytes reference the same underlying data. As a result, ints is overwritten by the next function's stack data and bytes lives on and references the overwritten data.
The only solution I can think of involves copying the data into a new byte slice. Is there away to avoid copying the data?
func pack() []byte {
ints := []int32{1,2,3,4,5} // This does not escape so it is allocated on the stack
bytes := intsToBytes(ints) // 'ints' and 'bytes' are different slice headers
return bytes
// After the return, the []int32{...} is deallocated and can be overwritten
// by the next function's stack data
}
func intsToBytes(i []int32) []byte {
const SizeOfInt32 = 4
// Get the slice header
header := *(*reflect.SliceHeader)(unsafe.Pointer(&i))
header.Len *= SizeOfInt32
header.Cap *= SizeOfInt32
// Convert slice header to an []byte
data := *(*[]byte)(unsafe.Pointer(&header))
/* Potentital Solution
outData := make([]byte, len(data))
copy(outData, data)
return outData
*/
return data
}
I'm trying to learn more regarding memory usage.
Doing some tests with interface{} and struct{} slices, I noticed that a slice of struct{} doesn't allocate any memory whereas a slice of interface{} does. It doesn't make so much sense to me, I'm actually expecting the same behavior (ie. both allocate nothing). Anyway I couldn't find any explanation regarding this particular case.
Could someone explain me why this happens?
package main
import (
"runtime"
"fmt"
)
func main() {
// Below is an example of using our PrintMemUsage() function
// Print our starting memory usage (should be around 0mb)
fmt.Println("Start")
PrintMemUsage()
fmt.Println("")
structContainer := make([]struct{}, 1000000)
for i := 0; i<1000000; i++ {
structContainer[i] = struct{}{}
}
fmt.Println("With 1kk struct{}")
PrintMemUsage()
fmt.Println("")
nilContainer := make([]interface{}, 1000000)
for i := 0; i<1000000; i++ {
nilContainer[i] = nil
}
fmt.Println("With 1kk nil interface{}")
PrintMemUsage()
fmt.Println("")
}
// PrintMemUsage outputs the current, total and OS memory being used. As well as the number
// of garage collection cycles completed.
func PrintMemUsage() {
var m runtime.MemStats
runtime.ReadMemStats(&m)
// For info on each, see: https://golang.org/pkg/runtime/#MemStats
fmt.Printf("Alloc = %v KiB", bToMb(m.Alloc))
fmt.Printf("\tTotalAlloc = %v KiB", bToMb(m.TotalAlloc))
fmt.Printf("\tSys = %v KiB", bToMb(m.Sys))
fmt.Printf("\tNumGC = %v\n", m.NumGC)
}
func bToMb(b uint64) uint64 {
return b / 1024
}
Playground link.
A variable of type interface{} can hold any value. E.g. it can hold the integer 8, it can hold the string value "hi", it can hold the struct value image.Point{X: 1, Y: 2} and pretty much everything else.
If you allocate a slice having interface{} as its element type, memory have to be allocated so that you can store any values in its elements. When using make() to allocate it, all its elements will get the zero value of the element type (which is nil for the interface{}), but memory still has to be allocated else you couldn't set elements later on.
On the other hand, the empty struct struct{} has no fields, it cannot hold any values (other than struct{}). When you allocate a slice having struct{} as its element type, memory does not need to be allocated because you won't be able to store anything in it that would require memory. So it's a simple and clever optimization not to allocate memory for such a type.
This is because an empty struct contains no value.
This is not very useful for arrays or slices. But it is useful for maps. A map without value is like a set. You can insert keys and test if they are present. The absence of value save space as you discovered.
I am trying to use Buffer package and copy the following code from Buffer documentation.
package main
import (
"bytes"
"fmt"
"os"
)
func main() {
var b bytes.Buffer // A Buffer needs no initialization.
b.Write([]byte("Hello "))
fmt.Fprintf(&b, "world!")
b.WriteTo(os.Stdout)
}
Why do Buffer here, not to be initialize?
As you can see here Buffer consists just of some ints, the buf slice and some arrays. All of them need no initialization, since go has zero values.
You can read more about slices and arrays and how they work here.
It is initialized. When you do not specifically initialize a variable, go will initialize it to its zero value. That means all the internal fields of a bytes.Buffer gets the value 0, or similar for the relevant types (e.g. nil for pointers).
The authors then implemented bytes.Buffer so all values being 0 is a meaningful starting point(It means an empty buffer), so programmers doesn't need to explicitly initialize it in order to start using a Buffer.
This due to the fact that when you call the Fprintf method, the bytes.Buffer.Write method is implicitely called, and as per the doc:
Write appends the contents of p to the buffer, growing the buffer as needed.
If you look at the source code, Write calls the grow function: func (b *Buffer) grow(n int) int.
This function recognizes that the buffer is empty, because it assumes that an empty buffer has 0 values for its internal fields, which is actually how a bytes.Buffer structure is initialized by default, just like every structure in go.
I'm trying to have some Go object implement io.Writer, but writes to a string instead of a file or file-like object. I thought bytes.Buffer would work since it implements Write(p []byte). However when I try this:
import "bufio"
import "bytes"
func main() {
var b bytes.Buffer
foo := bufio.NewWriter(b)
}
I get the following error:
cannot use b (type bytes.Buffer) as type io.Writer in function argument:
bytes.Buffer does not implement io.Writer (Write method has pointer receiver)
I am confused, since it clearly implements the interface. How do I resolve this error?
Pass a pointer to the buffer, instead of the buffer itself:
import "bufio"
import "bytes"
func main() {
var b bytes.Buffer
foo := bufio.NewWriter(&b)
}
package main
import "bytes"
import "io"
func main() {
var b bytes.Buffer
_ = io.Writer(&b)
}
You don't need use "bufio.NewWriter(&b)" to create an io.Writer. &b is an io.Writer itself.
Just use
foo := bufio.NewWriter(&b)
Because the way bytes.Buffer implements io.Writer is
func (b *Buffer) Write(p []byte) (n int, err error) {
...
}
// io.Writer definition
type Writer interface {
Write(p []byte) (n int, err error)
}
It's b *Buffer, not b Buffer. (I also think it is weird for we can call a method by a variable or its pointer, but we can't assign a pointer to a non-pointer type variable.)
Besides, the compiler prompt is not clear enough:
bytes.Buffer does not implement io.Writer (Write method has pointer receiver)
Some ideas, Go use Passed by value, if we pass b to buffio.NewWriter(), in NewWriter(), it is a new b (a new buffer), not the original buffer we defined, therefore we need pass the address &b.
bytes.Buffer is defined as:
type Buffer struct {
buf []byte // contents are the bytes buf[off : len(buf)]
off int // read at &buf[off], write at &buf[len(buf)]
bootstrap [64]byte // memory to hold first slice; helps small buffers avoid allocation.
lastRead readOp // last read operation, so that Unread* can work correctly.
}
using passed by value, the passed new buffer struct is different from the origin buffer variable.
I would like to understand how to store several byte slices separately in a slice. As hopefully illustrated below, I want the storage struct to store the result of the compressed result of n found in buf.
type storage struct
{
compressed []byte
}
func (s* storage) compress(n []byte) {
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
w.Write(n)
w.Close()
store := buf.Bytes()
s.compressed = append(s.compressed, store)
}
In your code compressed is a slice of bytes. If you want to store slices of bytes you need a slice of slices of bytes. So change the type of compressed to [][]byte