Please note this is pseudo code and I am summarising.I am reading some source code from inside a function:
maxKeyLen := 100 * 1024 * 1024
maxValueLen := 100 * 1024 * 1024
var klen, vlen uint32
binary.Read(p.buffer, binary.BigEndian, &klen)
if klen > maxKeyLen {
return nil, nil, fmt.Errorf("key exceeds max len %d, got %d bytes", maxKeyLen, klen)
}
At what point does the binary.Read stop? Because straight after this there is another read:
key := make([]byte, klen)
_, err := p.buffer.Read(key)
if err != nil {
return nil, nil, err
}
binary.Read(p.buffer, binary.BigEndian, &vlen)
if vlen > maxValueLen {
return nil, nil, fmt.Errorf("value exceeds max len %d, got %d bytes", maxValueLen, vlen)
}
Where p.buffer is defined via:
buff := new(bytes.Buffer)
io.Copy(buff, r)
p.buffer = buff
And r is some data that has been passed in.
At first I thought the answer was at 4 bytes it stops. But that's not true because the maxkeylen checks for greater than that. So how does the binary.read know when to stop as there is more data ahead, because the next binary read on for the vlen then finds stuff?
When questioning the superheros of Go, always refer to their actual source code in question:
https://golang.org/src/encoding/binary/binary.go?s=4201:4264#L132
142 func Read(r io.Reader, order ByteOrder, data interface{}) error {
143 // Fast path for basic types and slices.
144 if n := intDataSize(data); n != 0 {
Line 144 shows an example of reading the initial size of know types, and iterating or copying as needed later in that scope.
In your code example above, it will be the 4 byte length of klen which is an uint32. That is, it will read 4 bytes from p.buffer into klen.
It gives a hint in the documentation:
https://golang.org/pkg/encoding/binary/#Read
func Read(r io.Reader, order ByteOrder, data interface{}) error
Read reads structured binary data from r into data. Data must be a pointer to a fixed-size value or a slice of fixed-size values. Bytes read from r are decoded using the specified byte order and written to successive fields of the data.
Related
Without reading the contents of a file into memory, how can I read "x" bytes from the file so that I can specify what x is for every separate read operation?
I see that the Read method of various Readers takes a byte slice of a certain length and I can read from a file into that slice. But in that case the size of the slice is fixed, whereas what I would like to do, ideally, is something like:
func main() {
f, err := os.Open("./file.txt")
if err != nil {
panic(err)
}
someBytes := f.Read(2)
someMoreBytes := f.Read(4)
}
bytes.Buffer has a Next method which behaves very closely to what I would want, but it requires an existing buffer to work, whereas I'm hoping to read an arbitrary amount of bytes from a file without needing to read the whole thing into memory.
What is the best way to accomplish this?
Thank you for your time.
Use this function:
// readN reads and returns n bytes from the reader.
// On error, readN returns the partial bytes read and
// a non-nil error.
func readN(r io.Reader, n int) ([]byte, error) {
// Allocate buffer for result
b := make([]byte, n)
// ReadFull ensures buffer is filled or error is returned.
n, err := io.ReadFull(r, b)
return b[:n], err
}
Call like this:
someBytes, err := readN(f, 2)
if err != nil { /* handle error here */
someMoreBytes := readN(f, 4)
if err != nil { /* handle error here */
you can do something like this:
f, err := os.Open("/tmp/dat")
check(err)
b1 := make([]byte, 5)
n1, err := f.Read(b1)
check(err)
fmt.Printf("%d bytes: %s\n", n1, string(b1[:n1]))
for more reading please check site.
I need to send struct data with byte slice data type during socket communication.
type A struct {
header []byte
body []byte
}
So I wrote the following source code to convert the structure to bytes.
var a A
a.header = byte slice data...
a.body = byte slice data...
buf := new(bytes.Buffer)
binary.Write(buf, binary.BigEndian, a)
However, I get an error with the binary.Write function showing the following error:
binary.Write: invalid type main.A
I have found that fixed arrays solve the problem. But since the length of the data is constantly changing, I have to use a slice rather than a fixed array.
Is there a way to solve this problem?
If you write a variable length of byte slice, the other end would not know how many bytes it needs to read. You have to communicate the length too.
So one way to send a byte slice is to first write the length (number of bytes) using a fixed-size type, e.g. int32 or int64. Then simply write the byte slice.
For example:
var w io.Writer // This represents your connection
var a A
if err := binary.Write(w, binary.LittleEndian, int32(len(a.header))); err != nil {
// Handle error
}
if _, err := w.Write(a.header); err != nil {
// Handle error
}
You may use the same logic to send a.body too.
On the other end, this is how you could read it:
var r io.Reader // This represents your connection
var a A
var size int32
if err := binary.Read(r, binary.LittleEndian, &size); err != nil {
// Handle error
}
a.header = make([]byte, size)
if _, err := io.ReadFull(r, a.header); err != nil {
// Handle error
}
Try a working example on the Go Playground.
If you have to transfer more complex structs, consider using the encoding/gob which handles sending slices with ease. For an example and some insights, see Efficient Go serialization of struct to disk.
I have a large int array that I want to persist on the filesystem. My understanding is the best way to store something like this is to use the gob package to convert it to a byte array and then to compress it with gzip.
When I need it again, I reverse the process. I am pretty sure I am storing it correctly, however recovering it is failing with EOF. Long story short, I have some example code below that demonstrates the issue. (playground link here https://play.golang.org/p/v4rGGeVkLNh).
I am not convinced gob is needed, however reading around it seems that its more efficient to store it as a byte array than an int array, but that may not be true. Thanks!
package main
import (
"bufio"
"bytes"
"compress/gzip"
"encoding/gob"
"fmt"
)
func main() {
arry := []int{1, 2, 3, 4, 5}
//now gob this
var indexBuffer bytes.Buffer
writer := bufio.NewWriter(&indexBuffer)
encoder := gob.NewEncoder(writer)
if err := encoder.Encode(arry); err != nil {
panic(err)
}
//now compress it
var compressionBuffer bytes.Buffer
compressor := gzip.NewWriter(&compressionBuffer)
compressor.Write(indexBuffer.Bytes())
defer compressor.Close()
//<--- I think all is good until here
//now decompress it
buf := bytes.NewBuffer(compressionBuffer.Bytes())
fmt.Println("byte array before unzipping: ", buf.Bytes())
if reader, err := gzip.NewReader(buf); err != nil {
fmt.Println("gzip failed ", err)
panic(err)
} else {
//now ungob it...
var intArray []int
decoder := gob.NewDecoder(reader)
defer reader.Close()
if err := decoder.Decode(&intArray); err != nil {
fmt.Println("gob failed ", err)
panic(err)
}
fmt.Println("final int Array content: ", intArray)
}
}
You are using bufio.Writer which–as its name implies–buffers bytes written to it. This means if you're using it, you have to flush it to make sure buffered data makes its way to the underlying writer:
writer := bufio.NewWriter(&indexBuffer)
encoder := gob.NewEncoder(writer)
if err := encoder.Encode(arry); err != nil {
panic(err)
}
if err := writer.Flush(); err != nil {
panic(err)
}
Although the use of bufio.Writer is completely unnecessary as you're already writing to an in-memory buffer (bytes.Buffer), so just skip that, and write directly to bytes.Buffer (and so you don't even have to flush):
var indexBuffer bytes.Buffer
encoder := gob.NewEncoder(&indexBuffer)
if err := encoder.Encode(arry); err != nil {
panic(err)
}
The next error is how you close the gzip stream:
defer compressor.Close()
This deferred closing will only happen when the enclosing function (the main() function) returns, not a second earlier. But by that time you already wanted to read the zipped data, but that might still sit in an internal cache of gzip.Writer, and not in compressionBuffer, so you obviously can't read the compressed data from compressionBuffer. Close the gzip stream without using defer:
if err := compressor.Close(); err != nil {
panic(err)
}
With these changes, you program runs and outputs (try it on the Go Playground):
byte array before unzipping: [31 139 8 0 0 0 0 0 0 255 226 249 223 200 196 200 244 191 137 129 145 133 129 129 243 127 19 3 43 19 11 27 7 23 32 0 0 255 255 110 125 126 12 23 0 0 0]
final int Array content: [1 2 3 4 5]
As a side note: buf := bytes.NewBuffer(compressionBuffer.Bytes()) – this buf is also completely unnecessary, you can just start decoding compressionBuffer itself, you can read data from it that was previously written to it.
As you might have noticed, the compressed data is much larger than the initial, compressed data. There are several reasons: both encoding/gob and compress/gzip streams have significant overhead, and they (may) only make input smaller on a larger scale (5 int numbers don't qualify to this).
Please check related question: Efficient Go serialization of struct to disk
For small arrays, you may also consider variable-length encoding, see binary.PutVarint().
I have a connection, created like this:
conn, err = net.Dial("tcp", "127.0.0.1:20000")
I have tried reading from this connection in two ways. I think they both must work, but the first option doesn't.
Here is the first way of doing it:
var bytes []byte
for i := 0; i < 4; i++ {
conn.Read(bytes)
}
fmt.Printf("%v", bytes)
The output of this method is:
[]
And here is the same thing, done with bufio.Reader:
func readResponse(conn net.Conn) (response string, err error) {
reader := bufio.NewReader(conn)
_, err = reader.Discard(8)
if err != nil {
return
}
response, err = reader.ReadString('\n')
return
}
This function returns the response given by the server on the other end of the TCP connection.
Why does bufio.Reader.Read() work, but net.Conn.Read() doesn't?
The Conn.Read() method is to implement io.Reader, the general interface to read data from any source of bytes into a []byte. Quoting from the doc of Reader.Read():
Read reads up to len(p) bytes into p.
So Read() reads up to len(p) bytes but since you pass a nil slice, it won't read anything (length of a nil slice is 0). Please read the linked doc to know how Reader.Read() works.
Reader.Read() does not allocate a buffer ([]byte) where the read data will be stored, you have to create one and pass it, e.g.:
var buf = make([]byte, 100)
n, err := conn.Read(buf)
// n is the number of read bytes; don't forget to check err!
Don't forget to always check the returned error which may be io.EOF if end of data is reached. The general contract of io.Reader.Read() also allows returning some non-nil error (including io.EOF) and some read data (n > 0) at the same time. The number of read bytes will be in n, which means only the first n bytes of the buf is useful (in other words: buf[:n]).
Your other example using bufio.Reader works because you called Reader.ReadString() which doesn't require a []byte argument. If you would've used the bufio.Reader.Read() method, you would also had to pass a non-nil slice in order to actually get some data.
I am aware of the specific function in golang from the bufio package.
func (b *Reader) Peek(n int) ([]byte, error)
Peek returns the next n bytes without advancing the reader. The bytes
stop being valid at the next read call. If Peek returns fewer than n
bytes, it also returns an error explaining why the read is short. The
error is ErrBufferFull if n is larger than b's buffer size.
I need to be able to read a specific number of bytes from a Reader that will advance the reader. Basically, identical to the function above, but it advances the reader. Does anybody know how to accomplish this?
Note that the bufio.Read method calls the underlying io.Read at most once, meaning that it can return n < len(p), without reaching EOF. If you want to read exactly len(p) bytes or fail with an error, you can use io.ReadFull like this:
n, err := io.ReadFull(reader, p)
This works even if the reader is buffered.
func (b *Reader) Read(p []byte) (n int, err error)
http://golang.org/pkg/bufio/#Reader.Read
The number of bytes read will be limited to len(p)
TLDR:
my42bytes, err := ioutil.ReadAll(io.LimitReader(myReader, 42))
Full answer:
#monicuta mentioned io.ReadFull which works great. Here I provide another method. It works by chaining ioutil.ReadAll and io.LimitReader together. Let's read the doc first:
$ go doc ioutil.ReadAll
func ReadAll(r io.Reader) ([]byte, error)
ReadAll reads from r until an error or EOF and returns the data it read. A
successful call returns err == nil, not err == EOF. Because ReadAll is
defined to read from src until EOF, it does not treat an EOF from Read as an
error to be reported.
$ go doc io.LimitReader
func LimitReader(r Reader, n int64) Reader
LimitReader returns a Reader that reads from r but stops with EOF after n
bytes. The underlying implementation is a *LimitedReader.
So if you want to get 42 bytes from myReader, you do this
import (
"io"
"io/ioutil"
)
func main() {
// myReader := ...
my42bytes, err := ioutil.ReadAll(io.LimitReader(myReader, 42))
if err != nil {
panic(err)
}
//...
}
Here is the equivalent code with io.ReadFull
$ go doc io.ReadFull
func ReadFull(r Reader, buf []byte) (n int, err error)
ReadFull reads exactly len(buf) bytes from r into buf. It returns the number
of bytes copied and an error if fewer bytes were read. The error is EOF only
if no bytes were read. If an EOF happens after reading some but not all the
bytes, ReadFull returns ErrUnexpectedEOF. On return, n == len(buf) if and
only if err == nil. If r returns an error having read at least len(buf)
bytes, the error is dropped.
import (
"io"
)
func main() {
// myReader := ...
buf := make([]byte, 42)
_, err := io.ReadFull(myReader, buf)
if err != nil {
panic(err)
}
//...
}
Compared to io.ReadFull, an advantage is that you don't need to manually make a buf, where len(buf) is the number of bytes you want to read, then pass buf as an argument when you Read
Instead you simply tell io.LimitReader you want at most 42 bytes from myReader, and call ioutil.ReadAll to read them all, returning the result as a slice of bytes. If successful, the returned slice is guaranteed to be of length 42.
I am prefering Read() especially if you are going to read any type of files and it could be also useful in sending data in chunks, below is an example to show how it is used
fs, err := os.Open("fileName");
if err != nil{
fmt.Println("error reading file")
return
}
defer fs.Close()
reader := bufio.NewReader(fs)
buf := make([]byte, 1024)
for{
v, _ := reader.Read(buf) //ReadString and ReadLine() also applicable or alternative
if v == 0{
return
}
//in case it is a string file, you could check its content here...
fmt.Print(string(buf))
}
Pass a n-bytes sized buffer to the reader.
If you want to read the bytes from an io.Reader and into an io.Writer, then you can use io.CopyN
CopyN copies n bytes (or until an error) from src to dst. It returns the number of bytes copied and the earliest error encountered while copying.
On return, written == n if and only if err == nil.
written, err := io.CopyN(dst, src, n)
if err != nil {
// We didn't read the desired number of bytes
} else {
// We can proceed successfully
}
To do this you just need to create a byte slice and read the data into this slice with
n := 512
buff := make([]byte, n)
fs.Read(buff) // fs is your reader. Can be like this fs, _ := os.Open('file')
func (b *Reader) Read(p []byte) (n int, err error)
Read reads data into p. It returns the number of bytes read into p.
The bytes are taken from at most one Read on the underlying Reader,
hence n may be less than len(p)