I have a TCP packet connection (net.Conn) set up by listening on a port.
conn, err := ln.Accept()
I need to read the first UVarInt of the Conn.Read([]byte) buffer, which starts at index 0.
Previously, I would only need the first byte, which is easy to do using
packetSize := make([]byte, 1)
conn.Read(packetSize)
// Do stuff with packetSize[0]
However, as previously mentioned, I need to get the first UVarInt I can reach using the net.Conn.Read() method. Keep in mind that a UVarInt can have pretty much any length, which I cannot be sure of (the client doesn't send the size of the UVarInt). I do know however that the UVarInt starts at the very beginning of the buffer.
Wrap the connection with a bufio.Reader:
br := bufio.NewReader(conn)
Use the binary package to read an unsigned varint through the bufio.Reader:
n, err := binary. ReadUvarInt(br)
Because the bufio.Reader can buffer more than the varint, you should use the bufio.Reader for all subsequent reads on the connection.
Related
I would like to parse several times with gocal data I retrieve through a HTTP call. Since I would like to avoid making the call for each of the parsing, I would like to save this data and reuse it.
The Body I get from http.Get is of type io.ReadCloser. The gocal parser requires io.Reader so it works.
Since I can retrieve Body only once, I can save it with body, _ := io.ReadAll(get.Body) but then I do not know how to serve []byte as io.Reader back (to the gocal parser, several times to account for different parsing conditions)
As you have figured, the http.Response.Body is exposed as an io.Reader, this reader is not re usable because it is connected straight to the underlying connection* (might be tcp/utp/or any other stream like reader under the net package).
Once you read the bytes out of the connection, new bytes are sitting their waiting for another read.
In order to save the response, indeed, you need to drain it first, and save that result within a variable.
body, _ := io.ReadAll(get.Body)
To re use that slice of bytes many time using the Go programming language, the standard API provides a buffered reader bytes.NewReader.
This buffer adequately offers the Reset([]byte) method to reset the state of the buffer.
The bytes.Reader.Reset is very useful to read multiple times the same bytes buffer with no allocations. In comparison, bytes.NewReader allocates every time it is called.
Finally, between two consecutive calls to c.Parser, you should reset the buffer with bytes buffer you have collected previously.
such as :
buf := bytes.NewReader(body)
// initialize the parser
c.Parse()
// process the result
// reset the buf, parse again
buf.Reset(body)
c.Parse()
You can try this version https://play.golang.org/p/YaVtCTZHZEP It uses the strings.NewReader buffer, but the interface and behavior are similar.
not super obvious, that is the general principle, the transport reads the headers, and leave the body untouched unless you consume it. see also that.
var r bufio.Reader
How do I check if r has no more data (is empty, is depleted)?
I understand that this may need to block until that information is known.
Can't find anything searching Google. I thought the Peek function would be useful to see if there is more data, but this seems to only peek an underlying buffer if exists. I could also try to Read one byte and subsequently call UnreadByte but that's extremely messy and unclear, are there any better options?
If r.Peek(1) returns data, then the next call to Read will return data.
If there's no data in the buffer, then Peek calls to the underlying reader and will block until data is available or an error.
If I understand your question correctly, would this work?
// func (*Reader) Size() int
// Size returns the size of the underlying buffer in bytes.
size := r.Size()
// func (*Reader) Buffered() int
// Buffered returns the number of bytes that can be read from the current buffer
buffered := r.Buffered()
I wish to confirm there are no more bytes to be read from a buffered reader (neither from the internal buffer, nor from the underlying file object) by trying to read one more byte (and catching EOF).
Is using bufio.Read or bufio.ReadByte suitable for this purpose?
It's not clear from the bufio.Read documentation whether or not the integer returned can be zero, in non-EOF cases. Namely, is 0, nil a valid return value if len(p) > 0?
func (b *Reader) Read(p []byte) (n int, err error)
Read reads data into p. It returns the number of bytes read into p. The bytes are taken from at most one Read on the underlying Reader, hence n may be less than len(p). To read exactly len(p) bytes, use io.ReadFull(b, p). At EOF, the count will be zero and err will be io.EOF.
Similarly, the bufio.ReadByte documentation doesn't separate error cases from EOF cases very well, and it doesn't exactly define what it means by "available" (i.e. available in the internal buffer, or available in the underlying file)?
func (b *Reader) ReadByte() (byte, error)
ReadByte reads and returns a single byte. If no byte is available, returns an error.
Passing a buffer of length 1 to bufio.Read, when the reader is backed with an underlying os.File, will indeed return n==0, io.EOF if the file is at EOF.
The documentation is being a bit imprecise because some of the behavior depends on the underlying reader you pass to the bufio reader. The code for bufio.Read() draws a more accurate picture. I'll outline the logic.
bufio.Read: Only issues a Read to the underlying reader if all bytes in the internal buffer have been exhausted. So, presumably, if you've already read as many bytes from the buffered reader as the number of bytes in the underlying file, that internal buffer should be exhausted when you make the last call bufio.Read(buf[0:1]) to check for EOF.
When the internal buffer is exhausted, and you ask the bufio reader for more, bufio.Read will do at most one call to the underlying reader. The type of error you get then will depend on your underlying reader.
Asking to read for n > 0 bytes from an os.File when the read pointer is already at EOF should return 0, io.EOF (according to the doc on os.File File.Read). But if your underlying reader was something else, perhaps a custom type specific to your application designed to return 0, nil at EOF, then bufio.Read would echo that back instead.
bufio.ReadByte: The logic behind bufio.ReadByte is slightly different but the outcome should be the same as bufio.Read in cases where the underlying reader is an os.File. The main difference with bufio.Read is that bufio.ReadByte can make several attempts to refill the internal buffer. If an error is encountered during refilling (which will be the case for a os.File reader at EOF), it is returned after the first erroneous read attempt. So, if your underlying Reader is an os.File reader, then you'll get 0, io.EOF if and only if your underlying file is at EOF. If your underlying reader was a custom reader type that only returned 0, nil at EOF, then bufio.ReadByte would eventually emit a "NoProgress" error. I'm not sure why the retry logic is only in bufio.ReadByte, but the good news is that either option can be used if your underlying file behaves like an os.File.
Other info:
This is not directly applicable to golang, but you may find the following thread interesting: Can read(2) return zero bytes when not at EOF. Its topic is the read() system call's semantics (POSIX). Reads on non-blocking sockets/files, even when no data is ready, should return -1, not 0, and set errno EAGAIN (or EINTR when interrupted). Non-blocking sockets/files are not really a concept native to go (as far as i know), and the bufio module in particular will panic() whenever/if the underlying reader returns negative numbers, so you don't have to worry about it.
buff := bytes.NewBuffer(somebytes)
How to write on top of buff? Currently I'm creating a new buffer. Is this the right way?
newBuff := bytes.NewBuffer(otherbytes)
newBuff.ReadFrom(buff)
bytes.NewBuffer() returns a *Buffer. *Buffer implements io.Writer (and io.Reader) so you can simply write to it by calling its Write() or WriteString() methods.
Example:
somebytes := []byte("abc")
buff := bytes.NewBuffer(somebytes)
buff.Write([]byte("def"))
fmt.Println(buff)
Output as expected (try it on the Go Playground):
abcdef
If you want to start with an empty buffer, you can simply create an empty Buffer struct (and take its address):
buff := &bytes.Buffer{}
If you want to "overwrite" the current content of the buffer, you can use the Buffer.Reset() method or the equivalent Buffer.Truncate(0) call.
Note that resetting or truncating the buffer will throw away the content (or only a part of it in case of Truncate(). But the allocated buffer (byte slice) in the background is kept and reused.
Note:
What you really want to do is not possible directly: just imagine if you want to insert some data in front of an existing content, the existing content would have to be shifted every time you write / insert something in front of it. This is not really efficient.
Instead create your body in a Buffer. Once it's done, you will know what your header will be. Create the header in another Buffer, and when it's done, copy (write) the body (from the first Buffer) into the second already containing the header.
Or if you don't need to store the whole data, you don't need to create a 2nd Buffer for the header. Once the body is ready, write the header to your output, and then write the body from the Buffer.
I'm attempting to compose an image in memory and send it out through http.ResponseWriter without ever touching the file system.
I use the following to create a new file:
file := os.NewFile(0, "temp_destination.png")
However, I don't seem to be able to do anything at all with this file. Here is the function I'm using (which is being called within an http.HandleFunc, which just sends the file's bytes to the browser), which is intended to draw a blue rectangle on a temporary file and encode it as a PNG:
func ComposeImage() ([]byte) {
img := image.NewRGBA(image.Rect(0, 0, 640, 480))
blue := color.RGBA{0, 0, 255, 255}
draw.Draw(img, img.Bounds(), &image.Uniform{blue}, image.ZP, draw.Src)
// in memory destination file, instead of going to the file sys
file := os.NewFile(0, "temp_destination.png")
// write the image to the destination io.Writer
png.Encode(file, img)
bytes, err := ioutil.ReadAll(file)
if err != nil {
log.Fatal("Couldn't read temporary file as bytes.")
}
return bytes
}
If I remove the png.Encode call, and just return the file bytes, the server just hangs and does nothing forever.
Leaving the png.Encode call in results in the file bytes (encoded, includes some of the PNG chunks I'd expect to see) being vomited out to stderr/stdout (I can't tell which) and server hanging indefinitely.
I assume I'm just not using os.NewFile correctly. Can anyone point me in the right direction? Alternative suggestions on how to properly perform in-memory file manipulations are welcome.
os.NewFile is a low level function that most people will never use directly. It takes an already existing file descriptor (system representation of a file) and converts it to an *os.File (Go's representation).
If you never want the picture to touch your filesystem, stay out of the os package entirely. Just treat your ResponseWriter as an io.Writer and pass it to png.Encode.
png.Encode(yourResponseWriter, img)
If you insist on writing to an "in memory file", I suggest using bytes.Buffer:
buf := new(bytes.Buffer)
png.Encode(buf, img)
return buf.Bytes()
Please have a detailed read of the NewFile documentation. NewFile does not create a new file, not at all! It sets up a Go os.File which wraps around an existing file with the given file descriptor (0 in your case which is stdin I think).
Serving images without files is much easier: Just Encode your image to your ResponseWriter. That's what interfaces are there for. No need to write to ome magic "in memory file", no need to read it back with ReadAll, plain and simple: Write to your response.