I have the following code:
var buf []byte
read_len, err := conn.Read(buf)
if err != nil {
fmt.Println("Error reading:", err.Error())
}
buffer := make([]byte, read_len)
_, err = conn.Read(buffer)
if err != nil {
fmt.Println("Error reading:", err.Error())
}
The intention was to determine read_len with the first buf, then create a second buffer which is the exact length of an incoming json request. This just results in an error
unexpected end of JSON input
When I try to unmarshal
var request Device_Type_Request_Struct
err = json.Unmarshal(buffer, &request)
I'm assuming that this error occurs because the conn.Read(buffer) is returning nothing because another buffer has already read it (not sure though). How should I go about determining the length of json request while also being able to read it into a buffer (of the exact same length)?
Read returns the number of bytes read to the buffer. Because the length of the buffer passed to the first call to conn.Read is zero, the first call to conn.Read always returns zero.
There is no way to determine how much data a peer has sent without reading the data.
The easy solution to this problem is to use the JSON decoder:
d := json.NewDecoder(conn)
var request Device_Type_Request_Struct
if err := d.Decode(&request); err != nil {
// handle error
}
The decoder reads and decodes JSON values from a stream.
Related
I'm building my first go-libp2p application and trying to modify the echo example to read a []byte instead of a string as in the example.
In my code, I changed the doEcho function to run io.ReadAll(s) instead of bufio.NewReader(s) followed by ReadString('\n'):
// doEcho reads a line of data a stream and writes it back
func doEcho(s network.Stream) error {
b, err := io.ReadAll(s)
if err != nil {
return err
}
log.Printf("Number of bytes received: %d", len(b))
_, err = s.Write([]byte("thanks for the bytes"))
return err
}
When I run this and send a message, I do see the listener received new stream log but the doEcho function gets stuck after the io.ReadAll(s) call and never executes the reply.
So my questions are:
Why does my code not work and how can I make it work?
How does io.ReadAll(s) and bufio's ReadString('\n') work under the hood so that they cause this difference in behavior?
Edit:
As per #Stephan Schlecht suggestion I changed my code to this, but it still remains blocked as before:
func doEcho(s network.Stream) error {
buf := bufio.NewReader(s)
var data []byte
for {
b, err := buf.ReadByte()
if err != nil {
break
}
data = append(data, b)
}
log.Printf("Number of bytes received: %d", len(data))
_, err := s.Write([]byte("thanks for the bytes"))
return err
}
Edit 2: I forgot to clarify this, but I don't want to use ReadString('\n') or ReadBytes('\n') because I don't know anything about the []byte I'm receiving, so it might not end with \n. I want to read any []byte from the stream and then write back to the stream.
ReadString('\n') reads until the first occurrence of \n in the input and returns the string.
io.ReadAll(s) reads until an error or EOF and returns the data it read. So unless an error or EOF occurs it does not return.
In principle, there is no natural size for a data structure to be received on stream-oriented connections.
It depends on the remote sender.
If the remote sender sends binary data and closes the stream after sending the last byte, then you can simply read all data up to the EOF on the receiver side.
If the stream is not to be closed immediately and the data size is variable, there are further possibilities: One first sends a header that has a defined size and in the simplest case simply transmits the length of the data. Once you have received the specified amount of data, you know that this round of reception is complete and you can continue.
Alternatively, you can define a special character that marks the end of the data structure to be transmitted. This will not work if you want to transmit arbitrary binary data without encoding.
There are other options that are a little more complicated, such as splitting the data into blocks.
In the example linked in the question, a \n is sent at the end of the data just sent, but this would not work if you want to send arbitrary binary data.
Adapted Echo Example
In order to minimally modify the echo example linked in the question to first send a 1-byte header with the length of the payload and only then the actual payload, it could look something like the following:
Sending
In the function runSender line one could replace the current sending of the payload from:
log.Println("sender saying hello")
_, err = s.Write([]byte("Hello, world!\n"))
if err != nil {
log.Println(err)
return
}
to
log.Println("sender saying hello")
payload := []byte("Hello, world!")
header := []byte{byte(len(payload))}
_, err = s.Write(header)
if err != nil {
log.Println(err)
return
}
_, err = s.Write(payload)
if err != nil {
log.Println(err)
return
}
So we send one byte with the length of the payload before the actual payload.
Echo
The doEcho would then read the header first and afterwards the payload. It uses ReadFull, which reads exactly len(payload) bytes.
func doEcho(s network.Stream) error {
buf := bufio.NewReader(s)
header, err := buf.ReadByte()
if err != nil {
return err
}
payload := make([]byte, header)
n, err := io.ReadFull(buf, payload)
log.Printf("payload has %d bytes", n)
if err != nil {
return err
}
log.Printf("read: %s", payload)
_, err = s.Write(payload)
return err
}
Test
Terminal 1
2022/11/06 09:59:38 I am /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e
2022/11/06 09:59:38 listening for connections
2022/11/06 09:59:38 Now run "./echo -l 8089 -d /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e" on a different terminal
2022/11/06 09:59:55 listener received new stream
2022/11/06 09:59:55 payload has 13 bytes
2022/11/06 09:59:55 read: Hello, world!
Terminal 2
stephan#mac echo % ./echo -l 8089 -d /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e
2022/11/06 09:59:55 I am /ip4/127.0.0.1/tcp/8089/p2p/QmW6iSWiFBG5ugUUwBND14pDZzLDaqSNfxBG6yb8cmL3Di
2022/11/06 09:59:55 sender opening stream
2022/11/06 09:59:55 sender saying hello
2022/11/06 09:59:55 read reply: "Hello, world!"
s
This is certainly a fairly simple example and will certainly need to be customized to your actual requirements, but could perhaps be a first step in the right direction.
I am currently querying Druid and returning a large dataset back (roughly 4-5GBs). I would like to process this response and decode the JSON into a list of structs. I have it working fine when I change the query to return a smaller dataset but as soon as the response gets too large I get an unexpected EOF error.
I have tried reading the entire response body
bytes, err := ioutil.ReadAll(resp.Body)
Decoding the response body directly
var object []NewObject
err = json.NewDecoder(resp.Body).Decode(&object)
Creating a buffer and writing to a file
f, err := os.OpenFile("/tmp/test.txt", os.O_APPEND|os.O_WRONLY, 0600)
defer f.Close()
const oneMB = 1024 * 1024
bytesRead := 0
respBuf := make([]byte, oneMB)
// Read the response body
for {
n, err := resp.Body.Read(respBuf)
bytesRead += n
if err == io.EOF {
break
}
if err != nil {
fmt.Println("Error reading HTTP response: ", err.Error())
break
}
if _, err = f.Write(respBuf); err != nil {
panic(err)
}
}
All of these have ended with me getting an unexpected EOF error. I am using the default net/http, and encoding/json module which appear they should work fine. Is there anything else I can try?
I've figured out the problem. It appears to be that Druid has a hard query timeout (this link explains more about these configuration settings). What is interesting is it appears if you want the response as an object and it is too large it will just cut off mid byte stream, but if you want the result as array and it is too large it will send the bytes to finish the current element and then send a closing ] so when you parse the result it appears it all came through (no errors), though you may have been cut off midstream.
I'm using flatbuffer to send binary data over unix socket. The flatbuffer that I send is of dynamic length. The problem I'm facing is, how to know how many bytes I have to read for one table.
Is there something like a delimiter that can be appended while sending, which I can use to determine the end of the flatbuffer.
When I tried with a smaller size
buf := make([]byte, 512)
nr, err := c.Read(buf)
if err != nil {
fmt.Println("exit echo")
return
}
And if the flatbuffer that is bigger than 512 bytes is read, then this results in failure.
When I read by growing my buffer, then I'm not able to find the end of the read
var n, nr int
var err error
buf := make([]byte, 0, 4096) // big buffer
tmp := make([]byte, 512)
for {
n, err = c.Read(tmp)
if err != nil {
break
}
nr += n
if nr >= 4096 {
err = errOverrun
break
}
buf = append(buf, tmp[:n]...)
}
if err != nil {
fmt.Println("read error:", err)
break
}
FlatBuffers does not include a length field by design, since in most context the length is an implicit part of the storage or transfer of a buffer.
If you have no way to know the size of a buffer, or you are streaming buffers, the best is to simply pre-fix any buffer with a 32bit length field, so you can use that to read the rest of the data.
In the C++ API this is even built-in (see SizePrefixed functions), but this hasn't been ported to Go yet, so you'd have to do it manually.
I have a connection, created like this:
conn, err = net.Dial("tcp", "127.0.0.1:20000")
I have tried reading from this connection in two ways. I think they both must work, but the first option doesn't.
Here is the first way of doing it:
var bytes []byte
for i := 0; i < 4; i++ {
conn.Read(bytes)
}
fmt.Printf("%v", bytes)
The output of this method is:
[]
And here is the same thing, done with bufio.Reader:
func readResponse(conn net.Conn) (response string, err error) {
reader := bufio.NewReader(conn)
_, err = reader.Discard(8)
if err != nil {
return
}
response, err = reader.ReadString('\n')
return
}
This function returns the response given by the server on the other end of the TCP connection.
Why does bufio.Reader.Read() work, but net.Conn.Read() doesn't?
The Conn.Read() method is to implement io.Reader, the general interface to read data from any source of bytes into a []byte. Quoting from the doc of Reader.Read():
Read reads up to len(p) bytes into p.
So Read() reads up to len(p) bytes but since you pass a nil slice, it won't read anything (length of a nil slice is 0). Please read the linked doc to know how Reader.Read() works.
Reader.Read() does not allocate a buffer ([]byte) where the read data will be stored, you have to create one and pass it, e.g.:
var buf = make([]byte, 100)
n, err := conn.Read(buf)
// n is the number of read bytes; don't forget to check err!
Don't forget to always check the returned error which may be io.EOF if end of data is reached. The general contract of io.Reader.Read() also allows returning some non-nil error (including io.EOF) and some read data (n > 0) at the same time. The number of read bytes will be in n, which means only the first n bytes of the buf is useful (in other words: buf[:n]).
Your other example using bufio.Reader works because you called Reader.ReadString() which doesn't require a []byte argument. If you would've used the bufio.Reader.Read() method, you would also had to pass a non-nil slice in order to actually get some data.
Here's the error I get when I flood the server with too many packets per second:
2014/11/28 12:52:49 main.go:59: loading plugin: print
2014/11/28 12:52:49 main.go:86: starting server on 0.0.0.0:8080
2014/11/28 12:52:59 server.go:15: client has connected: 127.0.0.1:59146
2014/11/28 12:52:59 server.go:43: received data from client 127.0.0.1:59146: &main.Observation{SensorId:"1", Timestamp:1416492023}
2014/11/28 12:52:59 server.go:29: read error from 127.0.0.1:59146: zlib: invalid header
2014/11/28 12:52:59 server.go:18: closing connection to: 127.0.0.1:59146
It manages to decode one packet (sometimes, maybe 2 or 3) then errors out. Here's the code doing the flooding:
import socket
import struct
import json
import zlib
import time
def serialize(data):
data = json.dumps(data)
data = zlib.compress(data)
packet = struct.pack('!I', len(data))
packet += data
return len(data), packet
message = {
'sensor_id': '1',
'timestamp': 1416492023,
}
length, buffer = serialize([message])
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 8080))
while True:
client.send(buffer)
#time.sleep(0.0005)
When I uncomment the time.sleep() call, the server works fine. It seems too many packets/per second is killing the server. Why?
Here's the relevent Go code. The connection handler:
func (self *Server) handleConnection(connection net.Conn) {
for {
connection.SetReadDeadline(time.Now().Add(30 * time.Second))
observations, err := self.Protocol.Unserialize(connection)
if err != nil {
log.Printf("read error from %s: %s\n", connection.RemoteAddr(), err)
return
}
}
And here's the unserializer:
// Length Value protocol to read zlib compressed, JSON encoded packets.
type ProtocolV2 struct{}
func (self *ProtocolV2) Unserialize(packet io.Reader) ([]*Observation, error) {
var length uint32
if err := binary.Read(packet, binary.BigEndian, &length); err != nil {
return nil, err
}
buffer := make([]byte, length)
rawreader := bufio.NewReader(packet)
if _, err := rawreader.Read(buffer); err != nil {
return nil, err
}
bytereader := bytes.NewReader(buffer)
zreader, err := zlib.NewReader(bytereader)
if err != nil {
return nil, err
}
defer zreader.Close()
var observations []*Observation
decoder := json.NewDecoder(zreader)
if err := decoder.Decode(&observations); err != nil {
return nil, err
}
return observations, nil
}
It seems there is an error on client side in the Python script.
The return of client.send is not checked, so the script does not handle partial writes in a correct way. Basically, when the socket buffer is full, only part of the message will be written, resulting in the server unable to decode the message.
This code is broken, but adding the wait makes it work because it prevents the socket buffer to be full.
You can use client.sendall instead to ensure the write operations are complete.
More information in the Python documentation:
https://docs.python.org/2/library/socket.html
https://docs.python.org/2/howto/sockets.html#using-a-socket
Now in the Go server, there is also a similar problem. The documentation says:
Read reads data into p. It returns the number of bytes read into p. It calls Read at most once on the underlying Reader, hence n may be less than len(p). At EOF, the count will be zero and err will be io.EOF.
The rawreader.Read call may return less bytes than you expect. You may want to use the ReadFull() function of the io package to ensure the full message is read.