go-libp2p - receiving bytes from stream - go

I'm building my first go-libp2p application and trying to modify the echo example to read a []byte instead of a string as in the example.
In my code, I changed the doEcho function to run io.ReadAll(s) instead of bufio.NewReader(s) followed by ReadString('\n'):
// doEcho reads a line of data a stream and writes it back
func doEcho(s network.Stream) error {
b, err := io.ReadAll(s)
if err != nil {
return err
}
log.Printf("Number of bytes received: %d", len(b))
_, err = s.Write([]byte("thanks for the bytes"))
return err
}
When I run this and send a message, I do see the listener received new stream log but the doEcho function gets stuck after the io.ReadAll(s) call and never executes the reply.
So my questions are:
Why does my code not work and how can I make it work?
How does io.ReadAll(s) and bufio's ReadString('\n') work under the hood so that they cause this difference in behavior?
Edit:
As per #Stephan Schlecht suggestion I changed my code to this, but it still remains blocked as before:
func doEcho(s network.Stream) error {
buf := bufio.NewReader(s)
var data []byte
for {
b, err := buf.ReadByte()
if err != nil {
break
}
data = append(data, b)
}
log.Printf("Number of bytes received: %d", len(data))
_, err := s.Write([]byte("thanks for the bytes"))
return err
}
Edit 2: I forgot to clarify this, but I don't want to use ReadString('\n') or ReadBytes('\n') because I don't know anything about the []byte I'm receiving, so it might not end with \n. I want to read any []byte from the stream and then write back to the stream.

ReadString('\n') reads until the first occurrence of \n in the input and returns the string.
io.ReadAll(s) reads until an error or EOF and returns the data it read. So unless an error or EOF occurs it does not return.
In principle, there is no natural size for a data structure to be received on stream-oriented connections.
It depends on the remote sender.
If the remote sender sends binary data and closes the stream after sending the last byte, then you can simply read all data up to the EOF on the receiver side.
If the stream is not to be closed immediately and the data size is variable, there are further possibilities: One first sends a header that has a defined size and in the simplest case simply transmits the length of the data. Once you have received the specified amount of data, you know that this round of reception is complete and you can continue.
Alternatively, you can define a special character that marks the end of the data structure to be transmitted. This will not work if you want to transmit arbitrary binary data without encoding.
There are other options that are a little more complicated, such as splitting the data into blocks.
In the example linked in the question, a \n is sent at the end of the data just sent, but this would not work if you want to send arbitrary binary data.
Adapted Echo Example
In order to minimally modify the echo example linked in the question to first send a 1-byte header with the length of the payload and only then the actual payload, it could look something like the following:
Sending
In the function runSender line one could replace the current sending of the payload from:
log.Println("sender saying hello")
_, err = s.Write([]byte("Hello, world!\n"))
if err != nil {
log.Println(err)
return
}
to
log.Println("sender saying hello")
payload := []byte("Hello, world!")
header := []byte{byte(len(payload))}
_, err = s.Write(header)
if err != nil {
log.Println(err)
return
}
_, err = s.Write(payload)
if err != nil {
log.Println(err)
return
}
So we send one byte with the length of the payload before the actual payload.
Echo
The doEcho would then read the header first and afterwards the payload. It uses ReadFull, which reads exactly len(payload) bytes.
func doEcho(s network.Stream) error {
buf := bufio.NewReader(s)
header, err := buf.ReadByte()
if err != nil {
return err
}
payload := make([]byte, header)
n, err := io.ReadFull(buf, payload)
log.Printf("payload has %d bytes", n)
if err != nil {
return err
}
log.Printf("read: %s", payload)
_, err = s.Write(payload)
return err
}
Test
Terminal 1
2022/11/06 09:59:38 I am /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e
2022/11/06 09:59:38 listening for connections
2022/11/06 09:59:38 Now run "./echo -l 8089 -d /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e" on a different terminal
2022/11/06 09:59:55 listener received new stream
2022/11/06 09:59:55 payload has 13 bytes
2022/11/06 09:59:55 read: Hello, world!
Terminal 2
stephan#mac echo % ./echo -l 8089 -d /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e
2022/11/06 09:59:55 I am /ip4/127.0.0.1/tcp/8089/p2p/QmW6iSWiFBG5ugUUwBND14pDZzLDaqSNfxBG6yb8cmL3Di
2022/11/06 09:59:55 sender opening stream
2022/11/06 09:59:55 sender saying hello
2022/11/06 09:59:55 read reply: "Hello, world!"
s
This is certainly a fairly simple example and will certainly need to be customized to your actual requirements, but could perhaps be a first step in the right direction.

Related

How I can receive data for ever from TCP server

I try to create TCP client to receive data from TCP server,
but after server sending data only I receive data one even if server send many data, and I want to receive data forever, and I don't know what is my problem,and
Client:
func main() {
tcpAddr := "localhost:3333"
conn, err := net.DialTimeout("tcp", tcpAddr, time.Second*7)
if err != nil {
log.Println(err)
}
defer conn.Close()
// conn.Write([]byte("Hello World"))
connBuf := bufio.NewReader(conn)
for {
bytes, err := connBuf.ReadBytes('\n')
if err != nil {
log.Println("Rrecv Error:", err)
}
if len(bytes) > 0 {
fmt.Println(string(bytes))
}
time.Sleep(time.Second * 2)
}
}
I'm following this example to create TCP test server
Server:
// Handles incoming requests.
func handleRequest(conn net.Conn) {
// Make a buffer to hold incoming data.
buf := make([]byte, 1024)
// Read the incoming connection into the buffer.
_, err := conn.Read(buf)
if err != nil {
fmt.Println("Error reading:", err.Error())
}
fmt.Println(buf)
// Send a response back to person contacting us.
var msg string
fmt.Scanln(&msg)
conn.Write([]byte(msg))
// Close the connection when you're done with it.
conn.Close()
}
Read requires a Write on the other side of the connection
want to receive data forever
Then you have to send data forever. There's a for loop on the receiving end, but no looping on the sending end. The server writes its message once and closes the connection.
Server expects to get msg from client but client doesn't send it
// conn.Write([]byte("Hello World"))
That's supposed to provide the msg value to the server
_, err := conn.Read(buf)
So those two lines don't match.
Client expects a newline but server isn't sending one
fmt.Scanln expects to put each whitespace separated value into the corresponding argument. It does not capture the whitespace. So:
Only up to the first whitespace of what you type into server's stdin will be stored in msg
Newline will not be stored in msg.
But your client is doing
bytes, err := connBuf.ReadBytes('\n')
The \n never comes. The client never gets done reading that first msg.
bufio.NewScanner would be a better way to collect data from stdin, since you're likely to want to capture whitespace as well. Don't forget to append the newline to each line of text you send, because the client expects it!
Working code
I put these changes together into a working example on the playground. To get it working in that context, I had to make a few other changes too.
Running server and client in the same process
Hard coded 3 clients so the program ended in limited amount of time
Hard coded 10 receives in the client so program can end
Hard coded 3 server connections handled so program can end
Removed fmt.Scanln and have server just return the original message sent (because playground provides no stdin mechanism)
Should be enough to get you started.

How to determine the exact byte length of connection in golang?

I have the following code:
var buf []byte
read_len, err := conn.Read(buf)
if err != nil {
fmt.Println("Error reading:", err.Error())
}
buffer := make([]byte, read_len)
_, err = conn.Read(buffer)
if err != nil {
fmt.Println("Error reading:", err.Error())
}
The intention was to determine read_len with the first buf, then create a second buffer which is the exact length of an incoming json request. This just results in an error
unexpected end of JSON input
When I try to unmarshal
var request Device_Type_Request_Struct
err = json.Unmarshal(buffer, &request)
I'm assuming that this error occurs because the conn.Read(buffer) is returning nothing because another buffer has already read it (not sure though). How should I go about determining the length of json request while also being able to read it into a buffer (of the exact same length)?
Read returns the number of bytes read to the buffer. Because the length of the buffer passed to the first call to conn.Read is zero, the first call to conn.Read always returns zero.
There is no way to determine how much data a peer has sent without reading the data.
The easy solution to this problem is to use the JSON decoder:
d := json.NewDecoder(conn)
var request Device_Type_Request_Struct
if err := d.Decode(&request); err != nil {
// handle error
}
The decoder reads and decodes JSON values from a stream.

Golang reading from serial

I'm trying to read from a serial port (a GPS device on a Raspberry Pi).
Following the instructions from http://www.modmypi.com/blog/raspberry-pi-gps-hat-and-python
I can read from shell using
stty -F /dev/ttyAMA0 raw 9600 cs8 clocal -cstopb
cat /dev/ttyAMA0
I get well formatted output
$GNGLL,5133.35213,N,00108.27278,W,160345.00,A,A*65
$GNRMC,160346.00,A,5153.35209,N,00108.27286,W,0.237,,290418,,,A*75
$GNVTG,,T,,M,0.237,N,0.439,K,A*35
$GNGGA,160346.00,5153.35209,N,00108.27286,W,1,12,0.67,81.5,M,46.9,M,,*6C
$GNGSA,A,3,29,25,31,20,26,23,21,16,05,27,,,1.11,0.67,0.89*10
$GNGSA,A,3,68,73,83,74,84,75,85,67,,,,,1.11,0.67,0.89*1D
$GPGSV,4,1,15,04,,,34,05,14,040,21,09,07,330,,16,45,298,34*40
$GPGSV,4,2,15,20,14,127,18,21,59,154,30,23,07,295,26,25,13,123,22*74
$GPGSV,4,3,15,26,76,281,40,27,15,255,20,29,40,068,19,31,34,199,33*7C
$GPGSV,4,4,15,33,29,198,,36,23,141,,49,30,172,*4C
$GLGSV,3,1,11,66,00,325,,67,13,011,20,68,09,062,16,73,12,156,21*60
$GLGSV,3,2,11,74,62,177,20,75,53,312,36,76,08,328,,83,17,046,25*69
$GLGSV,3,3,11,84,75,032,22,85,44,233,32,,,,35*62
$GNGLL,5153.35209,N,00108.27286,W,160346.00,A,A*6C
$GNRMC,160347.00,A,5153.35205,N,00108.27292,W,0.216,,290418,,,A*7E
$GNVTG,,T,,M,0.216,N,0.401,K,A*3D
$GNGGA,160347.00,5153.35205,N,00108.27292,W,1,12,0.67,81.7,M,46.9,M,,*66
$GNGSA,A,3,29,25,31,20,26,23,21,16,05,27,,,1.11,0.67,0.89*10
$GNGSA,A,3,68,73,83,74,84,75,85,67,,,,,1.11,0.67,0.89*1D
$GPGSV,4,1,15,04,,,34,05,14,040,21,09,07,330,,16,45,298,34*40
(I've put some random data in)
I'm trying to read this in Go. Currently, I have
package main
import "fmt"
import "log"
import "github.com/tarm/serial"
func main() {
config := &serial.Config{
Name: "/dev/ttyAMA0",
Baud: 9600,
ReadTimeout: 1,
Size: 8,
}
stream, err := serial.OpenPort(config)
if err != nil {
log.Fatal(err)
}
buf := make([]byte, 1024)
for {
n, err := stream.Read(buf)
if err != nil {
log.Fatal(err)
}
s := string(buf[:n])
fmt.Println(s)
}
}
But this prints malformed data. I suspect that this is due to the buffer size or the value of Size in the config struct being wrong, but I'm not sure how to get those values from the stty settings.
Looking back, I think the issue is that I'm getting a stream and I want to be able to iterate over lines of the stty, rather than chunks. This is how the stream is outputted:
$GLGSV,3
,1,09,69
,10,017,
,70,43,0
69,,71,3
2,135,27
,76,23,2
32,22*6F
$GLGSV
,3,2,09,
77,35,30
0,21,78,
11,347,,
85,31,08
1,30,86,
72,355,3
6*6C
$G
LGSV,3,3
,09,87,2
4,285,30
*59
$GN
GLL,5153
.34919,N
,00108.2
7603,W,1
92901.00
,A,A*6A
The struct you get back from serial.OpenPort() contains a pointer to an open os.File corresponding to the opened serial port connection. When you Read() from this, the library calls Read() on the underlying os.File.
The documentation for this function call is:
Read reads up to len(b) bytes from the File. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF.
This means you have to keep track of how much data was read. You also have to keep track of whether there were newlines, if this is important to you. Unfortunately, the underlying *os.File is not exported, so you'll find it difficult to use tricks like bufio.ReadLine(). It may be worth modifying the library and sending a pull request.
As Matthew Rankin noted in a comment, Port implements io.ReadWriter so you can simply use bufio to read by lines.
stream, err := serial.OpenPort(config)
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(stream)
for scanner.Scan() {
fmt.Println(scanner.Text()) // Println will add back the final '\n'
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
Change
fmt.Println(s)
to
fmt.Print(s)
and you will probably get what you want.
Or did I misunderstand the question?
Two additions to Michael Hamptom's answer which can be useful:
line endings
You might receive data that is not newline-separated text. bufio.Scanner uses ScanLines by default to split the received data into lines - but you can also write your own line splitter based on the default function's signature and set it for the scanner:
scanner := bufio.NewScanner(stream)
scanner.Split(ownLineSplitter) // set custom line splitter function
reader shutdown
You might not receive a constant stream but only some packets of bytes from time to time. If no bytes arrive at the port, the scanner will block and you can't just kill it. You'll have to close the stream to do so, effectively raising an error. To not block any outer loops and handle errors appropriately, you can wrap the scanner in a goroutine that takes a context. If the context was cancelled, ignore the error, otherwise forward the error. In principle, this can look like
var errChan = make(chan error)
var dataChan = make(chan []byte)
ctx, cancelPortScanner := context.WithCancel(context.Background())
go func(ctx context.Context) {
scanner := bufio.NewScanner(stream)
for scanner.Scan() { // will terminate if connection is closed
dataChan <- scanner.Bytes()
}
// if execution reaches this point, something went wrong or stream was closed
select {
case <-ctx.Done():
return // ctx was cancelled, just return without error
default:
errChan <- scanner.Err() // ctx wasn't cancelled, forward error
}
}(ctx)
// handle data from dataChan, error from errChan
To stop the scanner, you would cancel the context and close the connection:
cancelPortScanner()
stream.Close()

Trouble programming a client that allows sending text messages to server

So I've programmed a server that receives text messages from a connecting client, reverses and capses them and sends them back.
Now I'm trying to program a client so that when I launch it it will keep running until I shut it down (ctrl + c) and allow me to input text lines and send them to the server.
I have a problem though - if I pass a, say, cyrillic symbol to the input, it will return a <nil> <nil> (type, value) error and will remain bugged unless I flush the memory somehow.
I also can't figure how to read the whole message (whole meaning the size of the slice (1024 bytes)) instead of each word separately.
Also, how do I figure out how to delay my 'enter your message' text? Depending on the length of the message I pass on to the server, it should wait longer or shorter. I don't want it popping all over the place if the message is split into a few messages, just once after the answer is received.
Here's the relevant code:
func client() {
// connect to the server
c, err := net.Dial("tcp", "127.0.0.1"+":"+port)
if err != nil {
log.Printf("Dial error: %T %+v", err, err)
return
}
// send the message
msg := ""
for {
fmt.Print("Enter your message:\n")
_, errs := fmt.Scan(&msg)
if errs != nil {
log.Printf("Scan error: %T %+v", errs, errs)
return
}
fmt.Println("Client sending:\n", msg)
_, errw := c.Write([]byte(msg))
if errw != nil {
log.Printf("Write error: %T %+v", errw, errw)
return
}
// handle the response
go handleServerResponse(c)
time.Sleep(1 * time.Second)
}
func main() {
port = "9999"
// launch client
done := make(chan bool)
go client()
<-done // Block forever
}
I've used the empty channel to block the main() from ending.
How should I approach the 2 problems explained above?
Question answered by #JimB:
You're using fmt.Scan which scans space separated values. Don't use
that if you don't want to read each value separately. You can use
Scanln to read a line, or just read directly from stdin.

Server failing to parse packets when flooded too fast

Here's the error I get when I flood the server with too many packets per second:
2014/11/28 12:52:49 main.go:59: loading plugin: print
2014/11/28 12:52:49 main.go:86: starting server on 0.0.0.0:8080
2014/11/28 12:52:59 server.go:15: client has connected: 127.0.0.1:59146
2014/11/28 12:52:59 server.go:43: received data from client 127.0.0.1:59146: &main.Observation{SensorId:"1", Timestamp:1416492023}
2014/11/28 12:52:59 server.go:29: read error from 127.0.0.1:59146: zlib: invalid header
2014/11/28 12:52:59 server.go:18: closing connection to: 127.0.0.1:59146
It manages to decode one packet (sometimes, maybe 2 or 3) then errors out. Here's the code doing the flooding:
import socket
import struct
import json
import zlib
import time
def serialize(data):
data = json.dumps(data)
data = zlib.compress(data)
packet = struct.pack('!I', len(data))
packet += data
return len(data), packet
message = {
'sensor_id': '1',
'timestamp': 1416492023,
}
length, buffer = serialize([message])
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 8080))
while True:
client.send(buffer)
#time.sleep(0.0005)
When I uncomment the time.sleep() call, the server works fine. It seems too many packets/per second is killing the server. Why?
Here's the relevent Go code. The connection handler:
func (self *Server) handleConnection(connection net.Conn) {
for {
connection.SetReadDeadline(time.Now().Add(30 * time.Second))
observations, err := self.Protocol.Unserialize(connection)
if err != nil {
log.Printf("read error from %s: %s\n", connection.RemoteAddr(), err)
return
}
}
And here's the unserializer:
// Length Value protocol to read zlib compressed, JSON encoded packets.
type ProtocolV2 struct{}
func (self *ProtocolV2) Unserialize(packet io.Reader) ([]*Observation, error) {
var length uint32
if err := binary.Read(packet, binary.BigEndian, &length); err != nil {
return nil, err
}
buffer := make([]byte, length)
rawreader := bufio.NewReader(packet)
if _, err := rawreader.Read(buffer); err != nil {
return nil, err
}
bytereader := bytes.NewReader(buffer)
zreader, err := zlib.NewReader(bytereader)
if err != nil {
return nil, err
}
defer zreader.Close()
var observations []*Observation
decoder := json.NewDecoder(zreader)
if err := decoder.Decode(&observations); err != nil {
return nil, err
}
return observations, nil
}
It seems there is an error on client side in the Python script.
The return of client.send is not checked, so the script does not handle partial writes in a correct way. Basically, when the socket buffer is full, only part of the message will be written, resulting in the server unable to decode the message.
This code is broken, but adding the wait makes it work because it prevents the socket buffer to be full.
You can use client.sendall instead to ensure the write operations are complete.
More information in the Python documentation:
https://docs.python.org/2/library/socket.html
https://docs.python.org/2/howto/sockets.html#using-a-socket
Now in the Go server, there is also a similar problem. The documentation says:
Read reads data into p. It returns the number of bytes read into p. It calls Read at most once on the underlying Reader, hence n may be less than len(p). At EOF, the count will be zero and err will be io.EOF.
The rawreader.Read call may return less bytes than you expect. You may want to use the ReadFull() function of the io package to ensure the full message is read.

Resources