Websockets Read/Write - websocket
Go/websockets noob here.
I've modified this working example in such way that instead of
io.Copy(ws,ws)
I am doing
msg := []byte{}
_, err := ws.Read(msg)
_, err = ws.Write(msg)
(printing errors omitted for brevity)
To me it looks the same, but it does not work. It does not read anything (though does not return error either). And on the client side (Chrome/FF) I get "WebSocket is already in CLOSING or CLOSED state" when trying to send second message.
In general, what is the approach for read msg -> do something -> write msg for ws in golang?
When doing this, you read and write 0 bytes, you need to allocate an actual buffer size. And also, it needs to be within a loop.
ws.Read() will not allocate a buffer for you and will read only len(msg), In your example, it is 0.
Cf io.Copy sources:
buf := make([]byte, 32*1024)
for {
nr, er := src.Read(buf)
if nr > 0 {
nw, ew := dst.Write(buf[0:nr])
Related
go-libp2p - receiving bytes from stream
I'm building my first go-libp2p application and trying to modify the echo example to read a []byte instead of a string as in the example.
In my code, I changed the doEcho function to run io.ReadAll(s) instead of bufio.NewReader(s) followed by ReadString('\n'):
// doEcho reads a line of data a stream and writes it back
func doEcho(s network.Stream) error {
b, err := io.ReadAll(s)
if err != nil {
return err
}
log.Printf("Number of bytes received: %d", len(b))
_, err = s.Write([]byte("thanks for the bytes"))
return err
}
When I run this and send a message, I do see the listener received new stream log but the doEcho function gets stuck after the io.ReadAll(s) call and never executes the reply.
So my questions are:
Why does my code not work and how can I make it work?
How does io.ReadAll(s) and bufio's ReadString('\n') work under the hood so that they cause this difference in behavior?
Edit:
As per #Stephan Schlecht suggestion I changed my code to this, but it still remains blocked as before:
func doEcho(s network.Stream) error {
buf := bufio.NewReader(s)
var data []byte
for {
b, err := buf.ReadByte()
if err != nil {
break
}
data = append(data, b)
}
log.Printf("Number of bytes received: %d", len(data))
_, err := s.Write([]byte("thanks for the bytes"))
return err
}
Edit 2: I forgot to clarify this, but I don't want to use ReadString('\n') or ReadBytes('\n') because I don't know anything about the []byte I'm receiving, so it might not end with \n. I want to read any []byte from the stream and then write back to the stream.
ReadString('\n') reads until the first occurrence of \n in the input and returns the string.
io.ReadAll(s) reads until an error or EOF and returns the data it read. So unless an error or EOF occurs it does not return.
In principle, there is no natural size for a data structure to be received on stream-oriented connections.
It depends on the remote sender.
If the remote sender sends binary data and closes the stream after sending the last byte, then you can simply read all data up to the EOF on the receiver side.
If the stream is not to be closed immediately and the data size is variable, there are further possibilities: One first sends a header that has a defined size and in the simplest case simply transmits the length of the data. Once you have received the specified amount of data, you know that this round of reception is complete and you can continue.
Alternatively, you can define a special character that marks the end of the data structure to be transmitted. This will not work if you want to transmit arbitrary binary data without encoding.
There are other options that are a little more complicated, such as splitting the data into blocks.
In the example linked in the question, a \n is sent at the end of the data just sent, but this would not work if you want to send arbitrary binary data.
Adapted Echo Example
In order to minimally modify the echo example linked in the question to first send a 1-byte header with the length of the payload and only then the actual payload, it could look something like the following:
Sending
In the function runSender line one could replace the current sending of the payload from:
log.Println("sender saying hello")
_, err = s.Write([]byte("Hello, world!\n"))
if err != nil {
log.Println(err)
return
}
to
log.Println("sender saying hello")
payload := []byte("Hello, world!")
header := []byte{byte(len(payload))}
_, err = s.Write(header)
if err != nil {
log.Println(err)
return
}
_, err = s.Write(payload)
if err != nil {
log.Println(err)
return
}
So we send one byte with the length of the payload before the actual payload.
Echo
The doEcho would then read the header first and afterwards the payload. It uses ReadFull, which reads exactly len(payload) bytes.
func doEcho(s network.Stream) error {
buf := bufio.NewReader(s)
header, err := buf.ReadByte()
if err != nil {
return err
}
payload := make([]byte, header)
n, err := io.ReadFull(buf, payload)
log.Printf("payload has %d bytes", n)
if err != nil {
return err
}
log.Printf("read: %s", payload)
_, err = s.Write(payload)
return err
}
Test
Terminal 1
2022/11/06 09:59:38 I am /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e
2022/11/06 09:59:38 listening for connections
2022/11/06 09:59:38 Now run "./echo -l 8089 -d /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e" on a different terminal
2022/11/06 09:59:55 listener received new stream
2022/11/06 09:59:55 payload has 13 bytes
2022/11/06 09:59:55 read: Hello, world!
Terminal 2
stephan#mac echo % ./echo -l 8089 -d /ip4/127.0.0.1/tcp/8088/p2p/QmVrjAX9QPqihfVFEPJ2apRSUxVCE9wnvqaWanBz2FLY1e
2022/11/06 09:59:55 I am /ip4/127.0.0.1/tcp/8089/p2p/QmW6iSWiFBG5ugUUwBND14pDZzLDaqSNfxBG6yb8cmL3Di
2022/11/06 09:59:55 sender opening stream
2022/11/06 09:59:55 sender saying hello
2022/11/06 09:59:55 read reply: "Hello, world!"
s
This is certainly a fairly simple example and will certainly need to be customized to your actual requirements, but could perhaps be a first step in the right direction.
Concurrently send bytes coming from stream to two destinations
What's the best way to broadcast values from a stream to two network destination simultaneously? Here is simplified code:
func main() {
resp, _ := http.Get("http://origin.com/image.jpeg")
var buf bytes.Buffer
// tee, when read, writes to &buf
respBodyTee := io.TeeReader(resp.Body, &buf)
sendToClient(respBodyTee)
uploadeToServer(&buf)
}
A stream cannot be read twice, so TeeReader is used to populate &buf with whatever is read from respo.Body.
However, the functions (sendToClient and uploadToServer) will run synchronously while I'd like to them to make their work concurrently.
Solution that I have on mind is to pass a channel to sendToClient that will populate channel with bytes already sent to client. Later have uploadToServer read from the same channel. Something along these lines:
func main() {
resp, _ := http.Get("http://origin.com/image.jpeg")
ch := make(chan byte)
go sendToClient(respBodyTee, ch) // pass 'ch' for writing and run in a goroutine
uploadeToServer(ch) // will read from 'ch' (synchronous)
}
I'm new to Go and am not sure if the above is the right direction.
in your scenario, it's better to have 2 independent byte streams for 2 network calls. If they rely on one source stream, when sendToClient stalls uploadeToServer will hang. channel wouldn't solve the problem above but introducing locking overhead.
you can try io.MultiWriter to make 2 independent byte streams
var buf1, buf2 bytes.Buffer
mw := io.MultiWriter(&buf1, &buf2)
if _, err := io.Copy(mw, r.Body); err != nil {
...
}
go sendToClient(buf1)
go uploadeToServer(buf2)
...
Golang reading from serial
I'm trying to read from a serial port (a GPS device on a Raspberry Pi).
Following the instructions from http://www.modmypi.com/blog/raspberry-pi-gps-hat-and-python
I can read from shell using
stty -F /dev/ttyAMA0 raw 9600 cs8 clocal -cstopb
cat /dev/ttyAMA0
I get well formatted output
$GNGLL,5133.35213,N,00108.27278,W,160345.00,A,A*65
$GNRMC,160346.00,A,5153.35209,N,00108.27286,W,0.237,,290418,,,A*75
$GNVTG,,T,,M,0.237,N,0.439,K,A*35
$GNGGA,160346.00,5153.35209,N,00108.27286,W,1,12,0.67,81.5,M,46.9,M,,*6C
$GNGSA,A,3,29,25,31,20,26,23,21,16,05,27,,,1.11,0.67,0.89*10
$GNGSA,A,3,68,73,83,74,84,75,85,67,,,,,1.11,0.67,0.89*1D
$GPGSV,4,1,15,04,,,34,05,14,040,21,09,07,330,,16,45,298,34*40
$GPGSV,4,2,15,20,14,127,18,21,59,154,30,23,07,295,26,25,13,123,22*74
$GPGSV,4,3,15,26,76,281,40,27,15,255,20,29,40,068,19,31,34,199,33*7C
$GPGSV,4,4,15,33,29,198,,36,23,141,,49,30,172,*4C
$GLGSV,3,1,11,66,00,325,,67,13,011,20,68,09,062,16,73,12,156,21*60
$GLGSV,3,2,11,74,62,177,20,75,53,312,36,76,08,328,,83,17,046,25*69
$GLGSV,3,3,11,84,75,032,22,85,44,233,32,,,,35*62
$GNGLL,5153.35209,N,00108.27286,W,160346.00,A,A*6C
$GNRMC,160347.00,A,5153.35205,N,00108.27292,W,0.216,,290418,,,A*7E
$GNVTG,,T,,M,0.216,N,0.401,K,A*3D
$GNGGA,160347.00,5153.35205,N,00108.27292,W,1,12,0.67,81.7,M,46.9,M,,*66
$GNGSA,A,3,29,25,31,20,26,23,21,16,05,27,,,1.11,0.67,0.89*10
$GNGSA,A,3,68,73,83,74,84,75,85,67,,,,,1.11,0.67,0.89*1D
$GPGSV,4,1,15,04,,,34,05,14,040,21,09,07,330,,16,45,298,34*40
(I've put some random data in)
I'm trying to read this in Go. Currently, I have
package main
import "fmt"
import "log"
import "github.com/tarm/serial"
func main() {
config := &serial.Config{
Name: "/dev/ttyAMA0",
Baud: 9600,
ReadTimeout: 1,
Size: 8,
}
stream, err := serial.OpenPort(config)
if err != nil {
log.Fatal(err)
}
buf := make([]byte, 1024)
for {
n, err := stream.Read(buf)
if err != nil {
log.Fatal(err)
}
s := string(buf[:n])
fmt.Println(s)
}
}
But this prints malformed data. I suspect that this is due to the buffer size or the value of Size in the config struct being wrong, but I'm not sure how to get those values from the stty settings.
Looking back, I think the issue is that I'm getting a stream and I want to be able to iterate over lines of the stty, rather than chunks. This is how the stream is outputted:
$GLGSV,3
,1,09,69
,10,017,
,70,43,0
69,,71,3
2,135,27
,76,23,2
32,22*6F
$GLGSV
,3,2,09,
77,35,30
0,21,78,
11,347,,
85,31,08
1,30,86,
72,355,3
6*6C
$G
LGSV,3,3
,09,87,2
4,285,30
*59
$GN
GLL,5153
.34919,N
,00108.2
7603,W,1
92901.00
,A,A*6A
The struct you get back from serial.OpenPort() contains a pointer to an open os.File corresponding to the opened serial port connection. When you Read() from this, the library calls Read() on the underlying os.File.
The documentation for this function call is:
Read reads up to len(b) bytes from the File. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF.
This means you have to keep track of how much data was read. You also have to keep track of whether there were newlines, if this is important to you. Unfortunately, the underlying *os.File is not exported, so you'll find it difficult to use tricks like bufio.ReadLine(). It may be worth modifying the library and sending a pull request.
As Matthew Rankin noted in a comment, Port implements io.ReadWriter so you can simply use bufio to read by lines.
stream, err := serial.OpenPort(config)
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(stream)
for scanner.Scan() {
fmt.Println(scanner.Text()) // Println will add back the final '\n'
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
Change fmt.Println(s) to fmt.Print(s) and you will probably get what you want. Or did I misunderstand the question?
Two additions to Michael Hamptom's answer which can be useful:
line endings
You might receive data that is not newline-separated text. bufio.Scanner uses ScanLines by default to split the received data into lines - but you can also write your own line splitter based on the default function's signature and set it for the scanner:
scanner := bufio.NewScanner(stream)
scanner.Split(ownLineSplitter) // set custom line splitter function
reader shutdown
You might not receive a constant stream but only some packets of bytes from time to time. If no bytes arrive at the port, the scanner will block and you can't just kill it. You'll have to close the stream to do so, effectively raising an error. To not block any outer loops and handle errors appropriately, you can wrap the scanner in a goroutine that takes a context. If the context was cancelled, ignore the error, otherwise forward the error. In principle, this can look like
var errChan = make(chan error)
var dataChan = make(chan []byte)
ctx, cancelPortScanner := context.WithCancel(context.Background())
go func(ctx context.Context) {
scanner := bufio.NewScanner(stream)
for scanner.Scan() { // will terminate if connection is closed
dataChan <- scanner.Bytes()
}
// if execution reaches this point, something went wrong or stream was closed
select {
case <-ctx.Done():
return // ctx was cancelled, just return without error
default:
errChan <- scanner.Err() // ctx wasn't cancelled, forward error
}
}(ctx)
// handle data from dataChan, error from errChan
To stop the scanner, you would cancel the context and close the connection:
cancelPortScanner()
stream.Close()
Go. Get error i/o timeout in server program
I wrote simply server program to received data form client. I little not understand what sometimes I get error read tcp4 IP:PORT i/o timeout from function
int, err := conn.Read([]byte) event time set in function SetDeadline() not was exceeded. I present some part of my code, but I think that this is will enough.
main loop where I receive data is below.
c := NewClient()
c.kickTime: time.Now()
func (c *Client) Listen(){
durationToClose := time.Minute*time.Duration(5),
c.conn.SetDeadline(c.kickTime.Add(c.durationToClose))
buffer := make([]byte, 1024)
for{
reqLen, err := c.conn.Read(buffer)
if err != nil || reqLen == 0 {
fmt.Printf(err)
break
}
if err = c.CheckData(buffer) ; err != nil{
fmt.Printf("something is bad")
}else{
result := c.PrepareDataToSendInOtherPlace(buffer)
go c.RecievedData(result)
}
c.conn.SetDeadline(c.kickTime.Add(c.durationToKick))
}
}
For me only suspicious can be additional function as PrepareDataToSendInOtherPlace() , CheckData() which may take some times CPU, and then new data was be send by client, and server at the time doing something else and rejects connect. This is only my supposition, but I'm not sure.
Syntax errors and undeclared variables aside, what you're showing us can't possibly be walking the Read/Write deadline forward indefinitely. The longest this could run is until a fixed duration after the first time.Now() (c.kickTime.Add(c.durationToKick)). You probably want something like: c.conn.SetDeadline(time.Now().Add(c.durationToKick))
Why does conn.Read() write nothing into a []byte, but bufio.Reader.ReadString() works?
I have a connection, created like this:
conn, err = net.Dial("tcp", "127.0.0.1:20000")
I have tried reading from this connection in two ways. I think they both must work, but the first option doesn't.
Here is the first way of doing it:
var bytes []byte
for i := 0; i < 4; i++ {
conn.Read(bytes)
}
fmt.Printf("%v", bytes)
The output of this method is:
[]
And here is the same thing, done with bufio.Reader:
func readResponse(conn net.Conn) (response string, err error) {
reader := bufio.NewReader(conn)
_, err = reader.Discard(8)
if err != nil {
return
}
response, err = reader.ReadString('\n')
return
}
This function returns the response given by the server on the other end of the TCP connection.
Why does bufio.Reader.Read() work, but net.Conn.Read() doesn't?
The Conn.Read() method is to implement io.Reader, the general interface to read data from any source of bytes into a []byte. Quoting from the doc of Reader.Read(): Read reads up to len(p) bytes into p. So Read() reads up to len(p) bytes but since you pass a nil slice, it won't read anything (length of a nil slice is 0). Please read the linked doc to know how Reader.Read() works. Reader.Read() does not allocate a buffer ([]byte) where the read data will be stored, you have to create one and pass it, e.g.: var buf = make([]byte, 100) n, err := conn.Read(buf) // n is the number of read bytes; don't forget to check err! Don't forget to always check the returned error which may be io.EOF if end of data is reached. The general contract of io.Reader.Read() also allows returning some non-nil error (including io.EOF) and some read data (n > 0) at the same time. The number of read bytes will be in n, which means only the first n bytes of the buf is useful (in other words: buf[:n]). Your other example using bufio.Reader works because you called Reader.ReadString() which doesn't require a []byte argument. If you would've used the bufio.Reader.Read() method, you would also had to pass a non-nil slice in order to actually get some data.