Go. Get error i/o timeout in server program - go
I wrote simply server program to received data form client. I little not understand what sometimes I get error read tcp4 IP:PORT i/o timeout from function
int, err := conn.Read([]byte) event time set in function SetDeadline() not was exceeded. I present some part of my code, but I think that this is will enough.
main loop where I receive data is below.
c := NewClient()
c.kickTime: time.Now()
func (c *Client) Listen(){
durationToClose := time.Minute*time.Duration(5),
c.conn.SetDeadline(c.kickTime.Add(c.durationToClose))
buffer := make([]byte, 1024)
for{
reqLen, err := c.conn.Read(buffer)
if err != nil || reqLen == 0 {
fmt.Printf(err)
break
}
if err = c.CheckData(buffer) ; err != nil{
fmt.Printf("something is bad")
}else{
result := c.PrepareDataToSendInOtherPlace(buffer)
go c.RecievedData(result)
}
c.conn.SetDeadline(c.kickTime.Add(c.durationToKick))
}
}
For me only suspicious can be additional function as PrepareDataToSendInOtherPlace() , CheckData() which may take some times CPU, and then new data was be send by client, and server at the time doing something else and rejects connect. This is only my supposition, but I'm not sure.
Syntax errors and undeclared variables aside, what you're showing us can't possibly be walking the Read/Write deadline forward indefinitely.
The longest this could run is until a fixed duration after the first time.Now() (c.kickTime.Add(c.durationToKick)). You probably want something like:
c.conn.SetDeadline(time.Now().Add(c.durationToKick))
Related
Golang ending (Binance) web service stream using go routine
I'm integrating Binance API into an existing system and while most parts a straight forward, the data streaming API hits my limited understanding of go-routines. I don't believe there is anything special in the golang SDK for Binance, but essentially I only need two functions, one that starts the data stream and processes events with the event handler given as a parameter and a second one that ends the data stream without actually shutting down the client as it would close all other connections. On a previous project, there were two message types for this, but the binance SDK uses an implementation that returns two go channels, one for errors and an another one, I guess from the name, for stopping the data stram.
The code I wrote for starting the data stream looks like this:
func startDataStream(symbol, interval string, wsKlineHandler futures.WsKlineHandler, errHandler futures.ErrHandler) (err error){
doneC, stopC, err := futures.WsKlineServe(symbol, interval, wsKlineHandler, errHandler)
if err != nil {
fmt.Println(err)
return err
}
return nil
}
This works as expected and streams data. A simple test verifies it:
func runWSDataTest() {
symbol := "BTCUSDT"
interval := "15m"
errHandler := func(err error) {fmt.Println(err)}
wsKlineHandler := func(event *futures.WsKlineEvent) {fmt.Println(event)}
_ = startDataStream(symbol, interval, wsKlineHandler, errHandler)
}
The thing that is not so clear to me, mainly due to incomplete understanding, really is how do I stop the stream. I think the returned stopC channel can be used to somehow issue a end singnal similar to, say, a sigterm on system level and then the stream should end.
Say, I have a stopDataStream function that takes a symbol as an argument
func stopDataStream(symbol){
}
Let's suppose I start 5 data streams for five symbols and now I want to stop just one of the streams. That begs the question of:
How do I track all those stopC channels?
Can I use a collection keyed with the symbol, pull the stopC channel, and then just issue a signal to end just that data stream?
How do I actually write into the stopC channel from the stop function?
Again, I don't think this is particularly hard, it's just I could not figure it out yet from the docs so any help would be appreciated.
Thank you
(Answer originally written by #Marvin.Hansen)
Turned out, just saving & closing the channel solved it all. I was really surprised how easy this is, but here is the code of the updated functions:
func startDataStream(symbol, interval string, wsKlineHandler futures.WsKlineHandler, errHandler futures.ErrHandler) (err error) {
_, stopC, err := futures.WsKlineServe(symbol, interval, wsKlineHandler, errHandler)
if err != nil {
fmt.Println(err)
return err
}
// just save the stop channel
chanMap[symbol] = stopC
return nil
}
And then, the stop function really becomes embarrassing trivial:
func stopDataStream(symbol string) {
stopC := chanMap[symbol] // load the stop channel for the symbol
close(stopC) // just close it.
}
Finally, testing it all out:
var (
chanMap map[string]chan struct{}
)
func runWSDataTest() {
chanMap = make(map[string]chan struct{})
symbol := "BTCUSDT"
interval := "15m"
errHandler := func(err error) { fmt.Println(err) }
wsKlineHandler := getKLineHandler()
println("Start stream")
_ = startDataStream(symbol, interval, wsKlineHandler, errHandler)
time.Sleep(3 * time.Second)
println("Stop stream")
stopDataStream(symbol)
time.Sleep(1 * time.Second)
}
This is it.
Why is my Go app not reading from sysfs like the busybox `cat` command?
Go 1.12 on Linux 4.19.93 armv6l.
Hardware is a raspberypi zero w (BCM2835) running a yocto linux image.
I've got a gpio driven SRF04 proximity sensor driven by the srf04 linux driver.
It works great over sysfs and the busybox shell.
# cat /sys/bus/iio/devices/iio:device0/in_distance_raw
1646
I've used Go before with IIO devices that support triggers and buffered output at high sample rates on this hardware platform. However for this application the srf04 driver doesn't implement those IIO features. Drat. I don't really feel like adding buffer / trigger support to the driver myself (at this time) since I do not have a need for a 'high' sample rate. A handful of pings per second should suffice for my purpose. I figure I'll calculate mean & std. dev. for a rolling window of data points and 'divine' the signal out of the noise.
So with that - I'd be perfectly happy to Read the bytes from the published sysfs file with Go.
Which brings me to the point of this post.
When I open the file for reading, and try to Read() any number of bytes, I always get a generic -EIO error.
func (s *Srf04) Read() (int, error) {
samp := make([]byte, 16)
f, err := os.OpenFile(s.readPath, OS.O_RDONLY, os.ModeDevice)
if err != nil {
return 0, err
}
defer f.Close()
n, err := f.Read(samp)
if err != nil {
// This block is always executed.
// The error is never a timeout, and always 'input/output error' (-EIO aka -5)
log.Fatal(err)
}
...
}
This seems like strange behavior to me.
So I decided to mess with using io.ReadFull. This yielded unreliable results.
func (s *Srf04) Read() (int, error) {
samp := make([]byte, 16)
f, err := os.OpenFile(s.readPath, OS.O_RDONLY, os.ModeDevice)
if err != nil {
return 0, err
}
defer f.Close()
for {
n, err := io.ReadFull(readFile, samp)
log.Println("ReadFull ", n, " bytes.")
if err == io.EOF {
break
}
if err != nil {
log.Println(err)
}
}
...
}
I ended up adding it to a loop, as I found behavior changes from 'one-off' reads to multiple read calls subsequent to one another. I have it exiting if it gets an EOF, and repeatedly trying to read otherwise.
The results are straight-up crazy unreliable, seemingly returning random results. Sometimes I get the -5, other times I read between 2 - 5 bytes from the device. Sometimes I get bytes without an eof file before the EOF. The bytes appear to represent character data for numbers (each rune is a rune between [0-9]) -- which I'd expect.
Aside: I expect this is related to file polling and the go blocking IO implementation, but I have no way to really tell.
As a temporary workaround, I decided try using os.exec, and now I get results I'd expect to see.
func (s *Srf04)Read() (int, error) {
out, err := exec.Command("cat", s.readPath).Output()
if err != nil {
return 0, err
}
return strconv.Atoi(string(out))
}
But Yick. os.exec. Yuck.
I'd try to run that cat whatever encantation under strace and then peer at what read(2) calls cat actually manages to do (including the number of bytes actually read), and then I'd try to re-create that behaviour in Go. My own sheer guess at the problem's cause is that the driver (or the sysfs layer) is not too well prepared to deal with certain access patterns. For a start, consider that GNU cat is not a simple-minded byte shoveler but is rather a reasonably tricky piece of software, which, among other things, considers optimal I/O block sizes for both input and output devices (if available), calls fadvise(2) etc. It's not that any of that gets actually used when you run it on your sysfs-exported file, but it may influence how the full stack (starting with the sysfs layer) performs in the case of using cat and with your code, respectively. Hence my advice: start with strace-ing the cat and then try to re-create its usage pattern in your Go code; then try to come up with a minimal subset of that, which works; then profoundly comment your code ;-)
I'm sure I've been looking at this too long tonight, and this code is probably terrible. That said, here's the snippet of what I came up with that works just as reliably as the busybox cat, but in Go.
The Srf04 struct carries a few things, the important bits are included below:
type Srf04 struct {
readBuf []byte `json:"-"`
readFile *os.File `json:"-"`
samples *ring.Ring `json:"-"`
}
func (s *Srf04) Read() (int, error) {
/** Reliable, but really really slow.
out, err := exec.Command("cat", s.readPath).Output()
if err != nil {
log.Fatal(err)
}
val, err := strconv.Atoi(string(out[:len(out) - 2]))
if err == nil {
s.samples.Value = val
s.samples = s.samples.Next()
}
*/
// Seek should tell us the new offset (0) and no err.
bytesRead := 0
_, err := s.readFile.Seek(0, 0)
// Loop until N > 0 AND err != EOF && err != timeout.
if err == nil {
n := 0
for {
n, err = s.readFile.Read(s.readBuf)
bytesRead += n
if os.IsTimeout(err) {
// bail out.
bytesRead = 0
break
}
if err == io.EOF {
// Success!
break
}
// Any other err means 'keep trying to read.'
}
}
if bytesRead > 0 {
val, err := strconv.Atoi(string(s.readBuf[:bytesRead-1]))
if err == nil {
fmt.Println(val)
s.samples.Value = val
s.samples = s.samples.Next()
}
return val, err
}
return 0, err
}
Golang reading from serial
I'm trying to read from a serial port (a GPS device on a Raspberry Pi).
Following the instructions from http://www.modmypi.com/blog/raspberry-pi-gps-hat-and-python
I can read from shell using
stty -F /dev/ttyAMA0 raw 9600 cs8 clocal -cstopb
cat /dev/ttyAMA0
I get well formatted output
$GNGLL,5133.35213,N,00108.27278,W,160345.00,A,A*65
$GNRMC,160346.00,A,5153.35209,N,00108.27286,W,0.237,,290418,,,A*75
$GNVTG,,T,,M,0.237,N,0.439,K,A*35
$GNGGA,160346.00,5153.35209,N,00108.27286,W,1,12,0.67,81.5,M,46.9,M,,*6C
$GNGSA,A,3,29,25,31,20,26,23,21,16,05,27,,,1.11,0.67,0.89*10
$GNGSA,A,3,68,73,83,74,84,75,85,67,,,,,1.11,0.67,0.89*1D
$GPGSV,4,1,15,04,,,34,05,14,040,21,09,07,330,,16,45,298,34*40
$GPGSV,4,2,15,20,14,127,18,21,59,154,30,23,07,295,26,25,13,123,22*74
$GPGSV,4,3,15,26,76,281,40,27,15,255,20,29,40,068,19,31,34,199,33*7C
$GPGSV,4,4,15,33,29,198,,36,23,141,,49,30,172,*4C
$GLGSV,3,1,11,66,00,325,,67,13,011,20,68,09,062,16,73,12,156,21*60
$GLGSV,3,2,11,74,62,177,20,75,53,312,36,76,08,328,,83,17,046,25*69
$GLGSV,3,3,11,84,75,032,22,85,44,233,32,,,,35*62
$GNGLL,5153.35209,N,00108.27286,W,160346.00,A,A*6C
$GNRMC,160347.00,A,5153.35205,N,00108.27292,W,0.216,,290418,,,A*7E
$GNVTG,,T,,M,0.216,N,0.401,K,A*3D
$GNGGA,160347.00,5153.35205,N,00108.27292,W,1,12,0.67,81.7,M,46.9,M,,*66
$GNGSA,A,3,29,25,31,20,26,23,21,16,05,27,,,1.11,0.67,0.89*10
$GNGSA,A,3,68,73,83,74,84,75,85,67,,,,,1.11,0.67,0.89*1D
$GPGSV,4,1,15,04,,,34,05,14,040,21,09,07,330,,16,45,298,34*40
(I've put some random data in)
I'm trying to read this in Go. Currently, I have
package main
import "fmt"
import "log"
import "github.com/tarm/serial"
func main() {
config := &serial.Config{
Name: "/dev/ttyAMA0",
Baud: 9600,
ReadTimeout: 1,
Size: 8,
}
stream, err := serial.OpenPort(config)
if err != nil {
log.Fatal(err)
}
buf := make([]byte, 1024)
for {
n, err := stream.Read(buf)
if err != nil {
log.Fatal(err)
}
s := string(buf[:n])
fmt.Println(s)
}
}
But this prints malformed data. I suspect that this is due to the buffer size or the value of Size in the config struct being wrong, but I'm not sure how to get those values from the stty settings.
Looking back, I think the issue is that I'm getting a stream and I want to be able to iterate over lines of the stty, rather than chunks. This is how the stream is outputted:
$GLGSV,3
,1,09,69
,10,017,
,70,43,0
69,,71,3
2,135,27
,76,23,2
32,22*6F
$GLGSV
,3,2,09,
77,35,30
0,21,78,
11,347,,
85,31,08
1,30,86,
72,355,3
6*6C
$G
LGSV,3,3
,09,87,2
4,285,30
*59
$GN
GLL,5153
.34919,N
,00108.2
7603,W,1
92901.00
,A,A*6A
The struct you get back from serial.OpenPort() contains a pointer to an open os.File corresponding to the opened serial port connection. When you Read() from this, the library calls Read() on the underlying os.File.
The documentation for this function call is:
Read reads up to len(b) bytes from the File. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF.
This means you have to keep track of how much data was read. You also have to keep track of whether there were newlines, if this is important to you. Unfortunately, the underlying *os.File is not exported, so you'll find it difficult to use tricks like bufio.ReadLine(). It may be worth modifying the library and sending a pull request.
As Matthew Rankin noted in a comment, Port implements io.ReadWriter so you can simply use bufio to read by lines.
stream, err := serial.OpenPort(config)
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(stream)
for scanner.Scan() {
fmt.Println(scanner.Text()) // Println will add back the final '\n'
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
Change fmt.Println(s) to fmt.Print(s) and you will probably get what you want. Or did I misunderstand the question?
Two additions to Michael Hamptom's answer which can be useful:
line endings
You might receive data that is not newline-separated text. bufio.Scanner uses ScanLines by default to split the received data into lines - but you can also write your own line splitter based on the default function's signature and set it for the scanner:
scanner := bufio.NewScanner(stream)
scanner.Split(ownLineSplitter) // set custom line splitter function
reader shutdown
You might not receive a constant stream but only some packets of bytes from time to time. If no bytes arrive at the port, the scanner will block and you can't just kill it. You'll have to close the stream to do so, effectively raising an error. To not block any outer loops and handle errors appropriately, you can wrap the scanner in a goroutine that takes a context. If the context was cancelled, ignore the error, otherwise forward the error. In principle, this can look like
var errChan = make(chan error)
var dataChan = make(chan []byte)
ctx, cancelPortScanner := context.WithCancel(context.Background())
go func(ctx context.Context) {
scanner := bufio.NewScanner(stream)
for scanner.Scan() { // will terminate if connection is closed
dataChan <- scanner.Bytes()
}
// if execution reaches this point, something went wrong or stream was closed
select {
case <-ctx.Done():
return // ctx was cancelled, just return without error
default:
errChan <- scanner.Err() // ctx wasn't cancelled, forward error
}
}(ctx)
// handle data from dataChan, error from errChan
To stop the scanner, you would cancel the context and close the connection:
cancelPortScanner()
stream.Close()
Trouble programming a client that allows sending text messages to server
So I've programmed a server that receives text messages from a connecting client, reverses and capses them and sends them back.
Now I'm trying to program a client so that when I launch it it will keep running until I shut it down (ctrl + c) and allow me to input text lines and send them to the server.
I have a problem though - if I pass a, say, cyrillic symbol to the input, it will return a <nil> <nil> (type, value) error and will remain bugged unless I flush the memory somehow.
I also can't figure how to read the whole message (whole meaning the size of the slice (1024 bytes)) instead of each word separately.
Also, how do I figure out how to delay my 'enter your message' text? Depending on the length of the message I pass on to the server, it should wait longer or shorter. I don't want it popping all over the place if the message is split into a few messages, just once after the answer is received.
Here's the relevant code:
func client() {
// connect to the server
c, err := net.Dial("tcp", "127.0.0.1"+":"+port)
if err != nil {
log.Printf("Dial error: %T %+v", err, err)
return
}
// send the message
msg := ""
for {
fmt.Print("Enter your message:\n")
_, errs := fmt.Scan(&msg)
if errs != nil {
log.Printf("Scan error: %T %+v", errs, errs)
return
}
fmt.Println("Client sending:\n", msg)
_, errw := c.Write([]byte(msg))
if errw != nil {
log.Printf("Write error: %T %+v", errw, errw)
return
}
// handle the response
go handleServerResponse(c)
time.Sleep(1 * time.Second)
}
func main() {
port = "9999"
// launch client
done := make(chan bool)
go client()
<-done // Block forever
}
I've used the empty channel to block the main() from ending.
How should I approach the 2 problems explained above?
Question answered by #JimB: You're using fmt.Scan which scans space separated values. Don't use that if you don't want to read each value separately. You can use Scanln to read a line, or just read directly from stdin.
How to place http.Serve in its own goroutine if it blocks?
http.Serve either returns an error as soon as it is called or blocks if successfully executing.
How can I make it so that if it blocks it does so in its own goroutine? I currently have the following code:
func serveOrErr(l net.Listener, handler http.Handler) error {
starting := make(chan struct{})
serveErr := make(chan error)
go func() {
starting <- struct{}{}
if err := http.Serve(l, handler); err != nil {
serveErr <- err
}
}()
<-starting
select {
case err := <-serveErr:
return err
default:
return nil
}
}
This seemed like a good start and works on my test machine but I believe that there are no guarantees that serveErr <- err would be called before case err := <-serveErr therefore leading to inconsistent results due to a data race if http.Serve were to produce an error.
http.Serve either returns an error as soon as it is called or blocks if successfully executing This assumption is not correct. And I believe it rarely occurs. http.Serve calls net.Listener.Accept in the loop – an error can occur any time (socket closed, too many open file descriptors etc.). It's http.ListenAndServe, usually being used for running http servers, which often fails early while binding listening socket (no permissions, address already in use). In my opinion what you're trying to do is wrong, unless really your net.Listener.Accept is failing on the first call for some reason. Is it? If you want to be 100% sure your server is working, you could try to connect to it (and maybe actually transmit something), but once you successfully bound the socket I don't see it really necessary.
You could use a timeout on your select statement, e.g.
timeout := time.After(5 * time.Millisecond) // TODO: ajust the value
select {
case err := <-serveErr:
return err
case _ := <- timeout:
return nil
}
This way your select will block until serveErr has a value or the specified timeout has elapsed. Note that the execution of your function will therefore block the calling goroutine for up to the duration of the specified timeout.
Rob Pike's excellent talk on go concurrency patterns might be helpful.