Go io string to int Atoi invalid argument - websocket

I'm trying to parse a string from WebSockets connection in Go language. I'm implementing both sides of the connection, so the specification of data format is depending only on me.
As this is a simple app (generally for learning purposes), I've come up with ActionId Data, where ActionId is a uint8. BackendHandler is a handler for every request in WebSocket Connection.
Platform information
kuba:~$ echo {$GOARCH,$GOOS,`6g -V`}
amd64 linux 6g version release.r60.3 9516
code:
const ( // Specifies ActionId's
SabPause = iota
)
func BackendHandler(ws *websocket.Conn) {
buf := make([]byte, 512)
_, err := ws.Read(buf)
if err != nil { panic(err.String()) }
str := string(buf)
tmp, _ := strconv.Atoi(str[:0])
data := str[2:]
fmt.Println(tmp, data)
switch tmp {
case SabPause:
// Here I get `parsing "2": invalid argument`
// when passing "0 2" to websocket connection
minutes, ok := strconv.Atoui(data)
if ok != nil {
panic(ok.String())
}
PauseSab(uint8(minutes))
default:
panic("Unmatched input for BackendHandler")
}
}
All the output: (note the Println that I used for inspecting)
0 2
panic: parsing "2": invalid argument [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
I couldn't find the code from which this error is launch, only where the error code is defined (dependent on platform). I'd appreciate general ideas for improving my code, but mainly I just want to solve the conversion problem.
Is this related to my buffer -> string conversion and slice-manipulation(I didn't want to use SplitAfter methods)?
Edit
This code reproduces the problem:
package main
import (
"strconv"
"io/ioutil"
)
func main() {
buf , _ := ioutil.ReadFile("input")
str := string(buf)
_, ok := strconv.Atoui(str[2:])
if ok != nil {
panic(ok.String())
}
}
The file input has to contain 0 2\r\n (depending on the file ending, it may look different on other OSes). This code can be fixed by adding the ending index for reslice, this way:
_, ok := strconv.Atoui(str[2:3])

You didn't provide a small compilable and runnable program to illustrate your problem. Nor did you provide full and meaningful print diagnostic messages.
My best guess is that you have a C-style null-terminated string. For example, simplifying your code,
package main
import (
"fmt"
"strconv"
)
func main() {
buf := make([]byte, 512)
buf = []byte("0 2\x00") // test data
str := string(buf)
tmp, err := strconv.Atoi(str[:0])
if err != nil {
fmt.Println(err)
}
data := str[2:]
fmt.Println("tmp:", tmp)
fmt.Println("str:", len(str), ";", str, ";", []byte(str))
fmt.Println("data", len(data), ";", data, ";", []byte(data))
// Here I get `parsing "2": invalid argument`
// when passing "0 2" to websocket connection
minutes, ok := strconv.Atoui(data)
if ok != nil {
panic(ok.String())
}
_ = minutes
}
Output:
parsing "": invalid argument
tmp: 0
str: 4 ; 0 2 ; [48 32 50 0]
data 2 ; 2 ; [50 0]
panic: parsing "2": invalid argument
runtime.panic+0xac /home/peter/gor/src/pkg/runtime/proc.c:1254
runtime.panic(0x4492c0, 0xf840002460)
main.main+0x603 /home/peter/gopath/src/so/temp.go:24
main.main()
runtime.mainstart+0xf /home/peter/gor/src/pkg/runtime/amd64/asm.s:78
runtime.mainstart()
runtime.goexit /home/peter/gor/src/pkg/runtime/proc.c:246
runtime.goexit()
----- goroutine created by -----
_rt0_amd64+0xc9 /home/peter/gor/src/pkg/runtime/amd64/asm.s:65
If you add my print diagnostic statements to your code, what do you see?
Note that your tmp, _ := strconv.Atoi(str[:0]) statement is probably wrong, since str[:0] is equivalent to str[0:0], which is equivalent to the empty string "".
I suspect that your problem is that you are ignoring the n return value from ws.Read. For example (including diagnostic messages), I would expect,
buf := make([]byte, 512)
buf = buf[:cap(buf)]
n, err := ws.Read(buf)
if err != nil {
panic(err.String())
}
fmt.Println(len(buf), n)
buf = buf[:n]
fmt.Println(len(buf), n)
Also, try using this code to set tmp,
tmp, err := strconv.Atoi(str[:1])
if err != nil {
panic(err.String())
}

Related

Multiple serial requests result in empty buffer

The first TCP connection running on localhost on osx always parses the binary sent to it correctly. Subsequent requests lose the binary data, only seeing the first byte [8]. How have I failed to set up my Reader?
package main
import (
"fmt"
"log"
"net"
"os"
"app/src/internal/handler"
"github.com/golang-collections/collections/stack"
)
func main() {
port := os.Getenv("SERVER_PORT")
s := stack.New()
ln, err := net.Listen("tcp", ":8080")
if err != nil {
log.Fatalf("net.Listen: %v", err)
}
fmt.Println("Serving on " + port)
for {
conn, err := ln.Accept()
// defer conn.Close()
if err != nil {
log.Fatal("ln.Accept")
}
go handler.Handle(conn, s)
}
}
package handler
import (
"fmt"
"io"
"log"
"net"
"github.com/golang-collections/collections/stack"
)
func Handle(c net.Conn, s *stack.Stack) {
fmt.Printf("Serving %s\n", c.RemoteAddr().String())
buf := make([]byte, 0, 256)
tmp := make([]byte, 128)
n, err := c.Read(tmp)
if err != nil {
if err != io.EOF {
log.Fatalf("connection Read() %v", err)
}
return
}
buf = append(buf, tmp[:n]...)
}
log:
Serving [::1]:51699
------------- value ---------------:QCXhoy5t
Buffer Length: 9. First Value: 8
Serving [::1]:51700
------------- value ---------------:
Buffer Length: 1. First Value: 8
Serving [::1]:51701
test sent over:
push random string:
QCXhoy5t
push random string:
GPh0EnbS
push random string:
4kJ0wN0R
The docs for Reader say:
Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n
<= len(p)) and any error encountered. Even if Read returns n < len(p), it may use
all of p as scratch space during the call. If some data is available but not
len(p) bytes, Read conventionally returns what is available instead of waiting
for more.
So the most likely cause of your issue is that Read is returning the data available (in this case a single character). You can fix this by using ioutil.ReadAll or performing the read in a loop (the fact the data is being added to a buffer makes it look like that was the original intention) with something like:
for {
n, err := c.Read(tmp)
if err != nil {
if err != io.EOF {
// Note that data might have also been received - you should process that
// if appropriate.
log.Fatalf("connection Read() %v", err)
return
}
break // All data received so process it
}
buf = append(buf, tmp[:n]...)
}
Note: There is no guarantee that any data is received; you should check the length before trying to access it (i.e. buf[0] may panic)

Farm out work to a slice but limit number of workers

I'm trying to improve the performance of an app.
One part of its code uploads a file to a server in chunks.
The original version simply does this in a sequential loop. However, it's slow and during the sequence it also needs to talk to another server before uploading each chunk.
The upload of chunks could simply be placed in a goroutine. It works, but is not a good solution because if the source file is extremely large it ends up using a large amount of memory.
So, I try to limit the number of active goroutines by using a buffered channel. Here is some code that shows my attempt. I've stripped it down to show the concept and you can run it to test for yourself.
package main
import (
"fmt"
"io"
"os"
"time"
)
const defaultChunkSize = 1 * 1024 * 1024
// Lets have 4 workers
var c = make(chan int, 4)
func UploadFile(f *os.File) error {
fi, err := f.Stat()
if err != nil {
return fmt.Errorf("err: %s", err)
}
size := fi.Size()
total := (int)(size/defaultChunkSize + 1)
// Upload parts
buf := make([]byte, defaultChunkSize)
for partno := 1; partno <= total; partno++ {
readChunk := func(offset int, buf []byte) (int, error) {
fmt.Println("readChunk", partno, offset)
n, err := f.ReadAt(buf, int64(offset))
if err != nil {
return n, err
}
return n, nil
}
// This will block if there are not enough worker slots available
c <- partno
// The actual worker.
go func() {
offset := (partno - 1) * defaultChunkSize
n, err := readChunk(offset, buf)
if err != nil && err != io.EOF {
return
}
err = uploadPart(partno, buf[:n])
if err != nil {
fmt.Println("Uploadpart failed:", err)
}
<-c
}()
}
return nil
}
func uploadPart(partno int, buf []byte) error {
fmt.Printf("Uploading partno: %d, buflen=%d\n", partno, len(buf))
// Actually upload the part. Lets test it by instead writing each
// buffer to another file. We can then use diff to compare the
// source and dest files.
// Open file. Seek to (partno - 1) * defaultChunkSize, write buffer
f, err := os.OpenFile("/home/matthewh/Downloads/out.tar.gz", os.O_CREATE|os.O_WRONLY, 0755)
if err != nil {
fmt.Printf("err: %s\n", err)
}
n, err := f.WriteAt(buf, int64((partno-1)*defaultChunkSize))
if err != nil {
fmt.Printf("err=%s\n", err)
}
fmt.Printf("%d bytes written\n", n)
defer f.Close()
return nil
}
func main() {
filename := "/home/matthewh/Downloads/largefile.tar.gz"
fmt.Printf("Opening file: %s\n", filename)
f, err := os.Open(filename)
if err != nil {
panic(err)
}
UploadFile(f)
}
It almost works. But there are several problems.
1) The final partno 22 is occuring 3 times. The correct length is actually 612545 as the file length isn't a multiple of 1MB.
// Sample output
...
readChunk 21 20971520
readChunk 22 22020096
Uploading partno: 22, buflen=1048576
Uploading partno: 22, buflen=612545
Uploading partno: 22, buflen=1048576
Another problem, the upload could fail and I am not familiar enough with go and how best to solve failure of the goroutine.
Finally, I want to ordinarily return some data from the uploadPart when it succeeds. Specifically, it'll be a string (an HTTP ETag header value). These etag values need to be collected by the main function.
What is a better way to structure this code in this instance? I've not yet found a good golang design pattern that correctly fulfills my needs here.
Skipping for the moment the question of how better to structure this code, I see a bug in your code which may be causing the problem you're seeing. Since the function you're running in the goroutine uses the variable partno, which changes with each iteration of the loop, your goroutine isn't necessarily seeing the value of partno at the time you invoked the goroutine. A common way of fixing this is to create a local copy of that variable inside the loop:
for partno := 1; partno <= total; partno++ {
partno := partno
// ...
}
Data race #1
Multiple goroutines are using the same buffer concurrently. Note that one gorouting may be filling it with a new chunk while another is still reading an old chunk from it. Instead, each goroutine should have it's own buffer.
Data race #2
As Andy Schweig has pointed, the value in partno is updated by the loop before the goroutine created in that iteration has a chance to read it. This is why the final partno 22 occurs multiple times. To fix it, you can pass partno as a argument to the anonymous function. That will ensure each goroutine has it's own part number.
Also, you can use a channel to pass the results from the workers. Maybe a struct type with the part number and error. That way, you will be able to observe the progress and retry failed uploads.
For an example of a good pattern check out this example from the GOPL book.
Suggested changes
As noted by dev.bmax buf moved into go routine, as noted by Andy Schweig partno is param to anon function, also added WaitGroup since UploadFile was exiting before uploads were complete. Also defer f.Close() file, good habit.
package main
import (
"fmt"
"io"
"os"
"sync"
"time"
)
const defaultChunkSize = 1 * 1024 * 1024
// wg for uploads to complete
var wg sync.WaitGroup
// Lets have 4 workers
var c = make(chan int, 4)
func UploadFile(f *os.File) error {
// wait for all the uploads to complete before function exit
defer wg.Wait()
fi, err := f.Stat()
if err != nil {
return fmt.Errorf("err: %s", err)
}
size := fi.Size()
fmt.Printf("file size: %v\n", size)
total := int(size/defaultChunkSize + 1)
// Upload parts
for partno := 1; partno <= total; partno++ {
readChunk := func(offset int, buf []byte, partno int) (int, error) {
fmt.Println("readChunk", partno, offset)
n, err := f.ReadAt(buf, int64(offset))
if err != nil {
return n, err
}
return n, nil
}
// This will block if there are not enough worker slots available
c <- partno
// The actual worker.
go func(partno int) {
// wait for me to be done
wg.Add(1)
defer wg.Done()
buf := make([]byte, defaultChunkSize)
offset := (partno - 1) * defaultChunkSize
n, err := readChunk(offset, buf, partno)
if err != nil && err != io.EOF {
return
}
err = uploadPart(partno, buf[:n])
if err != nil {
fmt.Println("Uploadpart failed:", err)
}
<-c
}(partno)
}
return nil
}
func uploadPart(partno int, buf []byte) error {
fmt.Printf("Uploading partno: %d, buflen=%d\n", partno, len(buf))
// Actually do the upload. Simulate long running task with a sleep
time.Sleep(time.Second)
return nil
}
func main() {
filename := "/home/matthewh/Downloads/largefile.tar.gz"
fmt.Printf("Opening file: %s\n", filename)
f, err := os.Open(filename)
if err != nil {
panic(err)
}
defer f.Close()
UploadFile(f)
}
I'm sure you can deal a little smarter with the buf situation. I'm just letting go deal with the garbage. Since you are limiting your workers to specific number 4 you really need only 4 x defaultChunkSize buffers. Please do share if you come up with something simple and shareworth.
Have fun!

Go: Server should block until a message from the client is received

I'm building some server/client application in Go (the language is new to me). I searched a lot and read a whole bunch of different examples but there is still one thing I can't find. Lets say I have a single server client up and running. The client will send some kind of a message to the server and vice versa. Encoding and decoding is done by the package gob.
This example is not my application, it is only a quick example:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type Message struct {
Sender string
Receiver string
Command uint8
Value int64
}
func (message *Message) Set(sender string, receiver string, command uint8, value int64) *Message {
message.Sender = sender
message.Receiver = receiver
message.Command = command
message.Value = value
return message
}
func main() {
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
message := new(Message).Set("first", "second", 10, -1)
err := enc.Encode(*message) // send message
if err != nil {
log.Fatal("encode error:", err)
}
var m Message
err = dec.Decode(&m) // receice message
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q %q %d %d\n", m.Sender, m.Receiver, m.Command, m.Value)
}
This works fine, but I want the server to block until a new message is received so I can put the receiving process inside a infinite for loop inside a goroutine.
Something like that:
for {
// The server blocks HERE until a message from the client is received
fmt.Println("Received message:")
// Decode the new message
var m Message
err = dec.Decode(&m) // receice message
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q %q %d %d\n", m.Sender, m.Receiver, m.Command, m.Value)
}
The gob decoder blocks until it has read a full message or there's an error. The read loop in the question works as is.
working example on the playground
add a length header to the raw tcp stream.
that means, send a 4-bytes-length-header information to server before send the real load. and in server side read 4 bytes, allocate buffer, full read total message, and finally decode.
assume you have a tcp connection conn, in server side we could have:
func getInt(v []byte) int {
var r uint
r = 0
r |= uint(v[0]) << 24
r |= uint(v[1]) << 16
r |= uint(v[2]) << 8
r |= uint(v[3]) << 0
return int(r)
}
buf := make([]byte, 4)
_, err := io.ReadFull(conn, buf)
if err != nil {
return
}
length := getInt(buf)
buf = make([]byte, length)
_, err = io.ReadFull(conn, buf)
if err != nil {
return
}
//do gob decode from `buf` here
you may know client side refer the the server side source I think.

Reading from serial port with while-loop

I’ve written a short program in Go to communicate with a sensor through a serial port:
package main
import (
"fmt"
"github.com/tarm/goserial"
"time"
)
func main() {
c := &serial.Config{Name: "/dev/ttyUSB0", Baud: 9600}
s, err := serial.OpenPort(c)
if err != nil {
fmt.Println(err)
}
_, err = s.Write([]byte("\x16\x02N0C0 G A\x03\x0d\x0a"))
if err != nil {
fmt.Println(err)
}
time.Sleep(time.Second/2)
buf := make([]byte, 40)
n, err := s.Read(buf)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(buf[:n]))
s.Close()
}
It works fine, but after writing to the port I have to wait about half a second before I can start reading from it. I would like to use a while-loop instead of time.Sleep to read all incoming data. My attempt doesn’t work:
buf := make([]byte, 40)
n := 0
for {
n, _ := s.Read(buf)
if n > 0 {
break
}
}
fmt.Println(string(buf[:n]))
I guess buf gets overwritten after every loop pass. Any suggestions?
Your problem is that Read() will return whenever it has some data - it won't wait for all the data. See the io.Reader specification for more info
What you want to do is read until you reach some delimiter. I don't know exactly what format you are trying to use, but it looks like maybe \x0a is the end delimiter.
In which case you would use a bufio.Reader like this
reader := bufio.NewReader(s)
reply, err := reader.ReadBytes('\x0a')
if err != nil {
panic(err)
}
fmt.Println(reply)
Which will read data until the first \x0a.
I guess buf gets overwritten after every loop pass. Any suggestions?
Yes, buf will get overwritten with every call to Read().
A timeout on the file handle would be the approach I would take.
s, _ := os.OpenFile("/dev/ttyS0", syscall.O_RDWR|syscall.O_NOCTTY|syscall.O_NONBLOCK, 0666)
t := syscall.Termios{
Iflag: syscall.IGNPAR,
Cflag: syscall.CS8 | syscall.CREAD | syscall.CLOCAL | syscall.B115200,
Cc: [32]uint8{syscall.VMIN: 0, syscall.VTIME: uint8(20)}, //2.0s timeout
Ispeed: syscall.B115200,
Ospeed: syscall.B115200,
}
// syscall
syscall.Syscall6(syscall.SYS_IOCTL, uintptr(s.Fd()),
uintptr(syscall.TCSETS), uintptr(unsafe.Pointer(&t)),
0, 0, 0)
// Send message
n, _ := s.Write([]byte("Test message"))
// Receive reply
for {
buf := make([]byte, 128)
n, err = s.Read(buf)
if err != nil { // err will equal io.EOF
break
}
fmt.Printf("%v\n", string(buf))
}
Also note, if there is no more data read and there is no error, os.File.Read() will return an error of io.EOF,
as you can see here.

How to flush Stdin after fmt.Scanf() in Go?

Here's an issue that's bedeviling me at the moment. When getting input from the user, I want to employ a loop to ask the user to retry until they enter valid input:
// user_input.go
package main
import (
"fmt"
)
func main() {
fmt.Println("Please enter an integer: ")
var userI int
for {
_, err := fmt.Scanf("%d", &userI)
if err == nil {
break
}
fmt.Println("Sorry, invalid input. Please enter an integer: ")
}
fmt.Println(userI)
}
Running the above, if the user enters valid input, no problem:
Please enter an integer:
3
3

exit code 0, process exited normally.
But try inputting a string instead?
Please enter an integer:

what?
Sorry, invalid input. Please enter an integer:

Sorry, invalid input. Please enter an integer:

Sorry...
Etc, and it keeps looping character by character until the string is exhausted.
Even inputting a single character loops twice, I assume as it parses the newline.
Anyways, there must be a way to flush Stdin in Go?
P.S. In the absence of such a feature, how would you work around it to provide equivalent functionality? I've failed even at that...
I would fix this by reading until the end of the line after each failure. This clears the rest of the text.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
stdin := bufio.NewReader(os.Stdin)
fmt.Println("Please enter an integer: ")
var userI int
for {
_, err := fmt.Fscan(stdin, &userI)
if err == nil {
break
}
stdin.ReadString('\n')
fmt.Println("Sorry, invalid input. Please enter an integer: ")
}
fmt.Println(userI)
}
Is it bad to wake up an old question?
I prefer to use fmt.Scanln because A) it doesn't require importing another library (e.g. reader) and B) it doesn't involve an explicit for loop.
func someFunc() {
fmt.Printf("Please enter an integer: ")
// Read in an integer
var i int
_, err := fmt.Scanln(&i)
if err != nil {
fmt.Printf("Error: %s", err.Error())
// If int read fails, read as string and forget
var discard string
fmt.Scanln(&discard)
return
}
fmt.Printf("Input contained %d", i)
}
However, it seems like there ought to be a more elegant solution. Particularly in the case of fmt.Scanln it seems odd that the read stops after the first non-number byte rather than "scanning the line".
I ran into a similar problem for getting user input but solved it in a slightly different way. Adding to the thread in case someone else finds this useful:
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// Get first word from stdin
func getFirstWord() (string) {
input := bufio.NewScanner(os.Stdin)
input.Scan()
ans := strings.Fields(input.Text())
if len(ans) == 0 {
return ""
} else {
return ans[0]
}
}
func main() {
fmt.Printf("Would you like to play a game?\n> ")
ans := getFirstWord()
fmt.Printf("Your answer: %s\n", ans)
}
I know this has already been answered but this was my implementation:
func flush (reader *bufio.Reader) {
var i int
for i = 0; i < reader.Buffered(); i++ {
reader.ReadByte()
}
}
This should work in every situation, including ones where "stdin.ReadString('\n')" cannot be used.
Sorry for digging this back up, but I ran into this today and wanted to improve on the existing answers by using new standard library functionality.
import (
"bufio"
"fmt"
"os"
)
func discardBuffer(r *bufio.Reader) {
r.Discard(r.Buffered())
}
stdin := bufio.NewReader(os.Stdin)
var i int
for true {
if _, err := fmt.Fscanln(stdin, &i); err != nil {
discardBuffer(stdin)
// Handle error, display message, etc.
continue
}
// Do your other value checks and validations
break
}
The basic idea is to always buffer your reads from stdin. When you encounter an error while scanning, just discard the buffer contents. That way you start with an empty buffer for your next scan.
Alternatively, you can discard the buffer before you scan, so any stray inputs by the user before then won't get picked up.
func fscanln(r *bufio.Reader, a ...interface{}) error {
r.Discard(r.Buffered())
_, err := fmt.Fscanln(r, a...)
return err
}
stdin := bufio.NewReader(os.Stdin)
var i int
if err := fscanln(stdin, &i); err != nil {
// Handle error
}
I use this snippet to filter unnecessary leading space/new line
in := bufio.NewReader(os.Stdin)
result, err = in.ReadString('\n')
for len(strings.TrimSpace(result)) == 0 {
result, err = in.ReadString('\n')
}
I usually use bufio.Scanner since the fmt.Scan funcs always split on whitespace.
func promptYN(msg string) bool {
s := bufio.NewScanner(os.Stdin)
for {
fmt.Printf("%s [y/n]: ", msg)
s.Scan()
input := strings.ToLower(s.Text())
if input == "y" || input == "n" {
return input == "y"
}
fmt.Println("Error: expected Y or N.")
}
}
func promptInt(msg string) int {
s := bufio.NewScanner(os.Stdin)
for {
fmt.Printf("%s [int]: ", msg)
s.Scan()
output, err := strconv.Atoi(s.Text())
if err == nil {
return output
}
fmt.Println("Error: expected an integer.")
}
}
Or you could make something more universal:
func prompt(msg string, check func(string) bool) {
s := bufio.NewScanner(os.Stdin)
for {
fmt.Printf("%s: ", msg)
s.Scan()
if check(s.Text()) {
return
}
}
}
Example:
var f float64
prompt("Enter a float", func(s string) bool {
f, err = strconv.ParseFloat(s, 64)
return err == nil
})

Resources