Can`t figure out how to use buffers with binary web socket - go

everyone!
I'm trying to get my go code work with openstack serial console. It`s exposed via web socket. And i have problems with it.
I found gorrilla websocket lib (which is great) and took this example as a reference
With a few tweaks, now i have a code like this:
package main
import (
"log"
"net/url"
"os"
"os/signal"
"time"
"net/http"
"github.com/gorilla/websocket"
)
func main() {
DialSettings := &websocket.Dialer {
Proxy: http.ProxyFromEnvironment,
HandshakeTimeout: 45 * time.Second,
Subprotocols: []string{"binary",},
ReadBufferSize: 4096,
WriteBufferSize: 4096,
}
log.SetFlags(0)
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
u, _ := url.Parse("ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5") //websocket url as it outputs from 'nova get-serial-console test' cmd
log.Printf("connecting to %s", u.String())
c, _, err := DialSettings.Dial(u.String(), nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("%s", message)
}
}()
c.WriteMessage(websocket.TextMessage, []byte("\n")) //just to force output to console
for {
select {
case <-done:
return
case <-interrupt:
log.Println("interrupt")
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
And i get output like this:
connecting to ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5
CentOS Linux 7
(C
ore)
K
erne
l
3.10.0-862.el7.x86_64
o
n an
x
86_64
centos
-test login:
Total mess...
I think it's because i recieve just a chunks of bytes with no way to delimit them. I need some buffer to store them and when do something like bufio.ReadLine. But i'm not most experienced go programmer, and i run out of ideas how to do this. At the end i just need strings to work with.

The log package writes each log message on a separate line. If the log message does not end with a newline, then the log package will add one.
These extra newlines are garbling the output. To fix the output, replace the call to log.Printf("%s", message) with a function that does not add newlines to the output. Here are some options:
Write the message to stderr (same destination as default log package config):
os.Stderr.Write(message)
Write the message to stdout (a more conventional location to write program output):
os.Stdout.Write(message)

Related

How to reset or discard the buffered data

Not able to reset or discard the buffer.
I am trying to get the data over the serial port where I am getting data packet of some fixed length for every 10 seconds. I have an infinite for loop to receive the data packets continuously. After receiving the new data packet I am resetting the buffer but when I receive the next data packet, it overwrites the buffer and I get mixed data packet.
Let say I should receive packet abcdef continuously for every n second. But when I try the following code I am receiving packet bcdefa then after n second cdefab then defabc and so on
package main
import (
"bufio"
"log"
"time"
"github.com/tarm/serial"
)
func main() {
c := &serial.Config{Name: "/dev/ttyUSB0", Baud: 57600}
s, err := serial.OpenPort(c)
if err != nil {
log.Println(err)
return
}
for {
time.Sleep(time.Second / 2)
reader := bufio.NewReader(s)
pck, err := reader.Peek(46)
if err != nil {
log.Println(err)
}
go parse(pck)
reader.Reset(s)
}
}
How do I reset or discard the buffer data effectively so that I will receive the exact data packet.
bare in mind i cant check what i m saying here...
1/ you must not instantiate the bufio reader at each iteration
2/ bufio.Reader.Peek does NOT advance the reader
https://golang.org/pkg/bufio/#Reader.Peek
3/ Unless you get a malformed packet, i think you dont need to reset at all.
4/ Please indent your code at play.golang.org
5/ You are not checking the read error for termination
6/ All package i can found to work with serial ports in go exposes an instance of io.Reader, so it might be useless to use an additional bufio.Reader. I suspect you r using https://godoc.org/github.com/tarm/serial#OpenPort
This is probably not the definitive answer, but it should help.
package main
import (
"io"
"log"
"time"
)
func main() {
s, err := serial.OpenPort(c)
if err != nil {
log.Fatal(err)
}
pck := make([]byte, 46)
for {
time.Sleep(time.Second / 2)
n, err := s.Read(pck)
if err != nil {
if err == io.EOF {
break
}
log.Println(err)
}
pck = pck[:n]
go parse(pck)
}
}

Error reading from external command: fatal error all goroutines are asleep - deadlock

I want to write a mime/multipart message in Python to standard output and read that message in Golang using the mime/multipart package. This is just a learning exercise.
I tried simulating this example.
output.py
#!/usr/bin/env python2.7
import sys
s = "--foo\r\nFoo: one\r\n\r\nA section\r\n" +"--foo\r\nFoo: two\r\n\r\nAnd another\r\n" +"--foo--\r\n"
print s
main.go
package main
import (
"io"
"os/exec"
"mime/multipart"
"log"
"io/ioutil"
"fmt"
"sync"
)
var wg sync.WaitGroup
func main() {
pr,pw := io.Pipe()
defer pw.Close()
cmd := exec.Command("python","output.py")
cmd.Stdout = pw
mr := multipart.NewReader(pr,"foo")
wg.Add(1)
go func() {
defer wg.Done()
for {
p, err := mr.NextPart()
if err == io.EOF {
fmt.Println("EOF")
return
}
if err != nil {
log.Fatal(err)
}
slurp, err := ioutil.ReadAll(p)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Part : %q\n", slurp)
return
}
}()
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
cmd.Wait()
wg.Wait()
}
Output of go run main.go:
fatal error: all goroutines are asleep - deadlock!
Other answers regarding this topic on StackOverflow are related to channels not being closed, but I am not even using a channel. I understand that somewhere, there is infinite loop or something similar, but I don't see it.
Try something like this (explanation below):
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"mime/multipart"
"os"
"os/exec"
"sync"
"github.com/pkg/errors"
)
func readCommand(cmdStdout io.ReadCloser, wg *sync.WaitGroup, resc chan<- []byte, errc chan<- error) {
defer wg.Done()
defer close(errc)
defer close(resc)
mr := multipart.NewReader(cmdStdout, "foo")
for {
part, err := mr.NextPart()
if err != nil {
if err == io.EOF {
fmt.Println("EOF")
} else {
errc <- errors.Wrap(err, "failed to get next part")
}
return
}
slurp, err := ioutil.ReadAll(part)
if err != nil {
errc <- errors.Wrap(err, "failed to read part")
return
}
resc <- slurp
}
}
func main() {
cmd := exec.Command("python", "output.py")
cmd.Stderr = os.Stderr
pr, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
var wg sync.WaitGroup
wg.Add(1)
resc := make(chan []byte)
errc := make(chan error)
go readCommand(pr, &wg, resc, errc)
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
for {
select {
case err, ok := <-errc:
if !ok {
errc = nil
break
}
if err != nil {
log.Fatal(errors.Wrap(err, "error from goroutine"))
}
case res, ok := <-resc:
if !ok {
resc = nil
break
}
fmt.Printf("Part from goroutine: %q\n", res)
}
if errc == nil && resc == nil {
break
}
}
cmd.Wait()
wg.Wait()
}
In no particular order:
Rather than using an io.Pipe() as the command's Stdout, just ask the command for it's StdoutPipe(). cmd.Wait() will ensure it's closed for you.
Set cmd.Stderr to os.Stderr so that you can see errors generated by your Python program.
I noticed this program was hanging anytime the Python program wrote to standard error. Now it doesn't :)
Don't make the WaitGroup a global variable; pass a reference to it to the goroutine.
Rather than log.Fatal()ing inside the goroutine, create an error channel to communicate errors back to main().
Rather than printing results inside the goroutine, create a result channel to communicate results back to main().
Ensure channels are closed to prevent blocking/goroutine leaks.
Separate out the goroutine into a proper function to make the code easier to read and follow.
In this example, we can create the multipart.Reader() inside our goroutine, since this is the only part of our code that uses it.
Note that I am using Wrap() from the errors package to add context to the error messages. This is, of course, not relevant to your question, but is a good habit.
The for { select { ... } } part may be confusing. This is one article I found introducing the concept. Basically, select is letting us read from whichever of these two channels (resc and errc) are currently readable, and then setting each to nil when the channel is closed. When both channels are nil, the loop exits. This lets us handle "either a result or an error" as they come in.
Edit: As johandalabacka said on the Golang Forum, it looks like the main issue here was that Python on Windows was adding an extra \r to the output, and that the problem is your Python program should omit the \r in the output string or sys.stdout.write() instead of print() ing. The output could also be cleaned up on the Golang side, but, aside from not being able to parse properly without modifying the Python side, this answer will still improve the concurrency mechanics of your program.

Capture Output and Errors of Goroutine Using Channels

I have a for-loop that calls a function runCommand() which runs a remote command on a switch and prints the output. The function is called in a goroutine on each iteration and I am using a sync.Waitgroup to synchronize the goroutines. Now, I need a way to capture the output and any errors of my runCommand() function into a channel. I have read many articles and watched a lot of videos on using channels with goroutines, but this is the first time I have ever written a concurrent application and I can't seem to wrap my head around the idea.
Basically, my program takes in a list of hostnames from the command line then asynchronously connects to each host, runs a configuration command on it, and prints the output. It is ok for my program to continue configuring the remaining hosts if one has an error.
How would I idiomatically send the output or error(s) of each call to runCommand() to a channel then receive the output or error(s) for printing?
Here is my code:
package main
import (
"fmt"
"golang.org/x/crypto/ssh"
"os"
"time"
"sync"
)
func main() {
hosts := os.Args[1:]
clientConf := configureClient("user", "password")
var wg sync.WaitGroup
for _, host := range hosts {
wg.Add(1)
go runCommand(host, &clientConf, &wg)
}
wg.Wait()
fmt.Println("Configuration complete!")
}
// Run a remote command
func runCommand(host string, config *ssh.ClientConfig, wg *sync.WaitGroup) {
defer wg.Done()
// Connect to the client
client, err := ssh.Dial("tcp", host+":22", config)
if err != nil {
fmt.Println(err)
return
}
defer client.Close()
// Create a session
session, err := client.NewSession()
if err != nil {
fmt.Println(err)
return
}
defer session.Close()
// Get the session output
output, err := session.Output("show lldp ne")
if err != nil {
fmt.Println(err)
return
}
fmt.Print(string(output))
fmt.Printf("Connection to %s closed.\n", host)
}
// Set up client configuration
func configureClient(user, password string) ssh.ClientConfig {
var sshConf ssh.Config
sshConf.SetDefaults()
// Append supported ciphers
sshConf.Ciphers = append(sshConf.Ciphers, "aes128-cbc", "aes256-cbc", "3des-cbc", "des-cbc", "aes192-cbc")
// Create client config
clientConf := &ssh.ClientConfig{
Config: sshConf,
User: user,
Auth: []ssh.AuthMethod{ssh.Password(password)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: time.Second * 5,
}
return *clientConf
}
EDIT: I got rid of the Waitgroup, as suggested, and now I need to keep track of which output belongs to which host by printing the hostname before printing its output and printing a Connection to <host> closed. message when the gorouttine completes. For example:
$ go run main.go host1[,host2[,...]]
Connecting to <host1>
[Output]
...
[Error]
Connection to <host1> closed.
Connecting to <host2>
...
Connection to <host2> closed.
Configuration complete!
I know the above won't necessarily process host1 and host2 in order, But I need to print the correct host value for the connecting and closing messages before and after the output/error(s), respectively. I tried defering printing the closing message in the runCommand() function, but the message is printed out before the output/error(s). And printing the closing message in the for-loop after each goroutine call doesn't work as expected either.
Updated code:
package main
import (
"fmt"
"golang.org/x/crypto/ssh"
"os"
"time"
)
type CmdResult struct {
Host string
Output string
Err error
}
func main() {
start := time.Now()
hosts := os.Args[1:]
clientConf := configureClient("user", "password")
results := make(chan CmdResult)
for _, host := range hosts {
go runCommand(host, &clientConf, results)
}
for i := 0; i < len(hosts); i++ {
output := <- results
fmt.Println(output.Host)
if output.Output != "" {
fmt.Printf("%s\n", output.Output)
}
if output.Err != nil {
fmt.Printf("Error: %v\n", output.Err)
}
}
fmt.Printf("Configuration complete! [%s]\n", time.Since(start).String())
}
// Run a remote command
func runCommand(host string, config *ssh.ClientConfig, ch chan CmdResult) {
// This is printing before the output/error(s).
// Does the same when moved to the bottom of this function.
defer fmt.Printf("Connection to %s closed.\n", host)
// Connect to the client
client, err := ssh.Dial("tcp", host+":22", config)
if err != nil {
ch <- CmdResult{host, "", err}
return
}
defer client.Close()
// Create a session
session, err := client.NewSession()
if err != nil {
ch <- CmdResult{host, "", err}
return
}
defer session.Close()
// Get the session output
output, err := session.Output("show lldp ne")
if err != nil {
ch <- CmdResult{host, "", err}
return
}
ch <- CmdResult{host, string(output), nil}
}
// Set up client configuration
func configureClient(user, password string) ssh.ClientConfig {
var sshConf ssh.Config
sshConf.SetDefaults()
// Append supported ciphers
sshConf.Ciphers = append(sshConf.Ciphers, "aes128-cbc", "aes256-cbc", "3des-cbc", "des-cbc", "aes192-cbc")
// Create client config
clientConf := &ssh.ClientConfig{
Config: sshConf,
User: user,
Auth: []ssh.AuthMethod{ssh.Password(password)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: time.Second * 5,
}
return *clientConf
}
If you use an unbuffered channel, you actually don't need the sync.WaitGroup, because you can call the receive operator on the channel once for every goroutine that will send on the channel. Each receive operation will block until a send statement is ready, resulting in the same behavior as a WaitGroup.
To make this happen, change runCommand to execute a send statement exactly once before the function exits, under all conditions.
First, create a type to send over the channel:
type CommandResult struct {
Output string
Err error
}
And edit your main() {...} to execute a receive operation on the channel the same number of times as the number of goroutines that will send to the channel:
func main() {
ch := make(chan CommandResult) // initialize an unbuffered channel
// rest of your setup
for _, host := range hosts {
go runCommand(host, &clientConf, ch) // pass in the channel
}
for x := 0; x < len(hosts); x++ {
fmt.Println(<-ch) // this will block until one is ready to send
}
And edit your runCommand function to accept the channel, remove references to WaitGroup, and execute the send exactly once under all conditions:
func runCommand(host string, config *ssh.ClientConfig, ch chan CommandResult) {
// do stuff that generates output, err; then when ready to exit function:
ch <- CommandResult{output, err}
}
EDIT: Question updated with stdout message order requirements
I'd like to get nicely formatted output that ignores the order of events
In this case, remove all print messages from runCommand, you're going to put all output into the element you're passing on the channel so it can be grouped together. Edit the CommandResult type to contain additional fields you want to organize, such as:
type CommandResult struct {
Host string
Output string
Err error
}
If you don't need to sort your results, you can just move on to printing the data received, e.g.
for x := 0; x < len(hosts); x++ {
r := <-ch
fmt.Printf("Host: %s----\nOutput: %s\n", r.Host, r.Output)
if r.Err != nil {
fmt.Printf("Error: %s\n", r.Err)
}
}
If you do need to sort your results, then in your main goroutine, add the elements received on the channel to a slice:
...
results := make([]CommandResult, 0, len(hosts))
for x := 0; x < len(hosts); x++ {
results = append(results, <-ch) // this will block until one is ready to send
}
Then you can use the sort package in the Go standard library to sort your results for printing. For example, you could sort them alphabetically by host. Or you could put the results into a map with host string as the key instead of a slice to allow you to print in the order of the original host list.

concurrency | goroutines | golang | buffered reader

Believe I am either misunderstanding how go routines work, how buffered readers work or both.
Expect an asynchronous execution of the goroutine ( a buffered reader with a for loop reading the buffer, waiting for a message from the server )
Try METHOD A to call go xyz() before the client dials the server; so xyz() creates the buffer and starts reading in the background. Then, the client dials the server; server sends message back; the client is reading the buffer so, it gets the message and prints to console
What Actually Happens the client to send the message to the server, but does not pick up anything on the buffer while reading for possible reply from server; so it is running concurrently because I know the for loop has not stopped, but it lets the next line of code execute ( client sending message to server ).
But When METHOD B I call xyz() NOT concurrently and after the client dials the server, all things work as expected. The client gets the message back from the server and prints to console.
METHOD A, we have the order :
///////////// steps 1 and 2 are in the goroutine called by go xyz()
creates the buffered reader
for loop -- reading the buffer for message from the server -- print out
client dials the server
go xyz(conn, p)
fmt.Fprintf(conn, "Give me a hash to work on ...")
METHOD B, we have the order :
///////////// steps 2 and 3 are in the goroutine called by xyz()
client dials the server
creates buffered reader
for loop -- reading the buffer for message from the server -- print out
fmt.Fprintf(conn, "Give me a hash to work on ...")
xyz(conn, p)
client.go
package main
import (
"fmt"
"net"
"bufio"
)
func xyz(conn net.Conn, p []byte) {
rd := bufio.NewReader(conn)
for {
_, err := rd.Read(p)
if err == nil {
fmt.Printf("SERVER : %s\n", p)
} else {
fmt.Printf("Some error %v\n", err)
}
}
}
func main() {
p := make([]byte, 2048)
conn, err := net.Dial("udp", "127.0.0.1:1234")
if err != nil {
fmt.Printf("Some error %v", err)
return
}
go xyz(conn, p)
fmt.Fprintf(conn, "Give me a hash to work on ...")
}
server.go
package main
import (
"fmt"
"net"
)
func sendResponse(conn *net.UDPConn, addr *net.UDPAddr, hash string) {
_,err := conn.WriteToUDP([]byte("Hello, here is the hash - " + hash), addr)
if err != nil {
fmt.Printf("Couldn't send response %v", err)
}
}
func main() {
hash := "36";
p := make([]byte, 2048)
addr := net.UDPAddr{
Port: 1234,
IP: net.ParseIP("127.0.0.1"),
}
ser, err := net.ListenUDP("udp", &addr)
if err != nil {
fmt.Printf("Some error %v\n", err)
return
}
for {
_, remoteaddr, err := ser.ReadFromUDP(p)
fmt.Printf("CLIENT : %v : %s\n", remoteaddr, p)
if err != nil {
fmt.Printf("Some error %v", err)
continue
}
go sendResponse(ser, remoteaddr, hash)
}
}
The Go Programming Language Specification
Go statements
A "go" statement starts the execution of a function call as an
independent concurrent thread of control, or goroutine, within the
same address space.
... unlike with a regular call, program execution does not wait for
the invoked function to complete.
client.go starts the goroutine xyz and then keeps going to the end of the main function which terminates the program. The program doesn't wait for the xyz goroutine to run or finish.

Golang: How to auto-restart process when binary updated?

I built the golang app in local, then scp to server. I need to stop the process and restart manually. Is there any way to auto-restart the process when binary updated?
While this is generally better to be implemented off-process using something like daemontools or similar, there are some cases when you want/need it to be done inside your program.
Doing it inside your program can be tricky depending on the program characteristics such as connections or files it may have open, etc.
Having said that, here you have an implementation which would work in most cases:
package main
import (
"log"
"os"
"syscall"
"time"
"github.com/fsnotify/fsnotify"
"github.com/kardianos/osext"
)
func setupWatcher() (chan struct{}, error) {
file, err := osext.Executable()
if err != nil {
return nil, err
}
log.Printf("watching %q\n", file)
w, err := fsnotify.NewWatcher()
if err != nil {
return nil, err
}
done := make(chan struct{})
go func() {
for {
select {
case e := <-w.Events:
log.Printf("watcher received: %+v", e)
err := syscall.Exec(file, os.Args, os.Environ())
if err != nil {
log.Fatal(err)
}
case err := <-w.Errors:
log.Printf("watcher error: %+v", err)
case <-done:
log.Print("watcher shutting down")
return
}
}
}()
err = w.Add(file)
if err != nil {
return nil, err
}
return done, nil
}
func main() {
log.Print("program starting")
watcher, err := setupWatcher()
if err != nil {
// do something sensible
log.Fatal(err)
}
// continue with app startup
time.Sleep(100 * time.Minute) // just for testing
// eventually you may need to end the watcher
close(watcher) // this way you can
}
Then you do
% go build main.go
% ./main
2016/12/29 14:15:06 program starting
2016/12/29 14:15:06 watching "/home/plalloni/tmp/v/main"
And here the output it produced after you run (in other terminal) some successive "go build main.go" (which "updates" the running binary).
2016/12/29 14:15:32 watcher received: "/home/plalloni/tmp/v/main": CHMOD
2016/12/29 14:15:32 program starting
2016/12/29 14:15:32 watching "/home/plalloni/tmp/v/main"
2016/12/29 14:15:38 watcher received: "/home/plalloni/tmp/v/main": CHMOD
2016/12/29 14:15:38 program starting
2016/12/29 14:15:38 watching "/home/plalloni/tmp/v/main"
Hope it helps.
You can use https://github.com/slayer/autorestart
package main
import "github.com/slayer/autorestart"
func main() {
autorestart.StartWatcher()
http.ListenAndServe(":8080", nil) // for example
}
Does it need to be sophisticated? You could have entr running and trigger an updater script when the binary changes.
http://entrproject.org/
e.g.
echo 'binary_path' | entr script.sh &
I have a resolution about this case.
See also.
https://github.com/narita-takeru/cmdrevive
example
cmdrevive ./htmls/ ".html$" (application) (arguments)
So, this case applicable.
cmdrevive "/(app path)" "(app filename)" (app full path) (arguments)
If (app filename) changed on (app path) directory, then restart (app full path) with (arguments).
How about this one?

Resources