Believe I am either misunderstanding how go routines work, how buffered readers work or both.
Expect an asynchronous execution of the goroutine ( a buffered reader with a for loop reading the buffer, waiting for a message from the server )
Try METHOD A to call go xyz() before the client dials the server; so xyz() creates the buffer and starts reading in the background. Then, the client dials the server; server sends message back; the client is reading the buffer so, it gets the message and prints to console
What Actually Happens the client to send the message to the server, but does not pick up anything on the buffer while reading for possible reply from server; so it is running concurrently because I know the for loop has not stopped, but it lets the next line of code execute ( client sending message to server ).
But When METHOD B I call xyz() NOT concurrently and after the client dials the server, all things work as expected. The client gets the message back from the server and prints to console.
METHOD A, we have the order :
///////////// steps 1 and 2 are in the goroutine called by go xyz()
creates the buffered reader
for loop -- reading the buffer for message from the server -- print out
client dials the server
go xyz(conn, p)
fmt.Fprintf(conn, "Give me a hash to work on ...")
METHOD B, we have the order :
///////////// steps 2 and 3 are in the goroutine called by xyz()
client dials the server
creates buffered reader
for loop -- reading the buffer for message from the server -- print out
fmt.Fprintf(conn, "Give me a hash to work on ...")
xyz(conn, p)
client.go
package main
import (
"fmt"
"net"
"bufio"
)
func xyz(conn net.Conn, p []byte) {
rd := bufio.NewReader(conn)
for {
_, err := rd.Read(p)
if err == nil {
fmt.Printf("SERVER : %s\n", p)
} else {
fmt.Printf("Some error %v\n", err)
}
}
}
func main() {
p := make([]byte, 2048)
conn, err := net.Dial("udp", "127.0.0.1:1234")
if err != nil {
fmt.Printf("Some error %v", err)
return
}
go xyz(conn, p)
fmt.Fprintf(conn, "Give me a hash to work on ...")
}
server.go
package main
import (
"fmt"
"net"
)
func sendResponse(conn *net.UDPConn, addr *net.UDPAddr, hash string) {
_,err := conn.WriteToUDP([]byte("Hello, here is the hash - " + hash), addr)
if err != nil {
fmt.Printf("Couldn't send response %v", err)
}
}
func main() {
hash := "36";
p := make([]byte, 2048)
addr := net.UDPAddr{
Port: 1234,
IP: net.ParseIP("127.0.0.1"),
}
ser, err := net.ListenUDP("udp", &addr)
if err != nil {
fmt.Printf("Some error %v\n", err)
return
}
for {
_, remoteaddr, err := ser.ReadFromUDP(p)
fmt.Printf("CLIENT : %v : %s\n", remoteaddr, p)
if err != nil {
fmt.Printf("Some error %v", err)
continue
}
go sendResponse(ser, remoteaddr, hash)
}
}
The Go Programming Language Specification
Go statements
A "go" statement starts the execution of a function call as an
independent concurrent thread of control, or goroutine, within the
same address space.
... unlike with a regular call, program execution does not wait for
the invoked function to complete.
client.go starts the goroutine xyz and then keeps going to the end of the main function which terminates the program. The program doesn't wait for the xyz goroutine to run or finish.
Related
Not able to reset or discard the buffer.
I am trying to get the data over the serial port where I am getting data packet of some fixed length for every 10 seconds. I have an infinite for loop to receive the data packets continuously. After receiving the new data packet I am resetting the buffer but when I receive the next data packet, it overwrites the buffer and I get mixed data packet.
Let say I should receive packet abcdef continuously for every n second. But when I try the following code I am receiving packet bcdefa then after n second cdefab then defabc and so on
package main
import (
"bufio"
"log"
"time"
"github.com/tarm/serial"
)
func main() {
c := &serial.Config{Name: "/dev/ttyUSB0", Baud: 57600}
s, err := serial.OpenPort(c)
if err != nil {
log.Println(err)
return
}
for {
time.Sleep(time.Second / 2)
reader := bufio.NewReader(s)
pck, err := reader.Peek(46)
if err != nil {
log.Println(err)
}
go parse(pck)
reader.Reset(s)
}
}
How do I reset or discard the buffer data effectively so that I will receive the exact data packet.
bare in mind i cant check what i m saying here...
1/ you must not instantiate the bufio reader at each iteration
2/ bufio.Reader.Peek does NOT advance the reader
https://golang.org/pkg/bufio/#Reader.Peek
3/ Unless you get a malformed packet, i think you dont need to reset at all.
4/ Please indent your code at play.golang.org
5/ You are not checking the read error for termination
6/ All package i can found to work with serial ports in go exposes an instance of io.Reader, so it might be useless to use an additional bufio.Reader. I suspect you r using https://godoc.org/github.com/tarm/serial#OpenPort
This is probably not the definitive answer, but it should help.
package main
import (
"io"
"log"
"time"
)
func main() {
s, err := serial.OpenPort(c)
if err != nil {
log.Fatal(err)
}
pck := make([]byte, 46)
for {
time.Sleep(time.Second / 2)
n, err := s.Read(pck)
if err != nil {
if err == io.EOF {
break
}
log.Println(err)
}
pck = pck[:n]
go parse(pck)
}
}
everyone!
I'm trying to get my go code work with openstack serial console. It`s exposed via web socket. And i have problems with it.
I found gorrilla websocket lib (which is great) and took this example as a reference
With a few tweaks, now i have a code like this:
package main
import (
"log"
"net/url"
"os"
"os/signal"
"time"
"net/http"
"github.com/gorilla/websocket"
)
func main() {
DialSettings := &websocket.Dialer {
Proxy: http.ProxyFromEnvironment,
HandshakeTimeout: 45 * time.Second,
Subprotocols: []string{"binary",},
ReadBufferSize: 4096,
WriteBufferSize: 4096,
}
log.SetFlags(0)
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
u, _ := url.Parse("ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5") //websocket url as it outputs from 'nova get-serial-console test' cmd
log.Printf("connecting to %s", u.String())
c, _, err := DialSettings.Dial(u.String(), nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("%s", message)
}
}()
c.WriteMessage(websocket.TextMessage, []byte("\n")) //just to force output to console
for {
select {
case <-done:
return
case <-interrupt:
log.Println("interrupt")
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
And i get output like this:
connecting to ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5
CentOS Linux 7
(C
ore)
K
erne
l
3.10.0-862.el7.x86_64
o
n an
x
86_64
centos
-test login:
Total mess...
I think it's because i recieve just a chunks of bytes with no way to delimit them. I need some buffer to store them and when do something like bufio.ReadLine. But i'm not most experienced go programmer, and i run out of ideas how to do this. At the end i just need strings to work with.
The log package writes each log message on a separate line. If the log message does not end with a newline, then the log package will add one.
These extra newlines are garbling the output. To fix the output, replace the call to log.Printf("%s", message) with a function that does not add newlines to the output. Here are some options:
Write the message to stderr (same destination as default log package config):
os.Stderr.Write(message)
Write the message to stdout (a more conventional location to write program output):
os.Stdout.Write(message)
I need to send echo aaa three times to redis server,
but its get stuck in the middle of process, I also check
if read and write operation get error message, but it doesn't.
so, why it get stuck in the middle of process?
package main
import (
"fmt"
"os"
"io"
"net"
"sync"
)
var (
wg = new(sync.WaitGroup)
)
func readFromServer(isWrite chan bool, r io.Reader) {
for {
select {
case <-isWrite:
_ , err := io.Copy(os.Stdout, r)
if err != nil {
panic(err)
}
}
}
}
func writeToServer(conn net.Conn , isWrite chan bool ){
defer wg.Done()
for i :=0; i<3; i++{
_ , err := conn.Write([]byte("*2\r\n$4\r\necho\r\n$3\r\naaa\r\n"))
if err != nil {
panic(err)
}
isWrite<- true
}
}
func main(){
wg.Add(1)
conn ,err := net.Dial("tcp","127.0.0.1:6379")
isWrite := make(chan bool)
if err != nil {
panic(err)
}
go readFromServer(isWrite, conn)
go writeToServer(conn , isWrite)
wg.Wait()
fmt.Println("finished...")
}
Output:
$3
aaa
$3
aaa
Stuck here...
The readFromServer function receives one value from the isWrite channel and then blocks in the call to io.Copy. The io.Copy function does not return until EOF or some error reading or writing data. All of the program output is from the single call to io.Copy.
The second send to isWrite in sendToServer blocks.
The isWrite channel is an unbuffered channel. A send on an unbuffered channel does not proceed until there's a receiver. There is no receiver on the channel because readFromServer is blocked in the call to io.Copy.
Possible fixes are:
The fix is to modify readFromServer to parse the RESP protocol and read exactly one message per iteration in the loop.
Replace for loop in readFromServer with a single call to io.Copy.
The isWrite channel is not needed.
The program does not ensure that readFromServer reads all of the responses from writeToServer before the program exits.
I have a for-loop that calls a function runCommand() which runs a remote command on a switch and prints the output. The function is called in a goroutine on each iteration and I am using a sync.Waitgroup to synchronize the goroutines. Now, I need a way to capture the output and any errors of my runCommand() function into a channel. I have read many articles and watched a lot of videos on using channels with goroutines, but this is the first time I have ever written a concurrent application and I can't seem to wrap my head around the idea.
Basically, my program takes in a list of hostnames from the command line then asynchronously connects to each host, runs a configuration command on it, and prints the output. It is ok for my program to continue configuring the remaining hosts if one has an error.
How would I idiomatically send the output or error(s) of each call to runCommand() to a channel then receive the output or error(s) for printing?
Here is my code:
package main
import (
"fmt"
"golang.org/x/crypto/ssh"
"os"
"time"
"sync"
)
func main() {
hosts := os.Args[1:]
clientConf := configureClient("user", "password")
var wg sync.WaitGroup
for _, host := range hosts {
wg.Add(1)
go runCommand(host, &clientConf, &wg)
}
wg.Wait()
fmt.Println("Configuration complete!")
}
// Run a remote command
func runCommand(host string, config *ssh.ClientConfig, wg *sync.WaitGroup) {
defer wg.Done()
// Connect to the client
client, err := ssh.Dial("tcp", host+":22", config)
if err != nil {
fmt.Println(err)
return
}
defer client.Close()
// Create a session
session, err := client.NewSession()
if err != nil {
fmt.Println(err)
return
}
defer session.Close()
// Get the session output
output, err := session.Output("show lldp ne")
if err != nil {
fmt.Println(err)
return
}
fmt.Print(string(output))
fmt.Printf("Connection to %s closed.\n", host)
}
// Set up client configuration
func configureClient(user, password string) ssh.ClientConfig {
var sshConf ssh.Config
sshConf.SetDefaults()
// Append supported ciphers
sshConf.Ciphers = append(sshConf.Ciphers, "aes128-cbc", "aes256-cbc", "3des-cbc", "des-cbc", "aes192-cbc")
// Create client config
clientConf := &ssh.ClientConfig{
Config: sshConf,
User: user,
Auth: []ssh.AuthMethod{ssh.Password(password)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: time.Second * 5,
}
return *clientConf
}
EDIT: I got rid of the Waitgroup, as suggested, and now I need to keep track of which output belongs to which host by printing the hostname before printing its output and printing a Connection to <host> closed. message when the gorouttine completes. For example:
$ go run main.go host1[,host2[,...]]
Connecting to <host1>
[Output]
...
[Error]
Connection to <host1> closed.
Connecting to <host2>
...
Connection to <host2> closed.
Configuration complete!
I know the above won't necessarily process host1 and host2 in order, But I need to print the correct host value for the connecting and closing messages before and after the output/error(s), respectively. I tried defering printing the closing message in the runCommand() function, but the message is printed out before the output/error(s). And printing the closing message in the for-loop after each goroutine call doesn't work as expected either.
Updated code:
package main
import (
"fmt"
"golang.org/x/crypto/ssh"
"os"
"time"
)
type CmdResult struct {
Host string
Output string
Err error
}
func main() {
start := time.Now()
hosts := os.Args[1:]
clientConf := configureClient("user", "password")
results := make(chan CmdResult)
for _, host := range hosts {
go runCommand(host, &clientConf, results)
}
for i := 0; i < len(hosts); i++ {
output := <- results
fmt.Println(output.Host)
if output.Output != "" {
fmt.Printf("%s\n", output.Output)
}
if output.Err != nil {
fmt.Printf("Error: %v\n", output.Err)
}
}
fmt.Printf("Configuration complete! [%s]\n", time.Since(start).String())
}
// Run a remote command
func runCommand(host string, config *ssh.ClientConfig, ch chan CmdResult) {
// This is printing before the output/error(s).
// Does the same when moved to the bottom of this function.
defer fmt.Printf("Connection to %s closed.\n", host)
// Connect to the client
client, err := ssh.Dial("tcp", host+":22", config)
if err != nil {
ch <- CmdResult{host, "", err}
return
}
defer client.Close()
// Create a session
session, err := client.NewSession()
if err != nil {
ch <- CmdResult{host, "", err}
return
}
defer session.Close()
// Get the session output
output, err := session.Output("show lldp ne")
if err != nil {
ch <- CmdResult{host, "", err}
return
}
ch <- CmdResult{host, string(output), nil}
}
// Set up client configuration
func configureClient(user, password string) ssh.ClientConfig {
var sshConf ssh.Config
sshConf.SetDefaults()
// Append supported ciphers
sshConf.Ciphers = append(sshConf.Ciphers, "aes128-cbc", "aes256-cbc", "3des-cbc", "des-cbc", "aes192-cbc")
// Create client config
clientConf := &ssh.ClientConfig{
Config: sshConf,
User: user,
Auth: []ssh.AuthMethod{ssh.Password(password)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Timeout: time.Second * 5,
}
return *clientConf
}
If you use an unbuffered channel, you actually don't need the sync.WaitGroup, because you can call the receive operator on the channel once for every goroutine that will send on the channel. Each receive operation will block until a send statement is ready, resulting in the same behavior as a WaitGroup.
To make this happen, change runCommand to execute a send statement exactly once before the function exits, under all conditions.
First, create a type to send over the channel:
type CommandResult struct {
Output string
Err error
}
And edit your main() {...} to execute a receive operation on the channel the same number of times as the number of goroutines that will send to the channel:
func main() {
ch := make(chan CommandResult) // initialize an unbuffered channel
// rest of your setup
for _, host := range hosts {
go runCommand(host, &clientConf, ch) // pass in the channel
}
for x := 0; x < len(hosts); x++ {
fmt.Println(<-ch) // this will block until one is ready to send
}
And edit your runCommand function to accept the channel, remove references to WaitGroup, and execute the send exactly once under all conditions:
func runCommand(host string, config *ssh.ClientConfig, ch chan CommandResult) {
// do stuff that generates output, err; then when ready to exit function:
ch <- CommandResult{output, err}
}
EDIT: Question updated with stdout message order requirements
I'd like to get nicely formatted output that ignores the order of events
In this case, remove all print messages from runCommand, you're going to put all output into the element you're passing on the channel so it can be grouped together. Edit the CommandResult type to contain additional fields you want to organize, such as:
type CommandResult struct {
Host string
Output string
Err error
}
If you don't need to sort your results, you can just move on to printing the data received, e.g.
for x := 0; x < len(hosts); x++ {
r := <-ch
fmt.Printf("Host: %s----\nOutput: %s\n", r.Host, r.Output)
if r.Err != nil {
fmt.Printf("Error: %s\n", r.Err)
}
}
If you do need to sort your results, then in your main goroutine, add the elements received on the channel to a slice:
...
results := make([]CommandResult, 0, len(hosts))
for x := 0; x < len(hosts); x++ {
results = append(results, <-ch) // this will block until one is ready to send
}
Then you can use the sort package in the Go standard library to sort your results for printing. For example, you could sort them alphabetically by host. Or you could put the results into a map with host string as the key instead of a slice to allow you to print in the order of the original host list.
I've implemented a demo tcp chat server in golang, it works fine, but every time a user disconnects and I try to write a message to the broadcast channel to let other users know a user has disconnected it blocks, and would not further process any new messages from other client because its a nonbuffered channel
I've commented by code and explained it you can go through it, I don't know why the code blocks, I've written msgs
I'm about to write to a channel
I've written to the channel
I've read from the channel
and messages are in perfect order still my msg channel blocks.
Ps: If I'm using buffered channel the code is not blocking, but I want to know where is my code getting stuck.
I also tried running my code with -race flag but no help
package main
import (
"fmt"
"io"
"net"
"sync"
)
func main() {
msg := make(chan string) //broadcast channel (making it buffered channel the problem goes away)
allConn := make(map[net.Conn]int) //Collection of incoming connections for broadcasting the message
disConn := make(chan net.Conn) //client disconnect channel
newConn := make(chan net.Conn) //new client connection channel
mutext := new(sync.RWMutex) //mux to assign unique id to incoming connections
i := 0
listener, err := net.Listen("tcp", "127.0.0.1:8081")
checkErr(err)
fmt.Println("Tcp server started at 127.0.0.1:8081")
//Accept incoming connections and store them in global connection store allConn
go func() {
for {
conn, err := listener.Accept()
checkErr(err)
mutext.Lock()
allConn[conn] = i
i++
mutext.Unlock()
newConn <- conn
}
}()
for {
select {
//Wait for a new client message to arrive and broadcast the message
case umsg := <-msg:
fmt.Println("Broadcast Channel: Already Read")
bmsg := []byte(umsg)
for conn1, _ := range allConn {
_, err := conn1.Write(bmsg)
checkErr(err)
}
//Handle client disconnection [disConn]
case conn := <-disConn:
mutext.RLock()
fmt.Println("user disconneting", allConn[conn])
mutext.RUnlock()
delete(allConn, conn)
fmt.Println("Disconnect: About to Write")
//this call results in deadlock even when channel is empty, buffered channel resolves the issue
//need to know why
msg <- fmt.Sprintf("Disconneting", allConn[conn])
fmt.Println("Disconnect: Already Written")
//Read client incoming message and put it on broadcasting channel and upon disconnect put on it disConn channel
case conn := <-newConn:
go func(conn net.Conn) {
for {
buf := make([]byte, 64)
n, err := conn.Read(buf)
if err != nil {
if err == io.EOF {
disConn <- conn
break
}
}
fmt.Println("Client: About to Write")
msg <- string(buf[0:n])
fmt.Println("Client: Already Written")
}
}(conn)
mutext.RLock()
fmt.Println("User Connected", allConn[conn])
mutext.RUnlock()
}
}
}
func checkErr(err error) {
if err != nil {
panic(err)
}
}
In Go, an unbuffered channel is a "synchronisation point". That is, if you have a channel c, and do c <- value, the goroutine blocks until someone is ready to do v = <- c (and the converse holds, receiving from a blocking channel without something to receive blocks until the value is available, but this is possibly less surprising). Specifically, for a blocking channel, the receive completes before the send completes.
Since you only have a single goroutine, it will be unable to loop back to reading from the channel and the write will block until something can read.
You could, in theory, get around this by doing something like: go func() { msg <- fmt.Sprintf("Disconneting", allConn[conn] }(), so essentially spawning a short-lived goroutine to do the write.