TCP to Redis server with channel stuck - go

I need to send echo aaa three times to redis server,
but its get stuck in the middle of process, I also check
if read and write operation get error message, but it doesn't.
so, why it get stuck in the middle of process?
package main
import (
"fmt"
"os"
"io"
"net"
"sync"
)
var (
wg = new(sync.WaitGroup)
)
func readFromServer(isWrite chan bool, r io.Reader) {
for {
select {
case <-isWrite:
_ , err := io.Copy(os.Stdout, r)
if err != nil {
panic(err)
}
}
}
}
func writeToServer(conn net.Conn , isWrite chan bool ){
defer wg.Done()
for i :=0; i<3; i++{
_ , err := conn.Write([]byte("*2\r\n$4\r\necho\r\n$3\r\naaa\r\n"))
if err != nil {
panic(err)
}
isWrite<- true
}
}
func main(){
wg.Add(1)
conn ,err := net.Dial("tcp","127.0.0.1:6379")
isWrite := make(chan bool)
if err != nil {
panic(err)
}
go readFromServer(isWrite, conn)
go writeToServer(conn , isWrite)
wg.Wait()
fmt.Println("finished...")
}
Output:
$3
aaa
$3
aaa
Stuck here...

The readFromServer function receives one value from the isWrite channel and then blocks in the call to io.Copy. The io.Copy function does not return until EOF or some error reading or writing data. All of the program output is from the single call to io.Copy.
The second send to isWrite in sendToServer blocks.
The isWrite channel is an unbuffered channel. A send on an unbuffered channel does not proceed until there's a receiver. There is no receiver on the channel because readFromServer is blocked in the call to io.Copy.
Possible fixes are:
The fix is to modify readFromServer to parse the RESP protocol and read exactly one message per iteration in the loop.
Replace for loop in readFromServer with a single call to io.Copy.
The isWrite channel is not needed.
The program does not ensure that readFromServer reads all of the responses from writeToServer before the program exits.

Related

timeout on function and goroutine leak

I wish to put a timeout on a function called foo. Consider the following
func fooWithTimeout(d time.Duration) error {
ch := make(chan error, 1)
go func() {
ch <- foo()
}()
select {
case err := <-ch:
return err
case <-time.After(d):
return errors.New("foo has timed out")
}
}
If foo has timed out, then will foo ever be able to write to channel ch or is there a risk the goroutine blocks or panics?
What happens to channel ch once fooWithTimeout has exited?
Is this code potentially problematic?
Should I add defer close(ch) within go func(){...}() just before calling foo?
Does it matter that I use a buffered (with size 1) or an unbuffered channel in this example?
After the timer tick, fooWithTimeout will return. The goroutine will continue running until foo returns.
If foo times out, it will write to channel ch because it is buffered.
The channel ch will be garbage collected eventually if foo returns.
You don't need to close the channel. Once it is out of scope, it will be garbage collected.
A large burst of calls fooWithTimeout will create large amount of resources. Each call creates two goroutines. The proper way of timing this out is to change foo to use a context.
Building on https://stackoverflow.com/a/73611534/1079543, here is foo with a context:
package main
import (
"context"
"fmt"
"log"
"time"
)
func foo(ctx context.Context) (string, error) {
ch := make(chan string, 1)
go func() {
fmt.Println("Sleeping...")
time.Sleep(time.Second * 1)
fmt.Println("Wake up...")
ch <- "foo"
}()
select {
case <-ctx.Done():
return "", fmt.Errorf("context cancelled: %w", ctx.Err())
case result := <-ch:
return result, nil
}
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*3)
defer cancel()
res, err := foo(ctx)
if err != nil {
log.Fatalf("foo failed: %v", err)
}
log.Printf("res: %s", res)
}

How to return multiple values from goroutine?

I found some example with "Catch values from Goroutines"-> link
There is show how to fetch value but if I want to return value from several goroutines, it wont work.So, does anybody know, how to do it?
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
)
// WaitGroup is used to wait for the program to finish goroutines.
var wg sync.WaitGroup
func responseSize(url string, nums chan int) {
// Schedule the call to WaitGroup's Done to tell goroutine is completed.
defer wg.Done()
response, err := http.Get(url)
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
// Send value to the unbuffered channel
nums <- len(body)
}
func main() {
nums := make(chan int) // Declare a unbuffered channel
wg.Add(1)
go responseSize("https://www.golangprograms.com", nums)
go responseSize("https://gobyexample.com/worker-pools", nums)
go responseSize("https://stackoverflow.com/questions/ask", nums)
fmt.Println(<-nums) // Read the value from unbuffered channel
wg.Wait()
close(nums) // Closes the channel
// >> loading forever
Also, this example, worker pools
Is it possible to get value from result: fmt.Println(<-results) <- will be error.
Yes, just read from the channel multiple times:
answerOne := <-nums
answerTwo := <-nums
answerThree := <-nums
Channels function like thread-safe queues, allowing you to queue up values and read them out one by one
P.S. You should either add 3 to the wait group or not have one at all. The <-nums will block until a value is available on nums so it is not necessary

Sending to channel doesn't happen if select has default

I am working on a personal project that will run on a Raspberry Pi with some sensors attached to it.
The function that read from the sensors and the function that handle the socket connection are executed in different goroutines, so, in order to send data on the socket when they are read from the sensors, I create a chan []byte in the main function and pass it to the goroutines.
My problem came out here: if I do multiple writes in a row, only the first data arrives to the client, but the others don't. But if I put a little time.Sleep in the sender function, all the data arrives correctly to the client.
Anyway, that's a simplified version of this little program :
package main
import (
"net"
"os"
"sync"
"time"
)
const socketName string = "./test_socket"
// create to the socket and launch the accept client routine
func launchServerUDS(ch chan []byte) {
if err := os.RemoveAll(socketName); err != nil {
return
}
l, err := net.Listen("unix", socketName)
if err != nil {
return
}
go acceptConnectionRoutine(l, ch)
}
// accept incoming connection on the socket and
// 1) launch the routine to handle commands from the client
// 2) launch the routine to send data when the server reads from the sensors
func acceptConnectionRoutine(l net.Listener, ch chan []byte) {
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
return
}
go commandsHandlerRoutine(conn, ch)
go autoSendRoutine(conn, ch)
}
}
// routine that sends data to the client
func autoSendRoutine(c net.Conn, ch chan []byte) {
for {
data := <-ch
if string(data) == "exit" {
return
}
c.Write(data)
}
}
// handle client connection and calls functions to execute commands
func commandsHandlerRoutine(c net.Conn, ch chan []byte) {
for {
buf := make([]byte, 1024)
n, err := c.Read(buf)
if err != nil {
ch <- []byte("exit")
break
}
// now, for sake of simplicity , only echo commands back to the client
_, err = c.Write(buf[:n])
if err != nil {
ch <- []byte("exit")
break
}
}
}
// write on the channel to the autosend routine so the data are written on the socket
func sendDataToClient(data []byte, ch chan []byte) {
select {
case ch <- data:
// if i put a little sleep here, no problems
// i i remove the sleep, only data1 is sent to the client
// time.Sleep(1 * time.Millisecond)
default:
}
}
func dummyReadDataRoutine(ch chan []byte) {
for {
// read data from the sensors every 5 seconds
time.Sleep(5 * time.Second)
// read first data and send it
sendDataToClient([]byte("dummy data1\n"), ch)
// read second data and send it
sendDataToClient([]byte("dummy data2\n"), ch)
// read third data and send it
sendDataToClient([]byte("dummy data3\n"), ch)
}
}
func main() {
ch := make(chan []byte)
wg := sync.WaitGroup{}
wg.Add(2)
go dummyReadDataRoutine(ch)
go launchServerUDS(ch)
wg.Wait()
}
I don't think it's correct to use a sleep to synchronize writes. How do I fix this while keeping the functions running on a different different goroutines.
The primary problem was in the function:
func sendDataToClient(data []byte, ch chan []byte) {
select {
case ch <- data:
// if I put a little sleep here, no problems
// if I remove the sleep, only data1 is sent to the client
// time.Sleep(1 * time.Millisecond)
default:
}
If the channel ch isn't ready at the moment the function is called, the default case will be taken and the data will never be sent. In this case you should eliminate the function and send to the channel directly.
Buffering the channel is orthogonal to the problem at hand, and should be done for the similar reasons as you would buffered IO, i.e. provide a "buffer" for writes that can't immediately progress. If the code were not able progress without a buffer, adding one only delays possible deadlocks.
You also don't need the exit sentinel value here, as you could range over the channel and close it when you're done. This however still ignores write errors, but again that requires some re-design.
for data := range ch {
c.Write(data)
}
You should also be careful passing slices over channels, as it's all too easy to lose track of which logical process has ownership and is going to modify the backing array. I can't say from the information given if passing the read+write data over channels improves the architecture, but this is not a pattern you will find in most go networking code.
JimB gave a good explanation, so I think his answer is the better one.
I have included my partial solution in this answer.
I was thinking that my code was clear and simplified, but as Jim said I can do it simpler and clearer. I leave my old code posted so people can understand better how you can post simpler code and not do a mess like I did.
As chmike said, my issue wasn't related to the socket like I was thinking, but was only related to the channel. Write on a unbuffered channel was one of the problems. After change the unbuffered channel to a buffered one, the issue was resolved. Anyway, this code is not "good code" and can be improved following the principles that JimB has written in his answer.
So here is the new code:
package main
import (
"net"
"os"
"sync"
"time"
)
const socketName string = "./test_socket"
// create the socket and accept clients connections
func launchServerUDS(ch chan []byte, wg *sync.WaitGroup) {
defer wg.Done()
if err := os.RemoveAll(socketName); err != nil {
return
}
l, err := net.Listen("unix", socketName)
if err != nil {
return
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
return
}
// this goroutine are launched when a client is connected
// routine that listen and echo commands
go commandsHandlerRoutine(conn, ch)
// routine to send data read from the sensors to the client
go autoSendRoutine(conn, ch)
}
}
// routine that sends data to the client
func autoSendRoutine(c net.Conn, ch chan []byte) {
for {
data := <-ch
if string(data) == "exit" {
return
}
c.Write(data)
}
}
// handle commands received from the client
func commandsHandlerRoutine(c net.Conn, ch chan []byte) {
for {
buf := make([]byte, 1024)
n, err := c.Read(buf)
if err != nil {
// if i can't read send an exit command to autoSendRoutine and exit
ch <- []byte("exit")
break
}
// now, for sake of simplicity , only echo commands back to the client
_, err = c.Write(buf[:n])
if err != nil {
// if i can't write back send an exit command to autoSendRoutine and exit
ch <- []byte("exit")
break
}
}
}
// this goroutine reads from the sensors and write to the channel , so data are sent
// to the client if a client is connected
func dummyReadDataRoutine(ch chan []byte, wg *sync.WaitGroup) {
x := 0
for x < 100 {
// read data from the sensors every 5 seconds
time.Sleep(1 * time.Second)
// read first data and send it
ch <- []byte("data1\n")
// read second data and send it
ch <- []byte("data2\n")
// read third data and send it
ch <- []byte("data3\n")
x++
}
wg.Done()
}
func main() {
// create a BUFFERED CHANNEL
ch := make(chan []byte, 1)
wg := sync.WaitGroup{}
wg.Add(2)
// launch the goruotines that handle the socket connections
// and read data from the sensors
go dummyReadDataRoutine(ch, &wg)
go launchServerUDS(ch, &wg)
wg.Wait()
}

Writing concurrently with channels

I wrote a short script to write a file concurrently.
One goroutine is supposed to write strings to a file while the others are supposed to send the messages through a channel to it.
However, for some really strange reason the file is created but no message is added to it through the channel.
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
func concurrent(n uint64) {
output <- fmt.Sprint(n)
defer wg.Done()
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666);
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg+"\n")
}
}
func main() {
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
printOutput()
}
The printOutput() goroutine is executed completely, if I tried to write something after the for loop it would actually get into the file. So this leads me to think that range output might be null
You need to have something taking from the output channel as it is blocking until something removes what you put on it.
Not the only/best way to do it but: I moved printOutput() to above the other funcs and run it as a go routine and it prevents the deadlock.
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
func concurrent(n uint64) {
output <- fmt.Sprint(n)
defer wg.Done()
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg + "\n")
}
}
func main() {
go printOutput()
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
}
One of the reason why you get a null output is because channels are blocking for both send/receive.
According to your flow, the code snippet below will never reach wg.Done(), as sending channel is expecting a receiving end to pull the data out. This is a typical deadlock example.
func concurrent(n uint64) {
output <- fmt.Sprint(n) // go routine is blocked until data in channel is fetched.
defer wg.Done()
}
Let's examine the main func:
func main() {
wg.Add(2)
go concurrent(1)
go concurrent(2)
wg.Wait() // the main thread will be waiting indefinitely here.
close(output)
printOutput()
}
My take on the problem:
package main
import (
"fmt"
"os"
"sync"
)
var wg sync.WaitGroup
var output = make(chan string)
var donePrinting = make(chan struct{})
func concurrent(n uint) {
defer wg.Done() // It only makes sense to defer
// wg.Done() before you do something.
// (like sending a string to the output channel)
output <- fmt.Sprint(n)
}
func printOutput() {
f, err := os.OpenFile("output.txt", os.O_CREATE|os.O_RDWR|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for msg := range output {
f.WriteString(msg + "\n")
}
donePrinting <- struct{}{}
}
func main() {
wg.Add(2)
go printOutput()
go concurrent(1)
go concurrent(2)
wg.Wait()
close(output)
<-donePrinting
}
Each concurrent function will deduct from the wait-group.
After the two concurrent goroutines finish, the wg.Wait() will unblock, and the next instruction (close(output)) will be executed. You have to wait for the two goroutines to finish before closing the channel. If, instead, you try the following:
go printOutput()
go concurrent(1)
go concurrent(2)
close(output)
wg.Wait()
you could end up with the close(output) instruction executing before any one of the concurrent goroutines concludes. If the channel closes before the concurrent goroutines run, they will crash at runtime, (while trying to write to a closed channel).
If, then, you don’t wait up for the printOutput() goroutine to finish, you could actually quit main() before printOutput() has gotten the chance to finish writing to its file.
Because I want to wait for the printOutput() goroutine to finish before I quit the program, I also created a separate channel just to signal that printOutput() is done.
The <-donePrinting instruction blocks until main receives something over the donePrinting channel.
Once main receives anything (even the empty structure that printOutput() sends), it will unblock and run to conclusion.
https://play.golang.org/p/nXJoYLI758m

Scanf in multiple goroutines giving unexpected results

I was simply experimenting in golang. I came across an interesting result. This is my code.
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
var str1, str2 string
wg.Add(2)
go func() {
fmt.Scanf("%s", &str1)
wg.Done()
}()
go func() {
fmt.Scanf("%s", &str2)
wg.Done()
}()
wg.Wait()
fmt.Printf("%s %s\n", str1, str2)
}
I gave the following input.
beat
it
I was expecting the result to be either
it beat
or
beat it
But I got.
eat bit
Can any one please help me figure out why it is so?
fmt.Scanf isn't an atomic operation. Here's the implementation : http://golang.org/src/pkg/fmt/scan.go#L1115
There's no semaphor, nothing preventing two parallel executions. So what happens is simply that the executions are really parallel, and as there's no buffering, any byte reading is an IO operation and thus a perfect time for the go scheduler to change goroutine.
The problem is that you are sharing a single resource (the stdin byte stream) across multiple goroutines.
Each goroutine could be spawn at different non-deterministic times. i.e:
first goroutine 1 read all stdin, then start goroutine 2
first goroutine 2 read all stdin, then start goroutine 1
first goroutine 1 block on read, then start goroutine 2 read one char and then restart goroutine 1
... and so on and on ...
In most cases is enough to use only one goroutine to access a linear resource as a byte stream and attach a channel to it and then spawn multiple consumers that listen to that channel.
For example:
package main
import (
"fmt"
"io"
"sync"
)
func main() {
var wg sync.WaitGroup
words := make(chan string, 10)
wg.Add(1)
go func() {
for {
var buff string
_, err := fmt.Scanf("%s", &buff)
if err != nil {
if err != io.EOF {
fmt.Println("Error: ", err)
}
break
}
words <- buff
}
close(words)
wg.Done()
}()
// Multiple consumers
for i := 0; i < 5; i += 1 {
go func() {
for word := range words {
fmt.Printf("%s\n", word)
}
}()
}
wg.Wait()
}

Resources