I have a go program that uses a goroutine to read UDP packets.
I wanted to use a select clause and a "stopping" channel to close the goroutine to shut down as soon as it is not needed anymore.
Here is a simple code example for the goroutine:
func Run(c chan string, q chan bool, conn *net.UDPConn) {
defer close(c)
buf := make([]byte, 1024)
for {
select {
case <- q:
return
default:
n, _, err := conn.ReadFromUDP(buf)
c <- string(buf[0:n])
fmt.Println("Received ", string(buf[0:n]))
if err != nil {
fmt.Println("Error: ", err)
}
}
}
}
The connection is created as:
conn, err := net.ListenUDP("udp",addr.Addr)
And the goroutine is supposed to terminate using:
close(q)
After closing the "stopping" channel ("q") the goroutine does not immediately stop. I need to send one more string via the UDP connection. When doing so the goroutine stops.
I simply do not understand this behaviour and I would be grateful if somebody could enlighten me.
Thank you in advance!
Your program is likely stopped at this line when you close the channel:
n, _, err := conn.ReadFromUDP(buf)
Because execution is blocked at a ReadFrom method, the select statement is not being evaluated, therefore the close on channel q is not immediately detected. When you do another send on the UDP connection, ReadFrom unblocks and (once that loop iteration finishes) control moves to the select statement: at that point the close on q is detected.
You can close the connection to unblock ReadFrom, as was suggested in a comment. See the PacketConn documentation in the net package, especially "Any blocked ReadFrom or WriteTo operations will be unblocked and return errors":
// Close closes the connection.
// Any blocked ReadFrom or WriteTo operations will be unblocked and return errors.
Close() error
Depending on your needs a timeout might be an option as well, see again PacketConn documentation in the net package:
// ReadFrom reads a packet from the connection,
// copying the payload into b. It returns the number of
// bytes copied into b and the return address that
// was on the packet.
// ReadFrom can be made to time out and return
// an Error with Timeout() == true after a fixed time limit;
// see SetDeadline and SetReadDeadline.
ReadFrom(b []byte) (n int, addr Addr, err error)
Related
I am writing a code, which reads data from network with timeouts. Initially I wrote this (much simplified skipped error checks too)
func read_msg(conn net.Conn, ch chan []byte) {
b := bufio.NewReader(conn)
msg,_ := b.ReadBytes(byte('\n'))
ch <- msg
close(ch)
}
func main() {
ln,_ := net.Listen("tcp", ":12345")
conn,_ := ln.Accept()
for {
ch := make(chan []byte)
go read_msg(conn,ch)
select {
case msg := <-ch:
fmt.Println("message received: ", msg)
case <-time.After(time.Duration(1)*time.Second):
fmt.Println("timeout")
}
}
}
When the timeout happens, the goroutine stays active, waiting on the bufio.ReadBytes(). Is there anyway to set a timeout on the bufio itself?
No, bufio does not have this feature (you can read the docs here).
The more appropriate solution is to use the SetReadDeadline method of the net.Conn value.
// A deadline is an absolute time after which I/O operations
// fail instead of blocking. The deadline applies to all future
// and pending I/O, not just the immediately following call to
// Read or Write.
...
// SetReadDeadline sets the deadline for future Read calls
// and any currently-blocked Read call.
// A zero value for t means Read will not time out.
For example:
conn.SetReadDeadline(time.Now().Add(time.Second))
// relay copies between left and right bidirectionally. Returns number of
// bytes copied from right to left, from left to right, and any error occurred.
func relay(left, right net.Conn) (int64, int64, error) {
type res struct {
N int64
Err error
}
ch := make(chan res)
go func() {
n, err := io.Copy(right, left)
right.SetDeadline(time.Now()) // wake up the other goroutine blocking on right
left.SetDeadline(time.Now()) // wake up the other goroutine blocking on left
ch <- res{n, err}
}()
n, err := io.Copy(left, right)
right.SetDeadline(time.Now()) // wake up the other goroutine blocking on right
left.SetDeadline(time.Now()) // wake up the other goroutine blocking on left
rs := <-ch
if err == nil {
err = rs.Err
}
return n, rs.N, err
}
This is from the go-shadowsocks2 project, however, I couldn't understand the left.SetDeadline(time.Now()) part, and what does the comment (wake up the other goroutine blocking on left) mean?
SetDeadline with argument of time.Now() seems unusual, can anyone help me understand this?
This might help, from the net.Conn documentation:
A deadline is an absolute time after which I/O operations
fail instead of blocking. The deadline applies to all future
and pending I/O, not just the immediately following call to
Read or Write. After a deadline has been exceeded, the
connection can be refreshed by setting a deadline in the future.
It appears like there are two goroutines copying data from one connection to the other. When the source connection for either of those operation close, that copy operation will terminate, but the other copy operation will be blocked. Based on the net.Conn documentation, by setting the timeout it will cause the blocked copy operation to fail, unblocking the gouroutine.
I know there is a function called SetReadDeadline that can set a timeout in socket(conn.net) reading, while io.Read not. There is a way that starts another routine as a timer to solve this problem, but it brings another problem that the reader routine(io.Read) still block:
func (self *TimeoutReader) Read(buf []byte) (n int, err error) {
ch := make(chan bool)
n = 0
err = nil
go func() { // this goroutime still exist even when timeout
n, err = self.reader.Read(buf)
ch <- true
}()
select {
case <-ch:
return
case <-time.After(self.timeout):
return 0, errors.New("Timeout")
}
return
}
This question is similar in this post, but the answer is unclear.
Do you guys have any good idea to solve this problem?
Instead of setting a timeout directly on the read, you can close the os.File after a timeout. As written in https://golang.org/pkg/os/#File.Close
Close closes the File, rendering it unusable for I/O. On files that support SetDeadline, any pending I/O operations will be canceled and return immediately with an error.
This should cause your read to fail immediately.
Your mistake here is something different:
When you read from the reader you just read one time and that is wrong:
go func() {
n, err = self.reader.Read(buf) // this Read needs to be in a loop
ch <- true
}()
Here is a simple example (https://play.golang.org/p/2AnhrbrhLrv)
buf := bytes.NewBufferString("0123456789")
r := make([]byte, 3)
n, err := buf.Read(r)
fmt.Println(string(r), n, err)
// Output: 012 3 <nil>
The size of the given slice is used when using the io.Reader. If you would log the n variable in your code you would see, that not the whole file is read. The select statement outside of your goroutine is at the wrong place.
go func() {
a := make([]byte, 1024)
for {
select {
case <-quit:
result <- []byte{}
return
default:
_, err = self.reader.Read(buf)
if err == io.EOF {
result <- a
return
}
}
}
}()
But there is something more! You want to implement the io.Reader interface. After the Read() method is called until the file ends you should not start a goroutine in here, because you just read chunks of the file.
Also the timeout inside the Read() method doesn't help, because that timeout works for each call and not for the whole file.
In addition to #apxp's point about looping over Read, you could use a buffer size of 1 byte so that you never block as long is there is data to read.
When interacting with external resources anything can happen. It is possible for any given io.Reader implementation to simply block forever. Here, I'll write one for you...
type BlockingReader struct{}
func (BlockingReader) Read(b []byte) (int, error) {
<-make(chan struct{})
return 0, nil
}
Remember anyone can implement an interface, so you can't make any assumptions that it will behave like *os.File or any other standard library io.Reader. In addition to asinine coding like mine above, an io.Reader could legitimately connect to a resources that can block forever.
You cannot kill gorountines, so if an io.Reader truly blocks forever the blocked goroutine will continue to consume resources until your application terminates. However, this shouldn't be a problem, a blocked goroutine does not consume much in the way of resources, and should be fine as long as you don't blindly retry blocked Reads by spawning more gorountines.
I've implemented a demo tcp chat server in golang, it works fine, but every time a user disconnects and I try to write a message to the broadcast channel to let other users know a user has disconnected it blocks, and would not further process any new messages from other client because its a nonbuffered channel
I've commented by code and explained it you can go through it, I don't know why the code blocks, I've written msgs
I'm about to write to a channel
I've written to the channel
I've read from the channel
and messages are in perfect order still my msg channel blocks.
Ps: If I'm using buffered channel the code is not blocking, but I want to know where is my code getting stuck.
I also tried running my code with -race flag but no help
package main
import (
"fmt"
"io"
"net"
"sync"
)
func main() {
msg := make(chan string) //broadcast channel (making it buffered channel the problem goes away)
allConn := make(map[net.Conn]int) //Collection of incoming connections for broadcasting the message
disConn := make(chan net.Conn) //client disconnect channel
newConn := make(chan net.Conn) //new client connection channel
mutext := new(sync.RWMutex) //mux to assign unique id to incoming connections
i := 0
listener, err := net.Listen("tcp", "127.0.0.1:8081")
checkErr(err)
fmt.Println("Tcp server started at 127.0.0.1:8081")
//Accept incoming connections and store them in global connection store allConn
go func() {
for {
conn, err := listener.Accept()
checkErr(err)
mutext.Lock()
allConn[conn] = i
i++
mutext.Unlock()
newConn <- conn
}
}()
for {
select {
//Wait for a new client message to arrive and broadcast the message
case umsg := <-msg:
fmt.Println("Broadcast Channel: Already Read")
bmsg := []byte(umsg)
for conn1, _ := range allConn {
_, err := conn1.Write(bmsg)
checkErr(err)
}
//Handle client disconnection [disConn]
case conn := <-disConn:
mutext.RLock()
fmt.Println("user disconneting", allConn[conn])
mutext.RUnlock()
delete(allConn, conn)
fmt.Println("Disconnect: About to Write")
//this call results in deadlock even when channel is empty, buffered channel resolves the issue
//need to know why
msg <- fmt.Sprintf("Disconneting", allConn[conn])
fmt.Println("Disconnect: Already Written")
//Read client incoming message and put it on broadcasting channel and upon disconnect put on it disConn channel
case conn := <-newConn:
go func(conn net.Conn) {
for {
buf := make([]byte, 64)
n, err := conn.Read(buf)
if err != nil {
if err == io.EOF {
disConn <- conn
break
}
}
fmt.Println("Client: About to Write")
msg <- string(buf[0:n])
fmt.Println("Client: Already Written")
}
}(conn)
mutext.RLock()
fmt.Println("User Connected", allConn[conn])
mutext.RUnlock()
}
}
}
func checkErr(err error) {
if err != nil {
panic(err)
}
}
In Go, an unbuffered channel is a "synchronisation point". That is, if you have a channel c, and do c <- value, the goroutine blocks until someone is ready to do v = <- c (and the converse holds, receiving from a blocking channel without something to receive blocks until the value is available, but this is possibly less surprising). Specifically, for a blocking channel, the receive completes before the send completes.
Since you only have a single goroutine, it will be unable to loop back to reading from the channel and the write will block until something can read.
You could, in theory, get around this by doing something like: go func() { msg <- fmt.Sprintf("Disconneting", allConn[conn] }(), so essentially spawning a short-lived goroutine to do the write.
It doesn't seem possible to have two way communication via channels with a goroutine which is performing file operations, unless you block the channel communication on the file operations. How can I work around the limits this imposes?
Another way to phrase this question...
If I have a loop similar to the following running in a goroutine, how can I tell it to close the connection and exit without blocking on the next Read?
func readLines(response *http.Response, outgoing chan string) error {
defer response.Body.Close()
reader := bufio.NewReader(response.Body)
for {
line, err := reader.ReadString('\n')
if err != nil {
return err
}
outgoing <- line
}
}
It's not possible for it to read from a channel that tells it when to close down because it's blocking on the network reads (in my case, that can take hours).
It doesn't appear to be safe to simply call Close() from outside the goroutine, since the Read/Close methods don't appear to be fully thread safe.
I could simply put a lock around references to response.Body that used inside/outside the routine, but would cause the external code to block until a pending read completes, and I specifically want to be able to interrupt an in-progress read.
To address this scenario, several io.ReadCloser implementations in the standard library support concurrent calls to Read and Close where Close interrupts an active Read.
The response body reader created by net/http Transport is one of those implementations. It is safe to concurrently call Read and Close on the response body.
You can also interrupt an active Read on the response body by calling the Transport CancelRequest method.
Here's how implement cancel using close on the body:
func readLines(response *http.Response, outgoing chan string, done chan struct{}) error {
cancel := make(chan struct{})
go func() {
select {
case <-done:
response.Body.Close()
case <-cancel:
return
}()
defer response.Body.Close()
defer close(cancel) // ensure that goroutine exits
reader := bufio.NewReader(response.Body)
for {
line, err := reader.ReadString('\n')
if err != nil {
return err
}
outgoing <- line
}
}
Calling close(done) from another goroutine will cancel reads on the body.