Keep-alive: dead peers detection - go

I run client and socket server written in Go (1.12) on macOS localhost.
Server sets SetKeepAlive and SetKeepAlivePeriod on net.TCPConn.
Client sends a packet and then closes connection (FIN) or client abruptly terminated.
Tcpdump shows that even after client closes the connection, server keeps sending keep-alive probes.
Shouldn't it detect that peer is "dead" and close the connection?
The question is generic, feel free to clarify if I'm missing some basics.
package main
import (
"flag"
"fmt"
"net"
"os"
"time"
)
func main() {
var client bool
flag.BoolVar(&client, "client", false, "")
flag.Parse()
if client {
fmt.Println("Client mode")
conn, err := net.Dial("tcp", "127.0.0.1:12345")
checkErr("Dial", err)
written, err := conn.Write([]byte("howdy"))
checkErr("Write", err)
fmt.Printf("Written: %v\n", written)
fmt.Println("Holding conn")
time.Sleep(60 * time.Second)
err = conn.Close()
checkErr("Close", err)
fmt.Println("Closed conn")
return
}
fmt.Println("Server mode")
l, err := net.Listen("tcp", "127.0.0.1:12345")
checkErr("listen", err)
defer l.Close()
for {
c, err := l.Accept()
checkErr("accept", err)
defer c.Close()
tcpConn := c.(*net.TCPConn)
err = tcpConn.SetKeepAlive(true)
checkErr("SetKeepAlive", err)
err = tcpConn.SetKeepAlivePeriod(5 * time.Second)
checkErr("SetKeepAlivePeriod", err)
b := make([]byte, 1024)
n, err := c.Read(b)
checkErr("read", err)
fmt.Printf("Received: %v\n", string(b[:n]))
}
}
func checkErr(location string, err error) {
if err != nil {
fmt.Printf("%v: %v\n", location, err)
os.Exit(-1)
}
}

The response to that question:
Sending keepalives is only necessary when you need the connection opened but idle. In that cases there is a risk that the connection is broken, so keep alive will try to detect broken connections.
If you had close the connection at server side with a proper con.Close() the keep alive would not be triggered (you did defer it to the end of the main function).
If you test your server code, it will start sending the keep alive after the timeout you set.
You notice that only after all keep alive proves (default 9 from kernel) and the time between the proves (8x), you get an io.EOF error on the server side Read (yes, the server stop sending)!
Currently the GO implementation is the same at Linux and OSX and it set both TCP_KEEPINTVL and TCP_KEEPIDLE to the value you pass to the setKeepAlivePeriod function, so, the behavior will depend of the kernel version.
func setKeepAlivePeriod(fd *netFD, d time.Duration) error {
// The kernel expects seconds so round to next highest second.
d += (time.Second - time.Nanosecond)
secs := int(d.Seconds())
if err := fd.pfd.SetsockoptInt(syscall.IPPROTO_TCP, syscall.TCP_KEEPINTVL, secs); err != nil {
return wrapSyscallError("setsockopt", err)
}
err := fd.pfd.SetsockoptInt(syscall.IPPROTO_TCP, syscall.TCP_KEEPIDLE, secs)
runtime.KeepAlive(fd)
return wrapSyscallError("setsockopt", err)
}
There is a request opened since 2014 to provide a way to set keepalive time and interval separately.
Some references:
rfc1122
net: enable TCP keepalive on new connections from net.Dial
net: enable TCP keepalives by default
TCP keep-alive to determine if client disconnected in netty
Using TCP keepalive with Go

Related

How to drop a UDP connection between server and client from server side in Golang?

I'm trying to find a standard way to close a connection between a client and server in a UDP connection.
Currently, I came up with the following solution, however, I'm not sure whether this is an idiomatic way or not?
Basically what I'm doing here on the server-side (handleClient function) is to conn.WriteTo(nil, Addr) which write nil to the UDP address. on the client side I check if the read() function retrieved any data or not, in case the number of the reading byte is zero, the client gives up on reading.
if n ==0 || err != nil {
break
}
Here is my simplified Server.go file:
func handleClient(conn *net.UDPConn) {
b := make([]byte, 1024)
n, Addr, err := conn.ReadFrom(b[0:])
if err != nil {
panic(err)
}
fmt.Println("read: ", n, " bytes", Addr.String())
// write the data back to the client, just for debug purpose
conn.WriteTo(b[0:n], Addr)
// let the client know that server has no more data
conn.WriteTo(nil, Addr)
}
And this is my client.go file
conn, err := net.Dial("udp", ":14000")
if err != nil {
panic(err)
}
defer conn.Close()
conn.Write([]byte("I'm client "))
for {
b := make([]byte, 512)
n, err := conn.Read(b)
if n ==0 || err != nil {
fmt.Println(">", n, err)
break
}
fmt.Println(string(b))
}
fmt.Println("finished.")
There is no standard way to do this. UDP is a stateless protocol so there is no "connection" like you would have with a TCP connection. DNS for example has no concept of a state, a client sends a request and the server will respond, no state is maintained, this is usually the selling point of UDP.
The applications are responsible for maintain state, so if you want a stateful protocol on top of UDP you also have to handle closing such a connection yourself. You might take inspiration from FTP or TFTP on how to implement this. Or consider using a stateful transport layer protocol like TCP or QUIC to handle stateful connections for you.

How to prevent SetReadDeadline from timing out connection between reads in go?

I'm writing a simple TLV style service using TCP in go, that should be able to handle multiple connections, and allow for multiple messages to be sent on the same connection.
I need to be able to ensure that a message with incorrect length doesn't block the connection indefinitely, but I need to ensure that subsequent messages can be sent on the same connection without timing out.
I am using io.ReadFull to read in a fixed number of bytes that contain the type and length, and then when I receive the length, I am calling io.ReadFull again and reading in length number of bytes.
If I am reading in 4 bytes for the type and length, but the client only sends 3 for whatever reason, io.ReadFull will hang. Similarly, if the client sends 300 for the length, but the length should only be 200, io.ReadFull will hang, blocking all communication on that channel.
I've tried using conn.SetReadDeadline() and setting it to 5 seconds for this example. This causes the connection to timeout if the improper length is sent, which is great. The issue is that the connection will timeout if the next request isn't sent until after >5 seconds.
// ...
for {
conn, err := lis.Accept()
if err != nil {
fmt.Println(err)
continue
}
fmt.Println("Connected")
go handleC(conn)
}
func handleC(conn net.Conn) {
for {
err := conn.SetReadDeadline(time.Now().Add(5 * time.Second))
if err != nil {
fmt.Println(err)
break
}
l, err := readTL(conn)
if err != nil {
if err, ok := err.(net.Error); ok && err.Timeout() {
fmt.Println("Timeout", err)
break
}
fmt.Println("Other error"), err
break
}
v, err := readV(conn, l)
if err != nil {
if err, ok := err.(net.Error); ok && err.Timeout() {
fmt.Println("Timeout", err)
break
}
fmt.Println("Other error"), err
break
}
// go doStuffWithv()
}
}
func readTL(conn net.Conn) (l int, err error) {
buf := make([]byte, 4)
n, err := io.ReadFull(conn, buf)
if err != nil {
return l, err
}
fmt.Printf("Read %d bytes\n", n)
// getLengthFromTL()
return l, err
}
func readV(conn net.Conn, l int) (v []byte, err error) {
buf := make([]byte, l)
n, err := io.ReadFull(conn, buf)
if err != nil {
return v, err
}
fmt.Printf("Read %d bytes\n", n)
return v, err
}
If a client sends one request with the proper TL, things work as intended.
However, if the same client doesn't send a second message for 10 seconds, the connection will timeout before then, with the error tls: use of closed connection
Is there a way to ensure that this doesn't occur?
One thing I've tried doing is in the event of a timeout, it simply continues, rather than breaking.
I added in another error check to see if it is EOF, and break if it is.
My first impression is that this works, but I'm not sure if there are instances where a connection timeout can mean that the connection is dead and shouldn't be used anymore or not, or if that would always return an EOF error.

Apparent deadlock between server and client

I have a test function which both creates a server and spawns a goroutine acting as a client. Now, simply sending a message from the client to the server works, but if I want to create an exchange, they seem to deadlock since the test never runs to completion (if no r/w deadlines are set). For example, I want the client to send a message to the server, the server to copy that message and send it back to the client, and then the client to verify that the received message was identical. Here is my test code:
func TestSendAwait(t *testing.T) {
m := "Hello World"
go func() {
conn, err := net.Dial("tcp", testingAddr)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
t.Log("client connected to server") // DEBUG
conn.SetDeadline(time.Now().Add(2 * time.Second))
conn.Write([]byte(m))
conn.SetDeadline(time.Now().Add(2 * time.Second))
buf, err := ioutil.ReadAll(conn)
if err != nil {
t.Fatal(err)
}
t.Log(string(buf))
}()
ln, err := net.Listen("tcp", testingAddr)
if err != nil {
t.Fatal(err)
}
defer ln.Close()
t.Log("server started") // DEBUG
conn, err := ln.Accept()
if err != nil {
t.Fatal(err)
}
defer conn.Close()
t.Log("server received connection") // DEBUG
buf, err := ioutil.ReadAll(conn)
if err != nil {
t.Fatal(err)
}
t.Logf("server read buffer: %v", buf) // DEBUG
_, err = conn.Write(buf)
if err != nil {
t.Fatal(err)
}
t.Log("server wrote to connection") // DEBUG
}
The deadlines are set on the connection because otherwise the deadlock would be indefinite. The output is as follows:
transmission_test.go:42: server started
transmission_test.go:24: client connected to server
transmission_test.go:49: server received connection
transmission_test.go:32: read tcp 127.0.0.1:41164->127.0.0.1:9090: i/o timeout
transmission_test.go:55: server read buffer: [72 101 108 108 111 32 87 111 114 108 100]
transmission_test.go:61: server wrote to connection
Process finished with exit code 1
I don't understand why the client is unable to read and exits, and only then the server decides to send data down the socket? This happens even if I increase the read deadline in the client.
The program blocks on the call to ioutil.ReadAll. This function reads until io.EOF or some other error is returned.
One fix is to shutdown write after writing data to the connection. This will cause read on the peer to return io.EOF and for ioutil.ReadAll to return successfully.
conn.Write(data)
cw, ok := conn.(interface{ CloseWrite() error })
if !ok {
// handle error
}
cw.CloseWrite()
playground example
The program in the question does not guarantee that the listener is opened before the connection is dialed or that client will print print the received message. The playground example corrects these issues.
Another approach is to frame the messages in some way:
Write newline or some other byte sequence not allowed in message after message. Read until this byte sequence is found.
Write message length before message. Read length and then specified number of bytes.

Can TCP clients receive data after a read timeout?

I create a tcp connection pool in golang and set every connection to keep alive, when I get a connection from pool, I will set a 10 seconds timeout by SetDeadline function. Now I want to know, if a timeout error occur after reading from a connection and server send message to me after this point, will the message be put into my connection receive buffer? Will I get the message in next reading? If so, how should I handle the timeout error? Close the connection and create a new one?
If a timeout error occur after reading from a connection and server send message to me after this point, will the message be put into my connection receive buffer?
If you don't close the connection, yes.
Will I get the message in next reading?
Yes, if you reset the deadline. Although this does beg the question why you set a timeout in the first place.
package main
import (
"fmt"
"io"
"log"
"net"
"time"
)
func main() {
l, err := net.Listen("tcp", "127.0.0.1:3001")
check(err)
go slowServer(l)
conn, err := net.Dial("tcp", "127.0.0.1:3001")
check(err)
conn.SetDeadline(time.Now().Add(time.Second))
b := make([]byte, 512)
n, err := conn.Read(b)
fmt.Printf("%q, %v\n", b[:n], err) // "", i/o timeout
// Reset deadline
conn.SetDeadline(time.Now().Add(2 * time.Second))
n, err = conn.Read(b)
fmt.Printf("%q, %v\n", b[:n], err) // "hello world", <nil>
}
func slowServer(l net.Listener) {
conn, err := l.Accept()
check(err)
time.Sleep(2 * time.Second)
io.WriteString(conn, "hello world")
conn.Close()
}
func check(err error) {
if err != nil {
log.Fatal(err)
}
}
// "", read tcp 127.0.0.1:50488->127.0.0.1:3001: i/o timeout
// "hello world", <nil>
Try it on the playground: https://play.golang.org/p/Id60hHK7tKF

How to set timeout while doing a net.DialTCP in golang?

As net.DialTCP seems like the only way to get net.TCPConn, I'm not sure how to set timeouts while doing the DialTCP. https://golang.org/pkg/net/#DialTCP
func connectAddress(addr *net.TCPAddr, wg *sync.WaitGroup) error {
start := time.Now()
conn, err := net.DialTCP("tcp", nil, addr)
if err != nil {
log.Printf("Dial failed for address: %s, err: %s", addr.String(), err.Error())
return err
}
elasped := time.Since(start)
log.Printf("Connected to address: %s in %dms", addr.String(), elasped.Nanoseconds()/1000000)
conn.Close()
wg.Done()
return nil
}
Use net.Dialer with either the Timeout or Deadline fields set.
d := net.Dialer{Timeout: timeout}
conn, err := d.Dial("tcp", addr)
if err != nil {
// handle error
}
A variation is to call Dialer.DialContext with a deadline or timeout applied to the context.
Type assert to *net.TCPConn if you specifically need that type instead of a net.Conn:
tcpConn, ok := conn.(*net.TCPConn)
One can use net.DialTimeout:
func DialTimeout(network, address string, timeout time.Duration) (Conn, error)
DialTimeout acts like Dial but takes a timeout.
The timeout includes name resolution, if required. When using TCP, and the
host in the address parameter resolves to multiple IP addresses, the timeout
is spread over each consecutive dial, such that each is given an appropriate
fraction of the time to connect.
See func Dial for a description of the network and address parameters.

Resources