According to the gRPC documentation, deadlines can be specified by clients to determine how long the client will wait on the server before exiting with a DEADLINE_EXCEEDED error. The documentation mentions that different languages have different implementations and that some languages do not have default values.
Indeed, a quick CTRL+F for "deadline" on the Go gRPC documentation reveals no results. What I did discover was a WithTimeout on the dialer for the TCP connection.
Implemented as follows (from the helloworld example):
package main
import (
"log"
"os"
"time"
"golang.org/x/net/context"
"google.golang.org/grpc"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
)
const (
address = "localhost:50051"
defaultName = "world"
deadline = 20
)
func main() {
// Set up a connection to the server with a timeout
conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithTimeout(time.Duration(deadline)*time.Second)
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
// Contact the server and print out its response.
name := defaultName
if len(os.Args) > 1 {
name = os.Args[1]
}
r, err := c.SayHello(context.Background(), &pb.HelloRequest{Name: name})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.Message)
}
The code will raise an error only if the client cannot connect after 20 seconds. The output will be something as follows:
2016/05/24 09:02:54 grpc: Conn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::1]:3265: getsockopt: connection refused"; Reconnecting to "localhost:3265"
2016/05/24 09:02:54 Failed to dial localhost:3265: grpc: timed out trying to connect; please retry.
2016/05/24 09:02:54 could not greet: rpc error: code = 2 desc = grpc: the client connection is closing
As noted in the question title, the system I'm working with is peer to peer, so there is no central, always up server and therefore the retry system that gRPC implements is wonderful. However, I'm actually looking for deadlines because if the remote does connect, but the server takes > 20 seconds to respond, no exception will be raised in the WithTimeout context.
A complete win for me would be a timeout/deadline system where:
if the client cannot connect, an error is returned after timeout
if the client connects, but the server doesn't respond before timeout, an error is returned.
if the client connects, but the connection drops before timeout, an error is returned.
My feeling though is that I will need some combination of connection management plus gRPC deadline management. Does anyone know how to implement deadlines in Go?
According to the WithTimeout example of context
package main
import (
"context"
"fmt"
"time"
)
func main() {
// Pass a context with a timeout to tell a blocking function that it
// should abandon its work after the timeout elapses.
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
select {
case <-time.After(1 * time.Second):
fmt.Println("overslept")
case <-ctx.Done():
fmt.Println(ctx.Err()) // prints "context deadline exceeded"
}
}
You can change the helloworld example client code for 100ms timeout:
ctx, _ := context.WithTimeout(context.Background(), 100 * time.Millisecond)
r, err := c.SayHello(ctx, &pb.HelloRequest{Name: name})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.Message)
You should look at the context package more closely. GRPC was built with contexts as a fundamental part of it. You might need the grpc.Dial context and client.SayHello context to be built off related info, but that should be fairly straight forward.
Related
In Go, how can I start the browser AFTER the server started listening?
Preferably the simplest way possible.
My code so far, super dumbed down to the point:
package main
import (
// Standard library packages
"fmt"
"net/http"
"github.com/skratchdot/open-golang/open"
// Third party packages
"github.com/julienschmidt/httprouter"
)
// go get github.com/toqueteos/webbrowser
func main() {
// Instantiate a new router
r := httprouter.New()
// Add a handler on /test
r.GET("/test", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
// Simply write some test data for now
fmt.Fprint(w, "Welcome!\n")
})
//open.Run("https://google.com/")
// open.Start("https://google.com")
// http://127.0.0.1:3000/test
// Fire up the server
http.ListenAndServe("localhost:3000", r)
fmt.Println("ListenAndServe is blocking")
open.RunWith("http://localhost:3000/test", "firefox")
fmt.Println("Done")
}
Open the listener, start the browser and then enter the server loop:
l, err := net.Listen("tcp", "localhost:3000")
if err != nil {
log.Fatal(err)
}
// The browser can connect now because the listening socket is open.
err := open.Start("http://localhost:3000/test")
if err != nil {
log.Println(err)
}
// Start the blocking server loop.
log.Fatal(http.Serve(l, r))
There's no need to poll as shown in another answer. The browser will connect if the listening socket is open before the browser is started.
ListenAndServe is a convenience function that opens a socket and calls Serve. The code in this answer splits out these steps so the browser can be opened after listening starts but before the blocking call to Serve.
If there is no error, http.ListenAndServe() will never return. So you shouldn't add code after that except code that handles failure.
You have to start a new goroutine, so ListenAndServe() is called in one goroutine, and code checking if it is up should run on the other goroutine.
And you can check if your server is up by making a simple HTTP GET call to it, for example using http.Get().
The following example delays startup for 7 seconds on purpose. The new goroutine starts an endless for loop that checks if server is up, sleeping 1 second between attempts.
Example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hi!"))
})
go func() {
for {
time.Sleep(time.Second)
log.Println("Checking if started...")
resp, err := http.Get("http://localhost:8081")
if err != nil {
log.Println("Failed:", err)
continue
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Println("Not OK:", resp.StatusCode)
continue
}
// Reached this point: server is up and running!
break
}
log.Println("SERVER UP AND RUNNING!")
}()
log.Println("Starting server...")
time.Sleep(time.Second * 7)
log.Fatal(http.ListenAndServe(":8081", nil))
Example output:
2015/09/23 13:53:03 Starting server...
2015/09/23 13:53:04 Checking if started...
2015/09/23 13:53:06 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:07 Checking if started...
2015/09/23 13:53:09 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:10 Checking if started...
2015/09/23 13:53:10 SERVER UP AND RUNNING!
The API is not absolutely terrible, but let's just say "It takes some getting used to". Here is how you use custom attributes on the Server struct:
s := &http.Server{
Addr: cnf.API_SERVER_ADDRESS,
Handler: h,
ReadTimeout: 0, // 1 * time.Minute,
WriteTimeout: 30 * time.Minute,
MaxHeaderBytes: 1 << 20,
}
go func() {
l, err := net.Listen("tcp", cnf.API_SERVER_ADDRESS)
if err != nil {
log.Fatal(err)
}
fmt.Println(`{"server_state":"listening"}`)
log.Fatal(s.Serve(l));
}()
because if you instead use:
http.Serve(l, handler)
then you can't define custom properties on the server
In Go, how can I start the browser AFTER the server started listening?
Preferably the simplest way possible.
My code so far, super dumbed down to the point:
package main
import (
// Standard library packages
"fmt"
"net/http"
"github.com/skratchdot/open-golang/open"
// Third party packages
"github.com/julienschmidt/httprouter"
)
// go get github.com/toqueteos/webbrowser
func main() {
// Instantiate a new router
r := httprouter.New()
// Add a handler on /test
r.GET("/test", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
// Simply write some test data for now
fmt.Fprint(w, "Welcome!\n")
})
//open.Run("https://google.com/")
// open.Start("https://google.com")
// http://127.0.0.1:3000/test
// Fire up the server
http.ListenAndServe("localhost:3000", r)
fmt.Println("ListenAndServe is blocking")
open.RunWith("http://localhost:3000/test", "firefox")
fmt.Println("Done")
}
Open the listener, start the browser and then enter the server loop:
l, err := net.Listen("tcp", "localhost:3000")
if err != nil {
log.Fatal(err)
}
// The browser can connect now because the listening socket is open.
err := open.Start("http://localhost:3000/test")
if err != nil {
log.Println(err)
}
// Start the blocking server loop.
log.Fatal(http.Serve(l, r))
There's no need to poll as shown in another answer. The browser will connect if the listening socket is open before the browser is started.
ListenAndServe is a convenience function that opens a socket and calls Serve. The code in this answer splits out these steps so the browser can be opened after listening starts but before the blocking call to Serve.
If there is no error, http.ListenAndServe() will never return. So you shouldn't add code after that except code that handles failure.
You have to start a new goroutine, so ListenAndServe() is called in one goroutine, and code checking if it is up should run on the other goroutine.
And you can check if your server is up by making a simple HTTP GET call to it, for example using http.Get().
The following example delays startup for 7 seconds on purpose. The new goroutine starts an endless for loop that checks if server is up, sleeping 1 second between attempts.
Example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hi!"))
})
go func() {
for {
time.Sleep(time.Second)
log.Println("Checking if started...")
resp, err := http.Get("http://localhost:8081")
if err != nil {
log.Println("Failed:", err)
continue
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Println("Not OK:", resp.StatusCode)
continue
}
// Reached this point: server is up and running!
break
}
log.Println("SERVER UP AND RUNNING!")
}()
log.Println("Starting server...")
time.Sleep(time.Second * 7)
log.Fatal(http.ListenAndServe(":8081", nil))
Example output:
2015/09/23 13:53:03 Starting server...
2015/09/23 13:53:04 Checking if started...
2015/09/23 13:53:06 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:07 Checking if started...
2015/09/23 13:53:09 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:10 Checking if started...
2015/09/23 13:53:10 SERVER UP AND RUNNING!
The API is not absolutely terrible, but let's just say "It takes some getting used to". Here is how you use custom attributes on the Server struct:
s := &http.Server{
Addr: cnf.API_SERVER_ADDRESS,
Handler: h,
ReadTimeout: 0, // 1 * time.Minute,
WriteTimeout: 30 * time.Minute,
MaxHeaderBytes: 1 << 20,
}
go func() {
l, err := net.Listen("tcp", cnf.API_SERVER_ADDRESS)
if err != nil {
log.Fatal(err)
}
fmt.Println(`{"server_state":"listening"}`)
log.Fatal(s.Serve(l));
}()
because if you instead use:
http.Serve(l, handler)
then you can't define custom properties on the server
I'm trying to debug a flaky unit test which is similar to the following:
package main
import (
"net"
"net/http"
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestFlaky(t *testing.T) {
// Mock an API
http.HandleFunc("/foo/bar", func(w http.ResponseWriter, r *http.Request) {
_, err := w.Write([]byte("foobar"))
require.NoError(t, err)
})
go func() {
require.NoError(t, http.ListenAndServe("localhost:7777", nil))
}()
// Wait (up to 1 second) for the mocked API to be available
conn, err := net.DialTimeout("tcp", "localhost:7777", time.Second)
require.NoError(t, err)
require.NoError(t, conn.Close())
}
However, from the require.NoError() line right after the DialTimeout error, I'm getting the following error (in a CI environment only):
--- FAIL: TestFlaky (0.00s)
main_test.go:24:
Error Trace: main_test.go:24
Error: Received unexpected error:
dial tcp [::1]:7777: connect: connection refused
Test: TestFlaky
Since the test fails immediately, I'm guessing this is not a matter of adjusting the timeout. How should I make this test non-flaky? I'm considering replacing the last three lines with a require.Eventually similar to the following:
var conn net.Conn
require.Eventually(t, func() bool {
var err error
conn, err = net.DialTimeout("tcp", "localhost:7777", time.Second)
if err != nil {
t.Logf("DialTimeout error: %v. Retrying...", err)
return false
}
return true
}, time.Second, 100*time.Millisecond)
require.NoError(t, conn.Close())
Would this suffice to remove test flakiness?
The code inside the goroutine does not guarantee to be executed before the Dial. (If you put a sleep just after the goroutine it should work but it is an ugly solution).
Also, notice that "dial with timeout" is waiting for tcp packets when connection is being stablished, but the refused connection is actually the RST packet.
Hint: see how httptest package works.
Edit: here is how httptest works: https://cs.opensource.google/go/go/+/refs/tags/go1.16.6:src/net/http/httptest/server.go;l=304
func (s *Server) goServe() {
s.wg.Add(1)
go func() {
defer s.wg.Done()
s.Config.Serve(s.Listener)
}()
}
Hi I'm trying to connect a gRPC client to the server but even though the connection is succesful I get the following error when querying it from the graphql resolvers. However if I dial directly from the resolver everything works so it has to do with the client not leaving the connection open.
rpc error: code = Canceled desc = grpc: the client connection is closing
client.go
var kacp = keepalive.ClientParameters{
Time: 10 * time.Second, // send pings every 10 seconds if there is no activity
Timeout: time.Second, // wait 1 second for ping back
PermitWithoutStream: true, // send pings even without active streams
}
func gqlHandler() http.HandlerFunc {
conn, err := grpc.Dial("127.0.0.1:50051", grpc.WithInsecure(),
grpc.WithKeepaliveParams(kacp),
grpc.WithBlock())
if err != nil {
panic(err)
}
defer conn.Close()
db := proto.NewPlatformDBClient(conn)
gh := handler.GraphQL(platform.NewExecutableSchema(platform.Config{Resolvers: &platform.Resolver{
DB: db,
}}))
gh = cors.Disable(gh)
return gh
}
Its because the
defer conn.Close()
command will be executed before the connection is even used.
From the go blog
A defer statement pushes a function call onto a list. The list of
saved calls is executed after the surrounding function returns.
So you would remove the line defer conn.Close() and close the connection after it is not used anymore.
Please check if the connection is not nil and then proceed to close it.
As per the error defined in the close() the ErrClientConnClosing error is occurring if the connection is nil .
Code ref : https://github.com/grpc/grpc-go/blob/master/clientconn.go#L1064
if c.conn != nil { errs := c.Close() }
I wrote this simple proxy server with net package, which I expected to proxy connections from a local server at 8001 to any incoming connections via 8000. When I go to the browser and try it, I get a refused to connect error.
package main
import (
"net"
"log"
"io"
)
func main() {
l, err := net.Listen("tcp", "localhost:8000")
if err != nil {
log.Fatal(err)
}
for {
conn, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go proxy(conn)
}
}
func proxy(conn net.Conn) {
defer conn.Close()
upstream, err := net.Dial("tcp","localhost:8001")
if err != nil {
log.Print(err)
return
}
defer upstream.Close()
io.Copy(upstream, conn)
io.Copy(conn, upstream)
}
But if I change the following lines in the proxy function
io.Copy(upstream, conn)
io.Copy(conn, upstream)
to
go io.Copy(upstream, conn)
io.Copy(conn, upstream)
then it works as expected. Shouldn't the io.Copy(upstream, conn) block the io.Copy(conn, upstream)? As per my understanding, the conn should be written only after upstream has responded back? And how does having a go routine for io.Copy(upstream, conn) part solve this?
Shouldn't io.Copy block?
Yes. "Copy copies from src to dst until either EOF is reached on src or an error occurs.". Since this is a network connection, this means it returns after the client closes the connection. If and when the client closes the connection depends on the application protocol. In HTTP it may never happen, for instance.
How does having a goroutine solve this?
Because then the second Copy can execute while the client is still connected, allowing the upstream to write its response. Without the goroutine nothing is reading from the upstream, so it is likely blocked on its write call.
The client (presumably) waits for a response, the proxy waits for the client to close the connection, and the upstream waits for the proxy to start reading the response: no one can make progress and you're in a deadlock.