Hi I'm trying to connect a gRPC client to the server but even though the connection is succesful I get the following error when querying it from the graphql resolvers. However if I dial directly from the resolver everything works so it has to do with the client not leaving the connection open.
rpc error: code = Canceled desc = grpc: the client connection is closing
client.go
var kacp = keepalive.ClientParameters{
Time: 10 * time.Second, // send pings every 10 seconds if there is no activity
Timeout: time.Second, // wait 1 second for ping back
PermitWithoutStream: true, // send pings even without active streams
}
func gqlHandler() http.HandlerFunc {
conn, err := grpc.Dial("127.0.0.1:50051", grpc.WithInsecure(),
grpc.WithKeepaliveParams(kacp),
grpc.WithBlock())
if err != nil {
panic(err)
}
defer conn.Close()
db := proto.NewPlatformDBClient(conn)
gh := handler.GraphQL(platform.NewExecutableSchema(platform.Config{Resolvers: &platform.Resolver{
DB: db,
}}))
gh = cors.Disable(gh)
return gh
}
Its because the
defer conn.Close()
command will be executed before the connection is even used.
From the go blog
A defer statement pushes a function call onto a list. The list of
saved calls is executed after the surrounding function returns.
So you would remove the line defer conn.Close() and close the connection after it is not used anymore.
Please check if the connection is not nil and then proceed to close it.
As per the error defined in the close() the ErrClientConnClosing error is occurring if the connection is nil .
Code ref : https://github.com/grpc/grpc-go/blob/master/clientconn.go#L1064
if c.conn != nil { errs := c.Close() }
Related
In Go, how can I start the browser AFTER the server started listening?
Preferably the simplest way possible.
My code so far, super dumbed down to the point:
package main
import (
// Standard library packages
"fmt"
"net/http"
"github.com/skratchdot/open-golang/open"
// Third party packages
"github.com/julienschmidt/httprouter"
)
// go get github.com/toqueteos/webbrowser
func main() {
// Instantiate a new router
r := httprouter.New()
// Add a handler on /test
r.GET("/test", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
// Simply write some test data for now
fmt.Fprint(w, "Welcome!\n")
})
//open.Run("https://google.com/")
// open.Start("https://google.com")
// http://127.0.0.1:3000/test
// Fire up the server
http.ListenAndServe("localhost:3000", r)
fmt.Println("ListenAndServe is blocking")
open.RunWith("http://localhost:3000/test", "firefox")
fmt.Println("Done")
}
Open the listener, start the browser and then enter the server loop:
l, err := net.Listen("tcp", "localhost:3000")
if err != nil {
log.Fatal(err)
}
// The browser can connect now because the listening socket is open.
err := open.Start("http://localhost:3000/test")
if err != nil {
log.Println(err)
}
// Start the blocking server loop.
log.Fatal(http.Serve(l, r))
There's no need to poll as shown in another answer. The browser will connect if the listening socket is open before the browser is started.
ListenAndServe is a convenience function that opens a socket and calls Serve. The code in this answer splits out these steps so the browser can be opened after listening starts but before the blocking call to Serve.
If there is no error, http.ListenAndServe() will never return. So you shouldn't add code after that except code that handles failure.
You have to start a new goroutine, so ListenAndServe() is called in one goroutine, and code checking if it is up should run on the other goroutine.
And you can check if your server is up by making a simple HTTP GET call to it, for example using http.Get().
The following example delays startup for 7 seconds on purpose. The new goroutine starts an endless for loop that checks if server is up, sleeping 1 second between attempts.
Example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hi!"))
})
go func() {
for {
time.Sleep(time.Second)
log.Println("Checking if started...")
resp, err := http.Get("http://localhost:8081")
if err != nil {
log.Println("Failed:", err)
continue
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Println("Not OK:", resp.StatusCode)
continue
}
// Reached this point: server is up and running!
break
}
log.Println("SERVER UP AND RUNNING!")
}()
log.Println("Starting server...")
time.Sleep(time.Second * 7)
log.Fatal(http.ListenAndServe(":8081", nil))
Example output:
2015/09/23 13:53:03 Starting server...
2015/09/23 13:53:04 Checking if started...
2015/09/23 13:53:06 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:07 Checking if started...
2015/09/23 13:53:09 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:10 Checking if started...
2015/09/23 13:53:10 SERVER UP AND RUNNING!
The API is not absolutely terrible, but let's just say "It takes some getting used to". Here is how you use custom attributes on the Server struct:
s := &http.Server{
Addr: cnf.API_SERVER_ADDRESS,
Handler: h,
ReadTimeout: 0, // 1 * time.Minute,
WriteTimeout: 30 * time.Minute,
MaxHeaderBytes: 1 << 20,
}
go func() {
l, err := net.Listen("tcp", cnf.API_SERVER_ADDRESS)
if err != nil {
log.Fatal(err)
}
fmt.Println(`{"server_state":"listening"}`)
log.Fatal(s.Serve(l));
}()
because if you instead use:
http.Serve(l, handler)
then you can't define custom properties on the server
As connection opening each time for publishing is costly I'm trying to implement some way to keep the connection alive and share it in my app to publish messages.
var (
Connection *amqp.Connection
Channel *amqp.Channel
err error
)
func Connect() {
Connection, err = amqp.Dial("amqp://guest:guest#localhost:5672")
FailOnError(err, "Failed to connect to RabbitMQ")
Channel, err = Connection.Channel()
FailOnError(err, "Failed to open a channel")
}
func CloseConnection() {
err = Channel.Close()
FailOnError(err, "Failed to close channel ")
err = Connection.Close()
FailOnError(err, "Failed to close connection ")
}
func KeepAlive() {
queue, err := Channel.QueueDeclare(
"hello", // name
false, // durable
false, // delete when unused
false, // exclusive
false, // no-wait
nil, // arguments
)
FailOnError(err, "couldn't publish tics")
tic := "tic"
for {
err := Channel.Publish(
"", // exchange
queue.Name, // routing key
false, // mandatory
false, // immediate
amqp.Publishing {
ContentType: "text/plain",
Body: []byte(tic),
Expiration: "5000",
})
FailOnError(err, "couldn't publish tics")
time.Sleep(5 *time.Second)
}
}
func FailOnError(err error, msg string) {
if err != nil {
log.Fatalf("%s: %s", msg, err)
}
}
The function KeepAlive is an infinite loop that keeps sending a dummy message every 5 secs and that message have a TTL of 5 secs too so it gets destroyed.
func main() {
rabbitmq.Connect()
defer rabbitmq.CloseConnection()
go func() {
//publisher connection to stay alive as long as application is running
rabbitmq.KeepAlive()
}()
data_layer.OpenDBConnection()
router := gin.Default()
router.POST("/whatever", whatever)
err := router.Run()
if err != nil {
log.Fatal(err.Error())
}
}
Here I'm creating the connection and calling KeepAlive in a goroutine so it can work in the background, keeping my connection alive all the time.
My questions:
I feel that this way is just a work around and although I've tried to search for examples how to keep it alive, it seems all of these examples are interested in the consumer side. Is there a cleaner way to keep my connection alive?
Is keeping my connection alive as long as my application is running bad? performance wise (network, memory usage)? note: I'm planning to monitor this with Prometheus to watch the performance but any note about what I might face would be helpful
Side note: these tics that are sent will be sent to a dummy queue since if I send it to my queue that I consume messages from by another service it will get stuck behind actual messages that doesn't have TTL and these tics will grow very large.
With streadway/amqp you don't need to implement the keepalive yourself. The library already provides this mechanism.
The method amqp.Dial constructs a Connection with a default heartbeat of 10 seconds. You can see the code here:
// connection.go
func Dial(url string) (*Connection, error) {
return DialConfig(url, Config{
Heartbeat: defaultHeartbeat,
Locale: defaultLocale,
})
}
This works by sending heartbeat frames on the open connection, which is going to be more efficient and maintainable than sending fake messages to a queue created only for that reason.
From the above follows that you can change the connection heartbeat with amqp.DialConfig:
conn, err := amqp.DialConfig(url, amqp.Config{
Heartbeat: 5 * time.Second,
})
What you might want to implement yourself is the reconnect-on-error logic. For that you can find some useful information here: How to check if the channel is still working in streadway/amqp RabbitMQ client?
I am getting back error websocket: close sent when I try to send data from my server side to client side(dashboard page). The error happen when I navigate to home page and back to dashboard page. Everything works fine initially with the Dashboard Page
My dashboard page code
let socket = new ReconnectingWebSocket("ws://127.0.0.1:8004/wsendpoint");
console.log("Attempting Connection...");
socket.onopen = () => {
console.log("Successfully Connected");
};
socket.onclose = event => {
console.log("Socket Closed Connection: ", event);
};
socket.onerror = error => {
console.log("Socket Error: ", error);
};
socket.onmessage = function (event) {
console.log("message received: " + event.data);
}
My Server side code (writer and reader
func wsEndpoint(w http.ResponseWriter, r *http.Request) {
upgrader.CheckOrigin = func(r *http.Request) bool { return true }
// upgrade this connection to a WebSocket connection
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println("upgrade error %s", err)
}
defer ws.Close()
go writer(ws)
go reader(ws)
}
Writer, get data from a channel when new data come in from database
func writer(conn *websocket.Conn) {
for {
singleset := <-singleorder // get data from channel
jsonString, err := json.Marshal(singleset )
err = conn.WriteMessage(1, []byte(jsonString))
if err != nil {
log.Println(err)
}
}
Reader, read data from client side
func reader(conn *websocket.Conn) {
for {
_, p, err := conn.ReadMessage() //what is message type?
if err != nil {
log.Println("there is errors%s", err)
return
}
}
}
I also got error WebSocket is closed before the connection is established. and reconnecting-websocket.min.js:1 WebSocket connection to 'ws://127.0.0.1:8004/wsendpoint' failed: WebSocket is closed before the connection is established.
As you can see all the code are very simple because I just start to learn websocket and follow simple tutorial. I tried to search on web but webworker seems little bit complex for me and saw about the ping pong method but I am not sure if its still valid if I navigate. Can I establish websocket on home page so the connection is not closed ? Since I only have two pages on client side.
Thanks in advance for any guidance on how to deal with these situation!
The basic strategy for dealing with navigation away from the page, page close or errors is this: The client creates the websocket connection on page load. The server expects client connections to come and go and cleans up resources when the client connection errors.
Here's why you get the error 'websocket: close sent': When the user navigates away from the page, the browser sends a close message to the server as part of the websocket closing handshake. The Gorilla package responds to the close message by sending another close message back to the client. From that point on, the server cannot send messages on the connection. The connection returns an error from write methods after the close message is sent.
Close messages are returned as errors from the websocket read methods. To fix the problem, modify the code to handle errors in general. There's no need to handle the closing handshake specifically.
Here's the updated code. The wsEndpoint function creates a channel for the reader to signal the writer that the reader is done. The defer ws.Close() statement is removed because the reader and writer goroutines will take responsibility for closing the connection.
func wsEndpoint(w http.ResponseWriter, r *http.Request) {
upgrader.CheckOrigin = func(r *http.Request) bool { return true }
// upgrade this connection to a WebSocket connection
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println("upgrade error %s", err)
}
done := make(chan struct{})
go writer(ws, done)
go reader(ws, done)
}
The reader closes the connection on return. It also closes the done channel to signal the writer that the reader is done.
func reader(conn *websocket.Conn, done chan struct{}) {
defer conn.Close()
defer close(done)
for {
_, p, err := conn.ReadMessage() //what is message type?
if err != nil {
log.Println("there is errors%s", err)
return
}
}
}
The writer also closes the connection on return. When the connection is closed, read on the connection immediately returns an error causing the read goroutine to complete. The writer waits on the done channel. Receive on the channel yields the zero value when the channel is closed by the reader. The writer returns on write errors instead of looping forever as in the question.
func writer(conn *websocket.Conn, done chan struct{}) {
defer conn.Close()
for {
select {
case <-done:
// the reader is done, so return
return
case singleset := <-singleorder: // get data from channel
jsonString, err := json.Marshal(singleset )
err = conn.WriteMessage(1, []byte(jsonString))
if err != nil {
log.Println(err)
return
}
}
}
}
An application should expect the connection to be closed for any number of reasons including the user navigating away from the page.
It's likely that the code that sends to channel singleorder needs to know that the connection was closed, but we cannot see that code here. I'll leave it to you to figure out how to handle that.
I am using a Go TCP Client to connect to our Go TCP Server.
I am able to connect to the Server and run commands properly, but every so often there will be an unusually high amount of consecutive TCP connection errors reported by my TCP Client when trying to either connect to our TCP Server or sending a message once connected:
dial tcp kubernetes_node_ip:exposed_kubernetes_port:
connectex: A connection attempt failed because the connected party did not properly
respond after a period of time, or established connection failed because connected
host has failed to respond.
read tcp unfamiliar_ip:unfamiliar_port->kubernetes_node_ip:exposed_kubernetes_port
wsarecv: A connection attempt failed because the connected party did not properly
respond after a period of time, or established connection failed because connected
host has failed to respond.
I say "unusually high" because I assume that the number of times these errors occur should be very minimal (about 5 or less within the hour). Note that I am not dismissing the possibility of this being caused by connection instabilities, as I have also noticed that it is possible to run several commands in rapid succession without any errors.
However, I am still going to post my code in case I am doing something wrong.
Below is the code that my TCP Client uses to connect to our server:
serverAddress, err := net.ResolveTCPAddr("tcp", kubernetes_ip+":"+kubernetes_port)
if err != nil {
fmt.Println(err)
return
}
// Never stop asking for commands from the user.
for {
// Connect to the server.
serverConnection, err := net.DialTCP("tcp", nil, serverAddress)
if err != nil {
fmt.Println(err)
continue
}
defer serverConnection.Close()
// Added to prevent connection timeout errors, but doesn't seem to be helping
// because said errors happen within just 1 or 2 minutes.
err = serverConnection.SetDeadline(time.Now().Add(10 * time.Minute))
if err != nil {
fmt.Println(err)
continue
}
// Ask for a command from the user and convert to JSON bytes...
// Send message to server.
_, err = serverConnection.Write(clientMsgBytes)
if err != nil {
err = merry.Wrap(err)
fmt.Println(merry.Details(err))
continue
}
err = serverConnection.CloseWrite()
if err != nil {
err = merry.Wrap(err)
fmt.Println(merry.Details(err))
continue
}
// Wait for a response from the server and print...
}
Below is the code that our TCP Server uses to accept client requests:
// We only supply the port so the IP can be dynamically assigned:
serverAddress, err := net.ResolveTCPAddr("tcp", ":"+server_port)
if err != nil {
return err
}
tcpListener, err := net.ListenTCP("tcp", serverAddress)
if err != nil {
return err
}
defer tcpListener.Close()
// Never stop listening for client requests.
for {
clientConnection, err := tcpListener.AcceptTCP()
if err != nil {
fmt.Println(err)
continue
}
go func() {
// Add client connection to Job Queue.
// Note that `clientConnections` is a buffered channel with a size of 1500.
// Since I am the only user connecting to our server right now, I do not think
// this is a channel blocking issue.
clientConnections <- clientConnection
}()
}
Below is the code that our TCP Server uses to process client requests:
defer clientConnection.Close()
// Added to prevent connection timeout errors, but doesn't seem to be helping
// because said errors happen within just 1 or 2 minutes.
err := clientConnection.SetDeadline(time.Now().Add(10 * time.Minute))
if err != nil {
return err
}
// Read full TCP message.
// Does not stop until an EOF is reported by `CloseWrite()`
clientMsgBytes, err := ioutil.ReadAll(clientConnection)
if err != nil {
err = merry.Wrap(err)
return nil, err
}
// Process the message bytes...
My questions are:
Am I doing something wrong in the above code, or is the above decent enough for basic TCP Client-Server operations?
Is it okay that both the TCP Client and TCP Server have code that defers closing their one connection?
I seem to recall that calling defer inside a loop does nothing. How do I properly close Client connections before starting new ones?
Some extra information:
Said errors are not logged by the TCP Server, so aside from
connection instabilities, this might also be a
Kubernetes/Docker-related issue.
It seems this piece of code does not act as you think it does. The defer statement on the connection close will only happen when the function returns, not when an iteration ends. So as far as I can see here, you are creating a lot of connections on the client side, it could be the problem.
serverAddress, err := net.ResolveTCPAddr("tcp", kubernetes_ip+":"+kubernetes_port)
if err != nil {
fmt.Println(err)
return
}
// Never stop asking for commands from the user.
for {
// Connect to the server.
serverConnection, err := net.DialTCP("tcp", nil, serverAddress)
if err != nil {
fmt.Println(err)
continue
}
defer serverConnection.Close()
// Added to prevent connection timeout errors, but doesn't seem to be helping
// because said errors happen within just 1 or 2 minutes.
err = serverConnection.SetDeadline(time.Now().Add(10 * time.Minute))
if err != nil {
fmt.Println(err)
continue
}
// Ask for a command from the user and send to the server...
// Wait for a response from the server and print...
}
I suggest to write it this way:
func start() {
serverAddress, err := net.ResolveTCPAddr("tcp", kubernetes_ip+":"+kubernetes_port)
if err != nil {
fmt.Println(err)
return
}
for {
if err := listen(serverAddress); err != nil {
fmt.Println(err)
}
}
}
func listen(serverAddress string) error {
// Connect to the server.
serverConnection, err := net.DialTCP("tcp", nil, serverAddress)
if err != nil {
fmt.Println(err)
continue
}
defer serverConnection.Close()
// Never stop asking for commands from the user.
for {
// Added to prevent connection timeout errors, but doesn't seem to be helping
// because said errors happen within just 1 or 2 minutes.
err = serverConnection.SetDeadline(time.Now().Add(10 * time.Minute))
if err != nil {
fmt.Println(err)
return err
}
// Ask for a command from the user and send to the server...
// Wait for a response from the server and print...
}
}
Also, you should keep a single connection open, or a pool of connections, instead of opening and closing the connection right away. Then when you send a message you get a connection from the pool (or the single connection), and you write the message and wait for the response, then you release the connection to the pool.
Something like that:
res, err := c.Send([]byte(`my message`))
if err != nil {
// handle err
}
// the implementation of send
func (c *Client) Send(msg []byte) ([]byte, error) {
conn, err := c.pool.Get() // returns a connection from the pool or starts a new one
if err != nil {
return nil, err
}
// send your message and wait for response
// ...
return response, nil
}
According to the gRPC documentation, deadlines can be specified by clients to determine how long the client will wait on the server before exiting with a DEADLINE_EXCEEDED error. The documentation mentions that different languages have different implementations and that some languages do not have default values.
Indeed, a quick CTRL+F for "deadline" on the Go gRPC documentation reveals no results. What I did discover was a WithTimeout on the dialer for the TCP connection.
Implemented as follows (from the helloworld example):
package main
import (
"log"
"os"
"time"
"golang.org/x/net/context"
"google.golang.org/grpc"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
)
const (
address = "localhost:50051"
defaultName = "world"
deadline = 20
)
func main() {
// Set up a connection to the server with a timeout
conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithTimeout(time.Duration(deadline)*time.Second)
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
// Contact the server and print out its response.
name := defaultName
if len(os.Args) > 1 {
name = os.Args[1]
}
r, err := c.SayHello(context.Background(), &pb.HelloRequest{Name: name})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.Message)
}
The code will raise an error only if the client cannot connect after 20 seconds. The output will be something as follows:
2016/05/24 09:02:54 grpc: Conn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::1]:3265: getsockopt: connection refused"; Reconnecting to "localhost:3265"
2016/05/24 09:02:54 Failed to dial localhost:3265: grpc: timed out trying to connect; please retry.
2016/05/24 09:02:54 could not greet: rpc error: code = 2 desc = grpc: the client connection is closing
As noted in the question title, the system I'm working with is peer to peer, so there is no central, always up server and therefore the retry system that gRPC implements is wonderful. However, I'm actually looking for deadlines because if the remote does connect, but the server takes > 20 seconds to respond, no exception will be raised in the WithTimeout context.
A complete win for me would be a timeout/deadline system where:
if the client cannot connect, an error is returned after timeout
if the client connects, but the server doesn't respond before timeout, an error is returned.
if the client connects, but the connection drops before timeout, an error is returned.
My feeling though is that I will need some combination of connection management plus gRPC deadline management. Does anyone know how to implement deadlines in Go?
According to the WithTimeout example of context
package main
import (
"context"
"fmt"
"time"
)
func main() {
// Pass a context with a timeout to tell a blocking function that it
// should abandon its work after the timeout elapses.
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
select {
case <-time.After(1 * time.Second):
fmt.Println("overslept")
case <-ctx.Done():
fmt.Println(ctx.Err()) // prints "context deadline exceeded"
}
}
You can change the helloworld example client code for 100ms timeout:
ctx, _ := context.WithTimeout(context.Background(), 100 * time.Millisecond)
r, err := c.SayHello(ctx, &pb.HelloRequest{Name: name})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.Message)
You should look at the context package more closely. GRPC was built with contexts as a fundamental part of it. You might need the grpc.Dial context and client.SayHello context to be built off related info, but that should be fairly straight forward.