I'm having trouble to estimate if there will be side effects of running http.ListenAndServe in goroutine.
To make it possible for prometheus to collect stats data from a /metrics endpoint of a service running a kafkaclient(running a kafka consumer in an infinite for-loop)
var addr = flag.String("listen-address", ":8070", "The address to listen on for HTTP requests.")
func main() {
flag.Parse()
http.Handle("/metrics", promhttp.Handler())
go http.ListenAndServe(*addr, nil)
for {....}
What would be the best practices to start the monitoring endpoint and run the infinite loop?
Best practice is to check for and handle errors. The error from http.ListenAndServe is ignored.
If return from http.ListenAndServe is fatal to the application, then use the following code or some variation on it to handle the error.
go func() {
log.Fatal(http.ListenAndServe(*addr, nil))
}()
The call to log.Fatal logs the error and exits the application.
Related
I'm referring to this article: Graceful Shutdowns on Cloud Run
The example outlines how to do this in Node.js.
How would one do this in Golang? Any issues with simply adding this to the func init() method?
func shutdownGracefully() {
c := make(chan os.Signal)
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
go func() {
<-c
// Do some cleanup stuff here...
os.Exit(0)
}()
}
We have sample Go code for proper signal handling at https://github.com/GoogleCloudPlatform/golang-samples/pull/1902. It's not merged yet but essentially highlights how to do this properly on Go.
Beware that when running locally, sending Ctrl+C to your application is a SIGINT, however on Cloud Run you will be getting SIGTERM. It is also important to pass cancellations properly and handling server shutdown gracefully while not dropping the ongoing requests (though, net/http.Server handles a decent portion of this for you as you’ll see in sample code).
How would one do this in Golang?
An idiomatic way to handle graceful shutdowns in go is having a select statement blocking the main goroutine listening for any signal. From there you can trigger all the proper shutdowns when necessary.
For instance:
select {
case err := <-anyOtherError:
// ...
case signal := <- c:
// Handle shutdown e.g. like http Server Shutdown
}
I like to set it inside a run function with other startup tasks like starting the server, configs, etc.
I want to listen for graceful server termination in grpc in my handler. When the server is being stopped gracefully, I want to add some logic in my code to close open ports, files, flush results etc. How can I do that?
How is it different in case of unary and streaming handler?
You can have shutdown hook by listening to the signals something like this
In your main function or where you start your server create channel for signals that you want to listen to
c := make(chan os.Signal, 1)
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
// call your cleanup method with this channel as a routine
go cleanup(c)
In your clean up method
func cleanup(c chan os.Signal) {
// Wait for termination
<- c
// Do your cleanups here
}
Create the signal channel and call the cleanup function as a go routine before you start the gRPC server. Whenever the application (gRPC server) stopped or interrupted this channel will get the signal in the cleanup function where you can do the necessary cleanups
For grpc servers u can do this
func waitForGracefulShutdown(srv *grpc.Server) {
fmt.Println("Grpc messaging server started ...")
interruptChan := make(chan os.Signal, 1)
signal.Notify(interruptChan, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
// Block until we receive our signal.
<-interruptChan
// Create a deadline to wait for.
_, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
srv.GracefulStop()
publisher, messagingError := messaging.GetPublisherInstance()
if messagingError.Error == nil {
publisher.Close()
}
log.Println("Shutting down grpc messaging server.")
os.Exit(0)
}
Currently there's no mechanism to signal a service handler about Graceful stop. Cleanups and other such global functions that must happen upon server exiting wouldn't usually happen inside of a service handler.
That said, if you think your design is better off with such cleanups happening inside of the service handler and a signal from graceful close is critical, we would love to hear more about your use case. Perhaps open an issue on our github repo and we can discuss it there.
Best,
Mak
I have studied the Godoc of the gorilla/websocket package.
In the Godoc it is clearly stated that
Concurrency
Connections support one concurrent reader and one concurrent writer.
Applications are responsible for ensuring that no more than one goroutine calls the write methods (NextWriter, SetWriteDeadline, WriteMessage, WriteJSON, EnableWriteCompression, SetCompressionLevel) concurrently and that no more than one goroutine calls the read methods (NextReader, SetReadDeadline, ReadMessage, ReadJSON, SetPongHandler, SetPingHandler) concurrently.
The Close and WriteControl methods can be called concurrently with all other
methods.
However, in one of the example provided by the package
func (c *Conn) readPump() {
defer func() {
hub.unregister <- c
c.ws.Close()
}()
c.ws.SetReadLimit(maxMessageSize)
c.ws.SetReadDeadline(time.Now().Add(pongWait))
c.ws.SetPongHandler(func(string) error {
c.ws.SetReadDeadline(time.Now().Add(pongWait)); return nil
})
for {
_, message, err := c.ws.ReadMessage()
if err != nil {
if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway) {
log.Printf("error: %v", err)
}
break
}
message = bytes.TrimSpace(bytes.Replace(message, newline, space, -1))
hub.broadcast <- message
}
}
Source: https://github.com/gorilla/websocket/blob/a68708917c6a4f06314ab4e52493cc61359c9d42/examples/chat/conn.go#L50
This line
c.ws.SetPongHandler(func(string) error {
c.ws.SetReadDeadline(time.Now().Add(pongWait)); return nil
})
and this line
_, message, err := c.ws.ReadMessage()
seems to be not synchronized because the first line is a callback function so it should be invoked in a Goroutine created in the package and the second line is executing in the Goroutine that invoke serveWs
More importantly, how should I ensure that no more than one goroutine calls the SetReadDeadline, ReadMessage, SetPongHandler, SetPingHandler concurrently?
I tries to use a Mutex lock and lock it whenever I call the above functions, and unlock it afterwards, but quickly I realize a problem. It is usual (also in the example) that ReadMessage is being called in a for-loop. But if the Mutext is locked before the ReadMessage, then no other Read-functions can acquire the lock and execute until next message is received
Is there any better way in handling this concurrency issue? Thanks in advance.
The best way to ensure that there are no concurrent calls to the read methods is to execute all of the read methods from a single goroutine.
All of the Gorilla websocket examples use this approach including the example pasted in the question. In the example, all calls to the read methods are from the readPump method. The readPump method is called once for a connection on a single goroutine. It follows that the connection read methods are not called concurrently.
The section of the documentation on control messages says that the application must read the connection to process control messages. Based on this and Gorilla's own examples, I think it's safe to assume that the ping, pong and close handlers will be called from the application's reading goroutine as it is in the current implementation. It would be nice if the documentation could be more explicit about this. Maybe file an issue?
If I'm using http.ListenAndServe to provide responses when the user hits a URL, should I be firing off the corresponding actions in the function as a goroutine ?
For instance, say I'm listening at /:
func main() {
http.HandleFunc("/", provideMainContent)
}
func provideMainContent(w http.ResponseWriter, r *http.Request) {
/// Bunch of code, looks up details in databases, parses, then returns
}
Should the bunch of code in provideMainContent be wrapped in a goroutine so it doesn't slow down any potential requests that come after the fact ?
Short answer, No
GoDoc from http.Serve :
Serve accepts incoming HTTP connections on the listener l, creating a new service goroutine for each. The service goroutines read requests and then call handler to reply to them.
However as mentioned in the question linked by #Mellow Marmot, there might be cases where you might want to spawn a goroutine to do some processing while you return from the handler so that the requester does not have to wait for all the processing to be done to get a response.
In order to test a server that I am writing, I want to be able to start and stop it in the testing framework.
To do so, I am hoping I can integrate the context package in with the http.Server struct. I want to be able to stop the server when I call the Done function and the ctx.Done() channel returns something.
What I would love to do would be to just modify the http.Server.Serve() method, to accept a context.Context and check if it is done, on each iteration of the for loop, like this:
func (srv *server) Serve(ctx context.Context, l net.Listener) error {
defer l.Close()
var tempDelay time.Duration // how long to sleep on accept failure
for {
select {
case <-ctx.Done():
return nil
default:
}
... rest is same as original
However it seems like if I wanna add that check inside the for loop, I would have to rewrite a lot of the methods, because this method calls other private methods (like http.server.srv), which in turn call other private methods....
I also notice that the for loop will stop when the Accept() method on the listener returns an error.
However, I can't seem to figure out a way to get a listener to output an error from it's accept method without accessing its private methods as well.
It seems like I am doing something very stupid and wrong if I have to copy and paste half the http library just to let the server stop using the context package.
I know there are lots of solutions around for supporting context canceling for the ServeHTTP function, but that isn't what I am talking about. I wanna pass a context to the whole server, not just to each incoming request.
Is this just impossible?
Use httptest.Server to create a server that you can start and stop in tests.
If you do use the http.Server directly, you can break the Serve loop by closing the net.Listener. When a net.Listener is closed, any blocked Accept operations are unblocked and return errors. The Serve function returns if Accept returns a permanent error.