golang resource ownership pattern (files, connections, close-ables) - go

What is the proper way to manage resource ownership in golang? Suppose I have the following:
db, err := sql.Open("mysql", "role#/test_db")
am := NewResourceManager(db)
am.DoWork()
db.Close()
Is it typical to always have the calling function maintain ownership and responsibility for closing resources? This feels a little weird to me because after closing, am still retains a reference and could try to use db if I or someone else is not careful later on (I guess this is a case for defer; however, if I want to pass the ResourceManager am back from this block, how would I even defer the closing of the file properly? I actually want it to stay open when this block finishes execution). I find that in other languages I often want to allow the instance to manage the resource and then clean it up when it's destructor is called, like this toy python example:
class Writer():
def __init__(self, filename):
self.f = open(filename, 'w+')
def __del__(self):
self.f.close()
def write(value):
self.f.write(value)
Unfortunately, there are no destructors in golang. I'm not sure how I would do this in go other than something like this:
type ResourceManager interface {
DoWork()
// Close() ?
}
type resourceManager struct {
db *sql.DB
}
func NewResourceManager(db *sql.DB) ResourceManager {
return &resourceManager{db}
}
db, err := sql.Open("mysql", "role#/test_db")
am := NewResourceManager(db)
am.DoWork()
am.Close() // using method shortening
But this seems less transparent, and I'm not sure how to communicate that the ResourceManager also needs to be Close()'d now. I'm finding this a frequent stumbling block, i.e. I also want to have a resource manager that holds a gRPC client connection, and if these types of resources aren't managed by resource managing objects, it seems like my main function is going to be cluttered with a lot of resource management, i.e. opening and closing. For instance, I could imagine a case where I wouldn't want main to know anything about the object and it's resources:
...
func NewResourceManager() ResourceManager {
db, err := sql.Open("mysql", "role#/test_db")
return &resourceManager{db}
}
...
// main elsewhere
am := NewResourceManager()
am.DoWork()

You chose a bad example, since you generally would reuse a database connection, instead of opening and closing one for each use. Hence, you would pass the db connection to functions using it and do the resource management in the caller without any need for an resource manager:
// Imports etc omitted for the sake of readability
func PingHandler(db *sql.DB) http.Handler (
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := db.ping(); err != nil {
http.Error(w,e.Error(),500)
}
})
)
func main(){
db,_ := sql.Open("superdb",os.Getenv("APP_DBURL"))
// Note the db connection will only be closed if main exits.
defer db.Close()
// Setup the server
http.Handle("/ping", PingHandler(db))
server := &http.Server{Addr: ":8080"}
// Create a channel for listening on SIGINT, -TERM and -QUIT
stop := make(chan os.Signal, 1)
// Register channel to be notified on said signals
signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
go func(){
// When we get the signal...
<- stop
ctx, _ := context.WithTimeout(context.Background(), 5*time.Second)
// ... we gracefully shut down the server.
// That ensures that no new connections, which potentially
// would use our db connection, are accepted.
if err := server.Shutdown(ctx); err != nil {
// handle err
}
}
// This blocks until the server is shut down.
// AFTER it is shut down, main exits, the deferred calls are executed.
// In this case, the database connection is closed.
// And it is closed only after the last handler call which uses the connection is finished.
// Mission accomplished.
server.ListenAndServe()
}
So in this example, there is no need for a resource manager and I honestly can not think of an example actually needing one. In the rare cases I needed something akin to one, I used sync.Pool.
However, for gRPC client connections, there is no need to maintain a pool, either:
[...]However, the ClientConn should manage the connections itself, so if a connection is broken, it will reconnect automatically. And if you have multiple backends, it's possible to connect to multiple of them and load balance between them. [...]
So the same principle applies: Create a connection (pool), pass it around as required, ensure that it is closed after all work is done.
Go proverb:
Clear is better than clever.
– Robert Pike
Instead of hiding resource management somewhere else, manage the resources as closely to the code where they are used and as explicitly as you possibly can.

Related

Goroutine safe channel close doesn't actually close webscoket

This one is a tricky issue that bugs me quite a bit.
Essentially, I wrote an integration microservice that provides data streams from Binance crypto exchange using the Go client. A client sends a start messages, starts data stream for a symbol, and at some point, sends a close message to stop the stream. My implementation looks basically like this:
func (c BinanceClient) StartDataStream(clientType bn.ClientType, symbol, interval string) error {
switch clientType {
case bn.SPOT_LIVE:
wsKlineHandler := c.handlers.klineHandler.SpotKlineHandler
wsErrHandler := c.handlers.klineHandler.ErrHandler
_, stopC, err := binance.WsKlineServe(symbol, interval, wsKlineHandler, wsErrHandler)
if err != nil {
fmt.Println(err)
return err
} else {
c.state.clientSymChanMap[clientType][symbol] = stopC
return nil
}
...
}
The clientSymChanMap stores the stopChannel in a nested hashmap so that I can retrieve the stop channel later to stop the data feed. The stop function has been implemented accordingly:
func (c BinanceClient) StopDataStream(clientType bn.ClientType, symbol string) {
//mtd := "StopDataStream: "
stopC := c.state.clientSymChanMap[clientType][symbol]
if isClosed(stopC) {
DbgPrint(" Channel is already closed. Do nothing for: " + symbol)
} else {
close(stopC)
}
// Delete channel from the map otherwise the next StopAll throws a NPE due to closing a dead channel
delete(c.state.clientSymChanMap[clientType], symbol)
return
}
To prevent panics from already closed channels, I use a check function that returns true in case the channel is already close.
func isClosed(ch <-chan struct{}) bool {
select {
case <-ch:
return true
default:
}
return false
}
Looks nice, but has a catch. When I run the code with starting data for just one symbol, it starts and closes the datafeed exactly as expected.
However, when starting multiple data feeds, then the above code somehow never closes the websocket and just keeps streaming data forever. Without the isClosed check, I get panics of trying to close a closed channel, but with the check in place, well, nothing gets closed.
When looking at the implementation of the above binance.WsKlineServe function, it's quite obvious that it just wraps a new websocket with each invocation and then returns the done & stop channel.
The documentation gives the following usage example:
wsKlineHandler := func(event *binance.WsKlineEvent) {
fmt.Println(event)
}
errHandler := func(err error) {
fmt.Println(err)
}
doneC, stopC, err := binance.WsKlineServe("LTCBTC", "1m", wsKlineHandler, errHandler)
if err != nil {
fmt.Println(err)
return
}
<-doneC
Because the doneC channel actually blocks, I removed it and thought that storing the stopC channel and then use it later to stop the datafeed would work. However, it only does so for one single instance. When multiple streams are open, this doesn't work anymore.
Any idea what that's the case and how to fix it?
Firstly, this is dangerous:
if isClosed(stopC) {
DbgPrint(" Channel is already closed. Do nothing for: " + symbol)
} else {
close(stopC) // <- can't be sure channel is still open
}
there is no guarantee that after your polling check of the channel state, that the channel will still be in that same state in the next line of code. So this code could in theory could panic if it's called concurrently.
If you want an asynchronous action to occur on the channel close - it's best to do this explicitly from its own goroutine. So you could try this:
go func() {
stopC := c.state.clientSymChanMap[clientType][symbol]
<-stopC
// stopC definitely closed now
delete(c.state.clientSymChanMap[clientType], symbol)
}()
P.S. you do need some sort of mutex on your map, since the delete is asynchronous - you need to ensure any adds to the map don't datarace with this.
P.P.S Channels are reclaimed by the GC when they go out of scope. If you are no longer reading from it - they do not need to be explicitly closed to be reclaimed by the GC.
Using channels for stopping a goroutine or closing something is very tricky. There are lots of things you can do wrong or forget to do.
context.WithCancel abstracts that complexity away, making the code more readable and maintainable.
Some code snippets:
ctx, cancel := context.WitchCancel(context.TODO())
TheThingToCancel(ctx, ...)
// Whenever you want to stop TheThingToCancel. Can be called multiple times.
cancel()
Then in a for loop you'd often have a select like this:
for {
select {
case <-ctx.Done():
return
default:
}
// do stuff
}
Here some code that is closer to your specific case of an open connection:
func TheThingToCancel(ctx context.Context) (context.CancelFunc, error) {
ctx, cancel := context.WithCancel(ctx)
conn, err := net.Dial("tcp", ":12345")
if err != nil {
cancel()
return nil, err
}
go func() {
<-ctx.Done()
_ = conn.Close()
}()
go func() {
defer func() {
_ = conn.Close()
// make sure context is always cancelled to avoid goroutine leak
cancel()
}()
var bts = make([]byte, 1024)
for {
n, err := conn.Read(bts)
if err != nil {
return
}
fmt.Println(bts[:n])
}
}()
return cancel, nil
}
It returns the cancel function to be able to close it from the outside.
Cancelling a context can be done many times over without a panic like would occur if a channel is closed multiple times. That is one advantage. Also you can derive contexts from other contexts and thereby close a lot of contexts that all stop different routines by closing a parent context. Carefully designed, this is very powerful for shutting down different routines belonging together that also need to be able to be shut down individually.

Go GRPC client disconnect terminates Go server

Bit of a newb to both Go and GRPC, so bear with me.
Using go version go1.14.4 windows/amd64, proto3, and latest grpc (1.31 i think). I'm trying to set up a bidi streaming connection that will likely be open for longer periods of time. Everything works locally, except if I terminate the client (or one of them) it kills the server as well with the following error:
Unable to trade data rpc error: code = Canceled desc = context canceled
This error comes out of this code server side
func (s *exchangeserver) Trade(stream proto.ExchageService_TradeServer) error {
endchan := make(chan int)
defer close(endchan)
go func() {
for {
req, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
log.Fatal("Unable to trade data ", err)
break
}
fmt.Println("Got ", req.GetNumber())
}
endchan <- 1
}()
go func() {
for {
resp := &proto.WordResponse{Word: "Hello again "}
err := stream.Send(resp)
if err != nil {
log.Fatal("Unable to send from server ", err)
break
}
time.Sleep(time.Duration(500 * time.Millisecond))
}
endchan <- 1
}()
<-endchan
return nil
}
And the Trade() RPC is so simple it isn't worth posting the .proto.
The error is clearly coming out of the Recv() call, but that call blocks until it sees a message, like the client disconnect, at which point I would expect it to kill the stream, not the whole process. I've tried adding a service handler with HandleConn(context, stats.ConnStats) and it does catch the disconnect before the server dies, but I can't do anything with it. I've even tried creating a global channel that the serve handler pushes a value into when HandleRPC(context, stats.RPCStats) is called and only allowing Recv() to be called when there's a value in the channel, but that can't be right, that's like blocking a blocking function for safety and it didn't work anyway.
This has to be one of those real stupid mistakes that beginner's make. Of what use would GPRC be if it couldn't handle a client disconnect without dying? Yet I have read probably a trillion (ish) posts from every corner of the internet and noone else is having this issue. On the contrary, the more popular version of this question is "My client stream stays open after disconnect". I'd expect that issue. Not this one.
Im not 100% sure how this is supposed to behave but I note that you are starting separate receive and send goroutines up at the same time. This might be valid but is not the typical approach. Instead you would usually receive what you want to process and then start a nested loop to handle the reply .
See an example of typical bidirectional streaming implementation from here: https://grpc.io/docs/languages/go/basics/
func (s *routeGuideServer) RouteChat(stream pb.RouteGuide_RouteChatServer) error {
for {
in, err := stream.Recv()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
key := serialize(in.Location)
... // look for notes to be sent to client
for _, note := range s.routeNotes[key] {
if err := stream.Send(note); err != nil {
return err
}
}
}
}
sending and receiving at the same time might be valid for your use case but if that is what you are trying to do then I believe your handling of the channels is incorrect. Either way, please read on to understand the issue as it is a common one in go.
You have a single channel which only blocks until it receives a single message, once it unblocks the function ends and the channel is closed (by defer).
You are trying to send to this channel from both your send and receive
loop.
When the last one to finish tries to send to the channel it will have been closed (by the first to finish) and the server will panic. Annoyingly, you wont actually see any sign of this as the server will exit before the goroutine can dump its panic (no clues - probably why you landed here)
see an example of the issue here (grpc code stripped out):
https://play.golang.org/p/GjfgDDAWNYr
Note: comment out the last pause in the main func to stop showing the panic reliably (as in your case)
So one simple fix would probably be to simply create two separate channels (one for send, one for receive) and block on both - this however would leave the send loop open necessarily if you don't get a chance to respond so probably better to structure like the example above unless you have good reason to pursue something different.
Another possibility is some sort server/request context mix up but I'm pretty sure the above will fix - drop an update with your server setup code if your still having issues after the above changes

How to use context.Context with tcp connection read

I am trying to create an intermediate layer between user and tcp, with Send and Receive functions. Currently, I am trying to integrate a context, so that the Send and Receive respects a context. However, I don't know how to make them respect the context's cancellation.
Until now, I got the following.
// c.underlying is a net.Conn
func (c *tcpConn) Receive(ctx context.Context) ([]byte, error) {
if deadline, ok := ctx.Deadline(); ok {
// Set the read deadline on the underlying connection according to the
// given context. This read deadline applies to the whole function, so
// we only set it once here. On the next read-call, it will be set
// again, or will be reset in the else block, to not keep an old
// deadline.
c.underlying.SetReadDeadline(deadline)
} else {
c.underlying.SetReadDeadline(time.Time{}) // remove the read deadline
}
// perform reads with
// c.underlying.Read(myBuffer)
return frameData, nil
}
However, as far as I understand that code, this only respects a context.WithTimeout or context.WithDeadline, and not a context.WithCancel.
If possible, I would like to pass that into the connection somehow, or actually abort the reading process.
How can I do that?
Note: If possible, I would like to avoid another function that reads in another goroutine and pushed a result back on a channel, because then, when calling cancel, and I am reading 2GB over the network, that doesn't actually cancel the read, and the resources are still used. If not possible in another way however, I would like to know if there is a better way of doing that than a function with two channels, one for a []byte result and one for an error.
EDIT:
With the following code, I can respect a cancel, but it doesn't abort the read.
// apply deadline ...
result := make(chan interface{})
defer close(result)
go c.receiveAsync(result)
select {
case res := <-result:
if err, ok := res.(error); ok {
return nil, err
}
return res.([]byte), nil
case <-ctx.Done():
return nil, ErrTimeout
}
}
func (c *tcpConn) receiveAsync(result chan interface{}) {
// perform the reads and push either an error or the
// read bytes to the result channel
If the connection can be closed on cancellation, you can setup a goroutine to shutdown the connection on cancellation within the Receive method. If the connection must be reused again later, then there is no way to cancel a Read in progress.
recvDone := make(chan struct{})
defer close(recvDone)
// setup the cancellation to abort reads in process
go func() {
select {
case <-ctx.Done():
c.underlying.CloseRead()
// Close() can be used if this isn't necessarily a TCP connection
case <-recvDone:
}
}()
It will be a little more work if you want to communicate the cancelation error back, but the CloseRead will provide a clean way to stop any pending TCP Read calls.

Go gRPC simple service Asynchronous and Synchronous explanation

I am trying to understand GoLang "Go" together with gRPC and to make a simple service scalable.
Lets say I have a client1 that calls a service1(adds numbers) that calls service2(determines if the result is prime), and service2 returns the result to service1 that returns the result to client1 all via gRPC.
When I use protocol buffers "proto3" and generate the Go code via protoc.
I get generated methods that call the service in one particular way.
I see no distinction to call the methods asynchronously "Go".
And the underlying call seems to be "Invoke" which I believe is synchronous,the call returns once a result is received.
How do I make service1 "performant", I know I can run this in a cluster and have copies, but that would mean I can only serve clients as per the amount of instances within the cluster.
I want a "single" service to be able to serve multiple clients(e.g. 1000) .
Here is a simple server and I am not sure if this is performant or not:
I do know that the getprime function does dial every time,
and this could probably be moved to make this dial persist and be re-used; But more importantly I want to make a simple performant scaleable service and get a good understanding.
(A)
Perhaps the whole design is incorrect and the service1 should just return
as soon as the instruction is received "ack", do the addition and send the next request to sercice2 which determines if the answer is prime or not; again service2 just responds with an acknowledgement of the request being received. Once prime is determined by the service2 a call is made to the client with an answer.
If (A) above is the better approach, then still please explain the bottlenecks below; what happens when multiple clients are processed?
The call to "Listen" does what, "blocks, or does not block", etc.
package main
import (
pb "demo/internal/pkg/proto_gen/calc"
"fmt"
"golang.org/x/net/context"
"google.golang.org/grpc"
"google.golang.org/grpc/reflection"
"log"
"net"
)
const (
port = ":8080"
)
type service struct {
}
func (s *service) Calculate(ctx context.Context, req *pb.Instruction) (*pb.Response, error) {
var answer float64
answer = req.Number1 + req.Number2
// call service prime
p := getprime(int(answer))
pa := pb.PrimeAnswer{Prime: p}
return &pb.Response{Answer: answer, Prime: &pa}, nil
}
const (
primeAddress = "127.0.0.1:8089"
)
func getprime(number int) bool {
conn, err := grpc.Dial(primeAddress, grpc.WithInsecure())
if err != nil {
log.Fatalf("Did not connect to prime service: %v", err)
}
defer conn.Close()
client := pb.NewPrimeServiceClient(conn)
p := pb.PrimeMessage{"", float64(number)}
r, err := client.Prime(context.Background(), &p)
if err != nil {
log.Fatalf("Call to prime service failed: %v", err)
}
return r.Prime
}
func main() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterCalculatorServer(s, &service{})
reflection.Register(s)
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
Thanks for your question. It is true that gRPC-Go is sync only; that is your Unary RPC(the one in your example) will return only when the RPC has finished (got a response from the server).
About performance:
The Dial operation establishes an underlying connection which may be expensive. So it not wise to do it every time getprime is called. A better way is to create a client, keep it around and make calls to the prime server on it. This way only first RPC incurs the cost of connection.
For each RPC request a server gets we launch a goroutine to process that request. So in general, this should scale fairly well.
About (A): It is not uncommon for a service handler to make an RPC call to yet another server and wait for its response before returning back.
Note that there's no way for a server to make call to the client.
To phrase what JimB said as an answer: "Synchronous" means that the function that makes the remote call waits for a reply before continuing, not that the whole server or client does. The server is normally multithreaded, even when processing synchronous calls; it can accept and work on a second call while it's responding to the first.
And similarly, if a client has multiple concurrent tasks that each have a gRPC call running, that won't block the process. Clients like that could include net/http servers serving end users, or gRPC servers handling multiple RPCs.
Where you might add explicit go statements is if you want to do something else from the specific function making the RPC call. For example, if you want to issue several RPC calls at once then wait for all their results to come in, you could write code following the examples of fan-out calls.

How to place http.Serve in its own goroutine if it blocks?

http.Serve either returns an error as soon as it is called or blocks if successfully executing.
How can I make it so that if it blocks it does so in its own goroutine? I currently have the following code:
func serveOrErr(l net.Listener, handler http.Handler) error {
starting := make(chan struct{})
serveErr := make(chan error)
go func() {
starting <- struct{}{}
if err := http.Serve(l, handler); err != nil {
serveErr <- err
}
}()
<-starting
select {
case err := <-serveErr:
return err
default:
return nil
}
}
This seemed like a good start and works on my test machine but I believe that there are no guarantees that serveErr <- err would be called before case err := <-serveErr therefore leading to inconsistent results due to a data race if http.Serve were to produce an error.
http.Serve either returns an error as soon as it is called or blocks if successfully executing
This assumption is not correct. And I believe it rarely occurs. http.Serve calls net.Listener.Accept in the loop – an error can occur any time (socket closed, too many open file descriptors etc.). It's http.ListenAndServe, usually being used for running http servers, which often fails early while binding listening socket (no permissions, address already in use).
In my opinion what you're trying to do is wrong, unless really your net.Listener.Accept is failing on the first call for some reason. Is it? If you want to be 100% sure your server is working, you could try to connect to it (and maybe actually transmit something), but once you successfully bound the socket I don't see it really necessary.
You could use a timeout on your select statement, e.g.
timeout := time.After(5 * time.Millisecond) // TODO: ajust the value
select {
case err := <-serveErr:
return err
case _ := <- timeout:
return nil
}
This way your select will block until serveErr has a value or the specified timeout has elapsed. Note that the execution of your function will therefore block the calling goroutine for up to the duration of the specified timeout.
Rob Pike's excellent talk on go concurrency patterns might be helpful.

Resources