I've written a REST web service that uses mongo as the backend data store. I was wondering at this stage (before deployment), what the best practices were, considering a service that essentially runs forever(ish).
Currently, I'm following this type of pattern:
// database.go
...
type DataStore struct {
mongoSession *mgo.Session
}
...
func (d *DataStore) OpenSession () {
... // read setup from environment
mongoSession, err = mgo.Dial(mongoURI)
if err != nil {}
...
}
func (d *DataStore) CloseSession() {...}
func (d *DataStore) Find (...) (results...) {
s := d.mongoSession.Copy()
defer s.Close()
// do stuff, return results
}
In main.go:
func main() {
ds := NewDataStore()
ds.OpenSession()
defer ds.CloseSession()
// Web Service Routes..
...
ws.Handle("/find/{abc}", doFindFunc)
...
}
My question is - what's the recommended practice for recovery from session that has timed out, lost connection (the mongo service provider I'm using is remote, so I assume that this will happen), so on any particular web service call, the database session may no longer work? How do people handle these cases to detect that the session is no longer valid and a "fresh" one should be established?
Thanks!
what you may want is to do the session .Copy() for each incoming HTTP request (with deffered .Close()), copy again from the new session in your handlers if ever needed..
connections and reconnections are managed by mgo, you can stop and restart MongoDB while making an HTTP request to your web service to see how its affected.
if there's a db connection problem while handling an HTTP request, a db operation will eventually timeout (timeout can be configured by using DialWithTimeout instead of the regular Dial, so you can respond with a 5xx HTTP error code in such case.
Related
I have an unary gRPC call that can take up to few minute to be processed on my Go gRPC server (involves human agent on the mobile APP). I would like to know if there is a way to check if the connection has been terminated on the client side before sending the response.
I found the solution for ServerStreaming case with Context Status.Done channel, but it does not work for my Unary RPC.
Below is the signature of the function where the control should be made:
func (*Server) EndpointName(ctx context.Context, in *pb.EndpointRequest) (*pb.EndpointResponse, error) {
As per the function definition shown in your question, the endpoint function is passed a context (ctx context.Context). If the connection drops the context will be cancelled.
For example I can modify the helloworld example so that it simulates a long running job:
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
select {
case <-ctx.Done():
fmt.Println("Context is done:", ctx.Err())
return nil, status.Error(codes.Canceled, "does not matter as nothing will ever get this anyway...")
case <-time.After(time.Minute):
// This simulates a long-running process. In reality the process itself should be checking the context
}
return &pb.HelloReply{}, nil
}
To test this I altered greeter_client to call the function and then panic() whilst waiting for the response; the server outputs:
2022/08/08 08:16:57 server listening at [::]:50051
Context is done: context canceled
I am not fully familiar with golang patterns / best practices with respect to database connections. I am trying to implement a simple web service in go, using jackc/pgx connection pool.
In a file (let us call it users.go, which is a web service with user registration methods), I am having a connection pool object at the module level. Every time a request lands on the app, I want to test if the connection pool was initialized,if yes return the pool, else initialize the pool and return it.
I am trying to write code as below..
var pool Pool
// the request handler fn
func registerUser (c *gin.Context) {
// get connection pool and insert into db.
pool, err = getConnPool()
...
}
// the getConnPool function
func getConnPool () {
// How to test if the pool was not initialized???
if pool == nil {
// initialize
pool, err := pgxpool.Connect(context.Background(), os.Getenv("DATABASE_URL"))
return pool, err
}
return pool, nil
}
How to test the pool variable if it is connected / not nil etc?
I am not sure, if this is the correct approach in go, while I have used this kind of lazy initialization in NodeJS frequently. Should I initialize the pool in the main file, or is there an alternative like above?
I reuse the http client connection to make external calls to a single endpoint. An excerpt of the program is shown below:
var AppCon MyApp
func New(user, pass string, platformURL *url.URL, restContext string) (*MyApp, error) {
if AppCon == (MyApp{}) {
AppCon = MyApp{
user: user,
password: pass,
URL: platformURL,
Client: &http.Client{Timeout: 30 * time.Second},
RESTContext: restContext,
}
cj, err := cookiejar.New(nil)
if err != nil {
return &AppCon, err
}
AppCon.cookie = cj
}
return &AppCon, nil
}
// This is an example only. There are many more functions which accept *MyApp as a pointer.
func(ma *MyApp) GetUser(name string) (string, error){
// Return user
}
func main(){
for {
// Get messages from a queue
// The message returned from the queue provide info on which methods to call
// 'm' is a struct with message metadata
c, err := New(m.un, m.pass, m.url)
go func(){
// Do something i.e c.GetUser("123456")
}()
}
}
I now have the requirement to set up a client connections with different endpoints/credentials received via queue messages.
The problem I foresee is I can't just simply modify AppCon with the new endpoint details since a pointer to MyApp is returned, resulting in resetting c. This can impact a goroutine making a HTTP call to an unintended endpoint. To make matters non trivial, the program is not meant to have awareness of the endpoints (I was considering using a switch statement) but rather receive what it needs via queue messages.
Given the issues I've called out are correct, are there any recommendations on how to solve it?
EDIT 1
Based on the feedback provided, I am inclined to believe this will solve my problem:
Remove the use of a Singleton of MyApp
Decouple the http client from MyApp which will enable it for reuse
var httpClient *http.Client
func New(user, pass string, platformURL *url.URL, restContext string) (*MyApp, error) {
AppCon = MyApp{
user: user,
password: pass,
URL: platformURL,
Client: func() *http.Client {
if httpClient == nil {
httpClient = &http.Client{Timeout: 30 * time.Second}
}
return httpClient
}()
RESTContext: restContext,
}
return &AppCon, nil
}
// This is an example only. There are many more functions which accept *MyApp as a pointer.
func(ma *MyApp) GetUser(name string) (string, error){
// Return user
}
func main(){
for {
// Get messages from a queue
// The message returned from the queue provide info on which methods to call
// 'm' is a struct with message metadata
c, err := New(m.un, m.pass, m.url)
// Must pass a reference
go func(c *MyApp){
// Do something i.e c.GetUser("123456")
}(c)
}
}
Disclaimer: this is not a direct answer to your question but rather an attempt to direct you to a proper way of solving your problem.
Try to avoid a singleton pattern for you MyApp. In addition, New is misleading, it doesn't actually create a new object every time. Instead you could be creating a new instance every time, while preserving the http client connection.
Don't use constructions like this: AppCon == (MyApp{}), one day you will shoot in your leg doing this. Use instead a pointer and compare it to nil.
Avoid race conditions. In your code you start a goroutine and immediately proceed to the new iteration of the for loop. Considering you re-use the whole MyApp instance, you essentially introduce a race condition.
Using cookies, you make your connection kinda stateful, but your task seems to require stateless connections. There might be something wrong in such an approach.
I am building service that needs to send events to all subscribed consumers in Pub/Sub manner eg. send one event to all currently connected clients.
I am using Protobuf for that with the following proto definition:
service EventsService {
rpc ListenForEvents (AgentProcess) returns (stream Event) {}
}
Both server & client are written in Go.
My problem is that when client initiates connection then the stream it is not long-lived, eg. when server returns from ListenForEvents method:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
//persist listener here so it can be used later when backend needs to send some messages to client
return nil
}
then the client almost instantly gets EOF error which means that server probably closed connection.
What do I do so that the client is subscribed for a long time to the server? The main problem is that I might not have anything to send to the client when it calls ListenForEvents method on the server, this is why I want this stream to be long lived to be able to send messages later.
The stream terminates when you return from the server function. Instead, you should receive events somehow, and send them to the client without returning from your server. There are probably many ways you can do this. Below is the sketch of one way of doing it.
This relies on the server connection running on a separate goroutine. There is a Broadcast() function that will send messages to all connected clients. It looks like this:
var allRegisteredClients map[*pb.AgentProcess]chan Message
var clientsLock sync.RWMutex{}
func Broadcast(msg Message) {
clientsLock.RLock()
for _,x:=range allRegisteredClients {
x<-msg
}
clientsLock.RUnlock()
}
Then, your clients have to register themselves, and process messages:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
clientsLock.Lock()
ch:=make(chan Message)
allRegisteredClients[process]=ch
clientsLock.Unlock()
for msg:=range ch {
// send message
// Deal with errors
// Deal with client terminations
}
clientsLock.Lock()
delete(allRegisteredClients,process)
clientsLock.Unlock()
}
As I said, this is only a sketch of the idea.
I have managed to nail it down.
Basically I never return from method ListenForEvents.
It creates channel, persists in global-like map of subscribed clients and keeps reading from that channel indefinitely.
The whole implementation of server logic:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
chans, exists := e.listeners[process.Hostname]
chanForThisClient := make(chan *pb.Event)
if !exists {
e.listeners[process.Hostname] = []chan *pb.Event{chanForThisClient}
} else {
e.listeners[process.Hostname] = append(chans, chanForThisClient)
}
for {
select {
case <-listener.Context().Done():
return nil
case res := <-chanForThisClient:
_ = listener.Send(res)
}
}
return nil
}
You need to provide keepalive settings for grpc client and server
See details here https://github.com/grpc/grpc/blob/master/doc/keepalive.md
Examples https://github.com/grpc/grpc-go/tree/master/examples/features/keepalive
I have a web app whose server creates a Client for each websocket connection. A Client acts as an intermediary between the websocket connection and a single instance of a Hub. The Hub maintains a set of registered clients and broadcasts messages to the clients. This works pretty well but the problem is that a client might miss events between when the server generates the initial state bundle that the client receives on connection and when the client is registered with the hub and starts receiving broadcast events.
My idea is to register the client with the hub before any information is fetched from the db. That would ensure that the client doesn't miss any broadcasts, though now it could receive messages that are already applied to the initial state it receives. To allow the client to disregard these messages I could include a monotonic timestamp in both the initial state bundle as well as broadcast events.
Can you think of a more elegant/simpler solution?
I have used a write-ahead-log in the past to do something like this. In short, keep a ring buffer of messages in the hub. Then replay messages that where send to existing clients while the new one was initialized.
You can expose this concept to the clients too if you wish. That way you can implement efficient re-connects (particularly nice for mobile connections). When clients loose the websocket connection they can reconnect and say "Hey there, it's me again. Looks like we got interrupted. The last message I've seen was number 42. What's new?"
The following is from memory, so take this only as an illustration of the idea, not a finished implementation. In the intererest of brevity I've omited the select statements around client.send, for instance.
package main
import (
"container/list"
"sync"
"github.com/gorilla/websocket"
)
type Client struct { // all unchanged
hub *Hub
conn *websocket.Conn
send chan []byte
}
type Hub struct {
mu *sync.RWMutex
wal list.List // List if recent messages
clients map[*Client]bool // Registered clients.
register chan Registration // not a chan *Client anymore
broadcast chan []byte
unregister chan *Client
}
type Registration struct {
client *Client
// init is a function that is executed before the client starts to receive
// broadcast messages. All messages that are broadcast while init is
// running will be sent after init returns.
init func()
}
func (h *Hub) run() {
for {
select {
case reg := <-h.register:
// Take note of the most recent message as of right now.
// initClient will replay all later messages
h.mu.RLock()
head := h.wal.Back()
h.mu.RUnlock()
go h.initClient(reg, head)
case client := <-h.unregister:
h.mu.Lock()
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
h.mu.Unlock()
case message := <-h.broadcast:
h.mu.Lock()
h.wal.PushBack(message)
// TODO: Trim list if too long by some metric (e.g. number of
// messages, age, total message size, etc.)
clients := make([]*Client, len(h.clients))
copy(clients, h.clients)
h.mu.Unlock()
for client := range clients {
// TODO: deal with backpressure
client.send <- message
}
}
}
}
func (h *Hub) initClient(reg Registration, head *list.Element) {
reg.init()
// send messages in h.wal after head
for {
h.mu.RLock()
head = head.Next()
if head == nil {
// caught up
h.clients[reg.client] = true
h.mu.RUnlock()
return
}
h.mu.RUnlock()
// TODO: deal with backpressure
reg.client.send <- head.Value.([]byte)
}
}