How to turn DataBase access into a Function idiomatically in Go - go

I have built a Backend API in Go, it works however I want refactor the code for the DB access layer into a function - idiomatically.
// Get the form data entered by client; FirstName, LastName, phone Number,
// assign the person a unique i.d
// check to see if that user isn't in the database already
// if they are send an error message with the a 'bad' response code
// if they aren't in db add to db and send a message with success
func CreateStudentAccountEndpoint(response http.ResponseWriter, request *http.Request){
client, err := mongo.NewClient("mongodb://localhost:27017")
if err != nil {
log.Fatalf("Error connecting to mongoDB client Host: Err-> %v\n ", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil {
log.Fatalf("Error Connecting to MongoDB at context.WtihTimeout: Err-> %v\n ", err)
}
response.Header().Set("Content-Type", "application/json")
studentCollection := client.Database(dbName).Collection("students")
_, err = studentCollection.InsertOne(context.Background(),data)
if err != nil {
response.WriteHeader(501)
response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
}
// encoding json object for returning to the client
jsonStudent, err := json.Marshal(student)
if err != nil {
http.Error(response, err.Error(), http.StatusInternalServerError)
}
response.Write(jsonStudent)
}
I understand that I can create a method which returns (*mongoClient, err) as I utilise the client local variable later on in the code.
However I am lost as to how to implement the defer cancel() part because it executes once the method CreateStudenAccountEndpoint is at the end. But I am at a loss on how to implement this defer section in a method that will recognise that I want the defer to happen at the end of the function that calls the DB access layer method e.g CreateStudentAccountEndpoint not the actual db access method itself.

As I understand it, the connection should be long-lived and set up as a part of a constructor, i.e. not part of the request flow.
This will typically look something like this:
type BackendAPI struct {
client *mongo.Client
}
func NewBackendAPI(mongoURI string) (*BackendAPI, error) {
client, err := mongo.NewClient(mongoURI)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil {
return nil, err
}
return &BackendAPI{client}, nil
}
func (api *BackendAPI) func CreateStudentAccountEndpoint(response http.ResponseWriter, request *http.Request) {
response.Header().Set("Content-Type", "application/json")
// note the use of the long-lived api.client, which is connected already.
studentCollection := api.client.Database(dbName).Collection("students")
_, err = studentCollection.InsertOne(context.Background() ,data)
if err != nil {
response.WriteHeader(501)
response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
return // at this point, the method should return
}
// encoding json object for returning to the client
jsonStudent, err := json.Marshal(student)
if err != nil {
http.Error(response, err.Error(), http.StatusInternalServerError)
}
response.Write(jsonStudent)
}
If you worry about losing the connection, you could implement a call to api.client.Ping in there, but in my opinion this should only be attempted if you encounter a failure you believe you can recover from by reconnecting.

Related

Pgxpool returns "pool closed" error on Scan

I'm trying to implement pgxpool in a new go app. I keep getting a "pool closed" error after attempting a scan into a struct.
The pgx logger into gives me this after connecting. I thought the pgxpool was meant to remain open.
{"level":"info","msg":"closed connection","pid":5499,"time":"2022-02-24T16:36:33+10:30"}
Here is my router code
func router() http.Handler {
var err error
config, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatalln(err)
}
log.Println(os.Getenv("DATABASE_URL"))
logrusLogger := &logrus.Logger{
Out: os.Stderr,
Formatter: new(logrus.JSONFormatter),
Hooks: make(logrus.LevelHooks),
Level: logrus.InfoLevel,
ExitFunc: os.Exit,
ReportCaller: false,
}
config.ConnConfig.Logger = NewLogger(logrusLogger)
db, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
--- minio connection
rs := newAppResource(db, mc)
Then, in a helper file I setup the resource
type appResource struct {
db *pgxpool.Pool
mc *minio.Client
}
// newAppResource function to pass global var
func newAppResource(db *pgxpool.Pool, mc *minio.Client) *appResource {
return &appResource{
db: db,
mc: mc,
}
}
There "pool closed" error occurs at the end of this code
func (rs *appResource) login(w http.ResponseWriter, r *http.Request) {
var user User
var login Login
d := json.NewDecoder(r.Body)
d.DisallowUnknownFields() // catch unwanted fields
err := d.Decode(&login)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err != nil {
fmt.Println("can't decode JSON", err)
}
if login.Email == "" {
log.Println("empty email")
return
}
log.Println(login.Email)
log.Println(login.Password)
if login.Password == "" {
log.Println("empty password")
return
}
// optional extra check
if d.More() {
http.Error(w, "extraneous data after JSON object", http.StatusBadRequest)
return
}
sqlStatement := "SELECT user_id, password FROM users WHERE active = 'true' AND email = ?"
row := rs.db.QueryRow(context.Background(), sqlStatement, login.Email)
err = row.Scan(&user.UserId, &user.Password)
if err == sql.ErrNoRows {
log.Println("user not found")
http.Error(w, err.Error(), http.StatusUnauthorized)
return
}
if err != nil {
log.Println(err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
It appears that you are doing something like the following:
func router() http.Handler {
db, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatalln(err)
}
defer db.Close()
return appResource{db: db}
}
The issue with this is that the defer db.Close() runs when the function router() ends and this is before the returned pgxPool.Pool is actually used (the http.Handler returned will be used later when http requests are processed). Attempting to use a closed pgxPool.Pool results in the error you are seeing.
The simplest solution is to simply remove the defer db.Close() however you might also consider calling db.Close() as part of a clean shutdown process (it needs to remain open as long as you are handling requests).
You are using pgxpool which does differ from the standard library; however I believe that the advice given in the standard library docs applies here:
It is rarely necessary to close a DB.

Unauthorized when using gocosmos to create a document

I got go-sql-driver for Azure CosmosDB from https://github.com/btnguyen2k/gocosmos.
It goes well when i call gocosmos.NewRestClient to get a rest client, CreateDatabase() to create database and CreateCollection() to create collection.
The problem is when i use CreateDocument(), i get response with statuscode 401 and body like this
{"code":"Unauthorized","message":"The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'post\ndocs\ndbs/ToDoList/colls/Items\nmon, 31 may 2021 13:31:44 gmt\n\n'\r\nActivityId: a9bbd729-3495-400f-9d79-ddec3737aa92, Microsoft.Azure.Documents.Common/2.11.0"}
i've tried all the solutions I've seen, but i haven't solved the problem.
I followed this tutorial and with this sample code, I can successfully create database, collection and document. Here's my testing result, can it help you?
// connects to MongoDB
func connect() *mongo.Client {
mongoDBConnectionString := os.Getenv(mongoDBConnectionStringEnvVarName)
if mongoDBConnectionString == "" {
log.Fatal("missing environment variable: ", mongoDBConnectionStringEnvVarName)
}
database = os.Getenv(mongoDBDatabaseEnvVarName)
if database == "" {
log.Fatal("missing environment variable: ", mongoDBDatabaseEnvVarName)
}
collection = os.Getenv(mongoDBCollectionEnvVarName)
if collection == "" {
log.Fatal("missing environment variable: ", mongoDBCollectionEnvVarName)
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
clientOptions := options.Client().ApplyURI(mongoDBConnectionString).SetDirect(true)
c, err := mongo.NewClient(clientOptions)
err = c.Connect(ctx)
if err != nil {
log.Fatalf("unable to initialize connection %v", err)
}
err = c.Ping(ctx, nil)
if err != nil {
log.Fatalf("unable to connect %v", err)
}
return c
}
// creates a todo
func create(desc string) {
c := connect()
ctx := context.Background()
defer c.Disconnect(ctx)
todoCollection := c.Database(database).Collection(collection)
r, err := todoCollection.InsertOne(ctx, Todo{Description: desc, Status: statusPending})
if err != nil {
log.Fatalf("failed to add todo %v", err)
}
fmt.Println("added todo", r.InsertedID)
}

"context deadline exceeded" AFTER first client-to-server request & response

In a simple gRPC example, I connect to the server, request to send a text message, and in return a success message is sent. Once the server is live, the first client request is successful, with no errors.
However, after the first try, each subsequent request (identical to the first) return this error, and does not return a response as the results (the text message is still sent, but the generated ID is not sent back):
rpc error: code = DeadlineExceeded desc = context deadline exceeded
After debugging a little bit, I found that the error
func (c *messageSchedulerClient) SendText(ctx context.Context, in *TextRequest, opts ...grpc.CallOption) (*MessageReply, error) {
...
err := c.cc.Invoke(ctx, "/communication.MessageScheduler/SendText", in, out, opts...)
...
return nil, err
}
returns
rpc error: code = DeadlineExceeded desc = context deadline exceeded
Here is my client code:
func main() {
// Set up a connection to the server.
conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())
if err != nil {
log.Fatalf("did not connect: %v", err)
}
c := pb.NewMessageSchedulerClient(conn)
var r *pb.MessageReply
r, err = pbSendText(c, false)
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.GetId())
err = conn.Close()
if err != nil {
log.Printf("Connection Close Error: %v", err)
}
return
}
func pbSendText(c pb.MessageSchedulerClient, instant bool) (*pb.MessageReply, error) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second * 5)
minute := timestamppb.New(time.Now().Add(time.Second * 30))
r, err := c.SendText(ctx, &pb.TextRequest{})
if err != nil {
log.Printf("DEBUG MESSAGE: Error after c.SendText(ctx, in): %v", err)
}
cancel()
return r, err
}
and the server code is...
func (s *MessageServer) SendText(ctx context.Context, in *pb.TextRequest) (*pb.MessageReply, error) {
return &pb.MessageReply{Id: GeneratePublicId()}, nil
}
func GrpcServe() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterMessageSchedulerServer(s, &MessageServer{})
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
return
}
const Alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHJKMNOPQRSTUVWXYZ0123457"
// executes instantly, O(n)
func GeneratePublicId() string {
return id.NewWithAlphabet(Alphabet)
}
I've tried changing the context to .TODO from .Background. Doesn't work. I am SURE it's something so simple that I'm missing.
Again, the first time I execute the application from the client side, it works. But after the first client application execution, meaning application execution and beyond -- until I restart the gRPC Server, it gives me the error.
The GeneratePublicId function executes almost instantly, as it is O(n) and uses a random number function.

How do I reuse a POST request and overwrites its payload?

I have use case where a Go Client with a file-watcher listens to changes in a file and sends these changes to a Go server. If the Client can't reach the server during the POST requests with the payload of the file changes, the Client will keep trying every 3 seconds to send the request until err := http.NewRequest() dosen't return a non-nil err
But If the Client is currently trying every 3 seconds to send a POST request but at the same time a new change occurs to the file under file-watch, I want the current POST requests's payload to be overwritten by the new payload(new changes from the file)
How Do I archive this best?
Client code for sending an HTTP requests
func makeRequest(method string, body io.Reader) (*http.Response, error) {
client := &http.Client{}
request, err := http.NewRequest(.., .., ..)
if err != nil {
log.Println("Error: Couldn't make a new Request:", err)
return nil, err
}
response, err := client.Do(request)
if err != nil {
log.Printf("Error: Couldn't execute %s request:%s", method, err)
}
return response, err
}
The function that retries until err !=nil
func autoRetry(f func() error) {
if err := backoff.Retry(f, getBackOff()); err != nil {
log.Println("Error: Couldn't execute exponential backOff retries: ", err)
}
}
autoRetry() is just a function which takes a function and uses ExponentialBackOff to calculate the amount of tries until err !=nil
The call to the method doing the POST request with retries
func postTodo() {
autoRetry(func() error {
r, err := makeRequest("POST", getFileData())
if err != nil {
return err
}
if r.StatusCode != 200 {
return errors.New("Error:" + r.Status)
}
return nil
})
}

Golang server "write tcp ... use of closed network connection"

I am beginner at Go, I had wrote small server to testing and deploy it on heroku platform. I have /logout request, which almost works, but sometimes I see something like this:
PANIC: write tcp 172.17.110.94:36641->10.11.189.195:9951: use of closed network connection
I don't know why it happens, and why sometimes it works perfectly.
My steps:
I send 1st POST request to /token-auth with body then generate token and send as response.
At 2nd I do /logout GET request with that token, and set token to Redis store
Here is full code of my redil_cli.go
package store
import (
"github.com/garyburd/redigo/redis"
)
type RedisCli struct {
conn redis.Conn
}
var instanceRedisCli *RedisCli = nil
func Connect() (conn *RedisCli) {
if instanceRedisCli == nil {
instanceRedisCli = new(RedisCli)
var err error
//this is works!!!
instanceRedisCli.conn, err = redis.Dial("tcp", "lab.redistogo.com:9951")
if err != nil {
panic(err)
}
if _, err := instanceRedisCli.conn.Do("AUTH", "password"); err != nil {
//instanceRedisCli.conn.Close()
panic(err)
}
}
return instanceRedisCli
}
func (redisCli *RedisCli) SetValue(key, value string, expiration ...interface{}) error {
_, err := redisCli.conn.Do("SET", key, value)
if err == nil && expiration != nil {
redisCli.conn.Do("EXPIRE", key, expiration[0])
}
return err
}
func (redisCli *RedisCli) GetValue(key string) (interface{}, error) {
data, err := redisCli.conn.Do("GET", key)
if err != nil{
panic(err)
}
return data, err
}
After that my function that checks Authorization header will panic while trying to do GetValue(key string) method
func (redisCli *RedisCli) GetValue(key string) (interface{}, error) {
data, err := redisCli.conn.Do("GET", key)
if err != nil{
panic(err)
}
return data, err
}
Can anyone point me, what I doing wrong?

Resources