How do I reuse a POST request and overwrites its payload? - go

I have use case where a Go Client with a file-watcher listens to changes in a file and sends these changes to a Go server. If the Client can't reach the server during the POST requests with the payload of the file changes, the Client will keep trying every 3 seconds to send the request until err := http.NewRequest() dosen't return a non-nil err
But If the Client is currently trying every 3 seconds to send a POST request but at the same time a new change occurs to the file under file-watch, I want the current POST requests's payload to be overwritten by the new payload(new changes from the file)
How Do I archive this best?
Client code for sending an HTTP requests
func makeRequest(method string, body io.Reader) (*http.Response, error) {
client := &http.Client{}
request, err := http.NewRequest(.., .., ..)
if err != nil {
log.Println("Error: Couldn't make a new Request:", err)
return nil, err
}
response, err := client.Do(request)
if err != nil {
log.Printf("Error: Couldn't execute %s request:%s", method, err)
}
return response, err
}
The function that retries until err !=nil
func autoRetry(f func() error) {
if err := backoff.Retry(f, getBackOff()); err != nil {
log.Println("Error: Couldn't execute exponential backOff retries: ", err)
}
}
autoRetry() is just a function which takes a function and uses ExponentialBackOff to calculate the amount of tries until err !=nil
The call to the method doing the POST request with retries
func postTodo() {
autoRetry(func() error {
r, err := makeRequest("POST", getFileData())
if err != nil {
return err
}
if r.StatusCode != 200 {
return errors.New("Error:" + r.Status)
}
return nil
})
}

Related

Is there a good way to do cache saves in goroutine?

Let's say I have a handler that makes a request and gets the latest data on the selected stock:
func (ss *stockService) GetStockInfo(ctx *gin.Context) {
code := ctx.Param("symbol")
ss.logger.Info("code", code)
url := fmt.Sprintf("URL/%v", code)
ss.logger.Info(url)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
errs.HTTPErrorResponse(ctx, &ss.logger, errs.New(errs.Internal, err))
return
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
errs.HTTPErrorResponse(ctx, &ss.logger, errs.New(errs.Internal, err))
return
}
defer resp.Body.Close()
var chart ChartResponse
err = json.NewDecoder(resp.Body).Decode(&chart)
if err != nil {
errs.HTTPErrorResponse(ctx, &ss.logger, errs.New(errs.Internal, err))
return
}
ctx.JSON(http.StatusOK, chart)
}
And I want to add caching here. Since I don't have a lot of experience right now, I'm interested in proper interaction with the cache.
I think that if, for example, it is not possible to save to the cache for some reason, then you can simply make a request to the api. Then I wonder if it would be right to save to the cache in a separate goroutine and immediately return the response:
func (ss *stockService) GetStockInfo(ctx *gin.Context) {
code := ctx.Param("symbol")
stockInfo, err := ss.cache.Get(code)
if err == nil {
// FIND
...
ctx.JSON(http.StatusOK, chart)
} else {
ss.logger.Info("code", code)
url := fmt.Sprintf("URL/%v", code)
ss.logger.Info(url)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
...
err = json.NewDecoder(resp.Body).Decode(&chart)
// IS IT A GOOD WAY ?
go ss.cache.Save(code,chart,expireAt)
ctx.JSON(http.StatusOK, chart)
}
}
I use redis as a cache.
I will be glad if someone says what is wrong with this approach.

"context deadline exceeded" AFTER first client-to-server request & response

In a simple gRPC example, I connect to the server, request to send a text message, and in return a success message is sent. Once the server is live, the first client request is successful, with no errors.
However, after the first try, each subsequent request (identical to the first) return this error, and does not return a response as the results (the text message is still sent, but the generated ID is not sent back):
rpc error: code = DeadlineExceeded desc = context deadline exceeded
After debugging a little bit, I found that the error
func (c *messageSchedulerClient) SendText(ctx context.Context, in *TextRequest, opts ...grpc.CallOption) (*MessageReply, error) {
...
err := c.cc.Invoke(ctx, "/communication.MessageScheduler/SendText", in, out, opts...)
...
return nil, err
}
returns
rpc error: code = DeadlineExceeded desc = context deadline exceeded
Here is my client code:
func main() {
// Set up a connection to the server.
conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock())
if err != nil {
log.Fatalf("did not connect: %v", err)
}
c := pb.NewMessageSchedulerClient(conn)
var r *pb.MessageReply
r, err = pbSendText(c, false)
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.GetId())
err = conn.Close()
if err != nil {
log.Printf("Connection Close Error: %v", err)
}
return
}
func pbSendText(c pb.MessageSchedulerClient, instant bool) (*pb.MessageReply, error) {
ctx, cancel := context.WithTimeout(context.Background(), time.Second * 5)
minute := timestamppb.New(time.Now().Add(time.Second * 30))
r, err := c.SendText(ctx, &pb.TextRequest{})
if err != nil {
log.Printf("DEBUG MESSAGE: Error after c.SendText(ctx, in): %v", err)
}
cancel()
return r, err
}
and the server code is...
func (s *MessageServer) SendText(ctx context.Context, in *pb.TextRequest) (*pb.MessageReply, error) {
return &pb.MessageReply{Id: GeneratePublicId()}, nil
}
func GrpcServe() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterMessageSchedulerServer(s, &MessageServer{})
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
return
}
const Alphabet = "abcdefghijklmnopqrstuvwxyzABCDEFGHJKMNOPQRSTUVWXYZ0123457"
// executes instantly, O(n)
func GeneratePublicId() string {
return id.NewWithAlphabet(Alphabet)
}
I've tried changing the context to .TODO from .Background. Doesn't work. I am SURE it's something so simple that I'm missing.
Again, the first time I execute the application from the client side, it works. But after the first client application execution, meaning application execution and beyond -- until I restart the gRPC Server, it gives me the error.
The GeneratePublicId function executes almost instantly, as it is O(n) and uses a random number function.

How to write functional tests

I have the requirement to test a Kafka consumer method which in turn creates a post call to another service. How can I write wiremock for the another service. The code is:
Kafka Consumer
func (handler FxmWorklogEventHandler) ProcessMessage(message kafka.Message) (swgykafka.Status, error) {
txn := util.TraceTransaction(handler.MessageType())
if txn != nil {
defer txn.End()
}
var fxmWorklogEvent event.FxmWorklogEvent
err := json.Unmarshal(message.Data, &fxmWorklogEvent)
if err != nil {
log.Error().Err(err).Msg("FxmWorklogEventHandler - Failed to un marshall the message")
return swgykafka.HardFailure, err
}
log.Info().Msgf("Processing fxm worklog event for order %s", fxmWorklogEvent.OrderId)
fxmEventToPersist := BuildFxmEventToPersist(fxmWorklogEvent)
_, err = handler.am.SaveFxmWorklog(fxmEventToPersist)
if err != nil {
log.Error().Err(err).Msgf("Failed to save fxm worklog event in am for order %s", fxmWorklogEvent.OrderId)
return swgykafka.SoftFailure, nil
}
return swgykafka.Success, nil
}
Post Call
func (am *AssistanceManagerClient) SaveFxmWorklog(r request.FxmWorklogDetails) (*response.BaseResponse, error) {
url := config.Client.GetString(amHost) + util.AssistanceManagerSaveFxmWorklog
return am.hc.Post(url, r, http.Header{}, util.AssistanceManagerCommand)
}
External Service Method
func SaveFxmWorklog(w http.ResponseWriter, r *http.Request) {
var rb entity.FxmWorklogDetails
err := json.NewDecoder(r.Body).Decode(&rb)
if err != nil {
log.Error().Err(err).Msgf("Failed to parse the fxm worklog request %s", util.ToJSON(rb))
util.ErrorResponse(w, err.Error(), http.StatusBadRequest)
return
}
log.Info().Msgf("Save fxm worklog request received for order %s", rb.OrderId)
response, err := service.FxmService.SaveFxmWorklogEvent(rb)
if err != nil {
log.Error().Err(err).Msgf("Failed to save fxm worklog event %s", util.ToJSON(rb))
util.ErrorResponse(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(response.HTTPStatusCode)
_ = json.NewEncoder(w).Encode(response)
}
How can I test this method by wiremocking the external service api call?

How to turn DataBase access into a Function idiomatically in Go

I have built a Backend API in Go, it works however I want refactor the code for the DB access layer into a function - idiomatically.
// Get the form data entered by client; FirstName, LastName, phone Number,
// assign the person a unique i.d
// check to see if that user isn't in the database already
// if they are send an error message with the a 'bad' response code
// if they aren't in db add to db and send a message with success
func CreateStudentAccountEndpoint(response http.ResponseWriter, request *http.Request){
client, err := mongo.NewClient("mongodb://localhost:27017")
if err != nil {
log.Fatalf("Error connecting to mongoDB client Host: Err-> %v\n ", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil {
log.Fatalf("Error Connecting to MongoDB at context.WtihTimeout: Err-> %v\n ", err)
}
response.Header().Set("Content-Type", "application/json")
studentCollection := client.Database(dbName).Collection("students")
_, err = studentCollection.InsertOne(context.Background(),data)
if err != nil {
response.WriteHeader(501)
response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
}
// encoding json object for returning to the client
jsonStudent, err := json.Marshal(student)
if err != nil {
http.Error(response, err.Error(), http.StatusInternalServerError)
}
response.Write(jsonStudent)
}
I understand that I can create a method which returns (*mongoClient, err) as I utilise the client local variable later on in the code.
However I am lost as to how to implement the defer cancel() part because it executes once the method CreateStudenAccountEndpoint is at the end. But I am at a loss on how to implement this defer section in a method that will recognise that I want the defer to happen at the end of the function that calls the DB access layer method e.g CreateStudentAccountEndpoint not the actual db access method itself.
As I understand it, the connection should be long-lived and set up as a part of a constructor, i.e. not part of the request flow.
This will typically look something like this:
type BackendAPI struct {
client *mongo.Client
}
func NewBackendAPI(mongoURI string) (*BackendAPI, error) {
client, err := mongo.NewClient(mongoURI)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil {
return nil, err
}
return &BackendAPI{client}, nil
}
func (api *BackendAPI) func CreateStudentAccountEndpoint(response http.ResponseWriter, request *http.Request) {
response.Header().Set("Content-Type", "application/json")
// note the use of the long-lived api.client, which is connected already.
studentCollection := api.client.Database(dbName).Collection("students")
_, err = studentCollection.InsertOne(context.Background() ,data)
if err != nil {
response.WriteHeader(501)
response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
return // at this point, the method should return
}
// encoding json object for returning to the client
jsonStudent, err := json.Marshal(student)
if err != nil {
http.Error(response, err.Error(), http.StatusInternalServerError)
}
response.Write(jsonStudent)
}
If you worry about losing the connection, you could implement a call to api.client.Ping in there, but in my opinion this should only be attempted if you encounter a failure you believe you can recover from by reconnecting.

Golang server "write tcp ... use of closed network connection"

I am beginner at Go, I had wrote small server to testing and deploy it on heroku platform. I have /logout request, which almost works, but sometimes I see something like this:
PANIC: write tcp 172.17.110.94:36641->10.11.189.195:9951: use of closed network connection
I don't know why it happens, and why sometimes it works perfectly.
My steps:
I send 1st POST request to /token-auth with body then generate token and send as response.
At 2nd I do /logout GET request with that token, and set token to Redis store
Here is full code of my redil_cli.go
package store
import (
"github.com/garyburd/redigo/redis"
)
type RedisCli struct {
conn redis.Conn
}
var instanceRedisCli *RedisCli = nil
func Connect() (conn *RedisCli) {
if instanceRedisCli == nil {
instanceRedisCli = new(RedisCli)
var err error
//this is works!!!
instanceRedisCli.conn, err = redis.Dial("tcp", "lab.redistogo.com:9951")
if err != nil {
panic(err)
}
if _, err := instanceRedisCli.conn.Do("AUTH", "password"); err != nil {
//instanceRedisCli.conn.Close()
panic(err)
}
}
return instanceRedisCli
}
func (redisCli *RedisCli) SetValue(key, value string, expiration ...interface{}) error {
_, err := redisCli.conn.Do("SET", key, value)
if err == nil && expiration != nil {
redisCli.conn.Do("EXPIRE", key, expiration[0])
}
return err
}
func (redisCli *RedisCli) GetValue(key string) (interface{}, error) {
data, err := redisCli.conn.Do("GET", key)
if err != nil{
panic(err)
}
return data, err
}
After that my function that checks Authorization header will panic while trying to do GetValue(key string) method
func (redisCli *RedisCli) GetValue(key string) (interface{}, error) {
data, err := redisCli.conn.Do("GET", key)
if err != nil{
panic(err)
}
return data, err
}
Can anyone point me, what I doing wrong?

Resources