Go Error handling on a REST API - go

I am very new to go and have deployed a small service with an API endpoint.
I have heard/read that go doesn't use try/catch so I am trying to figure out how I can "catch" any problems happening from my service call from my API and make sure that the resource server doesn't go down.
My code for my API looks like the following..
I have a routes.go file with the following
package main
import (
"net/http"
"github.com/gorilla/mux"
)
type Route struct {
Name string
Method string
Pattern string
HandlerFunc http.HandlerFunc
}
type Routes []Route
func NewRouter() *mux.Router {
router := mux.NewRouter().StrictSlash(true)
for _, route := range routes {
router.
Methods(route.Method).
Path(route.Pattern).
Name(route.Name).
Handler(route.HandlerFunc)
}
return router
}
var routes = Routes{
Route{
"CustomerLocationCreate",
"POST",
"/tracking/customer",
CustomerLocationCreate,
},
}
I have a handlers.go
package main
import (
"encoding/json"
"net/http"
"io"
"io/ioutil"
)
//curl -H "Content-Type: application/json" -d '{"userId":"1234"}' http://localhost:8181/tracking/customer
func CustomerLocationCreate(w http.ResponseWriter, r *http.Request) {
var location CustomerLocation
body, err := ioutil.ReadAll(io.LimitReader(r.Body, 1048576))
if err != nil {
panic(err)
}
if err := r.Body.Close(); err != nil {
panic(err)
}
if err := json.Unmarshal(body, &location); err != nil {
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.WriteHeader(422) // unprocessable entity
if err := json.NewEncoder(w).Encode(err); err != nil {
panic(err)
}
}
c := RepoCreateCustomerLocation(location)
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.WriteHeader(http.StatusCreated)
if err := json.NewEncoder(w).Encode(c); err != nil {
panic(err)
}
HandleCustomerLocationChange(c);
}
and I have a bus.go which has the HandleCustomerLocationChange(...) function.
func HandleCustomerLocationChange(custLoc CustomerLocation) {
endpoint := og.Getenv("RABBIT_ENDPOINT")
conn, err := amqp.Dial("amqp://guest:guest#" + endpoint)
failOnError(err, "Failed to connect to RabbitMQ")
defer conn.Close()
ch, err := conn.Channel()
failOnError(err, "Failed to open a channel")
defer ch.Close()
topic := "locationChange"
err = ch.ExchangeDeclare(
topic, // name
"topic", // type
true, // durable
false, // auto-deleted
false, // internal
false, // no-wait
nil, // arguments
)
failOnError(err, "Failed to declare an exchange")
// Create JSON from the instance data.
body, _ := json.Marshal(custLoc)
// Convert bytes to string.
err = ch.Publish(
topic, // exchange
"", // routing key
false, // mandatory
false, // immediate
amqp.Publishing{
ContentType: "text/plain",
Body: body,
})
failOnError(err, "Failed to publish a message")
log.Printf(" [x] Sent %s", body)
}
My question is how should I modify both the HandleCustomerLocationChange(...) function and if necessaryCustomerLocationChange(..)` handler to handle errors properly so that if an error occurs, my entire API doesn't go down?

Go suggests a different approach, that errors are not exceptional, they're normal events, just less common.
Taking an example from the code above:
body, err := ioutil.ReadAll(io.LimitReader(r.Body, 1048576))
if err != nil {
panic(err)
}
Here, a panic (without recovery) terminates the process, shutting down the web server. Seems an overly severe response to not fully reading a request.
What do you want to do? It may be appropriate to tell the client who made the request:
body, err := ioutil.ReadAll(io.LimitReader(r.Body, 1048576))
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
You might want to return a json encoded response, or give a generic message to the client avoid exposing too much, and log the specific error details.
For general functions it's idiomatic to return the error as the last return parameter. In the specific example you mentioned:
func HandleCustomerLocationChange(custLoc CustomerLocation)
...
conn, err := amqp.Dial(...)
failOnError(err, "Failed to connect to RabbitMQ")
Instead, check if the connection failed, and return the error to the caller. Handle it in the calling function, or add information and propagate it up the call stack.
func HandleCustomerLocationChange(custLoc CustomerLocation) error
...
conn, err := amqp.Dial(...)
if err != nil {
return fmt.Errorf("failed to connect to RabbitMQ: %s", err)
}
Propagating the error in this way gives a concise explanation of the root cause, like the 5 whys technique, eg:
"did not update client location: did not connect to rabbitmq: network address 1.2.3 unreachable"
Another convention is to deal with errors first and return early. This helps to reduce nesting.
See also the many error handling resources, like error handling in a web application, Go by Example, Error Handling and Go, errors are values and Defer, Panic & Recover. The source code of the error package is interesting, as is Russ Cox's comment on error handling, and Nathan Youngman's To Err is Human.
Also interesting is Upspin's concept of an operational trace, rather than a stack trace.

Related

GoLang net/http memory keeps increasing on contineous requests

I have the following code in GoLang
package main
import (
"bytes"
"encoding/json"
"io/ioutil"
"log"
"net/http"
"time"
)
func httpClient() *http.Client {
var transport http.RoundTripper = &http.Transport{
DisableKeepAlives: false,
}
client := &http.Client{Timeout: 60 * time.Second, Transport: transport}
return client
}
func sendRequest(client *http.Client, method string) []byte {
endpoint := "https://httpbin.org/post"
values := map[string]string{"foo": "baz"}
jsonData, err := json.Marshal(values)
req, err := http.NewRequest(method, endpoint, bytes.NewBuffer(jsonData))
if err != nil {
log.Fatalf("Error Occurred. %+v", err)
}
resp, err:= client.Do(req)
if err != nil {
defer resp.Body.Close()
log.Fatalf("Error sending request to API endpoint. %+v", err)
}
// Close the connection to reuse it
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("Couldn't parse response body. %+v", err)
}
return body
}
func main() {
// c should be re-used for further calls
c := httpClient()
for i := 1; i <= 60; i++ {
response := sendRequest(c, http.MethodPost)
log.Println("Response Body:", string(response))
response = nil
time.Sleep(time.Millisecond * 1000)
}
}
When executed, it keeps the memory size increasing and the growth goes to as much as 90mb in one hour. is the gc not working properly. Even though i am using same httpclient for multiple requests but it still looks like theres something thats increasing the size of memory footprint.
I advice you to use tools like pprof, these are very useful at troubleshooting precisely this kind of issues.
You have set DisableKeepAlives field to false, which means that it will keep open connections even after the requests have been made, leading to further memory leaks. You should also call defer resp.Body.Close() after calling ioutil.ReadAll(resp.Body). This is precisely the purpose of the defer keyword - preventing memory leaks. GC does not mean absolute memory safety.
Also, outside of main avoid using log.Fatal. Use leveled logger, like zap or zerolog instead, since log.Fatal calls os.Exit(1) with an immediate effect, which means your defer statements will take no effect, or call plain panic. See Should a Go package ever use log.Fatal and when?

Google PubSub and Go: create client outside or inside publish-function?

I'm new when it comes to Google PubSub(and pubsub applications in general). I'm also relatively new when it comes to Go.
I'm working on a pretty heavy backend service application that already has too many responsibilities. The service needs to fire off one message for each incoming request to a Google PubSub topic. It only needs to "fire and forget". If something goes wrong with the publishing, nothing will happen. The messages are not crucial(only used for analytics), but there will be many of them. We estimate between 50 and 100 messages per second for most of the day.
Now to the code:
func(p *publisher) Publish(message Message, log zerolog.Logger) error {
ctx := context.Background()
client, err := pubsub.NewClient(ctx, p.project)
defer client.Close()
if err != nil {
log.Error().Msgf("Error creating client: %v", err)
return err
}
marshalled, _ := json.Marshal(message)
topic := client.Topic(p.topic)
result := topic.Publish(ctx, &pubsub.Message{
Data: marshalled,
})
_, err = result.Get(ctx)
if err != nil {
log.Error().Msgf("Failed to publish message: %v", err)
return err
}
return nil
}
Disclaimer: p *publisher only contains configuration.
I wonder if this is the best way? Will this lead to the service creating and closing a client 100 times per second? If so, then I guess I should create the client once and pass it as an argument to the Publish()-function instead?
This is how the Publish()-function gets called:
defer func(publisher publish.Publisher, message Message, log zerolog.Logger) {
err := publisher.Publish(log, Message)
if err != nil {
log.Error().Msgf("Failed to publish message: %v", err)
}
}(publisher, message, logger,)
Maybe the way to go is to hold pubsubClient & pubsubTopic inside struct?
type myStruct struct {
pubsubClient *pubsub.Client
pubsubTopic *pubsub.Topic
logger *yourLogger.Logger
}
func newMyStruct(projectID string) (*myStruct, error) {
ctx := context.Background()
pubsubClient, err := pubusb.NewClient(ctx, projectID)
if err != nil {...}
pubsubTopic := pubsubClient.Topic(topicName)
return &myStruct{
pubsubClient: pubsubClient,
pubsubTopic: pubsubTopic,
logger: Logger,
// and whetever you want :D
}
}
And then for that struct create a method, which will take responsibility of marshalling the msg and sends it to Pub/sub
func (s *myStruct) request(ctx context.Context data yorData) {
marshalled, err := json.Marshal(message)
if err != nil {..}
res := s.pubsubTopic.Publish(ctx, &pubsub.Message{
Data: marshalled,
})
if _, err := res.Get(ctx); err !=nil {..}
return nil
}

Golang bufio from websocket breaking after first read

I am trying to stream JSON text from a websocket. However after an initial read I noticed that the stream seems to break/disconnect. This is from a Pleroma server (think: Mastodon). I am using the default Golang websocket library.
package main
import (
"bufio"
"fmt"
"log"
"golang.org/x/net/websocket"
)
func main() {
origin := "https://poa.st/"
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, err := websocket.Dial(url, "", origin)
if err != nil {
log.Fatal(err)
}
s := bufio.NewScanner(ws)
for s.Scan() {
line := s.Text()
fmt.Println(line)
}
}
After the initial JSON text response, the for-loop breaks. I would expect it to send a new message every few seconds.
What might be causing this? I am willing to switch to the Gorilla websocket library if I can use it with bufio.
Thanks!
Although x/net/websocket connection has a Read method with the same signature as the Read method in io.Reader, the connection does not work like an io.Reader. The connection will not work as you expect when wrapped with a bufio.Scanner.
The poa.st endpoint sends a stream of messages where each message is a JSON document. Use the following code to read the messages using the Gorilla package:
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
_, p, err := ws.ReadMessage()
if err != nil {
log.Fatal(err)
}
// p is a []byte containing the JSON document.
fmt.Printf("%s\n", p)
}
The Gorilla package has a helper method for decoding JSON messages. Here's an example of how to use that method.
url := "wss://poa.st/api/v1/streaming/?stream=public"
ws, _, err := websocket.DefaultDialer.Dial(url, nil)
if err != nil {
log.Fatal(err)
}
defer ws.Close()
for {
// The JSON documents are objects containing two fields,
// the event type and the payload. The payload is a JSON
// document itself.
var e struct {
Event string
Payload string
}
err := ws.ReadJSON(&e)
if err != nil {
log.Fatal(err)
}
// TODO: decode e.Payload based on e.Event
}

How to use self-describing message for protbuf

One of the use cases I'm working on while using protocol buffers is to deserialize the Protocol Buffers Kafka messages which I receive at the consumer end (using sarama library and Go).
The way how i'm doing currently is i defined the sample pixel.proto file as show below.
syntax = "proto3";
package saramaprotobuf;
message Pixel {
// Session identifier stuff
string session_id = 2;
}
i'm sending the message through sarama.Producer(by marshalling it) receiving it sarama.Consumer (unmarshalling message it by referencing with complied pixel.proto.pb). Code is as below.
import (
"github.com/Shopify/sarama"
"github.com/golang/protobuf/proto"
"log"
"os"
"os/signal"
"protobuftest/example"
"syscall"
"time"
)
func main() {
topic := "test_topic"
brokerList := []string{"localhost:9092"}
producer, err := newSyncProducer(brokerList)
if err != nil {
log.Fatalln("Failed to start Sarama producer:", err)
}
go func() {
ticker := time.NewTicker(time.Second)
for {
select {
case t := <-ticker.C:
elliot := &example.Pixel{
SessionId: t.String(),
}
pixelToSend := elliot
pixelToSendBytes, err := proto.Marshal(pixelToSend)
if err != nil {
log.Fatalln("Failed to marshal example:", err)
}
msg := &sarama.ProducerMessage{
Topic: topic,
Value: sarama.ByteEncoder(pixelToSendBytes),
}
producer.SendMessage(msg)
log.Printf("Pixel sent: %s", pixelToSend)
}
}
}()
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM)
partitionConsumer, err := newPartitionConsumer(brokerList, topic)
if err != nil {
log.Fatalln("Failed to create Sarama partition consumer:", err)
}
log.Println("Waiting for messages...")
for {
select {
case msg := <-partitionConsumer.Messages():
receivedPixel := &example.Pixel{}
err := proto.Unmarshal(msg.Value, receivedPixel)
if err != nil {
log.Fatalln("Failed to unmarshal example:", err)
}
log.Printf("Pixel received: %s", receivedPixel)
case <-signals:
log.Print("Received termination signal. Exiting.")
return
}
}
}
func newSyncProducer(brokerList []string) (sarama.SyncProducer, error) {
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForAll
config.Producer.Retry.Max = 5
config.Producer.Return.Successes = true
// TODO configure producer
producer, err := sarama.NewSyncProducer(brokerList, config)
if err != nil {
return nil, err
}
return producer, nil
}
func newPartitionConsumer(brokerList []string, topic string) (sarama.PartitionConsumer, error) {
conf := sarama.NewConfig()
// TODO configure consumer
consumer, err := sarama.NewConsumer(brokerList, conf)
if err != nil {
return nil, err
}
partitionConsumer, err := consumer.ConsumePartition(topic, 0, sarama.OffsetOldest)
if err != nil {
return nil, err
}
return partitionConsumer, err
}
In the code as you can see I have imported the .proto file and referencing it in the main function inorder to send and receive the message. The problem here is, the solution is not generic. I will receive the message of different .proto type at the consumer end.
How can I make it generic? I know there is something called as self-describing message(dynamic message) as the part of protobuf. I referred this link https://developers.google.com/protocol-buffers/docs/techniques?csw=1#self-description . But it doesn't has any explaination on how to embed this as the part of pixel.proto(example which i have used) so that at the consumer end i came directly deserialize it to required type.
You would define a generic container message type that would include a DescriptorSet and an Any fields.
When sending, you build an instance of that generic message type, setting the field of type Any with an instance of your Pixel message and setting the DescriptorSet field with the DescriptorSet of the Pixel type.
That would allow the receiver of such message to parse the Any contents using the DescriptorSet you are attaching. In practical terms, this is sending a piece of proto definition together with the message. So receivers wouldn't need pre-shared proto definitions or generated code.
Having said that, I'm not sure this is what you really want because if you are planning to share proto definitions or generated code with clients then I'd suggest simply using a oneof field in a container type would be much simpler to use.

Consume a DELETE endpoint from Go

I am working in a Go project, and I need to perform some operations over an external API: GET, PUT, POST and DELETE. Currently I am using net/http, and I created a &http.Client{} to make GET and PUT. That is working as expected.
Now I need to perform a DELETE and I cannot find anything about it. Is it supported? Basically, I need to call a URL like this:
somedomain.com/theresource/:id
Method: DELETE
How can I perform that?
Here is a small example of how to do it:
package main
import (
"fmt"
"io/ioutil"
"net/http"
)
func sendRequest() {
// Request (DELETE http://www.example.com/bucket/sample)
// Create client
client := &http.Client{}
// Create request
req, err := http.NewRequest("DELETE", "http://www.example.com/bucket/sample", nil)
if err != nil {
fmt.Println(err)
return
}
// Fetch Request
resp, err := client.Do(req)
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
// Read Response Body
respBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return
}
// Display Results
fmt.Println("response Status : ", resp.Status)
fmt.Println("response Headers : ", resp.Header)
fmt.Println("response Body : ", string(respBody))
}

Resources