Logging metrics for go grpc server - go

This is kindof an extension of my previous question Reuse log client in interceptor for Golang grpc server method.
Basically I have a grpc server (written in Go) that exposes three APIs:
SubmitJob
CancelJob
GetJobStatus
I am using Datadog to log metrics, so in each API, I have code like:
func (s *myServer) submitJob(ctx context.Context, request *submitJobRequest) (*submitJobResponse, error) {
s.dd_client.LogRequestCount("SubmitJob")
start_time := time.Now()
defer s.dd_client.LogRequestDuration("SubmitJob", time.Since(start_time))
sth, err:= someFunc1()
if err != nil {
s.dd_client.LogErrorCount("SubmitJob")
return nil, err
}
resp, err:= someFunc2(sth)
if err != nil {
s.dd_client.LogErrorCount("SubmitJob")
return nil, err
}
return resp, nil
}
This approach works but have several problems:
The LogRequestCount and LogRequestDuration is duplicated among all APIs
I am calling LogErrorCount in every places where errors are returned, which seems ugly
I learned that interceptor might help with logging, so I wrote an interceptor like
func (s *myServer) UnaryInterceptor(ctx context.Context,
request interface{},
info *grpc.UnaryServerInfo,
handler grpc.UnaryHandler,
) (interface{}, error) {
// Get method name e.g. SubmitJob, CancelJob, GetJobStatus
tmp := strings.Split(info.FullMethod, "/")
method := tmp[len(tmp)-1]
s.dd_client.LogRequestCount(method)
start_time := time.Now()
resp, err := handler(ctx, request)
server.dd_client.LogRequestDuration(method)
if err != nil {
s.dd_client.LogErrorCount(method)
}
return response, err
}
And set it in main() function:
server := grpc.NewServer(grpc.UnaryInterceptor(my_server.UnaryInterceptor))
This works for me, but I noticed two problems:
Here the interceptor takes myServer as a receiver, is this a good practice? I am doing this coz I want to reuse the Datadog client (dd_client) created within myServer. Other options would be create the Datadog client singleton which used by both interceptor and myServer, or create a interceptor struct and create a separate Datadog client there.
The interceptor could only handle logging for generic metrics e.g. request count, duration. But there could be metrics specific for each API, which means I still need to have logging related code in each API implementation. Then the question is, should I still use interceptor? Coz now the logging related code are splitted into two places (API implementation and interceptor).

Related

Manually extracting OpenTelemetry context from golang into a string?

I'm building a simple client server app which I want to trace across the client execution to a server microservice that calls a second server microservice.
Simply speaking, it's not more complicated than CLI -> ServiceA -> ServiceB.
The challenge I'm having is how to serialize the context - most of the docs I've looked at appear to do some form of automated HTTP header injection (e.g. https://opentelemetry.lightstep.com/core-concepts/context-propagation/) , but I do not have access to that. I need to serialize (I think) the context of the trace/span in the client and push it to the server, where I'll rehydrate it. (Mind you, I'd love this to be simpler, but I cannot figure it out).
So the object looks like this (called "job"):
args := &types.SubmitArgs{
SerializedOtelContext: serializedOtelContext,
}
job := &types.Job{}
tracer := otel.GetTracerProvider().Tracer("myservice.org")
_, span := tracer.Start(ctx, "Submitting Job to RPC")
err := system.JsonRpcMethod(rpcHost, rpcPort, "Submit", args, job)
The function to submit to JsonRpcMethod is here:
func JsonRpcMethod(
host string,
port int,
method string,
req, res interface{},
) error {
client, err := rpc.DialHTTP("tcp", fmt.Sprintf("%s:%d", host, port))
if err != nil {
return fmt.Errorf("Error in dialing. %s", err)
}
return client.Call(fmt.Sprintf("JobServer.%s", method), req, res)
}
And the function that receives it is here:
func (server *JobServer) Submit(args *types.SubmitArgs, reply *types.Job) error {
//nolint
job, err := server.RequesterNode.Scheduler.SubmitJob(args.Spec, args.Deal)
if err != nil {
return err
}
*reply = *job
return nil
}
My question is how do I, in the receiving function ("Submit" above) extract the trace/span from the sender?
Here is a small program to illustrate the usage. Hope this makes it clear.
package main
import (
"context"
"fmt"
"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
"go.opentelemetry.io/otel/propagation"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
func main() {
// common init
// You may also want to set them as globals
exp, _ := stdouttrace.New(stdouttrace.WithPrettyPrint())
bsp := sdktrace.NewSimpleSpanProcessor(exp) // You should use batch span processor in prod
tp := sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.AlwaysSample()),
sdktrace.WithSpanProcessor(bsp),
)
propgator := propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})
ctx, span := tp.Tracer("foo").Start(context.Background(), "parent-span-name")
defer span.End()
// Serialize the context into carrier
carrier := propagation.MapCarrier{}
propgator.Inject(ctx, carrier)
// This carrier is sent accros the process
fmt.Println(carrier)
// Extract the context and start new span as child
// In your receiving function
parentCtx := propgator.Extract(context.Background(), carrier)
_, childSpan := tp.Tracer("foo").Start(parentCtx, "child-span-name")
childSpan.AddEvent("some-dummy-event")
childSpan.End()
}

Gokit: Validate request/payload in transport layer

I am using go-kit to create an RPC endpoint. I am creating an endpoint like this
httptransport.NewServer(
endPoint.MakeGetBlogEndPoint(blogService),
transport.DecodeGetBlogRequest,
transport.EncodeGetBlogResponse
Below is my DecodeGetBlogRequest function
func DecodeGetBlogRequest(c context.Context, r *http.Request) (interface{}, error) {
vars := mux.Vars(r)
id, err := strconv.Atoi(vars["id"])
if err != nil {
return nil, err
}
req := endPoint.GetBlogRequest{
ID: id,
}
return req, nil
}
What I want to do is validate the HTTP request in this function and if found invalid, send a response with a valid error code from here only, without passing it to the service layer. i.e. If ID is not a valid number, return 400 Bad Request response from here.
But as I don't have a ResponseWriter reference in this function, I am not sure how to do it.
I am following this example from go-kit docs
https://gokit.io/examples/stringsvc.html
Is it a valid assumption that request/payload should be validated in the transport layer only and the service layer should only be called if the request/payload is valid? If yes, how to do so in this example?
You could use ServerErrorEncoder which returns Server options (can be found in github.com/go-kit/kit/transport/server.go).
Basically in your transport layer, apart from the Decode and Encode functions, you can define an YourErrorEncoderFunc() function which could look like the following. This will catch any error thrown in the transport layer.
YourErrorEncoderFunc(_ context.Context, err error, w http.ResponseWriter).
You will need to attach this function as an option in your endpoint registration like:
ABCOpts := []httptransport.ServerOption{
httptransport.ServerErrorEncoder(YourErrorEncoderFunc),
}
r.Methods("GET").Path("/api/v1/abc/def").Handler(httptransport.NewServer(
endpoints.GetDataEndpoint,
DecodeGetRequest,
EncodeGetResponse,
ABCOpts...,
))
This will stop at transport layer if your request validation is invalid and throw and error in the http response based of whatever format you've written in YourErrorEncoderFunc().
Not 100% sure if this applies to go-kit grpc as well:
You have an error return variable. Use that to indicate there was a problem. In the go grpc module there is a status package to return errors with status codes. If you return an error with a status code, the grpc layer will take the code from the error and send it back.
For example:
func DecodeGetBlogRequest(c context.Context, r *http.Request) (interface{}, error) {
vars := mux.Vars(r)
id, err := strconv.Atoi(vars["id"])
if err != nil {
return nil, status.Error(codes.InvalidArgument, err.Error())
}
req := endPoint.GetBlogRequest{
ID: id,
}
return req, nil
}
Note also that grpc uses different status codes. In Go they are located in the codes package.

How do you test a service that makes call to another API endpoint in Go?

I have a simple service created using infinite loop to call a certain HTTP API periodically, implemented in a package aservice. I created a Service struct there. Typically, to run that service, I expose a StartService method which is used to run that service synchronously. Users of the package can then run it using a goroutine. My question is, how do you write the tests for this kind of scenario?
Do you run the whole system and "mock" the API? I have heard that code that uses 3rd party services don't need to be tested, but the whole aservice package may only contain StartService and Shutdown methods. The rest of them are unexported functions/methods which are then cannot be tested individually. If that is the case then I can't write any tests at all?
With Go you will have awesome experience while mocking external http requests. Long story short just substitute base url with server url from net/http/httptest package.
You can mimic the way Google mocks their external requests for example exploring tests in google maps here.
server := mockServer(200, response)
defer server.Close()
c, _ := NewClient(WithAPIKey(apiKey), WithBaseURL(server.URL))
r := &DirectionsRequest{
Origin: "Google Sydney",
Destination: "Glebe Pt Rd, Glebe",
Mode: TravelModeTransit,
}
resp, _, err := c.Directions(context.Background(), r)
// your assertions goes here
// Create a mock HTTP Server that will return a response with HTTP code and body.
func mockServer(code int, body string) *httptest.Server {
server := mockServerForQuery("", code, body)
return server.s
}
func mockServerForQuery(query string, code int, body string) *countingServer {
server := &countingServer{}
server.s = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if query != "" && r.URL.RawQuery != query {
dmp := diffmatchpatch.New()
diffs := dmp.DiffMain(query, r.URL.RawQuery, false)
log.Printf("Query != Expected Query: %s", dmp.DiffPrettyText(diffs))
server.failed = append(server.failed, r.URL.RawQuery)
http.Error(w, "fail", 999)
return
}
server.successful++
w.WriteHeader(code)
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
fmt.Fprintln(w, body)
}))
return server
}

How to get http.ResponseWriter and http.Request in Revel controller

I am trying to implement an oauth server and the package I am using needs the complete http.ResponseWriter and http.Request types.
c.Response does not contain all the methods that http.ResponseWriter does and c.Request gives error incompatible type.
How do I get http.ResponseWriter and http.Request in a Revel controller?
type client struct {
ClientId string
ClientSecret string
}
type App struct {
*revel.Controller
}
func (c App) TokenRequest() {
r := c.Request
w := c.Response
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
log.Println(string(body))
var cli client
err = json.Unmarshal(body, &cli)
if err != nil {
panic(err)
}
log.Println(cli.ClientId)
err = OauthSrv.HandleTokenRequest(w, r)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
Warning
I am generally not fond of frameworks like Revel in Go, for reasons that I hope demonstrate themselves on this page. My first recommendation would be that you examine closely what you are actually getting out of Revel that merits the use of such a heavy abstraction layer, and if it's really that valuable, you may want to ask questions going in the other direction, such as how one might make OauthSrv work within Revel's customized ecosystem.
Using a Revel controller as a ResponseWriter
For something to be an http.ResponseWriter, it just needs to have these methods.
Header
You need a method named Header() that returns an http.Header, which you can build out of any map[string][]string. Revel provides similar functionality, but through several layers of abstraction. You will need to unravel them:
c.Response is a *Response, so it has a field named Out containing an OutResponse.
An OutResponse has a Header() method—but it doesn't return an http.Header. Instead, it returns a *RevelHeader.
A *RevelHeader has a GetAll(key string) []string method—which is very similar to the API already provided by the built-in map type, but isn't exactly the same. So, you will need to copy the returned values into a new map every time Header() is called, in order to fully satisfy the function signature requirements.
Also, GetAll() requires you to know the key name you are interested in, and *RevelHeader on its own does not provide a way to look up which keys are available. For now we can rely on the fact that the current implementation only has one field, a ServerHeader that does provide a GetKeys() []string method.
Putting all this together, we can build our Header method:
func (rrw RevelResponseWrapper) Header() http.Header {
revelHeader := rrw.Response.Out.Header()
keys := revelHeader.Server.GetKeys()
headerMap := make(map[string][]string)
for _, key := range keys {
headerMap[key] = revelHeader.GetAll(key)
}
return http.Header(headerMap)
}
Write and WriteHeader
You would use similar anti-patterns to expose rrw.Write([]byte) (int, error) so that it calls through to c.Response.Out.Write(data []byte) (int, error), and rrw.WriteHeader(int) error so that it calls c.Response.WriteHeader(int, string). Depending on what is considered appropriate for the framework, either panic on errors or fail silently, since their API doesn't expect WriteHeader errors to be handle-able.
Getting an http.Request from Revel
Unfortunately, the http.Request type is a struct, so you can't just simulate it. You basically have two options: reconstruct it using the net/http package from all the properties you are able to access, or hope that the *revel.Request you have is secretly an http.Request under the hood. In the latter case, you can use a type assertion:
revelReq, ok := c.Request.In.(*revel.GoRequest)
if !ok {
// handle this somehow
}
r := revelReq.Original

How to mock memcache in go lang for unit testing?

I want to mock memcache cache data in go lang to avoid authhorization
i tried with gomock but couldn't work out as i dont have any interface for it.
func getAccessTokenFromCache(accessToken string)
func TestSendData(t *testing.T) {
mockCtrl := gomock.NewController(t)
defer mockCtrl.Finish()
mockObj := mock_utils.NewMockCacheInterface(mockCtrl)
mockObj.EXPECT().GetAccessToken("abcd")
var jsonStr = []byte(`{
"devices": [
{"id": "avccc",
"data":"abcd/"
}
]
}`)
req, err := http.NewRequest("POST", "/send/v1/data",
bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "d958372f5039e28")
rr := httptest.NewRecorder()
handler := http.HandlerFunc(SendData)
handler.ServeHTTP(rr, req)
if status := rr.Code; status != 200 {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
expected := `{"error":"Invalid access token"}`
body, _ := ioutil.ReadAll(rr.Body)
if string(body) != expected {
t.Errorf("handler returned unexpected body: got %v want %v",
string(body), expected)
}
func SendData(w http.ResponseWriter, r *http.Request) {
accessToken := r.Header.Get(constants.AUTHORIZATION_HEADER_KEY)
t := utils.CacheType{At1: accessToken}
a := utils.CacheInterface(t)
isAccessTokenValid := utils.CacheInterface.GetAccessToken(a, accessToken)
if !isAccessTokenValid {
RespondError(w, http.StatusUnauthorized, "Invalid access token")
return
}
response := make(map[string]string, 1)
response["message"] = "success"
RespondJSON(w, http.StatusOK, response)
}
tried to mock using gomock
package mock_utils
gen mock for utils for get access controler
(1) Define an interface that you wish to mock.
(2) Use mockgen to generate a mock from the interface.
(3) Use the mock in a test:
You need to architect your code such that every such access to a service happens via an interface implementation. In your case, you should ideally create an interface like
type CacheInterface interface {
Set(key string, val interface{}) error
Get(key string) (interface{},error)
}
Your MemcacheStruct should implement this interface and all your memcache related calls should happen from there. Like in your case GetAccessToken should call cacheInterface.get(key) wherein your cacheInterface should refer to memcache implementation of this interface. This is a way better way to design your go programs and this would not only help you in writing tests but would also help in case let's say you want to use a different in memory database to help with caching. Like for ex., let's say in future if you want to use redis as your cache storage, then all you need to change is create a new implementation of this interface.

Resources