How do I distribute tracing between 2 Go microservices using Opentelemetry & Otelgin? - go

I am trying to distribute tracing between 2 Go microservices using Opentelemetry and Gin-Gonic.
Please help, I have come across Otelhttp examples, but I couldn't find examples with Otelgin.
Using "go.opentelemetry.io/otel/sdk/trace" as tracesdk
Microservice 1
func TracerProvider() (*tracesdk.TracerProvider, error) {
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
exporter, _:= stdouttrace.New(stdouttrace.WithPrettyPrint())
tp := tracesdk.NewTracerProvider(
tracesdk.WithSampler(tracesdk.AlwaysSample()),
tracesdk.WithBatcher(exporter),
tracesdk.WithResource(resource.NewWithAttributes(
semconv.ServiceNameKey.String("microservice-1"),
attribute.String("environment", "test"),
)),
)
otel.SetTracerProvider(tp)
return tp, nil
}
func main() {
TracerProvider()
resty := resty.New()
router := gin.Default()
router.Use(otelgin.Middleware("microservice-1"))
{
router.GET("/ping", func(c *gin.Context) {
result := Result{}
req := resty.R().SetHeader("Content-Type", "application/json")
ctx := req.Context()
span := trace.SpanFromContext(ctx)
defer span.End()
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))
resp, _ := req.Get("http://localhost:8088/pong")
json.Unmarshal([]byte(resp.String()), &result)
c.IndentedJSON(200, gin.H{
"message": result.Message,
})
})
}
router.Run(":8085")
}
// Microservice 2
//TracerProvider func is the same as Microservice 1
//main
TracerProvider()
router := gin.Default()
router.Use(otelgin.Middleware("microservice-2"))
{
router.GET("/pong", func(c *gin.Context) {
ctx := c.Request.Context()
span := trace.SpanFromContext(otel.GetTextMapPropagator().Extract(ctx, propagation.HeaderCarrier(c.Request.Header)))
defer span.End()
c.IndentedJSON(200, gin.H{
"message": "pong",
})
})
}
router.Run(":8088")

There is an example of go-gin instrumentation in the official repository of OpenTelemetry Go Contrib:
https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/github.com/gin-gonic/gin/otelgin/example
Basically, you need use the package:
"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
And you can also add the otelgin.HTML to render your response.
HTML will trace the rendering of the template as a child of the span in the given context.
This is a replacement for gin.Context.HTML function - it invokes the original function after setting up the span.
As detailed in its implementation: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/instrumentation/github.com/gin-gonic/gin/otelgin/gintrace.go

It might be a bit late but it will still be useful.
In the 1st microservice, you need to put:
req := resty.R().SetHeader("Content-Type", "application/json")
req.SetContext(c.Request.Context())
That will fix the problem.
On this screenshot you are able to see the 2 spans on the 1st microservice

Related

opentelemetry golang always give all 0 for traceID and spanID

I am totally new to the distributed world and trying to use open telemetry for distributed tracing. For phase 1, just trying to have a request id (traceID / spanID) going, so we can co-relate the request logs. Have not yet decided on collector/ exporter.
My span/ trace id is always zero, in below middleware code
Is it because I have not initialized the tracer with all the exporter/ collector etc?
func AddSpanId(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
values := []interface{}{}
spanID, traceID := getRequestIDs(r)
values = append(values, "spanID", spanID)
values = append(values, "traceID", traceID)
//code to wrap values into context
}
func getRequestIDs(r *http.Request) (string, string) {
_, span := otel.Tracer("somename").Start(r.Context(), "handler")
fmt.Print(span.SpanContext().SpanID().String())
return span.SpanContext().SpanID().String(), span.SpanContext().TraceID().String()
}
You need to instrument your http handler:
import (
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
)
// Package-level tracer.
// This should be configured in your code setup instead of here.
var tracer = otel.Tracer("github.com/full/path/to/mypkg")
func main() {
// Wrap your httpHandler function.
handler := http.HandlerFunc(httpHandler)
wrappedHandler := otelhttp.NewHandler(handler, "hello-instrumented")
http.Handle("/hello-instrumented", wrappedHandler)
// And start the HTTP serve.
log.Fatal(http.ListenAndServe(":3030", nil))
}
https://opentelemetry.io/docs/instrumentation/go/libraries/
You also rarely need to work directly with traceID and spanID. Starting Span API is typical building block to instrument the code.
_, span := otel.Tracer(name).Start(ctx, "Poll")
defer span.End()

Manually extracting OpenTelemetry context from golang into a string?

I'm building a simple client server app which I want to trace across the client execution to a server microservice that calls a second server microservice.
Simply speaking, it's not more complicated than CLI -> ServiceA -> ServiceB.
The challenge I'm having is how to serialize the context - most of the docs I've looked at appear to do some form of automated HTTP header injection (e.g. https://opentelemetry.lightstep.com/core-concepts/context-propagation/) , but I do not have access to that. I need to serialize (I think) the context of the trace/span in the client and push it to the server, where I'll rehydrate it. (Mind you, I'd love this to be simpler, but I cannot figure it out).
So the object looks like this (called "job"):
args := &types.SubmitArgs{
SerializedOtelContext: serializedOtelContext,
}
job := &types.Job{}
tracer := otel.GetTracerProvider().Tracer("myservice.org")
_, span := tracer.Start(ctx, "Submitting Job to RPC")
err := system.JsonRpcMethod(rpcHost, rpcPort, "Submit", args, job)
The function to submit to JsonRpcMethod is here:
func JsonRpcMethod(
host string,
port int,
method string,
req, res interface{},
) error {
client, err := rpc.DialHTTP("tcp", fmt.Sprintf("%s:%d", host, port))
if err != nil {
return fmt.Errorf("Error in dialing. %s", err)
}
return client.Call(fmt.Sprintf("JobServer.%s", method), req, res)
}
And the function that receives it is here:
func (server *JobServer) Submit(args *types.SubmitArgs, reply *types.Job) error {
//nolint
job, err := server.RequesterNode.Scheduler.SubmitJob(args.Spec, args.Deal)
if err != nil {
return err
}
*reply = *job
return nil
}
My question is how do I, in the receiving function ("Submit" above) extract the trace/span from the sender?
Here is a small program to illustrate the usage. Hope this makes it clear.
package main
import (
"context"
"fmt"
"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
"go.opentelemetry.io/otel/propagation"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
func main() {
// common init
// You may also want to set them as globals
exp, _ := stdouttrace.New(stdouttrace.WithPrettyPrint())
bsp := sdktrace.NewSimpleSpanProcessor(exp) // You should use batch span processor in prod
tp := sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.AlwaysSample()),
sdktrace.WithSpanProcessor(bsp),
)
propgator := propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})
ctx, span := tp.Tracer("foo").Start(context.Background(), "parent-span-name")
defer span.End()
// Serialize the context into carrier
carrier := propagation.MapCarrier{}
propgator.Inject(ctx, carrier)
// This carrier is sent accros the process
fmt.Println(carrier)
// Extract the context and start new span as child
// In your receiving function
parentCtx := propgator.Extract(context.Background(), carrier)
_, childSpan := tp.Tracer("foo").Start(parentCtx, "child-span-name")
childSpan.AddEvent("some-dummy-event")
childSpan.End()
}

HttpMock is not intercepting Resty call

I have a function that calls an external api that I want to mock out in the test.
func ApiWrapper(...) (...) {
client := resty.New()
var r apiResponse
apiPath := "..." // In the test will be http://localhost:{PORT}/path/to/endpoint
_, e := client.R().SetResult(&r).Get(apiPath)
...
return ...
}
The test looks like this:
func TestApiWrapper(t *testing.T) {
client := resty.New()
httpmock.ActivateNonDefault(client.GetClient())
defer httpmock.DeactivateAndReset()
mock_resp = `...`
responder := httpmock.NewStringResponder(200, mock_resp)
api_url := "same string used in the function"
httpmock.RegisterResponder("GET", api_url, responder)
res, e := ApiWrapper(...)
...
}
The issue I'm having is that the mock is not being used also the external api will not be available in our CI.
In the test the client has:
httpClient: *net/http.Client {
Transport: net/http.RoundTripper(*github.com/jarcoal/httpmock.MockTransport)
In the function the client has:
httpClient: *net/http.Client {
Transport: net/http.RoundTripper(*net/http.Transport)
I was able to get around the problem by using a function to inject the resty Client. Don't really like this approach as it leaves a couple of lines of code that are not executed during my test.
func ApiWrapper(...) {
client := resty.New()
resp, err := ApiWrapperWrapper(client)
return resp, err
}
func ApiWrapperWrapper(client *resty.Client) (...) {
copy all the code into here
}
Then in my test I just call ApiWrapperWrapper and pass in the mocked client.
func TestApiWrapper(...) {
...
// change last line in example to this
res, e := ApiWrapperWrapper(client)
}

How to mock memcache in go lang for unit testing?

I want to mock memcache cache data in go lang to avoid authhorization
i tried with gomock but couldn't work out as i dont have any interface for it.
func getAccessTokenFromCache(accessToken string)
func TestSendData(t *testing.T) {
mockCtrl := gomock.NewController(t)
defer mockCtrl.Finish()
mockObj := mock_utils.NewMockCacheInterface(mockCtrl)
mockObj.EXPECT().GetAccessToken("abcd")
var jsonStr = []byte(`{
"devices": [
{"id": "avccc",
"data":"abcd/"
}
]
}`)
req, err := http.NewRequest("POST", "/send/v1/data",
bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "d958372f5039e28")
rr := httptest.NewRecorder()
handler := http.HandlerFunc(SendData)
handler.ServeHTTP(rr, req)
if status := rr.Code; status != 200 {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
expected := `{"error":"Invalid access token"}`
body, _ := ioutil.ReadAll(rr.Body)
if string(body) != expected {
t.Errorf("handler returned unexpected body: got %v want %v",
string(body), expected)
}
func SendData(w http.ResponseWriter, r *http.Request) {
accessToken := r.Header.Get(constants.AUTHORIZATION_HEADER_KEY)
t := utils.CacheType{At1: accessToken}
a := utils.CacheInterface(t)
isAccessTokenValid := utils.CacheInterface.GetAccessToken(a, accessToken)
if !isAccessTokenValid {
RespondError(w, http.StatusUnauthorized, "Invalid access token")
return
}
response := make(map[string]string, 1)
response["message"] = "success"
RespondJSON(w, http.StatusOK, response)
}
tried to mock using gomock
package mock_utils
gen mock for utils for get access controler
(1) Define an interface that you wish to mock.
(2) Use mockgen to generate a mock from the interface.
(3) Use the mock in a test:
You need to architect your code such that every such access to a service happens via an interface implementation. In your case, you should ideally create an interface like
type CacheInterface interface {
Set(key string, val interface{}) error
Get(key string) (interface{},error)
}
Your MemcacheStruct should implement this interface and all your memcache related calls should happen from there. Like in your case GetAccessToken should call cacheInterface.get(key) wherein your cacheInterface should refer to memcache implementation of this interface. This is a way better way to design your go programs and this would not only help you in writing tests but would also help in case let's say you want to use a different in memory database to help with caching. Like for ex., let's say in future if you want to use redis as your cache storage, then all you need to change is create a new implementation of this interface.

Go lang Redis PubSub in different go routes for publish and subscribe

Am new to go programming language, and I have a requirement to create Redis PubSub with websocket.
My reference code is here
https://github.com/vortec/orchestrate
Am using following libraries
"golang.org/x/net/websocket"
"github.com/garyburd/redigo/redis"
Everything is working for me like this way, but I don't understand what is "websocket.Handler(handleWSConnection)" here.
I need 2 different go routes for /ws-subscribe and /ws-publish. I don't know anything wrong in this concept?
Doubts
Can I do this way, http.HandleFunc("/ws", handleWSConnection) // Tried this way but am getting "not enough arguments in call to handleWSConnection"
Is there any way to call "handleWSConnection()" as a normal function.
Any suggestions how to write /ws-publish as a different go route
Following is my code
main function
func (wsc *WSConnection) ReadWebSocket() {
for {
var json_data []byte
var message WSMessage
// Receive data from WebSocket
err := websocket.Message.Receive(wsc.socket, &json_data)
if err != nil {
return
}
// Parse JSON data
err = json.Unmarshal(json_data, &message)
if err != nil {
return
}
switch message.Action {
case "SUBSCRIBE":
wsc.subscribe.Subscribe(message.Channel)
case "UNSUBSCRIBE":
wsc.subscribe.Unsubscribe(message.Channel)
case "PUBLISH":
wsc.publish.Conn.Do("PUBLISH", message.Channel, message.Data)
}
}
}
func handleWSConnection(socket *websocket.Conn) {
wsc := &WSConnection {socket: socket}
defer wsc.Uninitialize()
wsc.Initialize()
go wsc.ProxyRedisSubscribe()
wsc.ReadWebSocket()
}
func serveWeb() {
http.Handle("/ws", websocket.Handler(handleWSConnection)) // I need to call this route as funciton
if err := http.ListenAndServe(":9000", nil); err != nil {
log.Fatal("ListenAndServe:", err)
}
}
Done following way, I dont know is it the proper way to do this
http.HandleFunc("/publish", publishHandler)
func publishHandler(conn http.ResponseWriter, req *http.Request) {
log.Println("PUBLISH HANDLER")
wsHandler := websocket.Handler(func(ws *websocket.Conn) {
handleWSConnection(ws)
})
wsHandler.ServeHTTP(conn, req)
}

Resources