I am totally new to the distributed world and trying to use open telemetry for distributed tracing. For phase 1, just trying to have a request id (traceID / spanID) going, so we can co-relate the request logs. Have not yet decided on collector/ exporter.
My span/ trace id is always zero, in below middleware code
Is it because I have not initialized the tracer with all the exporter/ collector etc?
func AddSpanId(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
values := []interface{}{}
spanID, traceID := getRequestIDs(r)
values = append(values, "spanID", spanID)
values = append(values, "traceID", traceID)
//code to wrap values into context
}
func getRequestIDs(r *http.Request) (string, string) {
_, span := otel.Tracer("somename").Start(r.Context(), "handler")
fmt.Print(span.SpanContext().SpanID().String())
return span.SpanContext().SpanID().String(), span.SpanContext().TraceID().String()
}
You need to instrument your http handler:
import (
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
)
// Package-level tracer.
// This should be configured in your code setup instead of here.
var tracer = otel.Tracer("github.com/full/path/to/mypkg")
func main() {
// Wrap your httpHandler function.
handler := http.HandlerFunc(httpHandler)
wrappedHandler := otelhttp.NewHandler(handler, "hello-instrumented")
http.Handle("/hello-instrumented", wrappedHandler)
// And start the HTTP serve.
log.Fatal(http.ListenAndServe(":3030", nil))
}
https://opentelemetry.io/docs/instrumentation/go/libraries/
You also rarely need to work directly with traceID and spanID. Starting Span API is typical building block to instrument the code.
_, span := otel.Tracer(name).Start(ctx, "Poll")
defer span.End()
Related
I'm trying to find a way to add a correlation/request id for logs in our project to make it easier to navigate through them and debug when some issues occur. I found this article. From the example there, there is a middleware to add the correlationID and then retrieve it in some handler function.
Middleware function:
const ContextKeyRequestID ContextKey = "requestID"
func reqIDMiddleware1(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
id := uuid.New()
ctx = context.WithValue(ctx, ContextKeyRequestID, id.String())
r = r.WithContext(ctx)
log.Debugf("Incoming request %s %s %s %s", r.Method, r.RequestURI, r.RemoteAddr, id.String())
next.ServeHTTP(w, r)
log.Debugf("Finished handling http req. %s", id.String())
})
}
Handler:
const LogFieldKeyRequestID = "requestID"
func handleSomeRequest() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
reqIDRaw := ctx.Value(ContextKeyRequestID) // reqIDRaw at this point is of type 'interface{}'
reqID, ok := reqIDRaw.(string)
if !ok {
// handler error
}
// if reached here, reqID is ready to be used
// let's use it with logrus FieldLogger!
logger := log.WithField(LogFieldKeyRequestID, reqID)
// Do something, then log what you did
logger.Debugf("What I just did!")
// Do more, log more. Handle this request seriously
}
}
But I was wondering if there is a way to achieve this without having to refactor all the existing handlers and changing the logging functionality, through some automatic configuration that would add id for each log, in my case our project is quite big, and doing it in the way described above would require a lot of changes.
Have you looked at WithContext and Hooks?
You still have to modify your code but you can centralize some behaviours.
https://go.dev/play/p/4YxJMK6Zl5D
I want to trace complete execution of an HTTP request in golang. For any non trivial operation the request will eventually call many different functions. I would like to have logs from the entire execution stack tagged with a unique request id. e.g.
http.Handle("/my-request", myrequesthandler)
func myrequestHandler(w http.ResponseWriter, r *http.Request) {
//debug print1
log.Printf("....")
myfunc()
}
func myfunc() {
//debug print2
log.Printf("....")
}
Here I need a way to identify print1 and print2 as part of same request. It looks like zerolog does have something like this, as described here. Like so:
....
c = c.Append(hlog.RequestIDHandler("req_id", "Request-Id"))
h := c.Then(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hlog.FromRequest(r).Info().
Msg("Something happened")
}))
http.Handle("/", h)
But if I understand it right, it will involve passing request object to each and every function. Is this the idiomatic way to solve this problem? What are the alternatives?
Set a unique id on a context.Context as soon as the request is received, and pass that context down the call stack. This is what contexts are for.
[Context] carries deadlines, cancellation signals, and other request-scoped values across API boundaries and between processes.
Example:
// You could place this in a separate helper package to improve encapsulation
type ctxKey struct{}
func myRequestHandler(w http.ResponseWriter, r *http.Request) {
uniqueID := // generate a unique identifier
// create a context with the http.Request context as parent
ctx := context.WithValue(r.Context(), ctxKey{}, uniqueID)
foo(ctx, ...)
bar(ctx, ...)
}
func foo(ctx context.Context, ...) {
uniqueID := ctx.Value(ctxKey{})
// do something with the unique id
baz(ctx, ...)
}
In particular:
Create the context with *http.Request.Context() as parent. This way if the request is canceled, e.g. due to client disconnection, cancellation will propagate to your sub-context
Consider setting the context value using an unexported struct as key. Unexported structs defined in your package will never conflict with other keys. If you use strings as keys instead, any package could in theory use the same key and overwrite your value (or you could overwrite others' values). YMMV.
Pass your context as the first argument of any function in your call stack, as the package documentation recommends.
For tracing and logging across applications, you might want to look into opentracing. Propagation of tracers is still done with Contexts as outlined above.
you can use context.Context and set request id on it via middleware function.
example:
type requestIDKey struct{}
func requestIDSetter(next http.HandlerFunc) http.HandlerFunc {
return func(rw http.ResponseWriter, r *http.Request) {
// use provided request id from incoming request if any
reqID := r.Header.Get("X-Request-ID")
if reqID == "" {
// or use some generated string
reqID = uuid.New().String()
}
ctx := context.WithValue(r.Context(), requestIDKey{}, reqID)
next(rw, r.WithContext(ctx))
}
}
then you need to modify your logger to accept context.Context
example:
func printfWithContext(ctx context.Context, format string, v ...interface{}) {
reqID := ctx.Value(requestIDKey{}).(string)
log.Printf(reqID+": "+format, v...)
}
and finally apply to your code
http.HandleFunc("/my-request", requestIDSetter(myrequestHandler))
func myrequestHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
//debug print1
printfWithContext(ctx, "....1")
myfunc(ctx)
}
func myfunc(ctx context.Context) {
//debug print2
printfWithContext(ctx, "....2")
}
I am building an API using gRPC and in server side, I want to receive a notification when a client disconnects, identify it and perform some tasks based on that.
So far, I was able to detect client disconnection using grpc.StatsHandler method HandleConn. I tried passing values using context, but they can't be accessed from server side.
Client side:
conn, err := grpc.DialContext(
context.WithValue(context.Background(), "user_id", 1234),
address,
grpc.WithInsecure(),
)
Server side:
// Build stats handler
type serverStats struct {}
func (h *serverStats) TagRPC(ctx context.Context, info *stats.RPCTagInfo) context.Context {
return ctx
}
func (h *serverStats) HandleRPC(ctx context.Context, s stats.RPCStats) {}
func (h *serverStats) TagConn(ctx context.Context, info *stats.ConnTagInfo) context.Context {
return context.TODO()
}
func (h *serverStats) HandleConn(ctx context.Context, s stats.ConnStats) {
fmt.Println(ctx.Value("user_id")) // Returns nil, can't access the value
switch s.(type) {
case *stats.ConnEnd:
fmt.Println("client disconnected")
break
}
}
// Build server
s := grpc.NewServer(grpc.StatsHandler(&serverStats{}))
I want to access the value passed from client side in server side. What is the right way to do it, or is there any other way to identify the client that has disconnected?
I don't think you may pass that value when you are dialling, but you may tag the connection and than all next client requests will have the same value in its context:
In your serverStats implementation:
func (h *serverStats) TagConn(ctx context.Context, info *stats.ConnTagInfo) context.Context {
return context.WithValue(ctx, "user_counter", userCounter++)
}
In your service implementation
func (s *service) SetUserId(ctx context.Context, v MyMessage) (r MyResponse, err Error) {
s.userIdMap[ctx.Value("user_counter")] = v.userId
...
}
In the other methods:
func (s *service) AnotherMethod(ctx context.Context, v MyMessage) (r MyResponse, err Error) {
userId := s.userIdMap[ctx.Value("user_counter")]
...
}
func (h *serverStats) HandleConn(ctx context.Context, s stats.ConnStats) {
switch s.(type) {
case *stats.ConnEnd:
fmt.Printf("client %d disconnected", s.userIdMap[ctx.Value("user_counter")])
break
}
}
Please let me know if you find a way to pass values in the dialling step.
I tried passing values using context, but they can't be accessed from server side.
You need to set metadata fields to the client context explicitly:
ctx := context.Background()
ctx = metadata.AppendToOutgoingContext(ctx, "user_id", "1234")
conn, err := grpc.DialContext(cxt, address)
On server side you can retrieve them like this:
md, ok := metadata.FromIncomingContext(ctx)
What is the right way to do it, or is there any other way to identify the client that has disconnected?
This really depends on your use case. I think that GRPC Stats API is good for some simple tasks (for example, to compute latency or aggregate network stats), but it's not very useful when some business logic should work when client leaves. I would suggest using defer calls in GRPC handlers directly for that. One more option is to implement custom GRPC interceptor.
My basic main setup:
muxRouter := mux.NewRouter()
v1Router.Router(muxRouter.PathPrefix("/v1").Subrouter())
http.Handle("/", muxRouter)
n := negroni.Classic()
n.Use(negroni.HandlerFunc(apiRouter.Middleware))
n.UseHandler(muxRouter)
s := &http.Server{
Addr: ":6060",
Handler: n,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
MaxHeaderBytes: 1 << 20,
}
log.Fatal(s.ListenAndServe())
Inside the apiRouter.Middleware I have set the following context:
context.Set(req, helperKeys.DomainName, "some-value")
However, in some handlerFunc within v1Router.Router when trying to Get the context's value, the result is nil:
domain := context.Get(req, helperKeys.DomainName)
fmt.Println("DomainName", domain)
Prints: DomainName <nil>
I know that the Set method is correct as getting the value immediately after setting it in the apiRouter.Middleware will return the correct string value.
I ended up using Go 1.7's built in Context:
context.Set(req, helperKeys.DomainName, "some-value")
// Replaced with:
ctx := req.Context()
ctx = context.WithValue(ctx, helperKeys.DomainName, "some-value")
req = req.WithContext(ctx)
AND
domain := context.Get(req, helperKeys.DomainName)
// Replaced with:
domain := req.Context().Value(helperKeys.DomainName).(string)
Based on your answer, it looks like you are trying to store a database in the context. I wouldn't suggest doing that. Instead try something like this:
type Something struct {
DB *sql.DB // or some other DB object
}
func (s *Something) CreateUser(w http.ResponseWriter, r *http.Request) {
// use s.DB to access the database
fmt.Fprintln(w, "Created a user...")
}
func main() {
db := ...
s := Something{db}
http.HandleFunc("/", s.CreateUser)
// ... everything else is pretty much like normal.
}
This gives your handlers access to the database while not having to set it on the context every single time. Context values should be reserved for things that you can't possibly set until runtime. For example, a request ID that is specific to that web request. Things that outlive the request don't typically fall into this category, and your DB connection will outlive the request.
If you do actually need context values, you should:
Use getters and setters that are typed
"packages should define keys as an unexported type to avoid collisions." - From the Go source code
An example of this is shown below, and I talk more about context values in general in this blog post:
type userCtxKeyType string
const userCtxKey userCtxKeyType = "user"
func WithUser(ctx context.Context, user *User) context.Context {
return context.WithValue(ctx, userCtxKey, user)
}
func GetUser(ctx context.Context) *User {
user, ok := ctx.Value(userCtxKey).(*User)
if !ok {
// Log this issue
return nil
}
return user
}
I'm trying to create a map that stores http.ResponseWriters so that I can write to them later, after a separate thread has done the relevant computation.
The map is defined in my main as follow:
jobs := make(map[uint32]http.ResponseWriter)
I then pass this map to a handle function like so:
r.HandleFunc("/api/{type}/{arg1}", func(w http.ResponseWriter, r *http.Request) {
typ, _ := strconv.Atoi(mux.Vars(r)["type"])
AddReqQueue(w, ReqQueue, typ, mux.Vars(r)["arg1"], jobs, ids)
}).Methods("get")
After that I process the reuqeuest and add it to a channel:
func AddReqQueue(w http.ResponseWriter, ReqQueue chan mssg.WorkReq, typ int, arg1 string, jobs map[uint32]http.ResponseWriter, ids []uint32) {
var id uint32
id, ids = ids[0], ids[1:] // get a free work id
jobs[id] = w
fmt.Println("Adding req to queue")
ReqQueue <- mssg.WorkReq{Type: uint8(typ), Arg1: arg1, WId: id}
}
Inside this function I have tested and am able to write data to the ReponseWriter, however later when I try to use the map in:
func SendResp(RespQueue chan mssg.WorkResp, jobs map[uint32]http.ResponseWriter) {
for {
resp := <-RespQueue
jobs[resp.WId].Header().Set("Content-Type", "text/plain")
_, err := jobs[resp.WId].Write(resp.Data) // ERROR IS COMING FROM HERE
if err != nil {
fmt.Fprintf(os.Stderr, "Fatal error: %s", err.Error())
}
}
}
It doesn't work. No matter what I pre-set the header too or try to write (even just a simple string I hardcoded) I get a err of
Conn.Write wrote more than the declared Content-Length
I know I'm accessing the right struct in the map, it just seems as if the ReponseWriter has gone out of context or been corrupted, also i know that the headers shouldn't really matter since it is the first time I am calling Write() and therefore it should create the headers for me.
#elithrar was correct. I wasn't aware that the http.ResponseWriter object became invalid after the handler exited. If I just force my handler to wait, it works fine.