I would like to ask how we should approach the issue with context propagation in Golang.
My application is an HTTP JSON API server.
I would use the context as a container of informative data (e.g. the request id, some things I unpack from requests or from the process).
One of the dumbest advantages is to vehicle data and tags useful for statistics and logging. E.g. Be able to add at each log line the transaction id in all the packages I own
The problem I'm facing is the following:
func handleActivityY(w http.ResponseWriter, r *http.Request) {
info, err := decodeRequest(r)
...
stepOne, err := stepOne(r.Context(), info)
...
stepTwo, err := stepTwo(r.Context(), stepOne)
...
}
The problem with this design is the fact the context is an immutable entity (each time we add something or we set a new timeout, we have a new context).
The context cannot be propagated except returning the context at each function call (together with the return value, if any and the error).
The only way to make this work would be:
func handleActivityY(w http.ResponseWriter, r *http.Request) {
ctx, info, err := decodeRequest(r)
...
ctx, stepOne, err := stepOne(ctx, info)
...
ctx, stepTwo, err := stepTwo(ctx, stepOne)
...
}
I've already polluted almost any function in my packages with the context.Context parameter. Returning it in addition to other parameters seems to me overkill.
Is there really no other more elegant way to do so?
I am currently using the gin framework, which has its own context and it is mutable. I don't want to add the dependency to Gin for that.
Early in your context pipeline, add a mutable pointer to a data struct:
type MyData struct {
// whatever you need
}
var MyDataKey = /* something */
ctx, cancel := context.WithValue(context.Background(), MyDataKey, &MyData{})
Then in your methods that need to modify your data structure, just do so:
data := ctx.Value(MyDataKey)
data.Foo = /* something */
All normal rules about concurrent access safety apply, so you may need to use mutexes or other protection mechanisms if multiple goroutines can read/set your data value simultaneously.
Is there really no other more elegant way to do so?
stepOne could return it's own data independent of context and isolated from how the caller may use its information (ie put it in a databag/context and pass it to other functions)
func handleActivityY(w http.ResponseWriter, r *http.Request) {
ctx, info, err := decodeRequest(r)
...
stepOne, err := stepOne(ctx, info)
...
ctx = context.WithValue(ctx, "someContextStepTwoNeeds", stepOne.Something())
stepTwo, err := stepTwo(ctx, stepOne)
...
}
IMO data being passed request scoped should be extremely minimal contextual information
Related
I'm new to golang and somewhat confused about context and how to use the context in golang applications.
Specifically im working on the cli application and just need to access mongo, for example.
Like - is this correct that I just create single shared ctx context variable, then use it for any operations that need context?
Would any operation that needs context restart the 5-second timer? or is this a shared timer?
package main
import (
"context"
"log"
"os"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func main() {
log.SetOutput(os.Stdout)
// Create a context with a timeout of 5 seconds
//This defines a timeout context that will be canceled after 5 seconds.
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
// always defer in case func returns early
defer cancel()
//Creates a new ClientOptions instance.
clientOptions := options.Client()
clientOptions = clientOptions.ApplyURI("mongodb+srv://127.0.0.1?retryWrites=true&w=majority")
//Connect to mongo
client, err := mongo.Connect(ctx, clientOptions)
defer client.Disconnect(ctx)
if err != nil {
log.Fatal(err)
}
//Test connection to the database
log.Println("I: test mongo connection using ping")
err = client.Ping(ctx, readpref.Primary())
if err != nil {
log.Fatal(err)
}
log.Println("I: Fin")
}
If you think about it, it makes no sense that a context.Context could be shared "horizontally" (meaning between operations not part of the same call stack). A golang Context provides the context within which an operation (including any nested operations below it in the call stack) are to be performed - such as "within X seconds," to protect against hanging due to communications delays, etc. So if you issue 10 requests in parallel, you should give each one its own context - you probably don't want the tenth one to fail because the first one did. If you are just using a context.Background() or context.TODO(), without further decoration, you probably don't need to store the Context in a variable the first time you create it - you can just create when you pass it to the first function in the call stack, and properly constructed code will pass it down the stack as appropriate, applying necessary decorations along the way:
func Execute() {
DoThing(context.Background())
// Other stuff
}
func DoThing(pctx context.Context) {
ctx, cancel := context.WithTimeout(pctx, 10 * time.Second) // Timeout after 10 seconds
defer cancel()
DoThingThatMayTakeAWhile(ctx)
select {
// You may want other stuff here
case <-ctx.Done():
// Context timed out, maybe log an error?
}
}
func DoThingThatMayTakeAWhile(pctx context.Context) {
DoThingNeedingInfoInContext(context.WithValue(pctx, "thisisakey", "value"))
}
func DoThingNeedingInfoInContext(ctx context.Context) {
val := ctx.Value("thisisakey")
// Do something with val, check that it isn't nil, etc.
}
If I were to make multiple calls to DoThingThatMayTakeAWhile(), I'd want to give each one a separate child context - I would not want to share ctx with each of them.
So in your code, every call to mongo.Connect() should receive a freshly created context.Context instance.
You create a new context variable for each operations that need context.
The timer of a context will never restart.
In your example, try to add time.Sleep(6*time.Second) after context.WithTimeout, you will see all operations return error context deadline exceeded.
I want to trace complete execution of an HTTP request in golang. For any non trivial operation the request will eventually call many different functions. I would like to have logs from the entire execution stack tagged with a unique request id. e.g.
http.Handle("/my-request", myrequesthandler)
func myrequestHandler(w http.ResponseWriter, r *http.Request) {
//debug print1
log.Printf("....")
myfunc()
}
func myfunc() {
//debug print2
log.Printf("....")
}
Here I need a way to identify print1 and print2 as part of same request. It looks like zerolog does have something like this, as described here. Like so:
....
c = c.Append(hlog.RequestIDHandler("req_id", "Request-Id"))
h := c.Then(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hlog.FromRequest(r).Info().
Msg("Something happened")
}))
http.Handle("/", h)
But if I understand it right, it will involve passing request object to each and every function. Is this the idiomatic way to solve this problem? What are the alternatives?
Set a unique id on a context.Context as soon as the request is received, and pass that context down the call stack. This is what contexts are for.
[Context] carries deadlines, cancellation signals, and other request-scoped values across API boundaries and between processes.
Example:
// You could place this in a separate helper package to improve encapsulation
type ctxKey struct{}
func myRequestHandler(w http.ResponseWriter, r *http.Request) {
uniqueID := // generate a unique identifier
// create a context with the http.Request context as parent
ctx := context.WithValue(r.Context(), ctxKey{}, uniqueID)
foo(ctx, ...)
bar(ctx, ...)
}
func foo(ctx context.Context, ...) {
uniqueID := ctx.Value(ctxKey{})
// do something with the unique id
baz(ctx, ...)
}
In particular:
Create the context with *http.Request.Context() as parent. This way if the request is canceled, e.g. due to client disconnection, cancellation will propagate to your sub-context
Consider setting the context value using an unexported struct as key. Unexported structs defined in your package will never conflict with other keys. If you use strings as keys instead, any package could in theory use the same key and overwrite your value (or you could overwrite others' values). YMMV.
Pass your context as the first argument of any function in your call stack, as the package documentation recommends.
For tracing and logging across applications, you might want to look into opentracing. Propagation of tracers is still done with Contexts as outlined above.
you can use context.Context and set request id on it via middleware function.
example:
type requestIDKey struct{}
func requestIDSetter(next http.HandlerFunc) http.HandlerFunc {
return func(rw http.ResponseWriter, r *http.Request) {
// use provided request id from incoming request if any
reqID := r.Header.Get("X-Request-ID")
if reqID == "" {
// or use some generated string
reqID = uuid.New().String()
}
ctx := context.WithValue(r.Context(), requestIDKey{}, reqID)
next(rw, r.WithContext(ctx))
}
}
then you need to modify your logger to accept context.Context
example:
func printfWithContext(ctx context.Context, format string, v ...interface{}) {
reqID := ctx.Value(requestIDKey{}).(string)
log.Printf(reqID+": "+format, v...)
}
and finally apply to your code
http.HandleFunc("/my-request", requestIDSetter(myrequestHandler))
func myrequestHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
//debug print1
printfWithContext(ctx, "....1")
myfunc(ctx)
}
func myfunc(ctx context.Context) {
//debug print2
printfWithContext(ctx, "....2")
}
I'm setting up a tcp server in a pet project I'm writing in go. I want to be able to maintain a slice of all connected clients, and then modify it whenever a new client connects or disconnects from my server.
My main mental obstacle right now is whether I should be declaring a package level slice, or just passing a slice into my handler.
My first thought was to declare my ClientList slice (I'm aware that a slice might not be my best option here, but I've decided to leave it as is for now) as a package level variable. While I think this would work, I've seen a number of posts discouraging the use of them.
My other thought was to declare ClientList as a slice in my main function, and then I pass ClientList to my HandleClient function, so whenever a client connects/disconnects I can call AddClient or RemoveClient and pass this slice in and add/remove the appropriate client.
This implementation is seen below. There are definitely other issues with the code, but I'm stuck trying to wrap my head around something that seems like it should be very simple.
type Client struct {
Name string
Conn net.Conn
}
type ClientList []*Client
// Identify is used to set the name of the client
func (cl *Client) Identify() error {
// code here to set the client's name in the based on input from client
}
// This is not a threadsafe way to do this - need to use mutex/channels
func (cList *ClientList) AddClient(cl *Client) {
*cList = append(*cList, cl)
}
func (cl *Client) HandleClient(cList *ClientList) {
defer cl.Conn.Close()
cList.AddClient(cl)
err := cl.Identify()
if err != nil {
log.Println(err)
return
}
for {
err := cl.Conn.SetDeadline(time.Now().Add(20 * time.Second))
if err != nil {
log.Println(err)
return
}
cl.Conn.Write([]byte("What command would you like to perform?\n"))
netData, err := bufio.NewReader(cl.Conn).ReadString('\n')
if err != nil {
log.Println(err)
return
}
cmd := strings.TrimSpace(string(netData))
if cmd == "Ping" {
cl.Ping() //sends a pong msg back to client
} else {
cl.Conn.Write([]byte("Unsupported command at this time\n"))
}
}
}
func main() {
arguments := os.Args
PORT := ":" + arguments[1]
l, err := net.Listen("tcp4", PORT)
if err != nil {
fmt.Println(err)
return
}
defer l.Close()
fmt.Println("Listening...")
// Create a new slice to store pointers to clients
var cList ClientList
for {
c, err := l.Accept()
if err != nil {
log.Println(err)
return
}
// Create client cl1
cl1 := Client{Conn: c}
// Go and handle the client
go cl1.HandleClient(&cList)
}
}
From my initial testing, this appears to work. I am able to print out my client list and I can see that new clients are being added, and their name is being added after Identify() is called as well.
When I run it with the -race flag, I do get data race warnings, so I know I will need a threadsafe way to handle adding clients. The same goes for removing clients when I add that in.
Are there any other issues I might be missing by passing my ClientList into HandleClient, or any benefits I would gain from declaring ClientList as a package level variable instead?
Several problems with this approach.
First, your code contains a data race: each TCP connection is served by a separate goroutine, and they all attempt to modify the slice concurrently.
You might try building your code with go build -race (or go install -race — whatever you're using), and see it crash by the enabled runtime checks.
This one is easy to fix. The most straightforward approach is to add a mutex variable into the ClientList type:
type ClientList struct {
mu sync.Mutex
clients []*Client
}
…and make the type's methods hold the mutex while they're mutating the clients field, like this:
func (cList *ClientList) AddClient(cl *Client) {
cList.mu.Lock()
defer cList.mu.Unlock()
cList.clients = append(cList.clients, o)
}
(If you will ever encounter the typical usage pattern of your ClientList type is to frequently call methods which only read the contained list, you may start using the sync.RWLock type instead, which allows multiple concurrent readers.)
Second, I'd split the part which "identifies" a client out of the handler function.
As of now, in the handler, if the identification fails, the handler exits but the client is not delisted.
I'd say it would be better to identify it up front and only run the handler once the client is beleived to be okay.
Also it supposedly worth adding a deferred call to something like RemoveClient at the top of the handler's body so that the client is properly delisted when the handler is done with it.
IOW, I'd expect to see something like this:
func (cl *Client) HandleClient(cList *ClientList) {
defer cl.Conn.Close()
err := cl.Identify()
if err != nil {
log.Println(err)
return
}
cList.AddClient(cl)
defer cList.RemoveClient(cl)
// ... the rest of the code
}
I am trying to implement an oauth server and the package I am using needs the complete http.ResponseWriter and http.Request types.
c.Response does not contain all the methods that http.ResponseWriter does and c.Request gives error incompatible type.
How do I get http.ResponseWriter and http.Request in a Revel controller?
type client struct {
ClientId string
ClientSecret string
}
type App struct {
*revel.Controller
}
func (c App) TokenRequest() {
r := c.Request
w := c.Response
body, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err)
}
log.Println(string(body))
var cli client
err = json.Unmarshal(body, &cli)
if err != nil {
panic(err)
}
log.Println(cli.ClientId)
err = OauthSrv.HandleTokenRequest(w, r)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
Warning
I am generally not fond of frameworks like Revel in Go, for reasons that I hope demonstrate themselves on this page. My first recommendation would be that you examine closely what you are actually getting out of Revel that merits the use of such a heavy abstraction layer, and if it's really that valuable, you may want to ask questions going in the other direction, such as how one might make OauthSrv work within Revel's customized ecosystem.
Using a Revel controller as a ResponseWriter
For something to be an http.ResponseWriter, it just needs to have these methods.
Header
You need a method named Header() that returns an http.Header, which you can build out of any map[string][]string. Revel provides similar functionality, but through several layers of abstraction. You will need to unravel them:
c.Response is a *Response, so it has a field named Out containing an OutResponse.
An OutResponse has a Header() method—but it doesn't return an http.Header. Instead, it returns a *RevelHeader.
A *RevelHeader has a GetAll(key string) []string method—which is very similar to the API already provided by the built-in map type, but isn't exactly the same. So, you will need to copy the returned values into a new map every time Header() is called, in order to fully satisfy the function signature requirements.
Also, GetAll() requires you to know the key name you are interested in, and *RevelHeader on its own does not provide a way to look up which keys are available. For now we can rely on the fact that the current implementation only has one field, a ServerHeader that does provide a GetKeys() []string method.
Putting all this together, we can build our Header method:
func (rrw RevelResponseWrapper) Header() http.Header {
revelHeader := rrw.Response.Out.Header()
keys := revelHeader.Server.GetKeys()
headerMap := make(map[string][]string)
for _, key := range keys {
headerMap[key] = revelHeader.GetAll(key)
}
return http.Header(headerMap)
}
Write and WriteHeader
You would use similar anti-patterns to expose rrw.Write([]byte) (int, error) so that it calls through to c.Response.Out.Write(data []byte) (int, error), and rrw.WriteHeader(int) error so that it calls c.Response.WriteHeader(int, string). Depending on what is considered appropriate for the framework, either panic on errors or fail silently, since their API doesn't expect WriteHeader errors to be handle-able.
Getting an http.Request from Revel
Unfortunately, the http.Request type is a struct, so you can't just simulate it. You basically have two options: reconstruct it using the net/http package from all the properties you are able to access, or hope that the *revel.Request you have is secretly an http.Request under the hood. In the latter case, you can use a type assertion:
revelReq, ok := c.Request.In.(*revel.GoRequest)
if !ok {
// handle this somehow
}
r := revelReq.Original
I noticed whilst writing some RESTful endpoints in Lambda, I had three models, and the handlers, were all identical. They take a request, fetch some data from a repository, and return the data as a response. The only thing which differed in each handler function, was the model instance. So I was thinking of ways of passing in the type and re-using the same functionality for marshalling the response into JSON etc.
If I have a pattern such as this:
type Handler struct {
model interface{}
repository
}
func (h *Handler) Fetch(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
model := h.model
if err := h.repository.Fetch(id, &model); err != nil {
return ErrResponse(err, http.StatusInternalServerError)
}
return Response(map[string]interface{}{
"resource": model,
}, http.StatusOK)
}
Then calling this:
repository := NewRepository(tableName, connection)
handler := &helpers.Handler{repository, new(models.Client)}
I'm re-assigning model := h.model so that the struct itself doesn't become 'stateful', otherwise I'd have to worry about locking to ensure its thread safe. So I figured I could just re-assign it in the scope of the handler to avoid that.
This pattern seems to work okay, but I wondered if there were any potential downsides or pitfalls?