I want use new relic APM in the mattermost application. In order to monitor the performance the application I have added the code (as mentioned in new relic) just above the createpost api request handler in api/post.go file.
func createPost(c *Context, w http.ResponseWriter, r *http.Request) {
config := newrelic.NewConfig("mylocalstarfp", "####12337")
app, err1 := newrelic.NewApplication(config)
fmt.Println("config")
fmt.Println(config)
if nil != err1 {
fmt.Println(err1)
// os.Exit(1)
}
txn := app.StartTransaction("mylocalstar",w, r)
defer txn.End()
post := model.PostFromJson(r.Body)
.....
.......
}
The application is displayed on new relic dashboard and attributes like CPU and Memory are displayed.But no Response time and Throughput attributes are displayed.
As per new relic documentation (https://github.com/newrelic/go-agent) this code has to be added in main /init block or just at start of function where we need to monitor the performance.
But I am not able to monitor as response time and throughput attributes are not being displayed.
May be I am adding the code at wrong place.
Also I have tried to add the code at beginning of main() function in mattermost.go file. But no success.
Please suggest as to where I have to add the code.
Secondly, they have also mentioned that:
If you are using the standard HTTP library package, you can create transactions by wrapping HTTP requests, as an alternative to instrumenting a function's code.
Here is a before-and-after example of an HTTP handler being wrapped:
Before:
http.HandleFunc("/users", usersHandler)
After:
http.HandleFunc(newrelic.WrapHandleFunc(app, "/users", usersHandler))
This automatically starts and ends a transaction with the request and response writer.
As per this where should I add the code in Mattermost?
You might try using the latest release (1.3) which has support for short lived processes and then adding the code section below
newrelic.NewConfig("mylocalstarfp", "####12337")
app, err1 := newrelic.NewApplication(config)`
to the mattermost.go, and passing the app variable to anywhere you want to monitor transactions.
That’s not a guarantee, however. Just a thought not backed up by any testing.
Got the solution, hence posting for others to refer.
Solved the issue to track each request by this code in mattermost:
BaseRoutes.NeedTeam.Handle(newrelic.WrapHandle(app, "/users", ApiAppHandler(usersHandler))).Methods("POST")
Related
I’m currently maintaining a few HTTP APIs based on the standard library and gorilla mux and running in kubernetes (GKE).
We’ve adopted the http.TimeoutHandler as our “standard” way to have a consistent timeout error management.
A typical endpoint implementation will use the following “chain”:
MonitoringMiddleware => TimeoutMiddleware => … => handler
so that we can monitor a few key metrics per endpoint.
One of our API is typically used in a “fire and forget” mode meaning that clients will push some data and not care for the API response. We are facing the issue that
the Golang standard HTTP server will cancel a request context when the client connection is no longer active (godoc)
the TimeoutHandler will return a “timeout” response whenever the request context is done (see code)
This means that we are not processing requests to completion when the client disconnects which is not what we want and I’m therefore looking for solutions.
The only discussion I could find that somewhat relates to my issue is https://github.com/golang/go/issues/18527; however
The workaround is your application can ignore the Handler's Request.Context()
would mean that the monitoring middleware would not report the "proper" status since the Handler would perform the request processing in its goroutine but the TimeoutHandler would be enforcing the status and observability would be broken.
For now, I’m not considering removing our middlewares as they’re helpful to have consistency across our APIs both in terms of behaviours and observability. My conclusion so far is that I need to “fork” the TimeoutHandler and use a custom context for when an handler should not depend on the client waiting for the response or not.
The gist of my current idea is to have:
type TimeoutHandler struct {
handler Handler
body string
dt time.Duration
// BaseContext optionally specifies a function that returns
// the base context for controling if the server request processing.
// If BaseContext is nil, the default is req.Context().
// If non-nil, it must return a non-nil context.
BaseContext func(*http.Request) context.Context
}
func (h *TimeoutHandler) ServeHTTP(w ResponseWriter, r *Request) {
reqCtx := r.Context()
if h.BaseContext != nil {
reqCtx = h.BaseContext(r)
}
ctx, cancelCtx := context.WithTimeout(reqCtx, h.dt)
defer cancelCtx()
r = r.WithContext(ctx)
...
case <-reqCtx.Done():
tw.mu.Lock()
defer tw.mu.Unlock()
w.WriteHeader(499) // write status for monitoring;
// no need to write a body since no client is listening.
case <-ctx.Done():
tw.mu.Lock()
defer tw.mu.Unlock()
w.WriteHeader(StatusServiceUnavailable)
io.WriteString(w, h.errorBody())
tw.timedOut = true
}
The middleware BaseContext callback would return context.Background() for requests to the “fire and forget” endpoint.
One thing I don’t like is that in doing so I’m losing any context keys written so this new middleware would have strong usage constraints. Overall I feel like this is more complex than it should be.
Am I completely missing something obvious?
Any feedback on API instrumentation (maybe our middlewares are an antipattern) /fire and forget implementations would be welcomed!
EDIT: as most comments are that a request for which the client does not wait for a response has unspecified behavior, I checked for more information on typical clients for which this happens.
From our logs, this happens for user agents that seem to be mobile devices. I can imagine that connections can be much more unstable and the problem will likely not disappear.
I would therefore not conclude that I shouldn't find a solution since this is currently creating false-positive alerts.
i have written a Golang app using GIN middleware with apm enabled and facing a problem which is random in nature.
for enabling the APM and elastic search monitoring i have used the middleware like below
func NewRouter() *gin.Engine {
gin.SetMode(gin.DebugMode)
router := gin.New()
router.Use(middlewares.Recover())
router.Use(apmgin.Middleware(router))
router.Use(middlewares.Logger(), gin.Logger())
//API Route Group
v2 := router.Group("/api")
//V1 Route
v2Routes(v2)
return router
}
uptill now , logging is going on fine on elastic search with all necessary parameters.
but now i want to send by POST request data also, which is not getting
logged currently by default
now inside my main handler function of the API , I am using a custom function to log my post request data to APM
func SendPostDataToAPM(ctx context.Context, data interface{}) {
defer RoutineRecovery()
tx := apm.TransactionFromContext(ctx)
tx.Context.SetCustom("postData", data)
Now the main problem that i am facing is that , the above function SendPostDataToAPM is giving me panic randomly, means it is working fine for some time and then randomly throws an panic, and again works fine for the next request
panic is coming specifically from the last line tx.Context.SetCustom("postData", data)
panic: runtime error: invalid memory address or nil pointer dereference
if anyone knows any workaround around this, kindly help
Updates
I know that there is some nil problem here so I have implemented a nil check before calling the SetCustom function, the solution worked , but still there were rare cases(like only 1 or 2 times in a day) , when even that too didn't work.
So I am trying to find a solution to this random behaviour, why this is panic is coming randomly.
I have small Golang package which does some work. This work suppose a high amount of errors could be produced and this is OK. Currently all errors are ignored. Yes it may look strange, but visit the link and check the main purpose of package.
I'd like to extend functionality of the package and provide ability to see errors occurred during runtime. But due to lack of software design skills I have some questions with no answers.
At first, I thought to implement logging inside the package using the existing logging (zerolog, zap or whatever else). But, will it be ok for package's users? Because they might want to use other logging packages and would like to modify output format.
Maybe it's possible to provide a way to user to inject it's own logging?
I'd like to achieve the ability to provide easy-configurable way for logging which could be switched on or off on users demands.
Some go lib use logging like this
in your packge definite a logger interface
type Yourlogging interface{
Errorf(...)
Warningf(...)
Infof(...)
Debugf(...)
}
and definite a variable for this interface
var mylogger Yourlogging
func SetLogger(l yourlogging)error{
mylogger = l
}
in your func, you can call them for logging
mylogger.Infof(..)
mylogger.Errorf(...)
you don't need implement the interface, but you can use them who implement this interface
for example:
SetLogger(os.Stdout) //logging output to stdout
SetLogger(logrus.New()) // logging output to logrus (github.com/sirupsen/logrus)
In Go, you will see some libraries implement logging interfaces like other answers have suggested. However, you could completely avoid your packages needing to log if you structured your application differently, for your example.
For example, in your example application you linked, your main application runtime calls idleexacts.Run(), which starts this function.
// startLoop starts workload using passed settings and database connection.
func startLoop(ctx context.Context, log log.Logger, pool db.DB, tables []string, jobs uint16, minTime, maxTime time.Duration) error {
rand.Seed(time.Now().UnixNano())
// Increment maxTime up to 1 due to rand.Int63n() never return max value.
maxTime++
// While running, keep required number of workers using channel.
// Run new workers only until there is any free slot.
guard := make(chan struct{}, jobs)
for {
select {
// Run workers only when it's possible to write into channel (channel is limited by number of jobs).
case guard <- struct{}{}:
go func() {
table := selectRandomTable(tables)
naptime := time.Duration(rand.Int63n(maxTime.Nanoseconds()-minTime.Nanoseconds()) + minTime.Nanoseconds())
err := startSingleIdleXact(ctx, pool, table, naptime)
if err != nil {
log.Warnf("start idle xact failed: %s", err)
}
// When worker finishes, read from the channel to allow starting another worker.
<-guard
}()
case <-ctx.Done():
return nil
}
}
}
The problem here is all of the orchestration of your logic is happening inside of your packages. Instead, this loop should be running in your main application, and this package should provide users with simple actions such as selectRandomTable() or createTempTable().
If the orchestration of code was in your main application and the package only provided simple actions. It would be much easier to return errors to the user as part of the function calls.
It would also make your packages easier for others to reuse because they have simple actions and open users to use them in other ways than you intended.
Hypothetically speaking, is it good practice to connect to a database for each request and close in when the request has completed?
I'm using mongodb with mgo for the database.
In my project, I would like to connect to a certain database by getting the database name from the request header (of course, this is combined with an authentication mechanism, e.g. JWT in my app). The flow goes something like:
User authentication:
POST to http://api.app.com/authenticate
// which checks the user in a "global" database,
// authenticates them and returns a signed JWT token
// The token is stored in bolt.db for the authentication mechanism
Some RESTful operations
POST to http://api.app.com/v1/blog/posts
// JWT middleware for each request to /v1* is set up
// `Client-Domain` in header is set to a database's name, e.g 'app-com'
// so we open a connection to that database and close when
// request finishes
So my questions are:
Is this feasible? - I've read about connection pools and reusing them but I haven't read much about them yet
Is there a better way of achieving the desired functionality?
How do I ensure the session is only closed when the request has completed?
The reason why I need to do this is because we have multiple vendors that have the same database collections with different entries with restricted access to their own databases.
Update / Solution
I ended up using Go's built in Context by Copying a session and using it anywhere I need to do any CRUD ops
Something like:
func main() {
...
// Configure connection and set in global var
model.DBSession, err = mgo.DialWithInfo(mongoDBDialInfo)
defer model.DBSession.Close()
...
n := negroni.Classic()
n.Use(negroni.HandlerFunc(Middleware))
...
}
func Middleware(res http.ResponseWriter, req *http.Request, next http.HandlerFunc) {
...
db := NewDataStore(clientDomain)
// db.Close() is an alias for ds.session.Close(), code for this function is not included in this post
// Im still experimenting with this, I need to make sure the session is only closed after a request has completed, currently it does not always do so
defer db.Close()
ctx := req.Context()
ctx = context.WithValue(ctx, auth.DataStore, db)
req = req.WithContext(ctx)
...
}
func NewDataStore(db string) *DataStore {
store := &DataStore{
db: DBSession.Copy().DB(db),
session: DBSession.Copy(),
}
return store
}
And then use it in a HandlerFunc, example /v1/system/users:
func getUsers(res http.ResponseWriter, req *http.Request) {
db := req.Context().Value(auth.DataStore).(*model.DataStore)
users := make([]SystemUser{}, 0)
// db.C() is an alias for ds.db.C(), code for this function is not included in this post
db.C("system_users").Find(nil).All(&users)
}
40% response time decrease over the original method I experimented with.
Hypothetically speaking is not a good practice because:
The database logic is scattered among several packages.
It's difficult to test
You can't apply DI (mainly it will be hard to maintain the code)
Replying to your questions:
Yes is feasible BUT you will not use the connection pool inside them go package (take a look to the code here if you want know more about Connection Pool)
A better way is to create a global variable that contains the database connection and close when the application is going to stop (and not close the connection every request)
How do I ensure the session is only closed when the request has complete<- you should checkout the answer fro your db query and then close the connection (but I don't recommend to close the connection after a request because you'll need to open again for another request and close again etc...)
I need to embed the default http.Server in my own server struct and customize the Serve method.
The server needs to short circuit the go c.serve() call and only run that line if it has the computing resources available to respond within 50ms. Otherwise the server is just going to send a 204 and move on.
This is almost straightforward.
type PragmaticServer struct {
http.Server
Addr string
Handler http.Handler
}
func (srv *PragmaticServer) Serve(l net.Listener) error {
defer l.Close()
var tempDelay time.Duration // how long to sleep on accept failure
for {
// SNIP for clarity
c, err := srv.newConn(rw)
if err != nil {
continue
}
c.setState(c.rwc, StateNew) // before Serve can return
go c.serve()
}
}
So, again. This almost works. Except that srv.newConn is an unexported method, as is c.serve and c.setState, which means that I end up having to copy and paste pretty much the entirety of net/http in order for this to compile. Which is basically a fork. Is there any better way to do this?
Unfortunately, you're not going to be able to do that without reimplementing most of the Server code. Short of that, we usually intercept the call either just before at conn.Accept, or just after at Handler.ServerHTTP.
The first method is to create a custom net.Listener that filters out connections before they are even handed off to the http.Server. While this can respond faster, and consume fewer resources, it however makes it less convenient to write http responses, and precludes you from limiting requests on already open connections.
The second way to handle this, is to just wrap the handlers and intercept the request before any real work has been done. You most likely want to create a http.Handler to filter the requests, and pass them through to your main handler. This can also be more flexible, since you can filter based on the route, or other request information if you so choose.