I am using a NewGaugeVec to report my metrics:
elapsed := prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "gogrinder_elapsed_ms",
Help: "Current time elapsed of gogrinder teststep",
}, []string{"teststep", "user", "iteration", "timestamp"})
prometheus.MustRegister(elapsed)
All works fine but I noticed that my custom exporter contains all metrics from prometheus/go_collector.go:
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.00041795300000000004
go_gc_duration_seconds{quantile="0.25"} 0.00041795300000000004
go_gc_duration_seconds{quantile="0.5"} 0.00041795300000000004
...
I suspect that this is kind of a default behavior but I did not find anything in the documentation on how to disable that. Any ideas on how to configure my custom exporter so that these default metrics disappear?
Well the topic is rather old but in case others have to deal with it.
The following code works fine with current codebase v0.9.0-pre1
// [...] imports, metric initialization ...
func main() {
// go get rid of any additional metrics
// we have to expose our metrics with a custom registry
r := prometheus.NewRegistry()
r.MustRegister(myMetrics)
handler := promhttp.HandlerFor(r, promhttp.HandlerOpts{})
// [...] update metrics within a goroutine
http.Handle("/metrics", handler)
log.Fatal(http.ListenAndServe(":12345", nil))
}
I would simply do it this way ->
// Register your collectors
elapsed := prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "gogrinder_elapsed_ms",
Help: "Current time elapsed of gogrinder teststep",
}, []string{"teststep", "user", "iteration", "timestamp"})
prometheus.MustRegister(elapsed)
// Remove Go collector
prometheus.Unregister(prometheus.NewGoCollector())
This solution worked from me. Idea is to create a custom registry and register our metrics. Making sure we pass False in handler options for open metrics will disable those default metrics
var httpDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "golang_api_http_duration_seconds",
Help: "Duration of HTTP requests.",
}, []string{"path", "host"})
promReg := prometheus.NewRegistry()
promReg.MustRegister(httpDuration)
handler := promhttp.HandlerFor(
promReg,
promhttp.HandlerOpts{
EnableOpenMetrics: false,
})
http.Handle("/metrics", handler)
log.Fatal(http.ListenAndServe(":12345", nil))
This is not currently possible in the Go client, once https://github.com/prometheus/client_golang/issues/46 is complete you'll have a way to do this.
In general you want your custom exporter to export these, the only ones I'm aware of where it doesn't currently make sense are the snmp and blackbox exporter.
Incidentally timestamp seems odd as a label, if you want that you should likely be using logging rather than metrics. See https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/
The Prometheus way would be to have the timestamp as a value, not as a label.
It's not really helpful as an answer to say "you'd have to go and do it yourself" but it seems like the only option for now.
Since Prometheus is open source and if you really need to do that; I believe you'd have to fork this one go_collector.go line #28 and the related sections, or better yet modify it to make all those metrics optional and make a PR so other people may also benefit from that in the future.
you can use --web.disable-exporter-metrics now.
https://github.com/prometheus/node_exporter/pull/1148
Related
I'm writing a custom terraform provider. I need to wrap the calls to the backed API in a "session", opening it before any calls are made and then closing it once all the terraform calls have completed.
Opening the session is straightforward, I can do that in the ConfigureContextFunc of the schema.Provider. Is there a way to set up a callback (or something) at the end of the application so I can close/"finalize" the session? I can imagine something specific to my resources, but that seems hacky. In my dream world I'd also be able to fail the apply if the close had an error.
Absent a nice finalize call is there a way to access the plan that I could use to determine that the current call is the last needed for the apply?
Update: I thought I could use a StopContext:
stopCtx, ok := schema.StopContext(ctx)
...
go func(ctx context.Context) {
// Wait for stop context cancellation
<-stopCtx.Done()
...
}
However, this is both deprecated and seems to only get called when stopping due to some outside trigger, like SIGINT, and not a regular exit (at least that's what I've been seeing).
After a fair amount of floundering I have what I believe to be a reasonable solution, and it even matches #Ben Hoyt's comment. We can defer and teardown in main.
func main() {
var prov *schema.Provider
defer func() {
xyz.PrividerTeardown(prov)
}()
plugin.Serve(&plugin.ServeOpts{
ProviderFunc: func() *schema.Provider {
prov = xyz.Provider()
return prov
},
})
}
I'll note that my comment on the question about not being around when the provider is made is incorrect. The provider is made in a main function in the provider code. In my (flimsy) defense I used the example scaffolding so that code came pre-written.
One thing that threw me was the docs for plugin.Serve
Serve serves a plugin. This function never returns and should be the final function called in the main function of the plugin.
It turns out that when the terraform action is done and the plugin is no longer needed, Serve does return and allows our defer to run. I did need to keep track of success or failure of all the calls that were made while Serve was active to know the status in my ProviderTeardown, but it is working.
I'm using golang logrus for logging and I'm having a wrapper with all the regular functions like Info(..),Infof(..) e.t.c I want to implement a wrapper function Audit(..) for logging to syslog.
I noticed logrus syslog hooks problem is, once it got hooked every log function is logging to syslog, also Infof(..) which I don't want them to.
Is there a way I can call syslog by demand? other than:
func (l *WrapLogger) Audit(msg string) {
l.logger.AddHook(syslogHook)
l.logger.Info(msg)
l.logger.ReplaceHooks(logrus.LevelHooks) // removing somehow the hook
}
Thanks
If you're trying to delegate what message to send by its log level then you can do it by setting the log levels the hook accepts.
For example:
log.AddHook(&writer.Hook{
Writer: os.Stderr,
LogLevels: []log.Level{ log.WarnLevel },
})
log.AddHook(lSyslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, ""))
log.Info("This will go to syslog")
log.Warn("This will go to stderr")
If you want to route this no according to the log level then what you suggested may work but it feels odd and may have race-conditions.
What I suggest you to do is create your own hook that gets hook list and route to the right hook(s) according to the message or to the fields that are passed when calling Info, Warn and etc.
I have small Golang package which does some work. This work suppose a high amount of errors could be produced and this is OK. Currently all errors are ignored. Yes it may look strange, but visit the link and check the main purpose of package.
I'd like to extend functionality of the package and provide ability to see errors occurred during runtime. But due to lack of software design skills I have some questions with no answers.
At first, I thought to implement logging inside the package using the existing logging (zerolog, zap or whatever else). But, will it be ok for package's users? Because they might want to use other logging packages and would like to modify output format.
Maybe it's possible to provide a way to user to inject it's own logging?
I'd like to achieve the ability to provide easy-configurable way for logging which could be switched on or off on users demands.
Some go lib use logging like this
in your packge definite a logger interface
type Yourlogging interface{
Errorf(...)
Warningf(...)
Infof(...)
Debugf(...)
}
and definite a variable for this interface
var mylogger Yourlogging
func SetLogger(l yourlogging)error{
mylogger = l
}
in your func, you can call them for logging
mylogger.Infof(..)
mylogger.Errorf(...)
you don't need implement the interface, but you can use them who implement this interface
for example:
SetLogger(os.Stdout) //logging output to stdout
SetLogger(logrus.New()) // logging output to logrus (github.com/sirupsen/logrus)
In Go, you will see some libraries implement logging interfaces like other answers have suggested. However, you could completely avoid your packages needing to log if you structured your application differently, for your example.
For example, in your example application you linked, your main application runtime calls idleexacts.Run(), which starts this function.
// startLoop starts workload using passed settings and database connection.
func startLoop(ctx context.Context, log log.Logger, pool db.DB, tables []string, jobs uint16, minTime, maxTime time.Duration) error {
rand.Seed(time.Now().UnixNano())
// Increment maxTime up to 1 due to rand.Int63n() never return max value.
maxTime++
// While running, keep required number of workers using channel.
// Run new workers only until there is any free slot.
guard := make(chan struct{}, jobs)
for {
select {
// Run workers only when it's possible to write into channel (channel is limited by number of jobs).
case guard <- struct{}{}:
go func() {
table := selectRandomTable(tables)
naptime := time.Duration(rand.Int63n(maxTime.Nanoseconds()-minTime.Nanoseconds()) + minTime.Nanoseconds())
err := startSingleIdleXact(ctx, pool, table, naptime)
if err != nil {
log.Warnf("start idle xact failed: %s", err)
}
// When worker finishes, read from the channel to allow starting another worker.
<-guard
}()
case <-ctx.Done():
return nil
}
}
}
The problem here is all of the orchestration of your logic is happening inside of your packages. Instead, this loop should be running in your main application, and this package should provide users with simple actions such as selectRandomTable() or createTempTable().
If the orchestration of code was in your main application and the package only provided simple actions. It would be much easier to return errors to the user as part of the function calls.
It would also make your packages easier for others to reuse because they have simple actions and open users to use them in other ways than you intended.
Besides a few tutorials on Go I have no actual experience in it. I'm trying to take a project written in Go and converting it into a windows service.
I honestly haven't tried anything besides trying to find things to read over. I have found a few threads and choosen the best library I felt covered all of our needs
https://github.com/golang/sys
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build windows
package main
import (
"fmt"
"strings"
"time"
"golang.org/x/sys/windows/svc"
"golang.org/x/sys/windows/svc/debug"
"golang.org/x/sys/windows/svc/eventlog"
)
var elog debug.Log
type myservice struct{}
func (m *myservice) Execute(args []string, r <-chan svc.ChangeRequest, changes chan<- svc.Status) (ssec bool, errno uint32) {
const cmdsAccepted = svc.AcceptStop | svc.AcceptShutdown | svc.AcceptPauseAndContinue
changes <- svc.Status{State: svc.StartPending}
fasttick := time.Tick(500 * time.Millisecond)
slowtick := time.Tick(2 * time.Second)
tick := fasttick
changes <- svc.Status{State: svc.Running, Accepts: cmdsAccepted}
loop:
for {
select {
case <-tick:
beep()
elog.Info(1, "beep")
case c := <-r:
switch c.Cmd {
case svc.Interrogate:
changes <- c.CurrentStatus
// Testing deadlock from https://code.google.com/p/winsvc/issues/detail?id=4
time.Sleep(100 * time.Millisecond)
changes <- c.CurrentStatus
case svc.Stop, svc.Shutdown:
// golang.org/x/sys/windows/svc.TestExample is verifying this output.
testOutput := strings.Join(args, "-")
testOutput += fmt.Sprintf("-%d", c.Context)
elog.Info(1, testOutput)
break loop
case svc.Pause:
changes <- svc.Status{State: svc.Paused, Accepts: cmdsAccepted}
tick = slowtick
case svc.Continue:
changes <- svc.Status{State: svc.Running, Accepts: cmdsAccepted}
tick = fasttick
default:
elog.Error(1, fmt.Sprintf("unexpected control request #%d", c))
}
}
}
changes <- svc.Status{State: svc.StopPending}
return
}
func runService(name string, isDebug bool) {
var err error
if isDebug {
elog = debug.New(name)
} else {
elog, err = eventlog.Open(name)
if err != nil {
return
}
}
defer elog.Close()
elog.Info(1, fmt.Sprintf("starting %s service", name))
run := svc.Run
if isDebug {
run = debug.Run
}
err = run(name, &myservice{})
if err != nil {
elog.Error(1, fmt.Sprintf("%s service failed: %v", name, err))
return
}
elog.Info(1, fmt.Sprintf("%s service stopped", name))
}
So I spent some time going over this code. Tested it out to see what it does. It performs as it should.
The question I have is we currently have a Go program that takes in arguments and for our service we pass in server. Which spins up our stuff on a localhost webpage.
I believe the code above may have something to do with that but I'm lost at how I would actually get it spin off our exe with the correct arguements. Is this the right spot to call main?
Im sorry if this is vague. I dont know exactly how to make this interact with our already exisiting exe.
I can get that modified if I know what needs to be changed. I appreacite any help.
OK, that's much clearer now. Well, ideally you should start with some tutorial on what constitutes a Windows service—I bet tihis might have solved the problem for you. But let's try anyway.
Some theory
A Windows service sort of has two facets: it performs some useful task and it communicates with the SCM facility. When you manipulate a service using the sc command or through the Control Panel, you have that piece of software to talk with SCM on your behalf, and SCM talks with that service.
The exact protocol the SCM and a service use is low-level and complicated
and the point of the Go package you're using is to hide that complexity from you
and offer a reasonably Go-centric interface to that stuff.
As you might gather from your own example, the Execute method of the type you've created is—for the most part—concerned with communicating with SCM: it runs an endless for loop which on each iteration sleeps on reading from the r channel, and that channel delivers SCM commands to your service.
So you basically have what could be called "an SCM command processing loop".
Now recall those two facets above. You already have one of them: your service interacts with SCM, so you need another one—the code which actually performs useful tasks.
In fact, it's already partially there: the example code you've grabbed creates a time ticker which provides a channel on which it delivers a value when another tick passes. The for loop in the Execute method reads from that channel as well, "doing work" each time another tick is signalled.
OK, this is fine for a toy example but lame for a real work.
Approaching the solution
So let's pause for a moment and think about our requirements.
We need some code running and doing our actual task(s).
We need the existing command processing loop to continue working.
We need these two pieces of code to work concurrently.
In this toy example the 3rd point is there "for free" because a time ticker carries out the task of waiting for the next tick automatically and fully concurrently with the rest of the code.
Your real code most probably won't have that luxury, so what do you do?
In Go, when you need to do something concurrently with something else,
an obvious answer is "use a goroutine".
So the first step is to grab your existing code, turn it into a callable function
and then call it in a separate goroutine right before entering the for loop.
This way, you'll have both pieces run concurrently.
The hard parts
OK, that wasn't hard.
The hard parts are:
How to configure the code which performs the tasks.
How to make the SCM command processing loop and the code carrying out tasks communicate.
Configuration
This one really depends on the policies at your $dayjob or of your $current_project, but there are few hints:
A Windows service may receive command-line arguments—either for a single run or permanently (passed to the service on each of its runs).
The downside is that it's not convenient to work with them from the UI/UX standpoint.
Typically Windows services used to read the registry.
These days (after the advent of .NET and its pervasive xml-ity) the services tend to read configuration files.
The OS environment most of the time is a bad fit for the task.
You may combine several of these venues.
I think I'd start with a configuration file but then again, you should pick the path of the least resistance, I think.
One of the things to keep in mind is that the reading and processing of the configuration should better be done before the service signals the SCM it started OK: if the configuration is invalid or cannot be loaded, the service should extensively log that and signal it failed, and not run the actual task processing code.
Communication between the command processing loop and the tasks carrying code
This is IMO the hardest part.
It's possible to write a whole book here but let's keep it simple for now.
To make it as simple as possible I'd do the following:
Consider pausing, stopping and shutting down mostly the same: all these signals must tell your task processing code to quit, and then wait for it to actually do that.
Consider the "continue" signal the same as starting the task processing function: run it again—in a new goroutine.
Have a one-directional communication: from the control loop to the tasks processing code, but not the other way—this will greatly simplify service state management.
This way, you may create a single channel which the task processing code listens on—or checks periodically, and when a value comes from that channel, the code stops running, closes the channel and exits.
The control loop, when the SCM tells it to pause or stop or shut down, sends anything on that channel then waits for it to close. When that happens, it knows the tasks processing code is finished.
In Go, a paradigm for a channel which is only used for signaling, is to have a channel of type struct{} (an empty struct).
The question of how to monitor this control channel in the tasks running code is an open one and heavily depends on the nature of the tasks it performs.
Any further help here would be reciting what's written in the Go books on concurrency so you should have that covered first.
There's also an interesting question of how to have the communication between the control loop and the tasks processing loop resilient to the possible processing stalls in the latter, but then again, IMO it's too early to touch upon that.
I want use new relic APM in the mattermost application. In order to monitor the performance the application I have added the code (as mentioned in new relic) just above the createpost api request handler in api/post.go file.
func createPost(c *Context, w http.ResponseWriter, r *http.Request) {
config := newrelic.NewConfig("mylocalstarfp", "####12337")
app, err1 := newrelic.NewApplication(config)
fmt.Println("config")
fmt.Println(config)
if nil != err1 {
fmt.Println(err1)
// os.Exit(1)
}
txn := app.StartTransaction("mylocalstar",w, r)
defer txn.End()
post := model.PostFromJson(r.Body)
.....
.......
}
The application is displayed on new relic dashboard and attributes like CPU and Memory are displayed.But no Response time and Throughput attributes are displayed.
As per new relic documentation (https://github.com/newrelic/go-agent) this code has to be added in main /init block or just at start of function where we need to monitor the performance.
But I am not able to monitor as response time and throughput attributes are not being displayed.
May be I am adding the code at wrong place.
Also I have tried to add the code at beginning of main() function in mattermost.go file. But no success.
Please suggest as to where I have to add the code.
Secondly, they have also mentioned that:
If you are using the standard HTTP library package, you can create transactions by wrapping HTTP requests, as an alternative to instrumenting a function's code.
Here is a before-and-after example of an HTTP handler being wrapped:
Before:
http.HandleFunc("/users", usersHandler)
After:
http.HandleFunc(newrelic.WrapHandleFunc(app, "/users", usersHandler))
This automatically starts and ends a transaction with the request and response writer.
As per this where should I add the code in Mattermost?
You might try using the latest release (1.3) which has support for short lived processes and then adding the code section below
newrelic.NewConfig("mylocalstarfp", "####12337")
app, err1 := newrelic.NewApplication(config)`
to the mattermost.go, and passing the app variable to anywhere you want to monitor transactions.
That’s not a guarantee, however. Just a thought not backed up by any testing.
Got the solution, hence posting for others to refer.
Solved the issue to track each request by this code in mattermost:
BaseRoutes.NeedTeam.Handle(newrelic.WrapHandle(app, "/users", ApiAppHandler(usersHandler))).Methods("POST")