I'm working on developing a k8s custom resource that as part of the business logic needs to reconcile its state when an external Job in the cluster have changed its own state.
Those Jobs aren't created by the custom resource itself but are externally created for a third party service, however I need to reconcile the state of the CRO for example when any of those external jobs have finished.
After reading bunch of documentation, I came up with setting a watcher for the controller, to watch Jobs like the following example
func (r *DatasetReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&datasetv1beta1.Dataset{}).
Watches(&source.Kind{Type: &batchv1.Job{}}, &handler.EnqueueRequestForObject{} /* filter by predicates, see https://pkg.go.dev/sigs.k8s.io/controller-runtime#v0.9.6/pkg/controller#Controller */).
Complete(r)
}
No I'm having my reconcile loop triggered for Jobs and my CRs with the corresponding name and namespace but I don't know anything about the object kind.
func (r *DatasetReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
l := log.FromContext(ctx)
l.Info("Enter Reconcile loop")
l.Info("Request", "Req", req)
//if this is triggered by my CR
dataset := &datasetv1beta1.Dataset{}
r.Get(ctx, types.NamespacedName{Name: req.Name, Namespace: req.Namespace}, dataset)
//whereas when triggered by a Job
job := &batchv1.Job{}
r.Get(ctx, types.NamespacedName{Name: req.Name, Namespace: req.Namespace}, job)
return ctrl.Result{}, nil
}
How can I check within Reconcile the object kind? so I can retrieve the full object data calling r.Get
By design, the event that triggered reconciliation is not passed to the reconciler so that you are forced to define and act on a state instead. This approach is referred to as level-based, as opposed to edge-based.
In your example you have two resources you are trying to keep track of. I would suggest either:
Using ownerReferences or labels if these resources are related. That way you can get all related Datasets for a given Job (or vice versa) and reconcile things that way.
If the two resources are not related, create a separate controller for each resource.
If you want to prevent reconciliation on certain events you can make use of predicates. From the event in the predicate function you can get the object type by e.Object.(*core.Pod) for example.
Related
I'm writing a custom terraform provider. I need to wrap the calls to the backed API in a "session", opening it before any calls are made and then closing it once all the terraform calls have completed.
Opening the session is straightforward, I can do that in the ConfigureContextFunc of the schema.Provider. Is there a way to set up a callback (or something) at the end of the application so I can close/"finalize" the session? I can imagine something specific to my resources, but that seems hacky. In my dream world I'd also be able to fail the apply if the close had an error.
Absent a nice finalize call is there a way to access the plan that I could use to determine that the current call is the last needed for the apply?
Update: I thought I could use a StopContext:
stopCtx, ok := schema.StopContext(ctx)
...
go func(ctx context.Context) {
// Wait for stop context cancellation
<-stopCtx.Done()
...
}
However, this is both deprecated and seems to only get called when stopping due to some outside trigger, like SIGINT, and not a regular exit (at least that's what I've been seeing).
After a fair amount of floundering I have what I believe to be a reasonable solution, and it even matches #Ben Hoyt's comment. We can defer and teardown in main.
func main() {
var prov *schema.Provider
defer func() {
xyz.PrividerTeardown(prov)
}()
plugin.Serve(&plugin.ServeOpts{
ProviderFunc: func() *schema.Provider {
prov = xyz.Provider()
return prov
},
})
}
I'll note that my comment on the question about not being around when the provider is made is incorrect. The provider is made in a main function in the provider code. In my (flimsy) defense I used the example scaffolding so that code came pre-written.
One thing that threw me was the docs for plugin.Serve
Serve serves a plugin. This function never returns and should be the final function called in the main function of the plugin.
It turns out that when the terraform action is done and the plugin is no longer needed, Serve does return and allows our defer to run. I did need to keep track of success or failure of all the calls that were made while Serve was active to know the status in my ProviderTeardown, but it is working.
I’m currently maintaining a few HTTP APIs based on the standard library and gorilla mux and running in kubernetes (GKE).
We’ve adopted the http.TimeoutHandler as our “standard” way to have a consistent timeout error management.
A typical endpoint implementation will use the following “chain”:
MonitoringMiddleware => TimeoutMiddleware => … => handler
so that we can monitor a few key metrics per endpoint.
One of our API is typically used in a “fire and forget” mode meaning that clients will push some data and not care for the API response. We are facing the issue that
the Golang standard HTTP server will cancel a request context when the client connection is no longer active (godoc)
the TimeoutHandler will return a “timeout” response whenever the request context is done (see code)
This means that we are not processing requests to completion when the client disconnects which is not what we want and I’m therefore looking for solutions.
The only discussion I could find that somewhat relates to my issue is https://github.com/golang/go/issues/18527; however
The workaround is your application can ignore the Handler's Request.Context()
would mean that the monitoring middleware would not report the "proper" status since the Handler would perform the request processing in its goroutine but the TimeoutHandler would be enforcing the status and observability would be broken.
For now, I’m not considering removing our middlewares as they’re helpful to have consistency across our APIs both in terms of behaviours and observability. My conclusion so far is that I need to “fork” the TimeoutHandler and use a custom context for when an handler should not depend on the client waiting for the response or not.
The gist of my current idea is to have:
type TimeoutHandler struct {
handler Handler
body string
dt time.Duration
// BaseContext optionally specifies a function that returns
// the base context for controling if the server request processing.
// If BaseContext is nil, the default is req.Context().
// If non-nil, it must return a non-nil context.
BaseContext func(*http.Request) context.Context
}
func (h *TimeoutHandler) ServeHTTP(w ResponseWriter, r *Request) {
reqCtx := r.Context()
if h.BaseContext != nil {
reqCtx = h.BaseContext(r)
}
ctx, cancelCtx := context.WithTimeout(reqCtx, h.dt)
defer cancelCtx()
r = r.WithContext(ctx)
...
case <-reqCtx.Done():
tw.mu.Lock()
defer tw.mu.Unlock()
w.WriteHeader(499) // write status for monitoring;
// no need to write a body since no client is listening.
case <-ctx.Done():
tw.mu.Lock()
defer tw.mu.Unlock()
w.WriteHeader(StatusServiceUnavailable)
io.WriteString(w, h.errorBody())
tw.timedOut = true
}
The middleware BaseContext callback would return context.Background() for requests to the “fire and forget” endpoint.
One thing I don’t like is that in doing so I’m losing any context keys written so this new middleware would have strong usage constraints. Overall I feel like this is more complex than it should be.
Am I completely missing something obvious?
Any feedback on API instrumentation (maybe our middlewares are an antipattern) /fire and forget implementations would be welcomed!
EDIT: as most comments are that a request for which the client does not wait for a response has unspecified behavior, I checked for more information on typical clients for which this happens.
From our logs, this happens for user agents that seem to be mobile devices. I can imagine that connections can be much more unstable and the problem will likely not disappear.
I would therefore not conclude that I shouldn't find a solution since this is currently creating false-positive alerts.
I have small Golang package which does some work. This work suppose a high amount of errors could be produced and this is OK. Currently all errors are ignored. Yes it may look strange, but visit the link and check the main purpose of package.
I'd like to extend functionality of the package and provide ability to see errors occurred during runtime. But due to lack of software design skills I have some questions with no answers.
At first, I thought to implement logging inside the package using the existing logging (zerolog, zap or whatever else). But, will it be ok for package's users? Because they might want to use other logging packages and would like to modify output format.
Maybe it's possible to provide a way to user to inject it's own logging?
I'd like to achieve the ability to provide easy-configurable way for logging which could be switched on or off on users demands.
Some go lib use logging like this
in your packge definite a logger interface
type Yourlogging interface{
Errorf(...)
Warningf(...)
Infof(...)
Debugf(...)
}
and definite a variable for this interface
var mylogger Yourlogging
func SetLogger(l yourlogging)error{
mylogger = l
}
in your func, you can call them for logging
mylogger.Infof(..)
mylogger.Errorf(...)
you don't need implement the interface, but you can use them who implement this interface
for example:
SetLogger(os.Stdout) //logging output to stdout
SetLogger(logrus.New()) // logging output to logrus (github.com/sirupsen/logrus)
In Go, you will see some libraries implement logging interfaces like other answers have suggested. However, you could completely avoid your packages needing to log if you structured your application differently, for your example.
For example, in your example application you linked, your main application runtime calls idleexacts.Run(), which starts this function.
// startLoop starts workload using passed settings and database connection.
func startLoop(ctx context.Context, log log.Logger, pool db.DB, tables []string, jobs uint16, minTime, maxTime time.Duration) error {
rand.Seed(time.Now().UnixNano())
// Increment maxTime up to 1 due to rand.Int63n() never return max value.
maxTime++
// While running, keep required number of workers using channel.
// Run new workers only until there is any free slot.
guard := make(chan struct{}, jobs)
for {
select {
// Run workers only when it's possible to write into channel (channel is limited by number of jobs).
case guard <- struct{}{}:
go func() {
table := selectRandomTable(tables)
naptime := time.Duration(rand.Int63n(maxTime.Nanoseconds()-minTime.Nanoseconds()) + minTime.Nanoseconds())
err := startSingleIdleXact(ctx, pool, table, naptime)
if err != nil {
log.Warnf("start idle xact failed: %s", err)
}
// When worker finishes, read from the channel to allow starting another worker.
<-guard
}()
case <-ctx.Done():
return nil
}
}
}
The problem here is all of the orchestration of your logic is happening inside of your packages. Instead, this loop should be running in your main application, and this package should provide users with simple actions such as selectRandomTable() or createTempTable().
If the orchestration of code was in your main application and the package only provided simple actions. It would be much easier to return errors to the user as part of the function calls.
It would also make your packages easier for others to reuse because they have simple actions and open users to use them in other ways than you intended.
I'm working on a suite of microservices writting in golang. I have a demo in a couple of months, and by next year these services should be in production. For now, I'm just hashing out all the basics and boilerplate, including calls to sentry.
All of the services make several async requests that set several processes in motion. If one thing fails, I don't want to panic or return; I want to continue execution, but I want to be able to go back and see what happened.
While developing, I don't really want to send anything to Sentry, but I want to see what the output to Sentry would be so I can make sure that the messages, breadcrumbs, stacktraces etc are all being captured as intended. Is anything like this possible? I tried running the local server but it's quite bloated and it fired up about 20 docker containers and consumed a LOT of memory. Just looking for something lightweight so I can see what's going on.
I came up with a solution -- the output is very verbose, but it's exactly what I was looking for (for now). I simply provided my own transport implementation and passed it in to ClientOptions:
type consoleTransport struct{}
func (t *consoleTransport) Configure(options sentry.ClientOptions) {
zap.L().Info("Sentry client initialized with an empty DSN. Using consoleTransport. No events will be delivered.")
}
func (t *consoleTransport) SendEvent(event *sentry.Event) {
b, _ := json.Marshal(event)
fmt.Println("[SENTRY CONSOLE] " + string(b))
}
func (t *consoleTransport) Flush(_ time.Duration) bool {
return true
}
I'm trying to build a little website in Go containing a report based on data collected from a web service. It uses an API to query the service for data. The service can only be queried once every few seconds.
However, I have to query it a number of times to get the complete report data. Right now I just hammer the API to update my whole data structure each time the http handler (http.HandleFunc) is called. Of course, this is bad because it triggers lots of queries to the external API that are throttled. So, my report comes up very, very, very, slowly.
What I want to do instead is to have a function to updateReportData in a non-blocking way and store that data in some variable that the http.HandleFunc() can just ingest without calling the external API.
But, I'm very new to Go (and things like closures, semaphores, concurrency, etc) and so I'm not really sure how to build this. Should I be using channels? Should I use timers? How can I get the updateReportData to not block the http.HandleFunc, but still run on a fixed interval?
To sum up, I want to have a background routine update a data structure on a fixed interval and I want to be able to use http.HandleFunc to serve whatever data is in the data structure any time i make an http request to the program. I just have no idea how to start. Any advice would be appreciated.
There are a few things you need to do:
Create a background service that polls for the data. This service can run as a goroutine that periodically checks for new data.
Create a channel that the background service uses to send new data to a centralized place to store the values. The background service should write data to this channel whenever it finds something new. Another option would be to protect the centralized data store with a mutex. Depending on the way the data is written and read, one option will be a better choice.
Create a HTTP handler that returns the current contents of the centralized data store.
Here is a simplified example showing how to use a goroutine and a sync.RWMutext to accomplish what you want:
package main
import (
"fmt"
"net/http"
"sync"
"time"
)
var (
timeSumsMu sync.RWMutex
timeSums int64
)
func main() {
// Start the goroutine that will sum the current time
// once per second.
go runDataLoop()
// Create a handler that will read-lock the mutext and
// write the summed time to the client
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
timeSumsMu.RLock()
defer timeSumsMu.RUnlock()
fmt.Fprint(w, timeSums)
})
http.ListenAndServe(":8080", nil)
}
func runDataLoop() {
for {
// Within an infinite loop, lock the mutex and
// increment our value, then sleep for 1 second until
// the next time we need to get a value.
timeSumsMu.Lock()
timeSums += time.Now().Unix()
timeSumsMu.Unlock()
time.Sleep(1 * time.Second)
}
}