I am trying to configure a RabbitMQ broker using the go-micro framework. I have noticed that the broker interface in go-micro has a broker.SubscriberOptions struct which allows configuring the parameters I am looking for (AutoAck, Queue name and so on) however I am unable to figure out how to pass this when starting a broker.
This is how a simple rabbit go-micro setup would look like
package main
import (
"log"
"github.com/micro/go-micro/server"
"github.com/micro/go-plugins/broker/rabbitmq"
micro "github.com/micro/go-micro"
)
func main() {
// Create a new service. Optionally include some options here.
service := micro.NewService(
micro.Name("go-micro-rabbit"),
micro.Broker(rabbitmq.NewBroker()),
)
// Init will parse the command line flags.
service.Init()
// Register handler
proto.RegisterGreeterHandler(service.Server(), new(Greeter))
micro.RegisterSubscriber("micro-exchange", service.Server(), myFunc, server.SubscriberQueue("my-queue"))
// Run the server
if err := service.Run(); err != nil {
log.Fatal(err)
}
}
The micro.RegisterSubscriber method takes in a list of server.SubscriberOption but does not allow me to set the broker.SubscriberOptions and the rabbitmq.NewBroker allows setting broker.Options but once again, not broker.SubscriberOptions
I have dug in the code of go-micro but have been unable to figure out how the broker.Subscribe method (Which exposes the correct struct) is called or by who.
Is this possible at all? Is it maybe something not yet fully fleshed out in the API?
Related
I have an existing TCP connection (net.Conn) on which I am trying to use (develop) an RPC client-server application.
So far, I explored gRPC and net/rpc in the standard lib; unless I am missing something, both seem to need full control over the connection(s).
I could just go for a custom implementation with gob to serialize requests/replies but since I also require a decent error handling (at least for errors I am generating on the server side) I am trying to see if there is some existing framework I could use.
Any pointers/ideas?
After some more investigation, I realized I was wrong about net/rpc. Apologies for that.
You can call rpc.ServeConn on server side and rpc.NewClient on client-side passing an existing connection, as long as it implements io.ReadWriteCloser
Example server:
package main
import (
"net/rpc"
"os"
"myapp/pkg"
)
func main() {
fsops := new(pkg.FSOps)
s := rpc.NewServer()
s.Register(fsops)
s.ServeConn(pkg.RWCloser{os.Stdin, os.Stdout})
}
Example client:
package main
import (
"net/rpc"
"os"
"myapp/pkg"
)
func main() {
client := rpc.NewClient(pkg.RWCloser{os.Stdin, os.Stdout})
var reply pkg.XXXReply
err := client.Call("FSOps.XXX", "args", &reply)
}
I am trying to integrate Elastic APM and Sentry into my website using Buffalo. The interesting files are as follows:
handlers/sentryHandler.go
package handlers
import (
sentryhttp "github.com/getsentry/sentry-go/http"
"github.com/gobuffalo/buffalo"
)
func SentryHandler(next buffalo.Handler) buffalo.Handler {
handler := buffalo.WrapBuffaloHandler(next)
sentryHandler := sentryhttp.New(sentryhttp.Options{})
return buffalo.WrapHandler(sentryHandler.Handle(handler))
}
handlers/elasticAPMHandler.go
package handlers
import (
"fmt"
"github.com/gobuffalo/buffalo"
"go.elastic.co/apm/module/apmhttp"
)
func ElasticAPMHandler(next buffalo.Handler) buffalo.Handler {
fmt.Println("AAA")
handler := apmhttp.Wrap(buffalo.WrapBuffaloHandler(next))
return buffalo.WrapHandler(handler)
}
actions/app.go
package actions
import (
"github.com/gobuffalo/buffalo"
"github.com/gobuffalo/envy"
forcessl "github.com/gobuffalo/mw-forcessl"
paramlogger "github.com/gobuffalo/mw-paramlogger"
"github.com/unrolled/secure"
"my_website/handlers"
"my_website/models"
"github.com/gobuffalo/buffalo-pop/pop/popmw"
csrf "github.com/gobuffalo/mw-csrf"
i18n "github.com/gobuffalo/mw-i18n"
"github.com/gobuffalo/packr/v2"
)
func App() *buffalo.App {
if app == nil {
app = buffalo.New(buffalo.Options{
Env: ENV,
SessionName: "_my_website_session",
})
// Automatically redirect to SSL
app.Use(forceSSL())
// Catch errors and send them to Sentry.
app.Use(handlers.SentryHandler)
// Get tracing information and send it to Elastic.
app.Use(handlers.ElasticAPMHandler)
// Other Buffalo middleware stuff goes here...
// Routing stuff goes here...
}
return app
}
The problem I'm running into is if I have the Sentry/APM handlers at the top, then I get errors like application.html: line 24: "showPagePath": unknown identifier. However, if I move it to just before I set up the routes, then I get a no transaction found error. So, I'm guessing that the handler wrappers are dropping the buffalo.Context information. So, what would I need to do to be able to integrate Sentry and Elastic in Buffalo asides from trying to reimplement their wrappers?
So, I'm guessing that the handler wrappers are dropping the buffalo.Context information.
That's correct. The problem is that buffalo.WrapHandler (Source) throws away all of the context other than the underlying http.Request/http.Response:
// WrapHandler wraps a standard http.Handler and transforms it
// into a buffalo.Handler.
func WrapHandler(h http.Handler) Handler {
return func(c Context) error {
h.ServeHTTP(c.Response(), c.Request())
return nil
}
}
So, what would I need to do to be able to integrate Sentry and Elastic in Buffalo asides from trying to reimplement their wrappers?
I can see two options:
Reimplement buffalo.WrapHandler/buffalo.WrapBuffaloHandler to stop throwing away the buffalo.Context. This would involve storing the buffalo.Context in the underlying http.Request's context, and then pulling it out again on the other side instead of creating a whole new context.
Implement Buffalo-specific middleware for Sentry and Elastic APM without using the Wrap* functions.
There's an open issue in the Elastic APM agent for the latter option: elastic/apm#39.
Currently running a go service on GCP however in the logs viewer every message is treated as an error.
Is there a generally advised way of logging to stderr and stdout depending on the log level. Ie errors to stderr and anything else to stdout.
I'm currently using the logrus package and have come across this implementation. Other ways i see achieving this while still using the same package is to pass the logger to each package that needs it or to create a global log object, neither of which i am too keen on.
https://github.com/microsoft/fabrikate/pull/252/commits/bd24d62d7c2b851ad6e7b36653eb0a6dc364474b#diff-ed0770fdbf87b0c6d536e33a99a8df9c
You can use Stackdriver library package for GoLang:
go get -u cloud.google.com/go/logging
Then you can use StandardLogger:
// Sample stdlogging writes log.Logger logs to the Stackdriver Logging.
package main
import (
"context"
"log"
"cloud.google.com/go/logging"
)
func main() {
ctx := context.Background()
// Sets your Google Cloud Platform project ID.
projectID := "YOUR_PROJECT_ID"
// Creates a client.
client, err := logging.NewClient(ctx, projectID)
if err != nil {
log.Fatalf("Failed to create client: %v", err)
}
defer client.Close()
// Sets the name of the log to write to.
logName := "my-log"
logger := client.Logger(logName).StandardLogger(logging.Info)
// Logs "hello world", log entry is visible at
// Stackdriver Logs.
logger.Println("hello world")
}
Here you can find documentation on Google Cloud website
Update:
Alternatively you could give a try GCP formatter for logrus
This will not tie your app to Google Cloud Platform. However, it does not mean that on another platform you will not need to change your code to format output.
Using StackDriver library is the recommended solution for Google Cloud.
We use https://github.com/rs/zerolog with the following method called in our Init() method to setup the logging options at global level:
package main
import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"os"
)
// initializeLogging sets up the logging configuration for the service.
// Invoke this method in your Init() method.
func initializeLogging() {
// Set logging options for production development
if os.Getenv("ENV") != "DEV" {
// change the level field name to ensure these are parsed correctly in Stackdriver
zerolog.LevelFieldName = "severity"
// UNIX Time is faster and smaller than most timestamps
zerolog.TimeFieldFormat = zerolog.TimeFormatUnix
} else {
// Set logging options for local development
log.Logger = log.Output(zerolog.ConsoleWriter{Out: os.Stderr})
zerolog.SetGlobalLevel(zerolog.DebugLevel)
}
// Example log
log.Info().Msg("This is how you log at Info level")
}
I recommend the https://github.com/apsystole/log. You can swap either log and logrus imports for it. It is a small zero-dependency module, hence much lighter than logrus.
I'm using the Operator SDK to build a custom Kubernetes operator. I have created a custom resource definition and a controller using the respective Operator SDK commands:
operator-sdk add api --api-version example.com/v1alpha1 --kind=Example
operator-sdk add controller --api-version example.com/v1alpha1 --kind=Example
Within the main reconciliation loop (for the example above, the auto-generated ReconcileExample.Reconcile method) I have some custom business logic that requires me to query the Kubernetes API for other objects of the same kind that have a certain field value. It's occurred to me that I might be able to use the default API client (that is provided by the controller) with a custom field selector:
func (r *ReconcileExample) Reconcile(request reconcile.Request) (reconcile.Result, error) {
ctx := context.TODO()
listOptions := client.ListOptions{
FieldSelector: fields.SelectorFromSet(fields.Set{"spec.someField": "someValue"}),
Namespace: request.Namespace,
}
otherExamples := v1alpha1.ExampleList{}
if err := r.client.List(ctx, &listOptions, &otherExamples); err != nil {
return reconcile.Result{}, err
}
// do stuff...
return reconcile.Result{}, nil
}
When I run the operator and create a new Example resource, the operator fails with the following error message:
{"level":"info","ts":1563388786.825384,"logger":"controller_example","msg":"Reconciling Example","Request.Namespace":"default","Request.Name":"example-test"}
{"level":"error","ts":1563388786.8255732,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"example-controller","request":"default/example-test","error":"Index with name field:spec.someField does not exist","stacktrace":"..."}
The most important part being
Index with name field:spec.someField does not exist
I've already searched the Operator SDK's documentation on the default API client and learned a bit about the inner workings of the client, but no detailed explanation on this error or how to fix it.
What does this error message mean, and how can I create this missing index to efficiently list objects by this field value?
The default API client that is provided by the controller is a split client -- it serves Get and List requests from a locally-held cache and forwards other methods like Create and Update directly to the Kubernetes API server. This is also explained in the respective documentation:
The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller's Reconciler will be populated with the Manager's Client which is a split-client. [...] A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.
To query values from the cache using a custom field selector, the cache needs to have a search index for this field. This indexer can be defined right after the cache has been set up.
To register a custom indexer, add the following code into the bootstrapping logic of the operator (in the auto-generated code, this is done directly in main). This needs to be done after the controller manager has been instantiated (manager.New) and also after the custom API types have been added to the runtime.Scheme:
package main
import (
k8sruntime "k8s.io/apimachinery/pkg/runtime"
"example.com/example-operator/pkg/apis/example/v1alpha1"
// ...
)
function main() {
// ...
cache := mgr.GetCache()
indexFunc := func(obj k8sruntime.Object) []string {
return []string{obj.(*v1alpha1.Example).Spec.SomeField}
}
if err := cache.IndexField(&v1alpha1.Example{}, "spec.someField", indexFunc); err != nil {
panic(err)
}
// ...
}
When a respective indexer function is defined, field selectors on spec.someField will work from the local cache as expected.
Coming from a Java background, I have some questions on how things are typically done in Golang. I am specifically talking about services and dao's/repositories.
In java, I would use dependency injection (probably as singleton/application-scoped), and have a Service injected into my rest endpoint / resource.
To give a bit more context. Imagine the following Golang code:
func main() {
http.ListenAndServe("localhost:8080", nil)
}
func init() {
r := httptreemux.New()
api := r.NewGroup("/api/v1")
api.GET("/blogs", GetAllBlogs)
http.Handle("/", r)
}
Copied this directly from my code, main and init are split because google app engine.
So for now I have one handler. In that handler, I expect to interact with a BlogService.
The question is, where, and in what scope should I instantiate a BlogService struct and a dao like datastructure?
Should I do it everytime the handler is triggered, or make it constant/global?
For completeness, here is the handler and blogService:
// GetAllBlogs Retrieves all blogs from GCloud datastore
func GetAllBlogs(w http.ResponseWriter, req *http.Request, params map[string]string) {
c := appengine.NewContext(req)
// need a reference to Blog Service at this point, where to instantiate?
}
type blogService struct{}
// Blog contains the content and meta data for a blog post.
type Blog struct {...}
// newBlogService constructs a new service to operate on Blogs.
func newBlogService() *blogService {
return &blogService{}
}
func (s *blogService) ListBlogs(ctx context.Context) ([]*Blog, error) {
// Do some dao-ey / repository things, where to instantiate BlogDao?
}
You can use context.Context to pass request scoped values into your handlers (available in Go 1.7) , if you build all your required dependencies during the request/response cycle (which you should to avoid race conditions, except for dependencies that manage concurrency on their own like sql.DB). Put all your services into a single container for instance, then query the context for that value :
container := request.Context.Value("container").(*Container)
blogs,err := container.GetBlogService().ListBlogs()
read the following material :
https://golang.org/pkg/context/
https://golang.org/pkg/net/http/#Request.Context