How can I achieve MDC Logging (Java) in GoLang?
I need to add UUIDs in all server logs in order to be able to trace concurrent requests.
Java MDC relies on thread local storage, something Go does not have.
The closest thing is to thread a Context through your stack.
This is what more and more libraries are doing in Go.
A somewhat typical way is to do this via a middleware package that adds a request id to the context of a web request, like:
req = req.WithContext(context.WithValue(req.Context(),"requestId",ID))
Then, assuming you pass the context around, you pull it out with ctx.Value("requestId") and use it wherever it makes sense.
Possibly making your own custom logger function like:
func logStuff(ctx context.Context, msg string) {
log.Println(ctx.Value("requestId"),msg) // call stdlib logger
}
There's a bunch of ways you may want to handle this, but that's a fairly simple form.
Related
TL;DR
What can I do to make two services (rabbitMQ consumer + HTTP server) share the same map?
More info
I'm new to Golang. Here's what I'm trying to achieve:
I have a RabbitMQ consumer receiving some json-format messages and store them into a concurrent map. On the other hand, I need an HTTP server that sends data from the concurrent map whenever a GET request arrives.
I kinda know that I need the"net/http" package for the HTTP server and the rabbitMQ client package.
However, I'm not sure how these two services can share the same map. Could anyone please offer some idea? Thank you in advance!
EDIT
One possible solution I can think of is to replace the concurrent map with Redis. So the running consumer will send the data to Redis server whenever a message arrives and then the http server will serve GET request from the data in Redis. But is there a better way to achieve my goal without adding this extra layer (Redis)?
Assuming that your two "services" live inside the same Go program, dependency injection. You can define a type that wraps your map (or provides equivalent functionality), instantiate it when your application starts, and inject it into both the HTTP handler and the MQ consumer.
The following code is meant to illustrate the concept:
package mymap
// imports
type MyMap struct {
// fields
}
package main
// imports
func main() {
...
// instantiate the shared map object
sharedMap := &MyMap{ /* init fields as appropriate */ }
mqconsumer := &mqpkg.Consumer{
SharedMap: sharedMap // inject the map obj into the mq consumer
// ...
}
// start the mq consumer
// inject the map obj into the http handler
http.HandleFunc("/foo", handlerWithMap(sharedMap))
log.Fatal(http.ListenAndServe(":8080", nil))
}
func handlerWithMap(mymap *mymap.MyMap) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// here the http handler is able to access the shared map object
}
}
With that said, unless your application has particular requirements, I would recommend to implement your own synchronized map. This isn't too difficult to accomplish with the sync package. The disadvantage of using third-party libraries is that you lose type safety, because their signatures must be designed to accept and return interface{}'s.
I'm using golang logrus for logging and I'm having a wrapper with all the regular functions like Info(..),Infof(..) e.t.c I want to implement a wrapper function Audit(..) for logging to syslog.
I noticed logrus syslog hooks problem is, once it got hooked every log function is logging to syslog, also Infof(..) which I don't want them to.
Is there a way I can call syslog by demand? other than:
func (l *WrapLogger) Audit(msg string) {
l.logger.AddHook(syslogHook)
l.logger.Info(msg)
l.logger.ReplaceHooks(logrus.LevelHooks) // removing somehow the hook
}
Thanks
If you're trying to delegate what message to send by its log level then you can do it by setting the log levels the hook accepts.
For example:
log.AddHook(&writer.Hook{
Writer: os.Stderr,
LogLevels: []log.Level{ log.WarnLevel },
})
log.AddHook(lSyslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, ""))
log.Info("This will go to syslog")
log.Warn("This will go to stderr")
If you want to route this no according to the log level then what you suggested may work but it feels odd and may have race-conditions.
What I suggest you to do is create your own hook that gets hook list and route to the right hook(s) according to the message or to the fields that are passed when calling Info, Warn and etc.
I have a large Go program that is spread across 50+ miscellaneous Go files in the root of my package folder. I know that this is considered terrible, so I've decided to embark upon splitting up the program into some subpackages for better organization.
Unfortunately, after splitting off the logical parts of my programs into subpackages, I'm running into the dreaded "import cycle not allowed" error. This is because the Go compiler refuses to compile anything with circular imports. But the different logical parts of my program need to communicate with each other...
I've done some research online and found some excellent resources, like this excellent StackOverflow question that attempts to explain what to think about to solve this problem at a high level.
My apologies, but this post is way over my head, and I was wondering if someone could spell out an exact solution for my specific code situation, and hopefully in simpler language aimed at a complete beginner to Go.
A brief description of how my code is organized and what it does:
It connects to 3 different servers using 3 different protocols (Twitch.tv, Discord, and a custom WebSocket server).
It seems obvious to make 3 subpackages, one for each server type, and then initialize all of them in a main.go file.
Each subpackage is not just an interface; it contains a collection of global variables (that track the connection + other things) and a bunch of functions. (Note that I can refactor this such that its all contained within one giant interface, if necessary.)
95% of the time, the subpackages receive messages from their individual servers and send messages back to their individual servers, so the subpackages are mostly compartmentalized.
However, sometimes the Twitch.tv module needs to send a message to the Discord server, and the Discord server needs to send a message to the Twitch.tv server. So the Discord server needs to be able to call the "Send()" functions inside the Twitch.tv subpackage, and the Twitch.tv subpackage needs to be able to call the "Send()" function of the Discord subpackage! So this is where my circular problem comes from.
It looks like you want to keep your protocol specific code in separate packages.
If you don't want much refactor, I'd suggest you to create a package with dispatcher. Each server imports dispatcher package and register a handler for specific protocol. When it needs to call another server, just send a message via dispatcher to specified handler.
In addition to the channel-based approaches proposed by TechSphinX and Oleg, you can use an interface-based approach and simple dependency injection.
You can use a setup function, probably in or called from main(), that creates instances of each service client. These should each implement Send() and have fields for the other clients they need to use. Create a Sender interface in its own package, and put your message struct in there as well.
After creating the instances, you can then set the other clients on each instance. This way they can send to whatever they need to send to, without circular dependencies. You can even put all the clients into a struct to make the injection easier.
For example:
// pkg sender
type Sender interface {
Send(m Message) error // or whatever it needs to be
}
type Message struct {
// Whatever goes in a message
}
type Dispatcher struct {
TwitchClient Sender
DiscordClient Sender
WebClient Sender
}
// pkg main
func setup() {
d := sender.Dispatcher{
TwitchClient: twitch.New(),
DiscordClient: discord.New(),
WebClient: web.New(),
}
d.TwitchClient.Dispatcher = d
d.DiscordClient.Dispatcher = d
d.WebClient.Dispatcher = d
}
// pkg twitch
type TwitchClient struct {
Dispatcher sender.Dispatcher
// other fields ...
}
func New() *TwitchClient {
return new(TwitchClient) // or whatever
}
func (t *TwitchClient) Send(m sender.Message) error {
// send twitch message...
// Need to send a Discord message?
t.Dispatcher.DiscordClient.Send(m)
}
Tailored to your particular case:
From what you describe the only reason for the packages to import each other is that they need to call each others Send() functions.
Channels to communicate
Create channel(s) in main and give it to both packages on init. Then they can communicate with each other without knowing of each others existence.
It sounds like the server/protocol packages are useful on their own, and the requirement to send a message from one kind of a server to another kind is a feature of your specific application. In other words, the server/protocol packages don't need to send messages to each other, your application needs to.
I usually put application-specific functionality into an app package. Package app can import all your protocol packages.
You can also do this in package main, but I've found an app package to be a more useful instrument. (My main package is usually just the single main.go file.)
There is this excellent blog post by Jack Lindamood How to correctly use context.Context in Go 1.7 which boils down to the following money quote:
Context.Value should inform, not control. This is the primary mantra that I feel should guide if you are using context.Value correctly. The
content of context.Value is for maintainers not users. It should never
be required input for documented or expected results.
Currently, I am using Context to transport the following information:
RequestID which is generated on the client-side passed to the Go backend and it solely travels through the command-chain and is then inserted in the response again. Without the RequestID in the response, the client-side would break though.
SessionID identifies the WebSocket session, this is important when certain responses are generated in asynchronous computations (e.g. worker queues) in order to identify on which WebSocket session the response should be send.
When taking the definition very seriously I would say both violate the intention of context.Context but then again their values do not change any behavior while the whole request is made, it's only relevant when generating the response.
What's the alternative? Having the context.Context for metadata in the server API actually helps to maintain lean method signatures because this data is really irrelevant to the API but only important for the transport layer which is why I am reluctant to create something like a request struct:
type Request struct {
RequestID string
SessionID string
}
and make it part of every API method which solely exists to be passed through before sending a response.
Based on my understanding context should be limited to passing things like request or session ID. In my application, I do something like below in one of my middleware. Helps with observability
if next != nil {
if requestID != "" {
b := context.WithValue(r.Context(), "requestId", requestID)
r = r.WithContext(b)
}
next.ServeHTTP(w, r)
}
In Go using the standard net/rpc functionality, I would like to determine what the IP address an inbound RPC request is coming from. The underlying http functionality appears to provide this in the http.Request object, but I cannot see any way of getting at that from the default RPC handler (set using rpc.HandleHTTP).
Is there some hidden mechanism for getting at the underlying http.Request, or do I have to do something fancier with setting up a different HTTP responder?
As far as I know, it is not possible to grab the address from somewhere in the default server.
The service call method, which calls the request receiving function, does not provide any access to the remote data stored in the codec.
If http handlers could be registered twice (which they can't), you could have overwritten the DefaultRPCPath for the HTTP Handler setup by HandleHTTP. But that's simply not possible today.
What you can do, without much fuss, is to build a RPC server based on the default one with your own ServeHTTP method:
import (
"log"
"net"
"net/http"
"net/rpc"
)
type myRPCServer struct {
*rpc.Server
}
func (r *myRPCServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
log.Println(req.RemoteAddr)
r.Server.ServeHTTP(w, req)
}
func (r *myRPCServer) HandleHTTP(rpcPath, debugPath string) {
http.Handle(rpcPath, r)
}
func main() {
srv := &myRPCServer{rpc.NewServer()}
srv.HandleHTTP(rpc.DefaultRPCPath, rpc.DefaultDebugPath)
// ...http listen code...
}
The downside of this, is of course, that you can't use rpc.Register anymore. You have to write srv.Register.
Edit: I forgot that you'd need to write your own HandleHTTP as well. The reason for this is, that if you embed the RPC server and you write srv.HandleHTTP it is called on the embedded instance, passing the embedded instance to http.Handle(), ignoring your own definition of ServeHTTP. This has the drawback, that you won't have the ability to debug your RPC server using the debug path, as the server's HandleHTTP uses a private debug handler (rpc.debugHTTP) which you can't access.
You can also use https://github.com/valyala/gorpc instead of net/rpc, which passes client address to RPC server handler - see http://godoc.org/github.com/valyala/gorpc#HandlerFunc for details.
The net/rpc package is at a higher level of abstraction than tcp or http. Since it can use multiple codecs it doesn't make sense for it to offer a way to get at the ip address of the inbound rpc. It's theoretically possible someone could implement a code that talks on unix sockets instead or using radio transmitters.
If you want access to specifics of the transport layer you will have to drop a level in the stack and use net or net/http directory to make your rpc service.
It seems that there is currently no way to do this in rpc function.
See this link for more info
Here is a summary.
Q:
Right now RemoteAddr() method can be called to get the RPC client's address only on
net.Conn when the client dials to server, but suppose that your server has multiple
clients connected and each of this clients are calling an RPC exported method. Is there
a way to implement a method to get the caller's remote address from inside the RPC
method?
func (t *Type) Method(args *Args, reply *string) error {
//something like
*reply = Caller.RemoteAddr().String()
// who called the method now?
return nil
}
A:
I'm skeptical. It would take an API change (not necessarily a backwards incompatible one
but still a bit of a redesign) to supply this.