I am looking for a way to rate-limit RPCs separately with high granularity, and to my dismay, there are not many options available for this issue. I am trying to replace a REST API with gRPC, and one of the most important features for me was the ability to add middleware for each route. Unfortunately, go-grpc-middleware only applies middleware to an entire server.
In my imagination, an ideal rate-limiting middleware for gRPC would use similar tricks as go-proto-validators, where the proto file would contain configurations for the ratelimiting itself.
Figured I could post a snippet for reference of how this would look like in practice, using go-grpc-middleware WithUnaryServerChain and a unary interceptor.
The idea is to add a grpc.UnaryInterceptor to the server, which will be invoked with an instance of *grpc.UnaryServerInfo. This object exports the field FullMethod, which holds the qualified name of the RPC method being called.
In the interceptor you can then implement arbitrary code before actually calling the RPC handler, including RPC-specific rate limiting logic.
// import grpc_middleware "github.com/grpc-ecosystem/go-grpc-middleware"
// import "google.golang.org/grpc"
grpcServer := grpc.NewServer(
// WithUnaryServerChain is a grpc.Server config option that accepts multiple unary interceptors.
grpc_middleware.WithUnaryServerChain(
// UnaryServerInterceptor provides a hook to intercept the execution of a unary RPC on the server. info
// contains all the information of this RPC the interceptor can operate on. And handler is the wrapper
// of the service method implementation. It is the responsibility of the interceptor to invoke handler
// to complete the RPC.
grpc.UnaryInterceptor(func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
// FullMethod is the full RPC method string, i.e., /package.service/method.
switch info.FullMethod {
case "/mypackage.someservice/DoThings":
// ... rate limiting code
// if all is good, then call the handler
return handler(ctx, req)
}
}),
// other `grpc.ServerOption` opts
),
)
Related
I’m currently maintaining a few HTTP APIs based on the standard library and gorilla mux and running in kubernetes (GKE).
We’ve adopted the http.TimeoutHandler as our “standard” way to have a consistent timeout error management.
A typical endpoint implementation will use the following “chain”:
MonitoringMiddleware => TimeoutMiddleware => … => handler
so that we can monitor a few key metrics per endpoint.
One of our API is typically used in a “fire and forget” mode meaning that clients will push some data and not care for the API response. We are facing the issue that
the Golang standard HTTP server will cancel a request context when the client connection is no longer active (godoc)
the TimeoutHandler will return a “timeout” response whenever the request context is done (see code)
This means that we are not processing requests to completion when the client disconnects which is not what we want and I’m therefore looking for solutions.
The only discussion I could find that somewhat relates to my issue is https://github.com/golang/go/issues/18527; however
The workaround is your application can ignore the Handler's Request.Context()
would mean that the monitoring middleware would not report the "proper" status since the Handler would perform the request processing in its goroutine but the TimeoutHandler would be enforcing the status and observability would be broken.
For now, I’m not considering removing our middlewares as they’re helpful to have consistency across our APIs both in terms of behaviours and observability. My conclusion so far is that I need to “fork” the TimeoutHandler and use a custom context for when an handler should not depend on the client waiting for the response or not.
The gist of my current idea is to have:
type TimeoutHandler struct {
handler Handler
body string
dt time.Duration
// BaseContext optionally specifies a function that returns
// the base context for controling if the server request processing.
// If BaseContext is nil, the default is req.Context().
// If non-nil, it must return a non-nil context.
BaseContext func(*http.Request) context.Context
}
func (h *TimeoutHandler) ServeHTTP(w ResponseWriter, r *Request) {
reqCtx := r.Context()
if h.BaseContext != nil {
reqCtx = h.BaseContext(r)
}
ctx, cancelCtx := context.WithTimeout(reqCtx, h.dt)
defer cancelCtx()
r = r.WithContext(ctx)
...
case <-reqCtx.Done():
tw.mu.Lock()
defer tw.mu.Unlock()
w.WriteHeader(499) // write status for monitoring;
// no need to write a body since no client is listening.
case <-ctx.Done():
tw.mu.Lock()
defer tw.mu.Unlock()
w.WriteHeader(StatusServiceUnavailable)
io.WriteString(w, h.errorBody())
tw.timedOut = true
}
The middleware BaseContext callback would return context.Background() for requests to the “fire and forget” endpoint.
One thing I don’t like is that in doing so I’m losing any context keys written so this new middleware would have strong usage constraints. Overall I feel like this is more complex than it should be.
Am I completely missing something obvious?
Any feedback on API instrumentation (maybe our middlewares are an antipattern) /fire and forget implementations would be welcomed!
EDIT: as most comments are that a request for which the client does not wait for a response has unspecified behavior, I checked for more information on typical clients for which this happens.
From our logs, this happens for user agents that seem to be mobile devices. I can imagine that connections can be much more unstable and the problem will likely not disappear.
I would therefore not conclude that I shouldn't find a solution since this is currently creating false-positive alerts.
TL;DR
What can I do to make two services (rabbitMQ consumer + HTTP server) share the same map?
More info
I'm new to Golang. Here's what I'm trying to achieve:
I have a RabbitMQ consumer receiving some json-format messages and store them into a concurrent map. On the other hand, I need an HTTP server that sends data from the concurrent map whenever a GET request arrives.
I kinda know that I need the"net/http" package for the HTTP server and the rabbitMQ client package.
However, I'm not sure how these two services can share the same map. Could anyone please offer some idea? Thank you in advance!
EDIT
One possible solution I can think of is to replace the concurrent map with Redis. So the running consumer will send the data to Redis server whenever a message arrives and then the http server will serve GET request from the data in Redis. But is there a better way to achieve my goal without adding this extra layer (Redis)?
Assuming that your two "services" live inside the same Go program, dependency injection. You can define a type that wraps your map (or provides equivalent functionality), instantiate it when your application starts, and inject it into both the HTTP handler and the MQ consumer.
The following code is meant to illustrate the concept:
package mymap
// imports
type MyMap struct {
// fields
}
package main
// imports
func main() {
...
// instantiate the shared map object
sharedMap := &MyMap{ /* init fields as appropriate */ }
mqconsumer := &mqpkg.Consumer{
SharedMap: sharedMap // inject the map obj into the mq consumer
// ...
}
// start the mq consumer
// inject the map obj into the http handler
http.HandleFunc("/foo", handlerWithMap(sharedMap))
log.Fatal(http.ListenAndServe(":8080", nil))
}
func handlerWithMap(mymap *mymap.MyMap) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// here the http handler is able to access the shared map object
}
}
With that said, unless your application has particular requirements, I would recommend to implement your own synchronized map. This isn't too difficult to accomplish with the sync package. The disadvantage of using third-party libraries is that you lose type safety, because their signatures must be designed to accept and return interface{}'s.
So I am using a mux that will determine what type of request is incoming (gRPC or REST) and route the request accordingly. For gRPC requests I am attaching a stats handler to collect some metadata but one of the interface methods TagRPC() which we use to tag our metadata is not being called.
app.httpMux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if isgRPC(r) {
app.gRPCServer.ServeHTTP(w, r)
} else {
app.rMux.ServeHTTP(w, r)
}
})
I think the problem is with gRPC's ServeHTTP method not supporting all the methods. If I were to attach TagGRPC() manually where would be a good place to do it?
stats handler is not fully support in ServeHTTP Handler now.
More details in https://github.com/grpc/grpc-go/issues/1784
(I just noticed that you also filed the issue, but this may help other people also interested in this)
I need to embed the default http.Server in my own server struct and customize the Serve method.
The server needs to short circuit the go c.serve() call and only run that line if it has the computing resources available to respond within 50ms. Otherwise the server is just going to send a 204 and move on.
This is almost straightforward.
type PragmaticServer struct {
http.Server
Addr string
Handler http.Handler
}
func (srv *PragmaticServer) Serve(l net.Listener) error {
defer l.Close()
var tempDelay time.Duration // how long to sleep on accept failure
for {
// SNIP for clarity
c, err := srv.newConn(rw)
if err != nil {
continue
}
c.setState(c.rwc, StateNew) // before Serve can return
go c.serve()
}
}
So, again. This almost works. Except that srv.newConn is an unexported method, as is c.serve and c.setState, which means that I end up having to copy and paste pretty much the entirety of net/http in order for this to compile. Which is basically a fork. Is there any better way to do this?
Unfortunately, you're not going to be able to do that without reimplementing most of the Server code. Short of that, we usually intercept the call either just before at conn.Accept, or just after at Handler.ServerHTTP.
The first method is to create a custom net.Listener that filters out connections before they are even handed off to the http.Server. While this can respond faster, and consume fewer resources, it however makes it less convenient to write http responses, and precludes you from limiting requests on already open connections.
The second way to handle this, is to just wrap the handlers and intercept the request before any real work has been done. You most likely want to create a http.Handler to filter the requests, and pass them through to your main handler. This can also be more flexible, since you can filter based on the route, or other request information if you so choose.
In Go using the standard net/rpc functionality, I would like to determine what the IP address an inbound RPC request is coming from. The underlying http functionality appears to provide this in the http.Request object, but I cannot see any way of getting at that from the default RPC handler (set using rpc.HandleHTTP).
Is there some hidden mechanism for getting at the underlying http.Request, or do I have to do something fancier with setting up a different HTTP responder?
As far as I know, it is not possible to grab the address from somewhere in the default server.
The service call method, which calls the request receiving function, does not provide any access to the remote data stored in the codec.
If http handlers could be registered twice (which they can't), you could have overwritten the DefaultRPCPath for the HTTP Handler setup by HandleHTTP. But that's simply not possible today.
What you can do, without much fuss, is to build a RPC server based on the default one with your own ServeHTTP method:
import (
"log"
"net"
"net/http"
"net/rpc"
)
type myRPCServer struct {
*rpc.Server
}
func (r *myRPCServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {
log.Println(req.RemoteAddr)
r.Server.ServeHTTP(w, req)
}
func (r *myRPCServer) HandleHTTP(rpcPath, debugPath string) {
http.Handle(rpcPath, r)
}
func main() {
srv := &myRPCServer{rpc.NewServer()}
srv.HandleHTTP(rpc.DefaultRPCPath, rpc.DefaultDebugPath)
// ...http listen code...
}
The downside of this, is of course, that you can't use rpc.Register anymore. You have to write srv.Register.
Edit: I forgot that you'd need to write your own HandleHTTP as well. The reason for this is, that if you embed the RPC server and you write srv.HandleHTTP it is called on the embedded instance, passing the embedded instance to http.Handle(), ignoring your own definition of ServeHTTP. This has the drawback, that you won't have the ability to debug your RPC server using the debug path, as the server's HandleHTTP uses a private debug handler (rpc.debugHTTP) which you can't access.
You can also use https://github.com/valyala/gorpc instead of net/rpc, which passes client address to RPC server handler - see http://godoc.org/github.com/valyala/gorpc#HandlerFunc for details.
The net/rpc package is at a higher level of abstraction than tcp or http. Since it can use multiple codecs it doesn't make sense for it to offer a way to get at the ip address of the inbound rpc. It's theoretically possible someone could implement a code that talks on unix sockets instead or using radio transmitters.
If you want access to specifics of the transport layer you will have to drop a level in the stack and use net or net/http directory to make your rpc service.
It seems that there is currently no way to do this in rpc function.
See this link for more info
Here is a summary.
Q:
Right now RemoteAddr() method can be called to get the RPC client's address only on
net.Conn when the client dials to server, but suppose that your server has multiple
clients connected and each of this clients are calling an RPC exported method. Is there
a way to implement a method to get the caller's remote address from inside the RPC
method?
func (t *Type) Method(args *Args, reply *string) error {
//something like
*reply = Caller.RemoteAddr().String()
// who called the method now?
return nil
}
A:
I'm skeptical. It would take an API change (not necessarily a backwards incompatible one
but still a bit of a redesign) to supply this.