I have an audit log middleware which trying to log details of each grpc function call. Currently I can get request method and url. But I'm not able to get the values in Post body.
Also this middleware is use to log multiple functions which contains different post body. Is this possible?
This is what I have now:
package main
func main() {
e := echo.New()
e.Use(grpcGatewayMiddleware(mux, func(c echo.Context) bool {
return strings.HasPrefix(c.Path(), "/skip")
}))
e.Use(logMiddleware)
...
}
func Middleware(config auth.Config) echo.MiddlewareFunc {
return func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) (err error) {
message := Message {
Method: c.Request().Method,
Url: c.Request().RequestURI,
Prams: c.Request().Form.Encode(),
}
log(message)
}
}
}
func grpcGatewayMiddleware(mux *runtime.ServeMux, skipper middleware.Skipper) echo.MiddlewareFunc {
if skipper == nil {
skipper = middleware.DefaultSkipper
}
return func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
if skipper(c) {
return next(c)
}
mux.ServeHTTP(c.Response(), c.Request())
return next(c)
}
}
}
Related
I am working on an existing application which is written in Go using framework such as gin, middleware. This application uses https://pkg.go.dev/log for logging.
I am trying to add a request id to the log for the API call trace.
main.go
// Creates a router without any middleware by default
r := gin.New()
r.Use(middleware.RequestID())
middleware.go
func (m *Middleware) CheckApiToken(allowWithoutAccessKey bool, validateTonce ...bool) gin.HandlerFunc {
return func(c *gin.Context) {
// Validate
.....
.....
logger.InitializeContext(c)
c.Next()
}
}
}
//RequestID is a middleware that injects a 'RequestID' into the context and header of each request.
func (m *Middleware) RequestID() gin.HandlerFunc {
return func(c *gin.Context) {
xRequestID := uuid.NewV4().String()
c.Request.Header.Set(logger.XRequestIDKey, xRequestID)
fmt.Printf("[GIN-debug] %s [%s] - \"%s %s\"\n", time.Now().Format(time.RFC3339), xRequestID, c.Request.Method, c.Request.URL.Path)
c.Next()
}
}
logger.go
const (
XRequestIDKey = "X-Request-ID"
)
var (
infoLogger *log.Logger
errorLogger *log.Logger
context *gin.Context
)
func init() {
infoLogger = log.New(os.Stdout, "", 0)
errorLogger = log.New(os.Stderr, "", 0)
}
// InitializeContext initialize golbal gin context to logger
func InitializeContext(c *gin.Context) {
context = c
}
//Check if the request id present in the context.
func getRequestID() interface{} {
if context != nil {
if context.Request != nil {
requestID := context.Request.Header.Get(XRequestIDKey)
if requestID != "" {
return requestID
}
}
}
return ""
}
func Info(entry Input) {
infoLogger.Println(getJSON(getRequestID(), msg))
}
This does not work in multi-threaded environment. How do I fix this solution to fix this in multi-threaded environment.
You cannot save the context in a global variable. Context is by definition local to that execution, and at any given moment, there will be multiple of them.
You can store the generated ID in the gin context:
func (m *Middleware) RequestID() gin.HandlerFunc {
return func(c *gin.Context) {
xRequestID := uuid.NewV4().String()
c.Set("requestId",xRequestID)
fmt.Printf("[GIN-debug] %s [%s] - \"%s %s\"\n", time.Now().Format(time.RFC3339), xRequestID, c.Request.Method, c.Request.URL.Path)
c.Next()
}
}
Then you can use the ID stored in the context with the custom log formatter:
router.Use(gin.LoggerWithFormatter(func(param gin.LogFormatterParams) string {
return fmt.Sprintf("%s ...",
param.Keys["requestId"],
...
)
}))
Or if you need to use a different logging library, you can write a wrapper:
func LogInfo(ctx *gin.Context,msg string) {
id:=ctx.Get("requestId")
// Log msg with ID
}
Many log libraries offer methods to get a logger with some parameters already set. For instance, if you use zerolog:
logger:=log.Info().With().Str("requestId",ctx.Get("requestId")).Logger()
logger.Info().Msg("This log msg will contain the request id")
I'm developing an API server with the Echo HTTP framework. I woudld like to filter some requests by IP address.
Later I can manager these URLs better.
It's my code:
func filterIP(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
fmt.Println("c.RealIP()=", c.RealIP())
fmt.Println("c.Path()", c.Path())
if isFilterIp(c.RealIP(), c.Path()) {
return echo.NewHTTPError(http.StatusUnauthorized,
fmt.Sprintf("IP address %s not allowed", c.RealIP()))
}
return next(c)
}
}
func main() {
e := echo.New()
filterGroup := e.Group("/filter")
filterGroup.Use(filterIP)
filterGroup.GET("/test", func(c echo.Context) error {
return c.String(http.StatusOK, "test filter")
})
noFilterGroup := e.Group("/noFilter")
noFilterGroup.GET("/test", func(c echo.Context) error {
return c.String(http.StatusOK, "test no filter")
})
e.Logger.Fatal(e.Start(":1323"))
}
And i want to filter Ip address in the url's level instead of group route.
eg: if there are two path: /filter/test01 and /filter/test02,and i want to filter only test01.
Is there some good way to do this?
You can add a middleware for this:
func filterIP(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
// Block requests from localhost.
if c.RealIP() == "127.0.0.1" {
return echo.NewHTTPError(http.StatusUnauthorized,
fmt.Sprintf("IP address %s not allowed", c.RealIP()))
}
return next(c)
}
}
func main() {
e := echo.New()
e.Use(filterIP)
e.GET("/", func(c echo.Context) error {
return c.String(http.StatusOK, "Hello, World!")
})
e.Logger.Fatal(e.Start(":1323"))
}
It's important to use the RealIP() function, otherwise you might end up with the IP address of a proxy.
I'm working with framework GIN and Gin-JWT in Golang.
So far so good, I was able to authorize and authenticate my REST API with JWT following the example in Gin-JWT package.
I'm trying now to implement some kind of Role in my API.
The flow would be:
Login and auth
Create the JWT with inside the userID and the RoleID
When I call a REST API I confront the role associated to the API with the RoleID in JWT to authorized
So far I have this in my main:
jwtAfp := InitJwtMiddleware(db)
afb := r.Group("api/v1/afb")
afb.Use(jwtAfp.MiddlewareFunc())
afb.GET("/ping", afbController.Ping)
and this for the InitJwtMiddleware using Gin-JWT
func InitJwtMiddleware(db *gorm.DB) *jwt.GinJWTMiddleware {
return &jwt.GinJWTMiddleware{
Realm: "afb",
Key: []byte("secret pwd"),
Timeout: time.Hour,
MaxRefresh: time.Hour,
PayloadFunc: func(data interface{}) jwt.MapClaims {
if v, ok := data.(*model.User); ok {
return jwt.MapClaims{
"afb": v.ID,
}
}
return jwt.MapClaims{}
},
Authenticator: func(c *gin.Context) (interface{}, error) {
var loginVals login
if err := c.Bind(&loginVals); err != nil {
return "", jwt.ErrMissingLoginValues
}
email := loginVals.Username
password := loginVals.Password
var u model.User
db.Where("email = ?", email).First(&u)
if service.CheckPasswordHash(password, u.Password) {
return &u, nil
}
return nil, jwt.ErrFailedAuthentication
},
Authorizator: func(data interface{}, c *gin.Context) bool {
claims := jwt.ExtractClaims(c)
v, ok := data.(float64)
if ok && v == claims["afb"] {
return true
}
return false
},
Unauthorized: func(c *gin.Context, code int, message string) {
c.JSON(code, gin.H{
"code": code,
"message": message,
})
},
TokenHeadName: "Bearer",
TimeFunc: time.Now,
}
}
I would like to add the checking on the Role in the Authorizator section but I'm struggling on how i can do this.
I come up with passing in the InitJwtMiddleware(db) function also the role, this will work but I don't like the idea to "instaziate" a GinJWTMiddleware for each ROLE/API. Or if I could know inside the middleware which function (controller) will be called later I can then figure out if authorize or not. But even this solutin sound awkward to me. I think there will be a most elegant solution, any ideas?
You can try this:
https://github.com/kyfk/gin-jwt
It's the simplest auth[orization/entication] library.
The VerifyPerm function could be helpful for role management.
There's a complete example
func main() {
auth, err := jwt.New(jwt.Auth{
SecretKey: []byte("must change here"),
// Authenticator authenticates a request and return jwt.MapClaims
// that contains a user information of the request.
Authenticator: func(c *gin.Context) (jwt.MapClaims, error) {
var loginForm LoginForm
if err := c.ShouldBind(&loginForm); err != nil {
return nil, jwt.ErrorAuthenticationFailed
}
u, ok := authenticate(req.Username, req.Password)
if ok {
return nil, jwt.ErrorAuthenticationFailed
}
return jwt.MapClaims{
"username": u.Username,
"role": u.Role,
}, nil
},
// UserFetcher takes a jwt.MapClaims and return a user object.
UserFetcher: func(c *gin.Context, claims jwt.MapClaims) (interface{}, error) {
username, ok := claims["username"].(string)
if !ok {
return nil, nil
}
return findByUsername(username)
},
})
// some lines
e.Use(jwt.ErrorHandler)
// issue authorization token
e.POST("/login", auth.AuthenticateHandler)
// refresh token expiration
e.POST("/auth/refresh_token", auth.RefreshHandler)
// role management
e.GET("/operator/hello", Operator(auth), SayHello) // this is only for Operator
e.GET("/admin/hello", Admin(auth), SayHello) // this is only for Admin
}
func Operator(m jwt.Auth) gin.HandlerFunc {
return m.VerifyPerm(func(claims jwt.MapClaims) bool {
return role(claims).IsOperator()
})
}
func Admin(m jwt.Auth) gin.HandlerFunc {
return m.VerifyPerm(func(claims jwt.MapClaims) bool {
return role(claims).IsAdmin()
})
}
I'm currently trying to write some middleware to work with fasthttp and fasthttprouter. And I'm stuck.
func jwt(h fasthttprouter.Handle) fasthttprouter.Handle {
myfunc := func(ctx *fasthttp.RequestCtx, _ fasthttprouter.Params) {
fmt.Println(string(ctx.Request.Header.Cookie("Authorization")))
}
return myfunc
}
How do I run the actual handler now? I feel like i'm missing something very simple.
I've read through this blog post: Middleware in Golang. But i'm lost.
Any ideas?
Regards
for example, let us create a middleware function that will handle CORS using:
github.com/buaazp/fasthttprouter and github.com/valyala/fasthttp
var (
corsAllowHeaders = "authorization"
corsAllowMethods = "HEAD,GET,POST,PUT,DELETE,OPTIONS"
corsAllowOrigin = "*"
corsAllowCredentials = "true"
)
func CORS(next fasthttp.RequestHandler) fasthttp.RequestHandler {
return func(ctx *fasthttp.RequestCtx) {
ctx.Response.Header.Set("Access-Control-Allow-Credentials", corsAllowCredentials)
ctx.Response.Header.Set("Access-Control-Allow-Headers", corsAllowHeaders)
ctx.Response.Header.Set("Access-Control-Allow-Methods", corsAllowMethods)
ctx.Response.Header.Set("Access-Control-Allow-Origin", corsAllowOrigin)
next(ctx)
}
}
Now we chain this middleware function on our Index handler and register it on the router.
func Index(ctx *fasthttp.RequestCtx) {
fmt.Fprint(ctx, "some-api")
}
func main() {
router := fasthttprouter.New()
router.GET("/", Index)
if err := fasthttp.ListenAndServe(":8181", CORS(router.Handler)); err != nil {
log.Fatalf("Error in ListenAndServe: %s", err)
}
}
Example of auth middleware for fasthttp & fasthttprouter (new versions)
type Middleware func(h fasthttp.RequestHandler) fasthttp.RequestHandler
type AuthFunc func(ctx *fasthttp.RequestCtx) bool
func NewAuthMiddleware(authFunc AuthFunc) Middleware {
return func(h fasthttp.RequestHandler) fasthttp.RequestHandler {
return func(ctx *fasthttp.RequestCtx) {
result, err: = authFunc(ctx)
if result {
h(ctx)
} else {
ctx.Response.SetStatusCode(fasthttp.StatusUnauthorized)
}
}
}
}
func AuthCheck(ctx *fasthttp.RequestCtx)(bool, error) {
return false; // for example ;)
}
// router
authMiddleware: = middleware.NewAuthMiddleware(security.AuthCheck)
...
router.GET("/protected", authMiddleware(handlers.ProtectedHandler))
I'm new to golang and wanted to experiment with grpc code to get a better understanding of it. In order to do so I followed the example shown here:
https://devicharan.wordpress.com/
The source code is here:
https://github.com/devicharan/basicwebapp
Unfortunately, when I run this code and do a go build I get an error message with the following:
# basicwebapp/proto
proto/CatalogService.pb.go:126: cannot use _CatalogService_GetProductCatalog_Handler (type func(interface {}, context.Context, []byte) (proto.Message, error)) as type grpc.methodHandler in field value
proto/RecommendationService.pb.go:99: cannot use _RecommendationService_GetRecommendations_Handler (type func(interface {}, context.Context, []byte) (proto.Message, error)) as type grpc.methodHandler in field value
I don't know what this means or what I need to change in order to begin finding a fix. Is it a problem with the code itself or with my Go configuration? Also, is there a good debugger for Go that someone can recommend?
Here is the code for CatalogService.pb.go:
// Code generated by protoc-gen-go.
// source: CatalogService.proto
// DO NOT EDIT!
/*
Package protos is a generated protocol buffer package.
It is generated from these files:
CatalogService.proto
Product.proto
RecommendationService.proto
It has these top-level messages:
Category
CatalogResponse
CatalogRequest
*/
package protos
import proto "github.com/golang/protobuf/proto"
import (
context "golang.org/x/net/context"
grpc "google.golang.org/grpc"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConn
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
type Category struct {
CategoryName string `protobuf:"bytes,1,opt,name=categoryName" json:"categoryName,omitempty"`
}
func (m *Category) Reset() { *m = Category{} }
func (m *Category) String() string { return proto.CompactTextString(m) }
func (*Category) ProtoMessage() {}
type CatalogResponse struct {
Products []*Product `protobuf:"bytes,1,rep,name=products" json:"products,omitempty"`
}
func (m *CatalogResponse) Reset() { *m = CatalogResponse{} }
func (m *CatalogResponse) String() string { return proto.CompactTextString(m) }
func (*CatalogResponse) ProtoMessage() {}
func (m *CatalogResponse) GetProducts() []*Product {
if m != nil {
return m.Products
}
return nil
}
type CatalogRequest struct {
Category *Category `protobuf:"bytes,1,opt,name=category" json:"category,omitempty"`
}
func (m *CatalogRequest) Reset() { *m = CatalogRequest{} }
func (m *CatalogRequest) String() string { return proto.CompactTextString(m) }
func (*CatalogRequest) ProtoMessage() {}
func (m *CatalogRequest) GetCategory() *Category {
if m != nil {
return m.Category
}
return nil
}
func init() {
}
// Client API for CatalogService service
type CatalogServiceClient interface {
GetProductCatalog(ctx context.Context, in *CatalogRequest, opts ...grpc.CallOption) (*CatalogResponse, error)
}
type catalogServiceClient struct {
cc *grpc.ClientConn
}
func NewCatalogServiceClient(cc *grpc.ClientConn) CatalogServiceClient {
return &catalogServiceClient{cc}
}
func (c *catalogServiceClient) GetProductCatalog(ctx context.Context, in *CatalogRequest, opts ...grpc.CallOption) (*CatalogResponse, error) {
out := new(CatalogResponse)
err := grpc.Invoke(ctx, "/protos.CatalogService/GetProductCatalog", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
// Server API for CatalogService service
type CatalogServiceServer interface {
GetProductCatalog(context.Context, *CatalogRequest) (*CatalogResponse, error)
}
func RegisterCatalogServiceServer(s *grpc.Server, srv CatalogServiceServer) {
s.RegisterService(&_CatalogService_serviceDesc, srv)
}
func _CatalogService_GetProductCatalog_Handler(srv interface{}, ctx context.Context, buf []byte) (proto.Message, error) {
in := new(CatalogRequest)
if err := proto.Unmarshal(buf, in); err != nil {
return nil, err
}
out, err := srv.(CatalogServiceServer).GetProductCatalog(ctx, in)
if err != nil {
return nil, err
}
return out, nil
}
var _CatalogService_serviceDesc = grpc.ServiceDesc{
ServiceName: "protos.CatalogService",
HandlerType: (*CatalogServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "GetProductCatalog",
Handler: _CatalogService_GetProductCatalog_Handler,
},
},
Streams: []grpc.StreamDesc{},
}
And this is RecommendationService.pg.go
// Code generated by protoc-gen-go.
// source: RecommendationService.proto
// DO NOT EDIT!
package protos
import proto "github.com/golang/protobuf/proto"
import (
context "golang.org/x/net/context"
grpc "google.golang.org/grpc"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConn
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
type RecommendationResponse struct {
Result []*RecommendationResponse_Recommendation `protobuf:"bytes,1,rep,name=result" json:"result,omitempty"`
}
func (m *RecommendationResponse) Reset() { *m = RecommendationResponse{} }
func (m *RecommendationResponse) String() string { return proto.CompactTextString(m) }
func (*RecommendationResponse) ProtoMessage() {}
func (m *RecommendationResponse) GetResult() []*RecommendationResponse_Recommendation {
if m != nil {
return m.Result
}
return nil
}
type RecommendationResponse_Recommendation struct {
Rating int32 `protobuf:"varint,1,opt,name=rating" json:"rating,omitempty"`
Productid int32 `protobuf:"varint,2,opt,name=productid" json:"productid,omitempty"`
}
func (m *RecommendationResponse_Recommendation) Reset() { *m = RecommendationResponse_Recommendation{} }
func (m *RecommendationResponse_Recommendation) String() string { return proto.CompactTextString(m) }
func (*RecommendationResponse_Recommendation) ProtoMessage() {}
func init() {
}
// Client API for RecommendationService service
type RecommendationServiceClient interface {
GetRecommendations(ctx context.Context, in *Product, opts ...grpc.CallOption) (*RecommendationResponse, error)
}
type recommendationServiceClient struct {
cc *grpc.ClientConn
}
func NewRecommendationServiceClient(cc *grpc.ClientConn) RecommendationServiceClient {
return &recommendationServiceClient{cc}
}
func (c *recommendationServiceClient) GetRecommendations(ctx context.Context, in *Product, opts ...grpc.CallOption) (*RecommendationResponse, error) {
out := new(RecommendationResponse)
err := grpc.Invoke(ctx, "/protos.RecommendationService/GetRecommendations", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
// Server API for RecommendationService service
type RecommendationServiceServer interface {
GetRecommendations(context.Context, *Product) (*RecommendationResponse, error)
}
func RegisterRecommendationServiceServer(s *grpc.Server, srv RecommendationServiceServer) {
s.RegisterService(&_RecommendationService_serviceDesc, srv)
}
func _RecommendationService_GetRecommendations_Handler(srv interface{}, ctx context.Context, buf []byte) (proto.Message, error) {
in := new(Product)
if err := proto.Unmarshal(buf, in); err != nil {
return nil, err
}
out, err := srv.(RecommendationServiceServer).GetRecommendations(ctx, in)
if err != nil {
return nil, err
}
return out, nil
}
var _RecommendationService_serviceDesc = grpc.ServiceDesc{
ServiceName: "protos.RecommendationService",
HandlerType: (*RecommendationServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "GetRecommendations",
Handler: _RecommendationService_GetRecommendations_Handler,
},
},
Streams: []grpc.StreamDesc{},
}
Basically your protoc-gen-go doesn't match the version of grpc. So sync them both to the latest version and reinstall protoc-gen-go will resolve the issue:
go get -u github.com/golang/protobuf/
cd github.com/golang/protobuf/
make
go get -u github.com/grpc/grpc-go
For anyone running into the same problem, what I did was simply change how I built the proto files. There is a comment on the blog page that highlights some missing steps, I followed it and did the following protoc command to generate code from the proto files:
protoc --go_out=plugins=grpc:. *.proto
I ran this command in the directory with my proto files and then did a go build on my main.go file, now everything is working fine.
I am having the same issue and reinstalling or the new command doesn't seem to work.
However: If you change
codec grpc.Codec, buf []byte
in the Handler Definition into:
dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor
and
codec.Unmarshal(buf, in)
into
dec(in)
(All in the Handler Definition in your .pb.go file)
it seems to work.
Changing this file is of course not optimal and every time you recompile it will override your changes, but at least there is a workaround until I figure out what I messed up.