I'm trying to use the Go Beam Sdk to create a pipeline processing pubsub messages.
github.com/apache/beam/sdks/v2/go/pkg/beam
I understand that the pubsubio connector is doing external calls working only on dataflow runner.
What if I want to test my pipeline locally ? How would you do that ?
I need to understand what is preventing me to write my own pubsub unbounded source ? (I may not understand how Beam works under the hood, like how does it serialize user defined code to send it to the runner ?)
Tried to do something like that:
package pubsubio
import (
"context"
"fmt"
cloud_pubsub "cloud.google.com/go/pubsub"
"github.com/apache/beam/sdks/v2/go/pkg/beam"
"github.com/apache/beam/sdks/v2/go/pkg/beam/log"
"github.com/apache/beam/sdks/v2/go/pkg/beam/register"
"github.com/apache/beam/sdks/v2/go/pkg/beam/util/pubsubx"
)
func init() {
register.DoFn3x1[context.Context, string, func(*cloud_pubsub.Message), error](&readFn{})
register.Emitter1[*cloud_pubsub.Message]()
}
type ReadConfig struct {
ProjectID string
TopicName string
SubscriptionName string
}
func Read(
scope beam.Scope,
cfg ReadConfig,
) beam.PCollection {
scope = scope.Scope("pubsubio.Read")
col := beam.Create(scope, cfg.SubscriptionName)
return beam.ParDo(scope, newReadFn(cfg.ProjectID, cfg.TopicName), col)
}
type readFn struct {
pubsubFn
TopicName string
}
func newReadFn(projectID, topicName string) *readFn {
return &readFn{
pubsubFn: pubsubFn{
ProjectID: projectID,
},
TopicName: topicName,
}
}
func (fn *readFn) ProcessElement(
ctx context.Context,
subscriptionName string,
emit func(message *cloud_pubsub.Message),
) error {
log.Info(ctx, "[pubsubio.ProcessElement] Reading from pubsub")
_, err := pubsubx.EnsureTopic(ctx, fn.client, fn.TopicName)
if err != nil {
return fmt.Errorf("cannot get topic: %w", err)
}
sub, err := pubsubx.EnsureSubscription(ctx, fn.client, fn.TopicName, subscriptionName)
if err != nil {
return fmt.Errorf("cannot get subscription: %w", err)
}
return sub.Receive(ctx, func(ctx context.Context, message *cloud_pubsub.Message) {
emit(message)
log.Debugf(ctx, "[pubsubio.ProcessElement] Emit msg: %s", message.ID)
message.Ack()
})
}
So basically I created a Read fn that never return, but the rest of my pipeline is never triggered (I must miss something)
go version: 1.19
gin version (or commit ref): 1.8.1
operating system: Ubuntu
I have a saas project which is based upon Rest APIs. All apis are developed in GO using gin package. When the user logs in then I set current user details in the request context so that I can access these details furthere to display some data. However I had a case in which 2 requests hits in parallel & the context values for the 1st request are override with the context values in the 2nd request. Due to this, my data is displaying wrong.
package main
import (
"fmt"
"strings"
"github.com/gin-gonic/gin"
"github.com/golang-jwt/jwt"
)
func main() {
g := gin.Default()
g.Use(ParseJWTToken)
g.GET("/hello/:name", hello)
g.Run(":9000")
}
func hello(c *gin.Context) {
c.Keys = make(map[string]interface{})
c.Keys["current_user_id"] = 10
c.Keys["current_user_name"] = c.Param("name")
fmt.Println(c.Keys)
c.String(200, "Hello %s", c.Param("name"))
}
var role, userName string
var userId float64
func ParseJWTToken(c *gin.Context) {
merchantDatabase := make(map[string]interface{})
if values, _ := c.Request.Header["Authorization"]; len(values) > 0 {
bearer := strings.Split(c.Request.Header["Authorization"][0], "Bearer")
bearerToken := strings.TrimSpace(bearer[1])
var userAgent string
var userAgentCheck bool
if values, _ := c.Request.Header["User-Agent"]; len(values) > 0 {
userAgent = values[0]
}
_ = config.InitKeys()
token, err := jwt.Parse(bearerToken, func(token *jwt.Token) (interface{}, error) {
return config.SignKey, nil
})
if err != nil {
c.Abort()
return
}
if !token.Valid {
c.Abort()
return
}
if len(token.Claims.(jwt.MapClaims)) > 0 {
for key, claim := range token.Claims.(jwt.MapClaims) {
if key == "user_agent" {
if claim == userAgent {
userAgentCheck = true
}
}
if key == "role" {
role = claim.(string)
}
if key == "id" {
userId = claim.(float64)
}
if key == "name" {
userName = claim.(string)
}
}
}
merchantDatabase["userid"] = userId
merchantDatabase["role"] = role
merchantDatabase["username"] = userName
c.Keys = merchantDatabase
if userAgentCheck {
c.Next()
} else {
c.Abort()
return
}
} else {
c.Abort()
return
}
}
This issue is not produced every time for parallel requests.
How can I fix that ?
I have used global variables for the details that were overridden. Declaring these inside the middleware fixed the issue. Find complete thread here: https://github.com/gin-gonic/gin/issues/3437
I have an audit log middleware which trying to log details of each grpc function call. Currently I can get request method and url. But I'm not able to get the values in Post body.
Also this middleware is use to log multiple functions which contains different post body. Is this possible?
This is what I have now:
package main
func main() {
e := echo.New()
e.Use(grpcGatewayMiddleware(mux, func(c echo.Context) bool {
return strings.HasPrefix(c.Path(), "/skip")
}))
e.Use(logMiddleware)
...
}
func Middleware(config auth.Config) echo.MiddlewareFunc {
return func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) (err error) {
message := Message {
Method: c.Request().Method,
Url: c.Request().RequestURI,
Prams: c.Request().Form.Encode(),
}
log(message)
}
}
}
func grpcGatewayMiddleware(mux *runtime.ServeMux, skipper middleware.Skipper) echo.MiddlewareFunc {
if skipper == nil {
skipper = middleware.DefaultSkipper
}
return func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
if skipper(c) {
return next(c)
}
mux.ServeHTTP(c.Response(), c.Request())
return next(c)
}
}
}
I'm currently trying to write some middleware to work with fasthttp and fasthttprouter. And I'm stuck.
func jwt(h fasthttprouter.Handle) fasthttprouter.Handle {
myfunc := func(ctx *fasthttp.RequestCtx, _ fasthttprouter.Params) {
fmt.Println(string(ctx.Request.Header.Cookie("Authorization")))
}
return myfunc
}
How do I run the actual handler now? I feel like i'm missing something very simple.
I've read through this blog post: Middleware in Golang. But i'm lost.
Any ideas?
Regards
for example, let us create a middleware function that will handle CORS using:
github.com/buaazp/fasthttprouter and github.com/valyala/fasthttp
var (
corsAllowHeaders = "authorization"
corsAllowMethods = "HEAD,GET,POST,PUT,DELETE,OPTIONS"
corsAllowOrigin = "*"
corsAllowCredentials = "true"
)
func CORS(next fasthttp.RequestHandler) fasthttp.RequestHandler {
return func(ctx *fasthttp.RequestCtx) {
ctx.Response.Header.Set("Access-Control-Allow-Credentials", corsAllowCredentials)
ctx.Response.Header.Set("Access-Control-Allow-Headers", corsAllowHeaders)
ctx.Response.Header.Set("Access-Control-Allow-Methods", corsAllowMethods)
ctx.Response.Header.Set("Access-Control-Allow-Origin", corsAllowOrigin)
next(ctx)
}
}
Now we chain this middleware function on our Index handler and register it on the router.
func Index(ctx *fasthttp.RequestCtx) {
fmt.Fprint(ctx, "some-api")
}
func main() {
router := fasthttprouter.New()
router.GET("/", Index)
if err := fasthttp.ListenAndServe(":8181", CORS(router.Handler)); err != nil {
log.Fatalf("Error in ListenAndServe: %s", err)
}
}
Example of auth middleware for fasthttp & fasthttprouter (new versions)
type Middleware func(h fasthttp.RequestHandler) fasthttp.RequestHandler
type AuthFunc func(ctx *fasthttp.RequestCtx) bool
func NewAuthMiddleware(authFunc AuthFunc) Middleware {
return func(h fasthttp.RequestHandler) fasthttp.RequestHandler {
return func(ctx *fasthttp.RequestCtx) {
result, err: = authFunc(ctx)
if result {
h(ctx)
} else {
ctx.Response.SetStatusCode(fasthttp.StatusUnauthorized)
}
}
}
}
func AuthCheck(ctx *fasthttp.RequestCtx)(bool, error) {
return false; // for example ;)
}
// router
authMiddleware: = middleware.NewAuthMiddleware(security.AuthCheck)
...
router.GET("/protected", authMiddleware(handlers.ProtectedHandler))
I'm new to golang and wanted to experiment with grpc code to get a better understanding of it. In order to do so I followed the example shown here:
https://devicharan.wordpress.com/
The source code is here:
https://github.com/devicharan/basicwebapp
Unfortunately, when I run this code and do a go build I get an error message with the following:
# basicwebapp/proto
proto/CatalogService.pb.go:126: cannot use _CatalogService_GetProductCatalog_Handler (type func(interface {}, context.Context, []byte) (proto.Message, error)) as type grpc.methodHandler in field value
proto/RecommendationService.pb.go:99: cannot use _RecommendationService_GetRecommendations_Handler (type func(interface {}, context.Context, []byte) (proto.Message, error)) as type grpc.methodHandler in field value
I don't know what this means or what I need to change in order to begin finding a fix. Is it a problem with the code itself or with my Go configuration? Also, is there a good debugger for Go that someone can recommend?
Here is the code for CatalogService.pb.go:
// Code generated by protoc-gen-go.
// source: CatalogService.proto
// DO NOT EDIT!
/*
Package protos is a generated protocol buffer package.
It is generated from these files:
CatalogService.proto
Product.proto
RecommendationService.proto
It has these top-level messages:
Category
CatalogResponse
CatalogRequest
*/
package protos
import proto "github.com/golang/protobuf/proto"
import (
context "golang.org/x/net/context"
grpc "google.golang.org/grpc"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConn
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
type Category struct {
CategoryName string `protobuf:"bytes,1,opt,name=categoryName" json:"categoryName,omitempty"`
}
func (m *Category) Reset() { *m = Category{} }
func (m *Category) String() string { return proto.CompactTextString(m) }
func (*Category) ProtoMessage() {}
type CatalogResponse struct {
Products []*Product `protobuf:"bytes,1,rep,name=products" json:"products,omitempty"`
}
func (m *CatalogResponse) Reset() { *m = CatalogResponse{} }
func (m *CatalogResponse) String() string { return proto.CompactTextString(m) }
func (*CatalogResponse) ProtoMessage() {}
func (m *CatalogResponse) GetProducts() []*Product {
if m != nil {
return m.Products
}
return nil
}
type CatalogRequest struct {
Category *Category `protobuf:"bytes,1,opt,name=category" json:"category,omitempty"`
}
func (m *CatalogRequest) Reset() { *m = CatalogRequest{} }
func (m *CatalogRequest) String() string { return proto.CompactTextString(m) }
func (*CatalogRequest) ProtoMessage() {}
func (m *CatalogRequest) GetCategory() *Category {
if m != nil {
return m.Category
}
return nil
}
func init() {
}
// Client API for CatalogService service
type CatalogServiceClient interface {
GetProductCatalog(ctx context.Context, in *CatalogRequest, opts ...grpc.CallOption) (*CatalogResponse, error)
}
type catalogServiceClient struct {
cc *grpc.ClientConn
}
func NewCatalogServiceClient(cc *grpc.ClientConn) CatalogServiceClient {
return &catalogServiceClient{cc}
}
func (c *catalogServiceClient) GetProductCatalog(ctx context.Context, in *CatalogRequest, opts ...grpc.CallOption) (*CatalogResponse, error) {
out := new(CatalogResponse)
err := grpc.Invoke(ctx, "/protos.CatalogService/GetProductCatalog", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
// Server API for CatalogService service
type CatalogServiceServer interface {
GetProductCatalog(context.Context, *CatalogRequest) (*CatalogResponse, error)
}
func RegisterCatalogServiceServer(s *grpc.Server, srv CatalogServiceServer) {
s.RegisterService(&_CatalogService_serviceDesc, srv)
}
func _CatalogService_GetProductCatalog_Handler(srv interface{}, ctx context.Context, buf []byte) (proto.Message, error) {
in := new(CatalogRequest)
if err := proto.Unmarshal(buf, in); err != nil {
return nil, err
}
out, err := srv.(CatalogServiceServer).GetProductCatalog(ctx, in)
if err != nil {
return nil, err
}
return out, nil
}
var _CatalogService_serviceDesc = grpc.ServiceDesc{
ServiceName: "protos.CatalogService",
HandlerType: (*CatalogServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "GetProductCatalog",
Handler: _CatalogService_GetProductCatalog_Handler,
},
},
Streams: []grpc.StreamDesc{},
}
And this is RecommendationService.pg.go
// Code generated by protoc-gen-go.
// source: RecommendationService.proto
// DO NOT EDIT!
package protos
import proto "github.com/golang/protobuf/proto"
import (
context "golang.org/x/net/context"
grpc "google.golang.org/grpc"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConn
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
type RecommendationResponse struct {
Result []*RecommendationResponse_Recommendation `protobuf:"bytes,1,rep,name=result" json:"result,omitempty"`
}
func (m *RecommendationResponse) Reset() { *m = RecommendationResponse{} }
func (m *RecommendationResponse) String() string { return proto.CompactTextString(m) }
func (*RecommendationResponse) ProtoMessage() {}
func (m *RecommendationResponse) GetResult() []*RecommendationResponse_Recommendation {
if m != nil {
return m.Result
}
return nil
}
type RecommendationResponse_Recommendation struct {
Rating int32 `protobuf:"varint,1,opt,name=rating" json:"rating,omitempty"`
Productid int32 `protobuf:"varint,2,opt,name=productid" json:"productid,omitempty"`
}
func (m *RecommendationResponse_Recommendation) Reset() { *m = RecommendationResponse_Recommendation{} }
func (m *RecommendationResponse_Recommendation) String() string { return proto.CompactTextString(m) }
func (*RecommendationResponse_Recommendation) ProtoMessage() {}
func init() {
}
// Client API for RecommendationService service
type RecommendationServiceClient interface {
GetRecommendations(ctx context.Context, in *Product, opts ...grpc.CallOption) (*RecommendationResponse, error)
}
type recommendationServiceClient struct {
cc *grpc.ClientConn
}
func NewRecommendationServiceClient(cc *grpc.ClientConn) RecommendationServiceClient {
return &recommendationServiceClient{cc}
}
func (c *recommendationServiceClient) GetRecommendations(ctx context.Context, in *Product, opts ...grpc.CallOption) (*RecommendationResponse, error) {
out := new(RecommendationResponse)
err := grpc.Invoke(ctx, "/protos.RecommendationService/GetRecommendations", in, out, c.cc, opts...)
if err != nil {
return nil, err
}
return out, nil
}
// Server API for RecommendationService service
type RecommendationServiceServer interface {
GetRecommendations(context.Context, *Product) (*RecommendationResponse, error)
}
func RegisterRecommendationServiceServer(s *grpc.Server, srv RecommendationServiceServer) {
s.RegisterService(&_RecommendationService_serviceDesc, srv)
}
func _RecommendationService_GetRecommendations_Handler(srv interface{}, ctx context.Context, buf []byte) (proto.Message, error) {
in := new(Product)
if err := proto.Unmarshal(buf, in); err != nil {
return nil, err
}
out, err := srv.(RecommendationServiceServer).GetRecommendations(ctx, in)
if err != nil {
return nil, err
}
return out, nil
}
var _RecommendationService_serviceDesc = grpc.ServiceDesc{
ServiceName: "protos.RecommendationService",
HandlerType: (*RecommendationServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "GetRecommendations",
Handler: _RecommendationService_GetRecommendations_Handler,
},
},
Streams: []grpc.StreamDesc{},
}
Basically your protoc-gen-go doesn't match the version of grpc. So sync them both to the latest version and reinstall protoc-gen-go will resolve the issue:
go get -u github.com/golang/protobuf/
cd github.com/golang/protobuf/
make
go get -u github.com/grpc/grpc-go
For anyone running into the same problem, what I did was simply change how I built the proto files. There is a comment on the blog page that highlights some missing steps, I followed it and did the following protoc command to generate code from the proto files:
protoc --go_out=plugins=grpc:. *.proto
I ran this command in the directory with my proto files and then did a go build on my main.go file, now everything is working fine.
I am having the same issue and reinstalling or the new command doesn't seem to work.
However: If you change
codec grpc.Codec, buf []byte
in the Handler Definition into:
dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor
and
codec.Unmarshal(buf, in)
into
dec(in)
(All in the Handler Definition in your .pb.go file)
it seems to work.
Changing this file is of course not optimal and every time you recompile it will override your changes, but at least there is a workaround until I figure out what I messed up.