Google PubSub and Go: create client outside or inside publish-function? - go

I'm new when it comes to Google PubSub(and pubsub applications in general). I'm also relatively new when it comes to Go.
I'm working on a pretty heavy backend service application that already has too many responsibilities. The service needs to fire off one message for each incoming request to a Google PubSub topic. It only needs to "fire and forget". If something goes wrong with the publishing, nothing will happen. The messages are not crucial(only used for analytics), but there will be many of them. We estimate between 50 and 100 messages per second for most of the day.
Now to the code:
func(p *publisher) Publish(message Message, log zerolog.Logger) error {
ctx := context.Background()
client, err := pubsub.NewClient(ctx, p.project)
defer client.Close()
if err != nil {
log.Error().Msgf("Error creating client: %v", err)
return err
}
marshalled, _ := json.Marshal(message)
topic := client.Topic(p.topic)
result := topic.Publish(ctx, &pubsub.Message{
Data: marshalled,
})
_, err = result.Get(ctx)
if err != nil {
log.Error().Msgf("Failed to publish message: %v", err)
return err
}
return nil
}
Disclaimer: p *publisher only contains configuration.
I wonder if this is the best way? Will this lead to the service creating and closing a client 100 times per second? If so, then I guess I should create the client once and pass it as an argument to the Publish()-function instead?
This is how the Publish()-function gets called:
defer func(publisher publish.Publisher, message Message, log zerolog.Logger) {
err := publisher.Publish(log, Message)
if err != nil {
log.Error().Msgf("Failed to publish message: %v", err)
}
}(publisher, message, logger,)

Maybe the way to go is to hold pubsubClient & pubsubTopic inside struct?
type myStruct struct {
pubsubClient *pubsub.Client
pubsubTopic *pubsub.Topic
logger *yourLogger.Logger
}
func newMyStruct(projectID string) (*myStruct, error) {
ctx := context.Background()
pubsubClient, err := pubusb.NewClient(ctx, projectID)
if err != nil {...}
pubsubTopic := pubsubClient.Topic(topicName)
return &myStruct{
pubsubClient: pubsubClient,
pubsubTopic: pubsubTopic,
logger: Logger,
// and whetever you want :D
}
}
And then for that struct create a method, which will take responsibility of marshalling the msg and sends it to Pub/sub
func (s *myStruct) request(ctx context.Context data yorData) {
marshalled, err := json.Marshal(message)
if err != nil {..}
res := s.pubsubTopic.Publish(ctx, &pubsub.Message{
Data: marshalled,
})
if _, err := res.Get(ctx); err !=nil {..}
return nil
}

Related

Testing around GRPC stream Send function in Go

I have a Go GRPC server-side streaming function:
func (server *Server) GetClients(req *iam.GetClientsRequest, client iam.IAM_GetClientsServer) error {
ctx := client.(interface{ Context() context.Context }).Context()
userID, err := getUserIDStream(client)
if err != nil {
return err
}
clients, err := server.db.QueryByUserID(ctx, userID)
if err != nil {
return grpc.Errorf(codes.Internal, apiutils.ServerError)
}
for _, value := range clients {
converted, err := server.fromInternalClient(value)
if err != nil {
return err
}
if err := client.Send(converted); err != nil {
return err
}
}
return nil
}
and I'm testing it like this:
It("GetClients - Send fails - Error", func() {
handler := createHandler(db)
lis := bufconn.Listen(bufSize)
server := grpc.NewServer()
iam.RegisterIAMServer(server, NewServer(handler))
go func() {
if err := server.Serve(lis); err != nil {
log.Fatalf("Server exited with error: %v", err)
}
}()
defer lis.Close()
defer server.GracefulStop()
conn, err := grpc.DialContext(context.Background(), "bufnet",
grpc.WithContextDialer(createBufDialier(lis)), grpc.WithInsecure())
Expect(err).ShouldNot(HaveOccurred())
defer conn.Close()
client := iam.NewIAMClient(conn)
cclient, _ := client.GetClients(addAccessToken(context.Background()), new(iam.GetClientsRequest))
resp, err := cclient.Recv()
Expect(resp).Should(BeNil())
Expect(err).Should(HaveOccurred())
Expect(err.Error()).Should(Equal(message))
})
My issue is that I'm not sure how to induce a failure on Send so I can test the response. Since I'm using an actual test server and client, I can't just mock out the object and I'd prefer not to go that route anyway. Is there a way I can do this?
Originally, I was trying to force Send to fail by setting bufSize to an artificially low value. However, this wasn't producing an error so I decided to try modifying the maxSendMessageSize on the server:
opts := []grpc.ServerOption{}
if sendFails {
opts = append(opts, grpc.MaxSendMsgSize(10))
}
lis := bufconn.Listen(bufSize)
server := grpc.NewServer(opts...)
And this worked in producing the error.

Send data in cloudevents format to Kafka topic

Right now I have this code and it works fine. (It sends some json format data to Kafka topic)
j, err := json.Marshal(data)
if err != nil {
log.Fatal(err)
}
msg := &sarama.ProducerMessage{
Topic: tName,
Value: sarama.StringEncoder(j),
}
_, _, err = producer.SendMessage(msg)
but somebody wishes ho have this data in cloudevents format. -> https://github.com/cloudevents/sdk-go
so what should I do because this Event structure can not be directly casted to string.
type Event struct {
Context EventContext
DataEncoded []byte
// DataBase64 indicates if the event, when serialized, represents
// the data field using the base64 encoding.
// In v0.3, this field is superseded by DataContentEncoding
DataBase64 bool
FieldErrors map[string]error
}
so this code won't even compile.
j, err := json.Marshal(data)
if err != nil {
log.Fatal(err)
}
//...
event := cloudevents.NewEvent()
event.SetSource("example/uri")
event.SetType("example.type")
event.SetData(cloudevents.ApplicationJSON, j)
producerMsg := &sarama.ProducerMessage{
Topic: s.outputTopic,
Value: sarama.StringEncoder(event),
}
_, _, err = s.producer.SendMessage(producerMsg)
What should I do to send this Event to Kafka? Try to cast event.DataEncoded to string or something like that?
btw.
Programming language is golang.
Did you see the section of the docs that serialized the event?
https://github.com/cloudevents/sdk-go#serializedeserialize-a-cloudevent
event := cloudevents.NewEvent()
event.SetSource("example/uri")
event.SetType("example.type")
// data here is a map[string] interface{}, or some other Struct type representing the "example.type" schema type above
event.SetData(cloudevents.ApplicationJSON, data)
bytes, err := json.Marshal(event)
if err != nil {
log.Fatal(err)
}
producerMsg := &sarama.ProducerMessage{
Topic: s.outputTopic,
Value: bytes, // you've already encoded the event
}
Otherwise, be sure to look at the sample code provided that uses the CloudEvent client https://github.com/cloudevents/sdk-go/blob/main/samples/kafka/sender/main.go
sender, err := kafka_sarama.NewSender([]string{"127.0.0.1:9092"}, saramaConfig, "test-topic")
if err != nil {
log.Fatalf("failed to create protocol: %s", err.Error())
}
defer sender.Close(context.Background())
c, err := cloudevents.NewClient(sender, cloudevents.WithTimeNow(), cloudevents.WithUUIDs())
if err != nil {
log.Fatalf("failed to create client, %v", err)
}
event := cloudevents.NewEvent()
event.Set...
c.Send(..., event)
...

How to use self-describing message for protbuf

One of the use cases I'm working on while using protocol buffers is to deserialize the Protocol Buffers Kafka messages which I receive at the consumer end (using sarama library and Go).
The way how i'm doing currently is i defined the sample pixel.proto file as show below.
syntax = "proto3";
package saramaprotobuf;
message Pixel {
// Session identifier stuff
string session_id = 2;
}
i'm sending the message through sarama.Producer(by marshalling it) receiving it sarama.Consumer (unmarshalling message it by referencing with complied pixel.proto.pb). Code is as below.
import (
"github.com/Shopify/sarama"
"github.com/golang/protobuf/proto"
"log"
"os"
"os/signal"
"protobuftest/example"
"syscall"
"time"
)
func main() {
topic := "test_topic"
brokerList := []string{"localhost:9092"}
producer, err := newSyncProducer(brokerList)
if err != nil {
log.Fatalln("Failed to start Sarama producer:", err)
}
go func() {
ticker := time.NewTicker(time.Second)
for {
select {
case t := <-ticker.C:
elliot := &example.Pixel{
SessionId: t.String(),
}
pixelToSend := elliot
pixelToSendBytes, err := proto.Marshal(pixelToSend)
if err != nil {
log.Fatalln("Failed to marshal example:", err)
}
msg := &sarama.ProducerMessage{
Topic: topic,
Value: sarama.ByteEncoder(pixelToSendBytes),
}
producer.SendMessage(msg)
log.Printf("Pixel sent: %s", pixelToSend)
}
}
}()
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM)
partitionConsumer, err := newPartitionConsumer(brokerList, topic)
if err != nil {
log.Fatalln("Failed to create Sarama partition consumer:", err)
}
log.Println("Waiting for messages...")
for {
select {
case msg := <-partitionConsumer.Messages():
receivedPixel := &example.Pixel{}
err := proto.Unmarshal(msg.Value, receivedPixel)
if err != nil {
log.Fatalln("Failed to unmarshal example:", err)
}
log.Printf("Pixel received: %s", receivedPixel)
case <-signals:
log.Print("Received termination signal. Exiting.")
return
}
}
}
func newSyncProducer(brokerList []string) (sarama.SyncProducer, error) {
config := sarama.NewConfig()
config.Producer.RequiredAcks = sarama.WaitForAll
config.Producer.Retry.Max = 5
config.Producer.Return.Successes = true
// TODO configure producer
producer, err := sarama.NewSyncProducer(brokerList, config)
if err != nil {
return nil, err
}
return producer, nil
}
func newPartitionConsumer(brokerList []string, topic string) (sarama.PartitionConsumer, error) {
conf := sarama.NewConfig()
// TODO configure consumer
consumer, err := sarama.NewConsumer(brokerList, conf)
if err != nil {
return nil, err
}
partitionConsumer, err := consumer.ConsumePartition(topic, 0, sarama.OffsetOldest)
if err != nil {
return nil, err
}
return partitionConsumer, err
}
In the code as you can see I have imported the .proto file and referencing it in the main function inorder to send and receive the message. The problem here is, the solution is not generic. I will receive the message of different .proto type at the consumer end.
How can I make it generic? I know there is something called as self-describing message(dynamic message) as the part of protobuf. I referred this link https://developers.google.com/protocol-buffers/docs/techniques?csw=1#self-description . But it doesn't has any explaination on how to embed this as the part of pixel.proto(example which i have used) so that at the consumer end i came directly deserialize it to required type.
You would define a generic container message type that would include a DescriptorSet and an Any fields.
When sending, you build an instance of that generic message type, setting the field of type Any with an instance of your Pixel message and setting the DescriptorSet field with the DescriptorSet of the Pixel type.
That would allow the receiver of such message to parse the Any contents using the DescriptorSet you are attaching. In practical terms, this is sending a piece of proto definition together with the message. So receivers wouldn't need pre-shared proto definitions or generated code.
Having said that, I'm not sure this is what you really want because if you are planning to share proto definitions or generated code with clients then I'd suggest simply using a oneof field in a container type would be much simpler to use.

How to turn DataBase access into a Function idiomatically in Go

I have built a Backend API in Go, it works however I want refactor the code for the DB access layer into a function - idiomatically.
// Get the form data entered by client; FirstName, LastName, phone Number,
// assign the person a unique i.d
// check to see if that user isn't in the database already
// if they are send an error message with the a 'bad' response code
// if they aren't in db add to db and send a message with success
func CreateStudentAccountEndpoint(response http.ResponseWriter, request *http.Request){
client, err := mongo.NewClient("mongodb://localhost:27017")
if err != nil {
log.Fatalf("Error connecting to mongoDB client Host: Err-> %v\n ", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil {
log.Fatalf("Error Connecting to MongoDB at context.WtihTimeout: Err-> %v\n ", err)
}
response.Header().Set("Content-Type", "application/json")
studentCollection := client.Database(dbName).Collection("students")
_, err = studentCollection.InsertOne(context.Background(),data)
if err != nil {
response.WriteHeader(501)
response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
}
// encoding json object for returning to the client
jsonStudent, err := json.Marshal(student)
if err != nil {
http.Error(response, err.Error(), http.StatusInternalServerError)
}
response.Write(jsonStudent)
}
I understand that I can create a method which returns (*mongoClient, err) as I utilise the client local variable later on in the code.
However I am lost as to how to implement the defer cancel() part because it executes once the method CreateStudenAccountEndpoint is at the end. But I am at a loss on how to implement this defer section in a method that will recognise that I want the defer to happen at the end of the function that calls the DB access layer method e.g CreateStudentAccountEndpoint not the actual db access method itself.
As I understand it, the connection should be long-lived and set up as a part of a constructor, i.e. not part of the request flow.
This will typically look something like this:
type BackendAPI struct {
client *mongo.Client
}
func NewBackendAPI(mongoURI string) (*BackendAPI, error) {
client, err := mongo.NewClient(mongoURI)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
err = client.Connect(ctx)
if err != nil {
return nil, err
}
return &BackendAPI{client}, nil
}
func (api *BackendAPI) func CreateStudentAccountEndpoint(response http.ResponseWriter, request *http.Request) {
response.Header().Set("Content-Type", "application/json")
// note the use of the long-lived api.client, which is connected already.
studentCollection := api.client.Database(dbName).Collection("students")
_, err = studentCollection.InsertOne(context.Background() ,data)
if err != nil {
response.WriteHeader(501)
response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
return // at this point, the method should return
}
// encoding json object for returning to the client
jsonStudent, err := json.Marshal(student)
if err != nil {
http.Error(response, err.Error(), http.StatusInternalServerError)
}
response.Write(jsonStudent)
}
If you worry about losing the connection, you could implement a call to api.client.Ping in there, but in my opinion this should only be attempted if you encounter a failure you believe you can recover from by reconnecting.

gRPC - GoLang - Stackdriver tracer

I am trying to get the stackdriver tracer to work with gRPC and I need some help. I have been looking at these two links for reference and I still can't get it to work:
https://medium.com/#harlow/tracing-grpc-calls-in-golang-with-google-stackdriver-b22495763a06#.81oa9q21v
https://rakyll.org/grpc-trace/
For simplicity, I am just working with the hello world gRPC example. Here's my client:
func main() {
// Set up a connection to the server.
conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithUnaryInterceptor(grpc.UnaryClientInterceptor(clientInterceptor)))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
ctx := context.Background()
tc, err := trace.NewClient(ctx, "{PROJECT-ID}")
if err != nil {
log.Fatal(err)
}
span := tc.NewSpan("/greeter/SayHello")
defer span.Finish()
ctx = trace.NewContext(ctx, span)
r, err := c.SayHello(ctx, &pb.HelloRequest{Name: "world"})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
println("Response:", r.Message)
}
func clientInterceptor(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
// trace current request w/ child span
span := trace.FromContext(ctx).NewChild(method)
defer span.Finish()
// new metadata, or copy of existing
md, ok := metadata.FromContext(ctx)
if !ok {
md = metadata.New(nil)
} else {
md = md.Copy()
}
// append trace header to context metadata
// header specification: https://cloud.google.com/trace/docs/faq
md["X-Cloud-Trace-Context"] = append(
md["X-Cloud-Trace-Context"], fmt.Sprintf("%s/%d;o=1", span.TraceID(), 0),
)
ctx = metadata.NewContext(ctx, md)
return invoker(ctx, method, req, reply, cc, opts...)
}
.. and my gRPC server:
// server is used to implement helloworld.GreeterServer.
type server struct{}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
println("HERE")
return &pb.HelloReply{Message: "Hello " + in.Name}, nil
}
func main() {
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
ctx := context.Background()
tc, err := trace.NewClient(ctx, "{PROJECT-ID}")
if err != nil {
log.Fatal(err)
}
s := grpc.NewServer(EnableGRPCTracingServerOption(tc))
pb.RegisterGreeterServer(s, &server{})
println("listening on :50051")
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
// EnableGRPCTracingServerOption enables parsing google trace header from metadata
// and adds a new child span to the incoming request context.
func EnableGRPCTracingServerOption(traceClient *trace.Client) grpc.ServerOption {
return grpc.UnaryInterceptor(serverInterceptor(traceClient))
}
func serverInterceptor(traceClient *trace.Client) grpc.UnaryServerInterceptor {
return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) {
// fetch metadata from request context
md, ok := metadata.FromContext(ctx)
if !ok {
md = metadata.New(nil)
}
header := strings.Join(md["X-Cloud-Trace-Context"], "")
// create new child span from google trace header, add to
// current request context
span := traceClient.SpanFromHeader(info.FullMethod, header)
defer span.Finish()
ctx = trace.NewContext(ctx, span)
return handler(ctx, req)
}
}
I when I run the client to initiate the trace, I get the error:
rpc error: code = 13 desc = stream terminated by RST_STREAM with error code: 1
I'm confused because I don't see anything else about authentication; only providing the project ID which can't be enough to initiate tracing for a specific project. What am I missing?
The issue was with:
defer span.Finish()
That call does not block so because I was just doing preliminary testing with one call my program was exiting before the traces could be uploaded. I contacted the author of https://rakyll.org/grpc-trace/ and she actually updated her post with the option of using:
defer span.FinishWait()
which blocks and that fixed it by allowing the traces to be successfully uploaded before the program exited.
Also, with a long running webserver this wouldn't have been an issue because the process wouldn't have been terminated.
I followed those same tutorials and ran into similar problems.
Header keys are converted to lowercase. If you retrieve it on the server side with header := strings.Join(md["x-cloud-trace-context"], "") you should be good.
You can also define your metadata headers with:
span := trace.FromContext(ctx).NewChild(method)
defer span.Finish()
md := metadata.Pairs(
"x-cloud-trace-context", fmt.Sprintf("%s/%d;o=1", span.TraceID(), 0),
)
ctx = metadata.NewContext(ctx, md)

Resources