I'm trying to access messageId of the Pub/Sub message triggering my Golang function. To do so, I'm trying to modify the PubSubMessage struct from the Cloud Functions documentation:
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
MessageId string `json:"messageId"`
}
The function compiles OK but the MessageID value comes empty. Changing the type doesn't help.
I wonder if there's a way to get the triggering message Id from within a function. Or maybe that's not passed to functions at all?
In the document you refer,
Event structure
Cloud Functions triggered from a Pub/Sub topic will be
sent events conforming to the PubsubMessage type, with the caveat that
publishTime and messageId are not directly available in the
PubsubMessage. Instead, you can access publishTime and messageId via
the event ID and timestamp properties of the event metadata. This
metadata is accessible via the context object that is passed to your
function when it is invoked.
You can get messageId like this.
import "cloud.google.com/go/functions/metadata"
func YourFunc(ctx context.Context, m PubSubMessage) error {
metadata, err := metadata.FromContext(ctx)
if err != nil {
// Handle Error
}
messageId := metadata.EventID
// Rest of your code below here.
}
Related
I want to create a basic test case without bootstrapping producer, consumer and an instance of kafka for a test. I'm stuck with creating a basic message somehow and cannot find my error.
This is the struct definition from the confluent-kafka-go sdk:
// Message represents a Kafka message
type Message struct {
TopicPartition TopicPartition
Value []byte
Key []byte
Timestamp time.Time
TimestampType TimestampType
Opaque interface{}
Headers []Header
}
My basic message creation looks like this.
I already verified that topicPartition struct and validImageUploadMessageAsBytes are valid objects.
kafkaMessage := kafka.Message{
TopicPartition: topicPartition,
Value: validImageUploadMessageAsBytes,
Key: messageKey,
Headers: nil,
}
I also tried the following approach to make sure it does not fail because of some data I provide into the message:
emptyMessage := new(kafka.Message)
emptyMessage.TopicPartition = topicPartition
emptyMessage.Value = []byte("")
emptyMessage.Key = []byte("")
This example produces the same output as in the picture below
When debugging it the test with GoLand (2021.3.3) I am presented with this variable value
The code is working properly, it is just a display issue by the IDE GoLand (2021.3.3)
I am a little bit confused about how to get custom types passed into my Lambda function using golang and sitting behind a HttpApi.
Consider the following go lambda handler, which is almost a copy of the example from the documentation.
type MyRequestType struct {
Name string `json:"name"`
Age int `json:"age"`
}
type MyResponseType struct {
Message string `json:"message"`
}
func handler(request MyRequestType) (MyResponseType, error) {
log.Printf("received request: %v", request)
return MyResponseType{Message: fmt.Sprintf("Hello %s, you are %d years old!", request.Name, request.Age)}, nil
}
func main() {
lambda.Start(handler)
}
The resulting message is always the following.
{
"message": "Hello , you are 0 years old!"
}
I have the dump feeling that this is not possible in Amazon API Gateway HTTP API.
But I also haven't found any documentation pointing out, that this is not possible. So I really wonder, if I am doing something wrong?
The documentation says also something about the valid signatures:
For example func (context.Context, TIn) (TOut, error)
If I am using a HTTP API with the Payload format version 2:
Is the context.Context the normal golang context or something special?
I am thinking of events.APIGatewayV2HTTPRequestContext or others.
What would be the right type of TIn and TOut => events.APIGatewayV2HTTPRequest and events.APIGatewayV2HTTPResponse?
Is the context.Context the normal golang context
Yes.
But you can get the Lambda context with lambdacontext.FromContext which contains additional lambda-specific metadata.
What would be the right type of TIn and TOut
It depends on who invokes the Lambda. When the Lambda is invoked by another AWS service, including the API Gateway, the so-called TIn and TOut are types from the lambda event package. Quote from the package introduction:
This package provides input types for Lambda functions that process AWS events.
In case of the API Gateway, that would be events.APIGatewayProxyRequest and Response, or presumably for the Payload format version 2 the events.APIGatewayV2HTTPRequest and Response — which were added in v1.16.0.
Further documentation (but not much) in the github repo README
My problem is I'm using single queue (as an entry-point to my service) and use Go consumer to handle incoming messages.
My consumer
message := pb.GetRequest{}
err := proto.Unmarshal(msg.Body, message)
My problems is my consumer is hard wired to handle GetRequests only. If I need to handle other type of message ie. AddRequest either
I need to define a new queue for each message or
I need to see if the first unmartial (GetRequest), and continue to test if it can be unmartialed to (AddRequest)
Is there any other good way of doing this (provided #1 is not a good option)
Use a switch on the RabbitMQ routing key.
The Channel.Consume method returns a Go channel of type <-chan amqp.Delivery, where amqp.Delivery contains the field RoutingKey.
The routing key is the identifier used to match published messages to consumer subscriptions. You should make sure that your publishers maintain a one-to-one association between routing keys and message types.
The publisher code will look like this:
msg := &pb.AddRequest{} // some protobuf generated type
body, _ := proto.Marshal(msg)
err := ch.Publish(
"my-exchange", // exchange name
"foo.bar.add", // routing key
true, // option: mandatory
true, // option: immediate
amqp.Publishing{
ContentType: "application/x-protobuf",
Body: body,
},
)
In the example above, you must ensure that all and only messages of type *pb.AddRequest are published with the routing key foo.bar.add, i.e. that your message types are deterministic.
If you can do that, then your consumer code can switch on the routing key and unmarshal the MQ payload into a variable of the correct type:
func formatEvent(payload amqp.Delivery) (proto.Message, error) {
var event proto.Message
// switch on the routing key
switch payload.RoutingKey {
case "foo.bar.add":
event = &pb.AddRequest{}
case "foo.bar.get":
event = &pb.GetRequest{}
default:
return nil, fmt.Errorf("unknown routingKey: %s", key)
}
// unmarshal the body into the event variable
if err := proto.Unmarshal(payload.Body, event); err != nil {
return nil, err
}
return event, nil
}
And then you can type-switch on the proto.Message instance to handle each concrete message type. (Of course you can also directly handle the concrete message in the routing key switch; that will depend on how you want to organize your code).
If your consumer is only able to handle some of the messages routed to the queue he consumes from and the consumer can't be extended to handle different types of messages, you will have to prevent the messages from reaching the queue in the first place. This is a job for the RabbitMQ server and possible the producer.
You don't provide enough information that allows us to suggest how to configure the RabbitMQ exchanges, queues and bindings. Maybe the messages carry some header information that allows the RabbitMQ server to distinguish different types of messages. If there is no such information, maybe the message producers can be extended to add such header information.
Simply rejecting (NACK) a message which your consumer can't handle is a bad idea. This will just place the message back into the same queue. If there is no other consumer that can handle it, this message will never be consumed successfully (ACK).
For Google Cloud Functions triggered from HTTP, it is possible to retrieve the execution id by inspecting the headers of the HTTP request ("Function-Execution-Id") :
package p
import (
"fmt"
"net/http"
)
func F(w http.ResponseWriter, r *http.Request) {
executionID := r.Header.Get("Function-Execution-Id")
fmt.Println(executionID)
}
However, for GCF triggered by PubSub events, I can't find how to retrieve this execution ID :
package p
import (
"context"
)
type PubSubMessage struct {
Data []byte `json:"data"`
}
func F(ctx context.Context, m PubSubMessage) error {
executionID := "" // ???
fmt.Println(executionID)
return nil
}
I have looked into the PubSubMessage (https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage), but it only contains data + an empty attributes map.
I have also checked if execution ID is in the metadata handled by the context. However, from my tests, and the docs (https://godoc.org/cloud.google.com/go/functions/metadata#FromContext), only EventID, Timestamp, EventType and Resource are present.
How can I retrieve the execution id of a GCF function triggered by a PubSub event?
No, it is currently not possible to get execution id from pubsub triggered events.
You can get the event id from context as Lauren stated but that does not match execution id.
Also, pubsub triggered events do have execution ids. You can see this by using a default logger to log the event id. In stackdriver there will be an attached execution id label and it will not match the event id. We have observed that event id is numeric while execution id is alphanumeric.
Further, if the function is retried, it will keep the same event id but get a different execution id.
This is a recent (undocumented) change but can be easily observed.
EDIT: This appears to no longer be accurate. See the other answer from ProGirlXOXO.
A Pub/Sub-triggered event does not have an execution ID; instead it has an EventID contained in the context metadata, which is a unique ID for the event.
You can access the EventID as follows:
import (
"context"
"log"
"cloud.google.com/go/functions/metadata"
)
func F(ctx context.Context, m PubSubMessage) error {
ctxMetadata, err := metadata.FromContext(ctx)
if err != nil {
log.Fatal(err);
}
log.Println("EventID: " + ctxMetadata.EventID)
return nil
}
I'm getting the following error:
./main.go:31: cannot use telegramService (type messaging.TelegramService) as type mypackage.MessagingService in argument to mypackage.RegisterMessagingService:
messaging.TelegramService does not implement mypackage.MessagingService (wrong type for HandleIncomingMessage method)
have HandleIncomingMessage(telegram.Message) error
want HandleIncomingMessage(mypackage.IncomingMessage) error
I have an interface that describes a messaging service like Telegram or WhatsApp, and an interface that describes an incoming message from one of those services:
// IncomingMessage is a message that comes in on a messaging service
type IncomingMessage interface {
Send() error
}
// MessagingService is a service on which messages can be send (like Telegram or FB Messenger)
type MessagingService interface {
Start()
HandleIncomingMessage(IncomingMessage) error
GetHTTPHandler() http.HandlerFunc
GetCommands() []MessagingCommand
}
The first implementation of MessagingService is for Telegram. The issue is the HandleIncomingMessage function, which currently doesn't really do anything and just looks like this:
// HandleIncomingMessage will take an incoming message and repond to it
func (s TelegramService) HandleIncomingMessage(msg *telegram.Message) error {
return nil
}
The issue is that this function accepts a telegram.Message, which the compiler says doesn't comply with the interface. The thing is, that telegram.Message is an implementation of IncomingMessage:
// Message is a Telegram message
type Message struct {
// Added the line below at some point, but it didn't work without it either
mypackage.IncomingMessage
MessageID uint64 `json:"message_id"`
FirstName string `json:"first_name"`
Username string `json:"username"`
Date uint64 `json:"date"`
Text string `json:"text"`
Chat Chat `json:"chat"`
From User `json:"from"`
}
// Send will take m and send it
func (m Message) Send() error {
// Do stuff
return nil
}
Initially IncomingMessage was an empty interface, which is where I first noticed the issue. I tried adding the function Send() which I was going to add anyway, as I thought maybe just giving it any struct wouldnt't work. However, I'm still getting this error.
I don't see any reason why telegram.Message doesn't implement the interface, it's pretty straight forward.
Can anyone explain why this doesn't work?
PS: My package isn't actually called mypackage, changed for clarity
HandleIncomingMessage must take an IncomingMessage argument since that's the way the interface is defined. You can't define an implementation of HandleIncomingMessage that takes some other type as the argument, even if that type implements IncomingMessage. You can define your function to take IncomingMessage and convert that to *telegram.Message using a type assertion:
func (s TelegramService) HandleIncomingMessage(im IncomingMessage) error {
msg := im.(*telegram.Message)
return nil
}
I'm assuming you actually want to be using a pointer to telegram.Message. If so, you need to change the definition of the Send method to take a pointer receiver.