I want to create a basic test case without bootstrapping producer, consumer and an instance of kafka for a test. I'm stuck with creating a basic message somehow and cannot find my error.
This is the struct definition from the confluent-kafka-go sdk:
// Message represents a Kafka message
type Message struct {
TopicPartition TopicPartition
Value []byte
Key []byte
Timestamp time.Time
TimestampType TimestampType
Opaque interface{}
Headers []Header
}
My basic message creation looks like this.
I already verified that topicPartition struct and validImageUploadMessageAsBytes are valid objects.
kafkaMessage := kafka.Message{
TopicPartition: topicPartition,
Value: validImageUploadMessageAsBytes,
Key: messageKey,
Headers: nil,
}
I also tried the following approach to make sure it does not fail because of some data I provide into the message:
emptyMessage := new(kafka.Message)
emptyMessage.TopicPartition = topicPartition
emptyMessage.Value = []byte("")
emptyMessage.Key = []byte("")
This example produces the same output as in the picture below
When debugging it the test with GoLand (2021.3.3) I am presented with this variable value
The code is working properly, it is just a display issue by the IDE GoLand (2021.3.3)
Related
I'm trying to access messageId of the Pub/Sub message triggering my Golang function. To do so, I'm trying to modify the PubSubMessage struct from the Cloud Functions documentation:
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
MessageId string `json:"messageId"`
}
The function compiles OK but the MessageID value comes empty. Changing the type doesn't help.
I wonder if there's a way to get the triggering message Id from within a function. Or maybe that's not passed to functions at all?
In the document you refer,
Event structure
Cloud Functions triggered from a Pub/Sub topic will be
sent events conforming to the PubsubMessage type, with the caveat that
publishTime and messageId are not directly available in the
PubsubMessage. Instead, you can access publishTime and messageId via
the event ID and timestamp properties of the event metadata. This
metadata is accessible via the context object that is passed to your
function when it is invoked.
You can get messageId like this.
import "cloud.google.com/go/functions/metadata"
func YourFunc(ctx context.Context, m PubSubMessage) error {
metadata, err := metadata.FromContext(ctx)
if err != nil {
// Handle Error
}
messageId := metadata.EventID
// Rest of your code below here.
}
I have my struct defining a Subscribers that map channels.
package ws
type SessionHandler struct {
Subscribers map[chan interface{}]bool
}
I wanna make it possible to instantiate it with any kind of channel, like this:
type WsSession struct {
handler *ws.SessionHandler
}
handler := &ws.SessionHandler{
Subscribers: make(map[chan WsResponse]bool),
}
The code example I provided doesn't work (cannot use make(map[chan WsResponse]bool) (value of type map[chan WsResponse]bool) as map[chan interface{}]bool value in struct literal), but how could I update it to my purposes?
What I was trying to accomplish will be possible in Go v1.18, as commented by #torek in my question.
For now, what I ended doing was this:
handler := &ws.SessionHandler{
Subscribers: make(map[chan json.RawMessage]bool),
}
I stopped relying on a go struct and communicated json.RawMessage in my channels. It's not totally clean because I need to marshal/unmarshal my message to give the proper treatment, but its "generic" and I could accomplish what I was trying to do.
My problem is I'm using single queue (as an entry-point to my service) and use Go consumer to handle incoming messages.
My consumer
message := pb.GetRequest{}
err := proto.Unmarshal(msg.Body, message)
My problems is my consumer is hard wired to handle GetRequests only. If I need to handle other type of message ie. AddRequest either
I need to define a new queue for each message or
I need to see if the first unmartial (GetRequest), and continue to test if it can be unmartialed to (AddRequest)
Is there any other good way of doing this (provided #1 is not a good option)
Use a switch on the RabbitMQ routing key.
The Channel.Consume method returns a Go channel of type <-chan amqp.Delivery, where amqp.Delivery contains the field RoutingKey.
The routing key is the identifier used to match published messages to consumer subscriptions. You should make sure that your publishers maintain a one-to-one association between routing keys and message types.
The publisher code will look like this:
msg := &pb.AddRequest{} // some protobuf generated type
body, _ := proto.Marshal(msg)
err := ch.Publish(
"my-exchange", // exchange name
"foo.bar.add", // routing key
true, // option: mandatory
true, // option: immediate
amqp.Publishing{
ContentType: "application/x-protobuf",
Body: body,
},
)
In the example above, you must ensure that all and only messages of type *pb.AddRequest are published with the routing key foo.bar.add, i.e. that your message types are deterministic.
If you can do that, then your consumer code can switch on the routing key and unmarshal the MQ payload into a variable of the correct type:
func formatEvent(payload amqp.Delivery) (proto.Message, error) {
var event proto.Message
// switch on the routing key
switch payload.RoutingKey {
case "foo.bar.add":
event = &pb.AddRequest{}
case "foo.bar.get":
event = &pb.GetRequest{}
default:
return nil, fmt.Errorf("unknown routingKey: %s", key)
}
// unmarshal the body into the event variable
if err := proto.Unmarshal(payload.Body, event); err != nil {
return nil, err
}
return event, nil
}
And then you can type-switch on the proto.Message instance to handle each concrete message type. (Of course you can also directly handle the concrete message in the routing key switch; that will depend on how you want to organize your code).
If your consumer is only able to handle some of the messages routed to the queue he consumes from and the consumer can't be extended to handle different types of messages, you will have to prevent the messages from reaching the queue in the first place. This is a job for the RabbitMQ server and possible the producer.
You don't provide enough information that allows us to suggest how to configure the RabbitMQ exchanges, queues and bindings. Maybe the messages carry some header information that allows the RabbitMQ server to distinguish different types of messages. If there is no such information, maybe the message producers can be extended to add such header information.
Simply rejecting (NACK) a message which your consumer can't handle is a bad idea. This will just place the message back into the same queue. If there is no other consumer that can handle it, this message will never be consumed successfully (ACK).
I'm currently working on an image handler in Go which is intended to reject the upload of unsupported files. My intentions are for the Go program to return an error back via the servers http.ResponceWritter detailing which case for rejection has been met, as json, for the upload service to use.
How errors are set up in the server code:
type Error struct {
ImgHandlerError `json:"error"`
}
type ImgHandlerError struct {
Message string `json:"message"`
Code string `json:"code"`
}
func MakeError(message, code string) *Error {
errorMessage := &Error{
ImgHandlerError: ImgHandlerError{
Message: message,
Code: code,
},
}
return errorMessage
}
func ThrowErr(errorMessage Error, writer http.ResponseWriter) {
jData, err := json.Marshal(errorMessage)
if err != nil {
}
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusForbidden)
writer.Write(jData)
return
}
How an error is generated and written as a HTTP response:
errorMes := *MakeError("PSD files are not supported", "DEF-IMG-0006")
ThrowErr(errorMes, res)
Currently this works as expected, for example when a .psd is uploaded the current HTTP response comes back as JSON.
{"error":{"message":"PSD files are not supported","code":"DEF-IMG-0006"}}
My problem is now testing this as I'm out of my depth, as this is my first Go program. I'm using the Ginko test suite. I've tried setting up the http.ResponseRecorder (as I think thats where the solution lies) in the test suite but I'm having no luck. Also of my current tests in the test suite only test functions which don't require the writer http.ResponseWriter as a param so I'm not sure if I need to start a server in the test suite to have a response writer to read from.
Any help is greatly appreciated
My problem is now testing this as I'm out of my depth, as this is my
first Go program.
So you want to do some end-to-end testing. In your test files you could do something like this.
req := httptest.NewRequest("POST", "my/v1/endpoint", bytes.NewBuffer(payload))
responseRecorder = httptest.NewRecorder()
router.ServeHTTP(responseRecorder, request)
Take a look at httptest.NewRequest and httptest.ResponseRecorder.
Essentially you need to set up your router, personally I use mux and make requests to it just as you would if you were a real consumer of your API.
Your responseRecorder after the request will have fields like Code for reading the response code and things like Body for reading the response body among other fields.
e.g.
if responseRecorder.Code != 200{
t.FailNow()
}
I'm getting the following error:
./main.go:31: cannot use telegramService (type messaging.TelegramService) as type mypackage.MessagingService in argument to mypackage.RegisterMessagingService:
messaging.TelegramService does not implement mypackage.MessagingService (wrong type for HandleIncomingMessage method)
have HandleIncomingMessage(telegram.Message) error
want HandleIncomingMessage(mypackage.IncomingMessage) error
I have an interface that describes a messaging service like Telegram or WhatsApp, and an interface that describes an incoming message from one of those services:
// IncomingMessage is a message that comes in on a messaging service
type IncomingMessage interface {
Send() error
}
// MessagingService is a service on which messages can be send (like Telegram or FB Messenger)
type MessagingService interface {
Start()
HandleIncomingMessage(IncomingMessage) error
GetHTTPHandler() http.HandlerFunc
GetCommands() []MessagingCommand
}
The first implementation of MessagingService is for Telegram. The issue is the HandleIncomingMessage function, which currently doesn't really do anything and just looks like this:
// HandleIncomingMessage will take an incoming message and repond to it
func (s TelegramService) HandleIncomingMessage(msg *telegram.Message) error {
return nil
}
The issue is that this function accepts a telegram.Message, which the compiler says doesn't comply with the interface. The thing is, that telegram.Message is an implementation of IncomingMessage:
// Message is a Telegram message
type Message struct {
// Added the line below at some point, but it didn't work without it either
mypackage.IncomingMessage
MessageID uint64 `json:"message_id"`
FirstName string `json:"first_name"`
Username string `json:"username"`
Date uint64 `json:"date"`
Text string `json:"text"`
Chat Chat `json:"chat"`
From User `json:"from"`
}
// Send will take m and send it
func (m Message) Send() error {
// Do stuff
return nil
}
Initially IncomingMessage was an empty interface, which is where I first noticed the issue. I tried adding the function Send() which I was going to add anyway, as I thought maybe just giving it any struct wouldnt't work. However, I'm still getting this error.
I don't see any reason why telegram.Message doesn't implement the interface, it's pretty straight forward.
Can anyone explain why this doesn't work?
PS: My package isn't actually called mypackage, changed for clarity
HandleIncomingMessage must take an IncomingMessage argument since that's the way the interface is defined. You can't define an implementation of HandleIncomingMessage that takes some other type as the argument, even if that type implements IncomingMessage. You can define your function to take IncomingMessage and convert that to *telegram.Message using a type assertion:
func (s TelegramService) HandleIncomingMessage(im IncomingMessage) error {
msg := im.(*telegram.Message)
return nil
}
I'm assuming you actually want to be using a pointer to telegram.Message. If so, you need to change the definition of the Send method to take a pointer receiver.