My problem is I'm using single queue (as an entry-point to my service) and use Go consumer to handle incoming messages.
My consumer
message := pb.GetRequest{}
err := proto.Unmarshal(msg.Body, message)
My problems is my consumer is hard wired to handle GetRequests only. If I need to handle other type of message ie. AddRequest either
I need to define a new queue for each message or
I need to see if the first unmartial (GetRequest), and continue to test if it can be unmartialed to (AddRequest)
Is there any other good way of doing this (provided #1 is not a good option)
Use a switch on the RabbitMQ routing key.
The Channel.Consume method returns a Go channel of type <-chan amqp.Delivery, where amqp.Delivery contains the field RoutingKey.
The routing key is the identifier used to match published messages to consumer subscriptions. You should make sure that your publishers maintain a one-to-one association between routing keys and message types.
The publisher code will look like this:
msg := &pb.AddRequest{} // some protobuf generated type
body, _ := proto.Marshal(msg)
err := ch.Publish(
"my-exchange", // exchange name
"foo.bar.add", // routing key
true, // option: mandatory
true, // option: immediate
amqp.Publishing{
ContentType: "application/x-protobuf",
Body: body,
},
)
In the example above, you must ensure that all and only messages of type *pb.AddRequest are published with the routing key foo.bar.add, i.e. that your message types are deterministic.
If you can do that, then your consumer code can switch on the routing key and unmarshal the MQ payload into a variable of the correct type:
func formatEvent(payload amqp.Delivery) (proto.Message, error) {
var event proto.Message
// switch on the routing key
switch payload.RoutingKey {
case "foo.bar.add":
event = &pb.AddRequest{}
case "foo.bar.get":
event = &pb.GetRequest{}
default:
return nil, fmt.Errorf("unknown routingKey: %s", key)
}
// unmarshal the body into the event variable
if err := proto.Unmarshal(payload.Body, event); err != nil {
return nil, err
}
return event, nil
}
And then you can type-switch on the proto.Message instance to handle each concrete message type. (Of course you can also directly handle the concrete message in the routing key switch; that will depend on how you want to organize your code).
If your consumer is only able to handle some of the messages routed to the queue he consumes from and the consumer can't be extended to handle different types of messages, you will have to prevent the messages from reaching the queue in the first place. This is a job for the RabbitMQ server and possible the producer.
You don't provide enough information that allows us to suggest how to configure the RabbitMQ exchanges, queues and bindings. Maybe the messages carry some header information that allows the RabbitMQ server to distinguish different types of messages. If there is no such information, maybe the message producers can be extended to add such header information.
Simply rejecting (NACK) a message which your consumer can't handle is a bad idea. This will just place the message back into the same queue. If there is no other consumer that can handle it, this message will never be consumed successfully (ACK).
Related
I tried writing a subscriber for a NATS limit queue:
sub, err := js.SubscribeSync(fullSubject, nats.Context(ctx))
if err != nil {
return err
}
msg, err := sub.NextMsgWithContext(ctx)
if err != nil {
if errors.Is(err, nats.ErrSlowConsumer) {
log.Printf("Slow consumer error returned. Waiting for reset...")
time.Sleep(50 * time.Millisecond)
continue
} else {
return err
}
}
msg.InProgress()
var message pnats.NatsMessage
if err := conn.unmarshaller(msg.Data, &message); err != nil {
msg.Term()
return err
}
actualSubject := message.Context.FullSubject()
handler, ok := callbacks[message.Context.Category]
if !ok {
msg.Nak()
continue
}
callback, err := handler(&message)
if err == nil {
msg.Ack()
msg.Term()
} else {
msg.Nak()
return err
}
callback(ctx)
The goal of this code is consume any message on a number of subjects and call a callback function associated with the subject. This code works but the issue I'm running into is that I'd like the message to be deleted after the call to handler if that function doesn't return an error. I thought that's what msg.Term was doing but I still see all the messages in the queue.
I had originally designed this around a work queue but I wanted it to work with multiple subscribers so I had to redesign it. Is there any way to make this work?
Based on the code provided, I assume that you are not providing stream and consumer info when creating a subscription with the JetStream library.
In the documentation for the SubscribeSync method, it says that when stream and consumer information is not provided, the library will create an ephemeral consumer and the name of the consumer is picked by the server. It also attempts to figure out which stream the subscription is for.
Here is what I believe happens in your code:
When you call the SubscribeSync method, an ephemeral consumer is created, with your provided topic.
When msg.Ack and msg.Term are called, you do acknowledge the message, but only for that current consumer.
The next time you call the SubscribeSync method, a new ephemeral consumer is created, containing the message that you already deleted on another consumer. Which is how the Jetstream concepts of streams, consumers, and subscriptions work by design.
Based on what you want to accomplish, here are some suggestions:
Use the plain NATS Core library to work with either a pub/sub or a queue. Don't use JetStream. The NATS Core library works with topics directly, whereas the Jetstream library creates additional things (streams and consumers) under the hood if the information is not provided.
Use JetStream but create a stream and a durable consumer yourself, either through code or directly on the NATS server. This way, with a stream and a consumer already defined, you should be able to make it work as intended.
I'm trying to access messageId of the Pub/Sub message triggering my Golang function. To do so, I'm trying to modify the PubSubMessage struct from the Cloud Functions documentation:
// PubSubMessage is the payload of a Pub/Sub event.
// See the documentation for more details:
// https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
type PubSubMessage struct {
Data []byte `json:"data"`
MessageId string `json:"messageId"`
}
The function compiles OK but the MessageID value comes empty. Changing the type doesn't help.
I wonder if there's a way to get the triggering message Id from within a function. Or maybe that's not passed to functions at all?
In the document you refer,
Event structure
Cloud Functions triggered from a Pub/Sub topic will be
sent events conforming to the PubsubMessage type, with the caveat that
publishTime and messageId are not directly available in the
PubsubMessage. Instead, you can access publishTime and messageId via
the event ID and timestamp properties of the event metadata. This
metadata is accessible via the context object that is passed to your
function when it is invoked.
You can get messageId like this.
import "cloud.google.com/go/functions/metadata"
func YourFunc(ctx context.Context, m PubSubMessage) error {
metadata, err := metadata.FromContext(ctx)
if err != nil {
// Handle Error
}
messageId := metadata.EventID
// Rest of your code below here.
}
I have an ecommerce app where I'm sending a message to a kafka server every time a user adds something to a cart. I can send the message and consume it from a client, however, I am curious about error handling. Once in a while, my Go server fails because of a network error or some other reasons. Adding to cart functionality will be an essential part of the app, so I don't want kafka producer to fail that functionality or become dependent on it. I tried separate them by creating a separate function for kafka Producer and I think the kafka.Produce() function is non-blocking, so even if that fails user still should be able to add items to a cart. Here's a sample code (I put the full code for kafka part, but I trimmed the implementation of adding to cart for readability). Is there a way to quit from kafka function if something goes wrong or if it is longer than couple of seconds-timeout? So, the adding to cart functionality wouldn't hang or cause the server to fail. I'm not very experienced with channels and concurrency in Go, so I can't really tell if this could become an issue with this current design.
ADD TO CART
func addToCart(c *context.Context, rw web.ResponseWriter, req *web.Request) {
cartID := req.PathParams["id"]
var items []map[string]interface{}
if err := json.NewDecoder(req.Body).Decode(&items); err != nil {
errors.Write(rw, 400, "Unable to parse request body JSON or invalid data format.")
return
}
//MAKE SOME OPERATIONS AND SAVE IT TO DATABASE
cart, jsonErr := saveToDB(c, cartID, items)
if jsonErr != nil {
jsonErr.Write(rw)
return
}
webLib.Write204(rw)
deliveryChan := make(chan kafka.Event)
kafkaMessage("cart_topic", "sample-cart-event-message", deliveryChan, rw, rq)
return
}
KAFKA
func kafkaMessage(topic string, message []byte, deliveryChan chan kafka.Event, rw web.ResponseWriter, req *web.Request) {
err := c.KafkaProducer.Produce(&kafka.Message{
TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
Value: message,
}, deliveryChan)
if err != nil {
c.Log("error:%s", err)
return
}
e, ok := <-deliveryChan
if !ok{
c.Log("Channel is closed for kafka producer")
return
}
m, ok := e.(*kafka.Message)
if !ok{
c.Log("There has been an error obtaining the kafka message")
return
}
if m.TopicPartition.Error != nil {
fmt.Printf("Delivery failed: %v\n", m.TopicPartition.Error)
} else {
c.Log("Delivered message to topic %s [%d] at offset %v\n",
*m.TopicPartition.Topic, m.TopicPartition.Partition, m.TopicPartition.Offset)
}
}
So the send to kafka is async but you're, in effect, turning it into a sync function by waiting for a "success" message.
A couple quick options.
1: You can totally disregard the status of the async message by passing a nil channel as deliveryChan. Then you really get a "fire and forget" async model. It sounds like this may be what you are looking for.
2: You can run kafkaMessage in a goroutine by simply changing to
deliveryChan := make(chan kafka.Event)
go kafkaMessage("cart_topic", "sample-cart-event-message", deliveryChan, rw, rq)
return
Then you can keep your waiting for a message, logging, etc in that function. You can even add retries if you want! Be aware that in this case you can get a backlog of goroutines waiting on response messages / retrying / etc since you're essentially queuing up operations as goroutines. For most applications this won't be a problem as well as you're not continually falling behind in processing, but still - something to keep an eye on with monitoring!
There are lots of other patterns to follow here, but these are fairly low lift and give you a few options.
I am building service that needs to send events to all subscribed consumers in Pub/Sub manner eg. send one event to all currently connected clients.
I am using Protobuf for that with the following proto definition:
service EventsService {
rpc ListenForEvents (AgentProcess) returns (stream Event) {}
}
Both server & client are written in Go.
My problem is that when client initiates connection then the stream it is not long-lived, eg. when server returns from ListenForEvents method:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
//persist listener here so it can be used later when backend needs to send some messages to client
return nil
}
then the client almost instantly gets EOF error which means that server probably closed connection.
What do I do so that the client is subscribed for a long time to the server? The main problem is that I might not have anything to send to the client when it calls ListenForEvents method on the server, this is why I want this stream to be long lived to be able to send messages later.
The stream terminates when you return from the server function. Instead, you should receive events somehow, and send them to the client without returning from your server. There are probably many ways you can do this. Below is the sketch of one way of doing it.
This relies on the server connection running on a separate goroutine. There is a Broadcast() function that will send messages to all connected clients. It looks like this:
var allRegisteredClients map[*pb.AgentProcess]chan Message
var clientsLock sync.RWMutex{}
func Broadcast(msg Message) {
clientsLock.RLock()
for _,x:=range allRegisteredClients {
x<-msg
}
clientsLock.RUnlock()
}
Then, your clients have to register themselves, and process messages:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
clientsLock.Lock()
ch:=make(chan Message)
allRegisteredClients[process]=ch
clientsLock.Unlock()
for msg:=range ch {
// send message
// Deal with errors
// Deal with client terminations
}
clientsLock.Lock()
delete(allRegisteredClients,process)
clientsLock.Unlock()
}
As I said, this is only a sketch of the idea.
I have managed to nail it down.
Basically I never return from method ListenForEvents.
It creates channel, persists in global-like map of subscribed clients and keeps reading from that channel indefinitely.
The whole implementation of server logic:
func (e EventsService) ListenForEvents(process *pb.AgentProcess, listener pb.EventsService_ListenForEventsServer) error {
chans, exists := e.listeners[process.Hostname]
chanForThisClient := make(chan *pb.Event)
if !exists {
e.listeners[process.Hostname] = []chan *pb.Event{chanForThisClient}
} else {
e.listeners[process.Hostname] = append(chans, chanForThisClient)
}
for {
select {
case <-listener.Context().Done():
return nil
case res := <-chanForThisClient:
_ = listener.Send(res)
}
}
return nil
}
You need to provide keepalive settings for grpc client and server
See details here https://github.com/grpc/grpc/blob/master/doc/keepalive.md
Examples https://github.com/grpc/grpc-go/tree/master/examples/features/keepalive
I have a web app whose server creates a Client for each websocket connection. A Client acts as an intermediary between the websocket connection and a single instance of a Hub. The Hub maintains a set of registered clients and broadcasts messages to the clients. This works pretty well but the problem is that a client might miss events between when the server generates the initial state bundle that the client receives on connection and when the client is registered with the hub and starts receiving broadcast events.
My idea is to register the client with the hub before any information is fetched from the db. That would ensure that the client doesn't miss any broadcasts, though now it could receive messages that are already applied to the initial state it receives. To allow the client to disregard these messages I could include a monotonic timestamp in both the initial state bundle as well as broadcast events.
Can you think of a more elegant/simpler solution?
I have used a write-ahead-log in the past to do something like this. In short, keep a ring buffer of messages in the hub. Then replay messages that where send to existing clients while the new one was initialized.
You can expose this concept to the clients too if you wish. That way you can implement efficient re-connects (particularly nice for mobile connections). When clients loose the websocket connection they can reconnect and say "Hey there, it's me again. Looks like we got interrupted. The last message I've seen was number 42. What's new?"
The following is from memory, so take this only as an illustration of the idea, not a finished implementation. In the intererest of brevity I've omited the select statements around client.send, for instance.
package main
import (
"container/list"
"sync"
"github.com/gorilla/websocket"
)
type Client struct { // all unchanged
hub *Hub
conn *websocket.Conn
send chan []byte
}
type Hub struct {
mu *sync.RWMutex
wal list.List // List if recent messages
clients map[*Client]bool // Registered clients.
register chan Registration // not a chan *Client anymore
broadcast chan []byte
unregister chan *Client
}
type Registration struct {
client *Client
// init is a function that is executed before the client starts to receive
// broadcast messages. All messages that are broadcast while init is
// running will be sent after init returns.
init func()
}
func (h *Hub) run() {
for {
select {
case reg := <-h.register:
// Take note of the most recent message as of right now.
// initClient will replay all later messages
h.mu.RLock()
head := h.wal.Back()
h.mu.RUnlock()
go h.initClient(reg, head)
case client := <-h.unregister:
h.mu.Lock()
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
h.mu.Unlock()
case message := <-h.broadcast:
h.mu.Lock()
h.wal.PushBack(message)
// TODO: Trim list if too long by some metric (e.g. number of
// messages, age, total message size, etc.)
clients := make([]*Client, len(h.clients))
copy(clients, h.clients)
h.mu.Unlock()
for client := range clients {
// TODO: deal with backpressure
client.send <- message
}
}
}
}
func (h *Hub) initClient(reg Registration, head *list.Element) {
reg.init()
// send messages in h.wal after head
for {
h.mu.RLock()
head = head.Next()
if head == nil {
// caught up
h.clients[reg.client] = true
h.mu.RUnlock()
return
}
h.mu.RUnlock()
// TODO: deal with backpressure
reg.client.send <- head.Value.([]byte)
}
}