I am looking at processing thousand of messages coming from multiple RabbitMQ queues (between 5 and 10) and push the processed messages by batch into ELK.
What would be the best generic way to process n queues with the streadway/amqp library?
What exactly should be included in each goroutine, in term of amqp.Connection, amqp.Channel and amqp.Consumer.
I mainly see 3 designs:
A) 1 Connection - 1 Channel - n Consumers
B) 1 Connection - n Channels - 1 Consumer
C) n Connection - 1 Channel - 1 Consummer
A) does not work for me:
Failed to register a consumer: Exception (504) Reason: "channel/connection is not open"
Each of those goroutine will then buffer x messages and make a BatchRequest to ELK independently from the others.
For now, starting 1 connection per queue (C) seems to work even if I have to deal with high memory consumption from the server. Is it really the most effective design or should I keep 1 connection per worker handling all 5 to 10 channels?
Here is (C) with one connection per queue.
func main() {
queues := []string{"q1", "q2", "q3", "q4"}
forever := make(chan bool)
for _, queue := range queues {
go processQueue(queue)
}
<-forever
}
func processQueue(name string) {
conn, _ := amqp.Dial("amqp://guest:guest#localhost:5672/")
defer conn.Close()
ch, _ := conn.Channel()
defer ch.Close()
msgs, _ := ch.Consume(name, "test-dev", false, false, false, false, nil)
go func() {
for d := range msgs {
log.Printf("[%s] %s", name, d.RoutingKey)
d.Ack(true)
}
}()
}
Related
I have a queue in RabbitMQ in the consumer-producer fashion, which works properly as a basic round robin queue.
My issue is I am trying to limit the number of requests that are processed per second because when I dequeue an item, I make a request to a DO space that will block my IP if I make 750 requests or more in a second. I use goroutines to concurrently dequeue items, but I only want to dequeue 500 items at a time per second to avoid hitting that limit. This needs to factor in items that are currently being dequeued (i.e I can't just pull 500 items from the queue then delay until the next second), basically before it runs the dequeue code, it needs to wait to be sure that there are not already over 500 requests being dequeued within that second. I have this code so far, but it doesn't seem to be working properly (Note I am testing with 2 requests per second instead of 500 for now). It will have very long delays (like 20+ seconds) every once in a while and I am not sure it is calculating the limit properly. Note that I am pretty sure the prefetch option is not what I need here because that limits the number of messages coming in per second, here I just want to limit the messages being dequeued concurrently per second.
import (
"os"
"fmt"
"github.com/streadway/amqp"
"golang.org/x/time/rate"
"context"
)
// Rate-limit => 2 req/s
const (
workers = 2
)
func failOnErrorWorker(err error, msg string) {
if err != nil {
fmt.Println(msg)
fmt.Println(err)
}
}
func main() {
// Get the env variables for the queue name and connection string
queueName := os.Getenv("QUEUE_NAME")
connectionString := os.Getenv("CONNECTION_STRING")
// Set up rate limiter and context
limiter := rate.NewLimiter(2, 1)
ctx := context.Background()
// Connect to the rabbitmq instance
conn, err := amqp.Dial(connectionString)
failOnErrorWorker(err, "Failed to connect to RabbitMQ")
defer conn.Close()
// Open a channel for the queue
ch, err := conn.Channel()
failOnErrorWorker(err, "Failed to open a channel")
defer ch.Close()
// Consume the messages from this queue
msgs, err := ch.Consume(
queueName, // queue
"", // consumer
false, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
failOnErrorWorker(err, "Failed to register a consumer")
forever := make(chan bool)
go func() {
for d := range msgs {
// Wait until there are less than 2 workers per second
limiter.Wait(ctx)
go func() {
// Dequeue the item and acknowledge the message
DeQueue(d.Body)
d.Ack(false)
} ()
}
}()
fmt.Println(" [*] Waiting for messages. To exit press CTRL+C")
// Continually run the worker
<-forever
}
I was following the chat client/server example for the gorilla websocket library.
https://github.com/gorilla/websocket/blob/master/examples/chat/hub.go#L36
I tried modifying the code to notify other clients when a new client connects, like so:
for {
select {
case client := <-h.register:
h.clients[client] = true
// My addition. Hangs after this (no further register/unregister events are processed):
h.broadcast <- []byte("Another client connected!")
case client := <-h.unregister:
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
case message := <-h.broadcast:
for client := range h.clients {
select {
case client.send <- message:
default:
close(client.send)
delete(h.clients, client)
}
}
}
}
}
My understanding is on the next iteration of the outer for loop, the broadcast channel should receive that data and follow the logic in the message case, but it just hangs.
Why? I can't spot any reason. No further channel events are processed (nothing on register/unregister or broadcast), which makes me think it's some kind of unbuffered channel mechanism it's stuck on, but I don't see how?
The hub's broadcast channel is unbuffered. Communication on an unbuffered channel waits for a ready sender and a ready receiver. The hub goroutine blocks because the goroutine cannot be ready to send and receive at the same time.
Changing the channel from an unbuffered channel to a buffered channel does not fix the problem. Consider the case where the buffer capacity is one:
return &Hub{
broadcast: make(chan []byte, 1),
...
}
with this timeline:
1 clientA: client.hub.register <- client
2 clientB: c.hub.broadcast <- message
3 hub: case client := <-h.register:
4 hub: h.broadcast <- []byte("Another client connected!")
The hub blocks at #4 because the channel was filled to capacity at #2. Increasing the channel capacity to two or more does not fix the problem because any number of clients can broadcast a message while another client is registering.
To fix the problem, move the broadcast code to a function and call that function from both cases in the select:
// sendAll sends message to all registered clients.
// This method must only be called by Hub.run.
func (h *Hub) sendAll(message []byte) {
for client := range h.clients {
select {
case client.send <- message:
default:
close(client.send)
delete(h.clients, client)
}
}
}
func (h *Hub) run() {
for {
select {
case client := <-h.register:
h.clients[client] = true
h.sendAll([]byte("Another client connected!"))
case client := <-h.unregister:
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
case message := <-h.broadcast:
h.sendAll(message)
}
}
}
Your channels are unbuffered, this means that each read/write blocks until an other goroutine performs the opposite operation on the same channel.
When you try to write to h.broadcast the goroutine stops, waiting for a reader. But the same goroutine is supposed to act as a reader of this channel, which never happens because the goroutine is blocked by the write. Thus the program deadlock.
Yea, this won't work. You cannot send/receive on the same unbuffered channel in the same go routine.
The line h.broadcast <- []byte("Another client connected!") blocks until another go routine pops off that queue. An easy solution is to make the broadcast channel have a length 1 buffer. broadcast := make(chan []byte, 1)
You can see it in this playground example
// c := make(chan int) <- This will hang
c := make(chan int, 1)
c <- 1
fmt.Println(<-c)
Remove the length 1 buffer and the whole system deadlocks. One issue you can run into is if 2 clients register at the same time, then you can have the case where 2 items are trying to be stuffed into the broadcast channel, and we're back to the same problem with the unbuffered channel. You can avoid this and keep 1 go routine like so:
for {
select {
case message := <-h.broadcast:
// ...
default:
}
select { // This select statement can only add 1 item to broadcast at most
case client := <-h.register:
// ...
h.broadcast <- []byte("Another client connected!")
}
}
}
However, this will still break if another go routine is also adding to the broadcast channel. So I'd go with Cerise Limon's solution, or buffer the channel enough that other go routines won't ever fill the buffer.
I've never used kafka before. I have two test Go programs accessing a local kafka instance: a reader and a writer. I'm trying to tweak my producer, consumer, and kafka server settings to get a particular behavior.
My writer:
package main
import (
"fmt"
"math/rand"
"strconv"
"time"
"github.com/confluentinc/confluent-kafka-go/kafka"
)
func main() {
rand.Seed(time.Now().UnixNano())
topics := []string{
"policymanager-100",
"policymanager-200",
"policymanager-300",
}
progress := make(map[string]int)
for _, t := range topics {
progress[t] = 0
}
producer, err := kafka.NewProducer(&kafka.ConfigMap{
"bootstrap.servers": "localhost",
"group.id": "0",
})
if err != nil {
panic(err)
}
defer producer.Close()
fmt.Println("producing messages...")
for i := 0; i < 30; i++ {
index := rand.Intn(len(topics))
topic := topics[index]
num := progress[topic]
num++
fmt.Printf("%s => %d\n", topic, num)
msg := &kafka.Message{
Value: []byte(strconv.Itoa(num)),
TopicPartition: kafka.TopicPartition{
Topic: &topic,
},
}
err = producer.Produce(msg, nil)
if err != nil {
panic(err)
}
progress[topic] = num
time.Sleep(time.Millisecond * 100)
}
fmt.Println("DONE")
}
There are three topics that exist on my local kafka: policymanager-100, policymanager-200, policymanager-300. They each only have 1 partition to ensure all messages are sorted by the time kafka receives them. My writer will randomly pick one of those topics and issue a message consisting of a number that increments solely for that topic. When it's done running, I expect the queues to look something like this (topic names shortened for legibility):
100: 1 2 3 4 5 6 7 8 9 10 11
200: 1 2 3 4 5 6 7
300: 1 2 3 4 5 6 7 8 9 10 11 12
So far so good. I'm trying to configure things so that any number of consumers can be spun up and consume these messages in order. By "in-order" I mean that no consumer should get message 2 for topic 100 until message 1 is COMPLETED (not just started). If message 1 for topic 100 is being worked on, consumers are free to consume from other topics that currently don't have a message being processed. If a message of a topic has been sent to a consumer, that entire topic should become "locked" until either a timeout assumes that the consumer failed or the consumer commits the message, then the topic is "unlocked" to have it's next message made available to be consumed.
My reader:
package main
import (
"fmt"
"time"
"github.com/confluentinc/confluent-kafka-go/kafka"
)
func main() {
count := 2
for i := 0; i < count; i++ {
go consumer(i + 1)
}
fmt.Println("cosuming...")
// hold this thread open indefinitely
select {}
}
func consumer(id int) {
c, err := kafka.NewConsumer(&kafka.ConfigMap{
"bootstrap.servers": "localhost",
"group.id": "0", // strconv.Itoa(id),
"enable.auto.commit": "false",
})
if err != nil {
panic(err)
}
c.SubscribeTopics([]string{`^policymanager-.+$`}, nil)
for {
msg, err := c.ReadMessage(-1)
if err != nil {
panic(err)
}
fmt.Printf("%d) Message on %s: %s\n", id, msg.TopicPartition, string(msg.Value))
time.Sleep(time.Second)
_, err = c.CommitMessage(msg)
if err != nil {
fmt.Printf("ERROR commiting: %+v\n", err)
}
}
}
From my current understanding, the way I'm likely to achieve this is by setting up my consumer properly. I've tried many different variations of this program. I've tried having all my goroutines share the same consumer. I've tried using a different group.id for each goroutine. None of these was the right configuration to get the behavior I'm after.
What the posted code does is empty out one topic at a time. Despite having multiple goroutines, the process will read all of 100 then move to 200 then 300 and only one goroutine will actually do all the reading. When I let each goroutine have a different group.id then messages get read by multiple goroutines which I would like to prevent.
My example consumer is simply breaking things up with goroutines but when I begin working this project into my use case at work, I'll need this to work across multiple kubernetes instances that won't be talking to each other so using anything that interacts between goroutines won't work as soon as there are 2 instances on 2 kubes. That's why I'm hoping to make kafka do the gatekeeping I want.
Generally speaking, you cannot. Even if you had a single consumer that consumed all the partitions for the topic, the partitions would be consumed in a non-deterministic order and your total ordering across all partitions would not be guaranteed.
Try Keyed Messages, think you may find this of good use for your use case.
I'm trying to find a good method to consume asynchronously from an input queue, process the content using several workers and then publish to an output queue. So far I've tried a number of examples, most recently using the code from here and here as inspiration.
My current code doesn't appear to be doing what it should be however, increasing the number of workers doesn't increase performance (msg/s consumed or published) and the number of goroutines remains fairly static whilst running.
main:
func main() {
maxWorkers := 10
// channel for jobs
in := make(chan []byte)
out := make(chan []byte)
// start workers
wg := &sync.WaitGroup{}
wg.Add(maxWorkers)
for i := 1; i <= maxWorkers; i++ {
log.Println(i)
defer wg.Done()
go processor(in, out)
}
// add jobs
go collector(in)
go sender(out)
// wait for workers to complete
wg.Wait()
}
The collector is basically the example from the RabbitMQ site with a goroutine that collects messages from the queue and places them on the 'in' channel:
forever := make(chan bool)
go func() {
for d := range msgs {
in <- d.Body
d.Ack(false)
}
}()
log.Printf("[*] Waiting for messages. To exit press CTRL+C")
<-forever
The processor receives an 'in' and 'out' channel, unmarshals JSON, performs a series of regexes and then places the output into the 'out' channel:
func processor(in chan []byte, out chan []byte) {
var (
// list of regexes declared here
)
for {
body := <-in
jsonIn := &Data{}
err := json.Unmarshal(body, jsonIn)
if err != nil {
log.Fatalln("Failed to decode:", err)
}
content := jsonIn.Content
//process regexes using:
//jsonIn.a = r1.FindAllString(content, -1)
jsonOut, _ := json.Marshal(jsonIn)
out <- jsonOut
}
}
And finally the sender is simply the code from the RabbitMQ site, setting up a connection, reading from the 'out' channel and then publishing to a RMQ queue:
for {
jsonOut := <-out
err = ch.Publish(
"", // exchange
q.Name, // routing key
false, // mandatory
false,
amqp.Publishing{
DeliveryMode: amqp.Persistent,
ContentType: "text/json",
Body: []byte(jsonOut),
})
failOnError(err, "Failed to publish a message")
}
This is a pattern that I'll be using quite a lot, so I'm spending a lot of time trying to find something that works correctly (and well) - any advice or help would be appreciated (and in case it isn't obvious, I'm new to Go).
There are a couple of things that jump out:
Done within main function
wg.Add(maxWorkers)
for i := 1; i <= maxWorkers; i++ {
log.Println(i)
defer wg.Done()
go processor(in, out)
}
The defer here is executed when main returns so it's not actually indicating when processing is complete. I don't think this'll have an effect on the performance profile of your program though.
To address this you could pass in wg *sync.WaitGroup to your processor so your processor can indicate when it's done.
CPU Bound Processing
Parsing messages and performing Regex is a cpu intensive workload. How many cores is your machine? How is throughput affected if you run your program on two separate machines, does throughput 2x? What if you double your amount of cores? What about running your program with 1 worker vs 2 processor workers? does that double throughput? Are you maxing out your rabbitmq local instance? is it the bottleneck??
Setting up benchmarking and load testing harnesses should allow you to setup experiments to see where your bottle necks are :)
For queue based services it's pretty easy to setup a test harness to fill rabbitmq with a set backlog and benchmark how fast you can process those messages, or to setup a load generator to send x messages/second to rabbitmq and observe if you can keep up.
Does rabbitmq have good visibility into message processing throughput? If not I frequently add a counter to go code and then log the overall averaged throughput on an interval to get a rough idea of performance:
start := time.Now()
updateInterval := time.Tick(1 * time.Second)
numIn := 0
for {
select {
case <-updateInterval:
log.Infof("IN - Count: %d", numIn)
log.Infof("IN - Througput: %.0f events/second",
float64(numIn)/(time.Now().Sub(start)).Seconds())
case e := <-msgs:
numIn++
in <- d.Body
d.Ack(false)
}
}
I've implemented a demo tcp chat server in golang, it works fine, but every time a user disconnects and I try to write a message to the broadcast channel to let other users know a user has disconnected it blocks, and would not further process any new messages from other client because its a nonbuffered channel
I've commented by code and explained it you can go through it, I don't know why the code blocks, I've written msgs
I'm about to write to a channel
I've written to the channel
I've read from the channel
and messages are in perfect order still my msg channel blocks.
Ps: If I'm using buffered channel the code is not blocking, but I want to know where is my code getting stuck.
I also tried running my code with -race flag but no help
package main
import (
"fmt"
"io"
"net"
"sync"
)
func main() {
msg := make(chan string) //broadcast channel (making it buffered channel the problem goes away)
allConn := make(map[net.Conn]int) //Collection of incoming connections for broadcasting the message
disConn := make(chan net.Conn) //client disconnect channel
newConn := make(chan net.Conn) //new client connection channel
mutext := new(sync.RWMutex) //mux to assign unique id to incoming connections
i := 0
listener, err := net.Listen("tcp", "127.0.0.1:8081")
checkErr(err)
fmt.Println("Tcp server started at 127.0.0.1:8081")
//Accept incoming connections and store them in global connection store allConn
go func() {
for {
conn, err := listener.Accept()
checkErr(err)
mutext.Lock()
allConn[conn] = i
i++
mutext.Unlock()
newConn <- conn
}
}()
for {
select {
//Wait for a new client message to arrive and broadcast the message
case umsg := <-msg:
fmt.Println("Broadcast Channel: Already Read")
bmsg := []byte(umsg)
for conn1, _ := range allConn {
_, err := conn1.Write(bmsg)
checkErr(err)
}
//Handle client disconnection [disConn]
case conn := <-disConn:
mutext.RLock()
fmt.Println("user disconneting", allConn[conn])
mutext.RUnlock()
delete(allConn, conn)
fmt.Println("Disconnect: About to Write")
//this call results in deadlock even when channel is empty, buffered channel resolves the issue
//need to know why
msg <- fmt.Sprintf("Disconneting", allConn[conn])
fmt.Println("Disconnect: Already Written")
//Read client incoming message and put it on broadcasting channel and upon disconnect put on it disConn channel
case conn := <-newConn:
go func(conn net.Conn) {
for {
buf := make([]byte, 64)
n, err := conn.Read(buf)
if err != nil {
if err == io.EOF {
disConn <- conn
break
}
}
fmt.Println("Client: About to Write")
msg <- string(buf[0:n])
fmt.Println("Client: Already Written")
}
}(conn)
mutext.RLock()
fmt.Println("User Connected", allConn[conn])
mutext.RUnlock()
}
}
}
func checkErr(err error) {
if err != nil {
panic(err)
}
}
In Go, an unbuffered channel is a "synchronisation point". That is, if you have a channel c, and do c <- value, the goroutine blocks until someone is ready to do v = <- c (and the converse holds, receiving from a blocking channel without something to receive blocks until the value is available, but this is possibly less surprising). Specifically, for a blocking channel, the receive completes before the send completes.
Since you only have a single goroutine, it will be unable to loop back to reading from the channel and the write will block until something can read.
You could, in theory, get around this by doing something like: go func() { msg <- fmt.Sprintf("Disconneting", allConn[conn] }(), so essentially spawning a short-lived goroutine to do the write.