Limit tasks dequeued per second for rabbitmq - go

I have a queue in RabbitMQ in the consumer-producer fashion, which works properly as a basic round robin queue.
My issue is I am trying to limit the number of requests that are processed per second because when I dequeue an item, I make a request to a DO space that will block my IP if I make 750 requests or more in a second. I use goroutines to concurrently dequeue items, but I only want to dequeue 500 items at a time per second to avoid hitting that limit. This needs to factor in items that are currently being dequeued (i.e I can't just pull 500 items from the queue then delay until the next second), basically before it runs the dequeue code, it needs to wait to be sure that there are not already over 500 requests being dequeued within that second. I have this code so far, but it doesn't seem to be working properly (Note I am testing with 2 requests per second instead of 500 for now). It will have very long delays (like 20+ seconds) every once in a while and I am not sure it is calculating the limit properly. Note that I am pretty sure the prefetch option is not what I need here because that limits the number of messages coming in per second, here I just want to limit the messages being dequeued concurrently per second.
import (
"os"
"fmt"
"github.com/streadway/amqp"
"golang.org/x/time/rate"
"context"
)
// Rate-limit => 2 req/s
const (
workers = 2
)
func failOnErrorWorker(err error, msg string) {
if err != nil {
fmt.Println(msg)
fmt.Println(err)
}
}
func main() {
// Get the env variables for the queue name and connection string
queueName := os.Getenv("QUEUE_NAME")
connectionString := os.Getenv("CONNECTION_STRING")
// Set up rate limiter and context
limiter := rate.NewLimiter(2, 1)
ctx := context.Background()
// Connect to the rabbitmq instance
conn, err := amqp.Dial(connectionString)
failOnErrorWorker(err, "Failed to connect to RabbitMQ")
defer conn.Close()
// Open a channel for the queue
ch, err := conn.Channel()
failOnErrorWorker(err, "Failed to open a channel")
defer ch.Close()
// Consume the messages from this queue
msgs, err := ch.Consume(
queueName, // queue
"", // consumer
false, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
failOnErrorWorker(err, "Failed to register a consumer")
forever := make(chan bool)
go func() {
for d := range msgs {
// Wait until there are less than 2 workers per second
limiter.Wait(ctx)
go func() {
// Dequeue the item and acknowledge the message
DeQueue(d.Body)
d.Ack(false)
} ()
}
}()
fmt.Println(" [*] Waiting for messages. To exit press CTRL+C")
// Continually run the worker
<-forever
}

Related

Using rate.NewLimiter rate limiter with channels

I'm using a rate limiter to throttle the number of requests that are routed
The requests are sent to a channel, and I want to limit the number that are processed per second but i'm struggling to understand if i'm setting this correctly, I don't get an error, but i'm unsure if i'm even using the rate limiter
This is what is being added to the channel:
type processItem struct {
itemString string
}
Here's the channel and limiter:
itemChannel := make(chan processItem, 5)
itemThrottler := rate.NewLimiter(4, 1) //4 a second, no more per second (1)
var waitGroup sync.WaitGroup
Items are added to the channel:
case "newItem":
waitGroup.Add(1)
itemToExec := new(processItem)
itemToExec.itemString = "item string"
itemChannel <- *itemToExec
Then a go routine is used to process everything that is added to the channel:
go func() {
defer waitGroup.Done()
err := itemThrottler.Wait(context.Background())
if err != nil {
fmt.Printf("Error with limiter: %s", err)
return
}
for item := range itemChannel {
execItem(item.itemString) // the processing function
}
defer func() { <-itemChannel }()
}()
waitGroup.Wait()
Can someone confirm that the following occurs:
The execItem function is run on each member of the channel 4 times a second
I don't understand what "err := itemThrottler.Wait(context.Background())" is doing in the code, how is this being invoked?
... i'm unsure if i'm even using the rate limiter
Yes, you are using the rate-limiter. You are rate-limiting the case "newItem": branch of your code.
I don't understand what "err := itemThrottler.Wait(context.Background())" is doing in the code
itemThrottler.Wait(..) will just stagger requests (4/s i.e. every 0.25s) - it does not refuse requests if the rate is exceeded. So what does this mean? If you receive a glut of 1000 requests in 1 second:
4 requests will be handled immediately; but
996 requests will create a backlog of 996 go-routines that will block
The 996 will unblock at a rate of 4/s and thus the backlog of pending go-routines will not clear for another 4 minutes (or maybe longer if more requests come in). A backlog of go-routines may or may not be what you want. If not, you may want to use Limiter.Allow - and if false is returned, then refuse the request (i.e. don't create a go-routine) and return a 429 error (if this is a HTTP request).
Finally, if this is a HTTP request, you should use it's imbedded context when calling Wait e.g.
func (a *app) myHandler(w http.ResponseWriter, r *http.Request) {
// ...
err := a.ratelimiter(r.Context())
if err != nil {
// client http request most likely canceled (i.e. caller disconnected)
}
}

RabbitMQ consumer perfomance - prefetch vs concurency

I have a Go application processing events from a single RabbitMQ queue. I use the github.com/streadway/amqp RabbitMQ Client Library.
The Go application processes every message in ~2-3 seconds. It's possible to process ~1000 or even more messages in parallel, if I feed them from memory.
But, unfortunately, RabbitMQ performance is worse.
So, I want to consume messages from queue faster.
So, the question is: how to consume messages in most effective manner using github.com/streadway/amqp?
As far as I understand, there are two approaches:
set high prefetch
https://godoc.org/github.com/streadway/amqp#Channel.Qos.
Use single consumer goroutine
Example code:
conn, err := amqp.Dial("amqp://guest:guest#localhost:5672/")
failOnError(err, "Failed to connect to RabbitMQ")
defer conn.Close()
ch, err := conn.Channel()
failOnError(err, "Failed to open a channel")
defer ch.Close()
ch.Qos(
10000, // prefetch count
0, // prefetch size
false, // global
)
msgs, err := ch.Consume(
q.Name, // queue
"", // consumer
false, // NO auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
for d := range msgs {
log.Printf("Received a message: %s", d.Body)
err:= processMessage(d)
if err != nil {
log.Printf("%s : while consuming task", err)
d.Nack(false, true)
} else {
d.Ack(false)
}
continue // consume other messages
}
But DO the processMessage will be called here in parallel?
spawn many channels and use multiple consumers
conn, err := amqp.Dial("amqp://guest:guest#localhost:5672/")
failOnError(err, "Failed to connect to RabbitMQ")
defer conn.Close()
var i = 0
for i = 0; i<=100; i++ {
go func(){
ch, err := conn.Channel()
failOnError(err, "Failed to open a channel")
defer ch.Close()
ch.Qos(
10, // prefetch count
0, // prefetch size
false, // global
)
msgs, err := ch.Consume(
q.Name, // queue
"", // consumer
false, // NO auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
for d := range msgs {
log.Printf("Received a message: %s", d.Body)
err:= processMessage(d)
if err != nil {
log.Printf("%s : while consuming task", err)
d.Nack(false, true)
} else {
d.Ack(false)
}
continue // consume other messages
}
}()
}
But is this a RAM friendly approach? Isn't spawning a new channel for each worker is quite dramatic for RabbitMQ?
So, question is, which variant is better? Better performance, better memory usage, etc.
So, what is the optimal usage of RabbitMQ here?
Update: currently, I encountered a case when my worker consumes all RAM on VPS, and is OOM-killed. I used second approach for it. So, better in my case is ability to keep my worker without OOM killing after few minutes of work.
Update 2: nack when worker failed to process message, and ack when worker processed message is very important. All messages has to be processed (its customers analytics), but sometimes worker cannot process it, so it have to nack message to pass it to other workers (currently, some 3rd party api used to process messages sometimes simply returns 503 status code, in this case message should be passed to other worker or retried).
SO, using auto-ack is unfortunately not an option.
I suppose each processMessage() run in a new goroutine.
Which variant is better?
I prefer the first one, because open/close channel is a little bit expensive (2 + 2 TCP packets). I think your OOM problem is not related to too many gorutine, gorutine is very light, just cost about 5KB. So the problem is probably caused by your processMessage().
I think the github.com/streadway/amqp channel consume operation is thread/gorutine-safe, so it is safe to share channel between goruntine if you just do some consume operation.

How can I ensure my consumers process messages in kafka topics in order, only once?

I've never used kafka before. I have two test Go programs accessing a local kafka instance: a reader and a writer. I'm trying to tweak my producer, consumer, and kafka server settings to get a particular behavior.
My writer:
package main
import (
"fmt"
"math/rand"
"strconv"
"time"
"github.com/confluentinc/confluent-kafka-go/kafka"
)
func main() {
rand.Seed(time.Now().UnixNano())
topics := []string{
"policymanager-100",
"policymanager-200",
"policymanager-300",
}
progress := make(map[string]int)
for _, t := range topics {
progress[t] = 0
}
producer, err := kafka.NewProducer(&kafka.ConfigMap{
"bootstrap.servers": "localhost",
"group.id": "0",
})
if err != nil {
panic(err)
}
defer producer.Close()
fmt.Println("producing messages...")
for i := 0; i < 30; i++ {
index := rand.Intn(len(topics))
topic := topics[index]
num := progress[topic]
num++
fmt.Printf("%s => %d\n", topic, num)
msg := &kafka.Message{
Value: []byte(strconv.Itoa(num)),
TopicPartition: kafka.TopicPartition{
Topic: &topic,
},
}
err = producer.Produce(msg, nil)
if err != nil {
panic(err)
}
progress[topic] = num
time.Sleep(time.Millisecond * 100)
}
fmt.Println("DONE")
}
There are three topics that exist on my local kafka: policymanager-100, policymanager-200, policymanager-300. They each only have 1 partition to ensure all messages are sorted by the time kafka receives them. My writer will randomly pick one of those topics and issue a message consisting of a number that increments solely for that topic. When it's done running, I expect the queues to look something like this (topic names shortened for legibility):
100: 1 2 3 4 5 6 7 8 9 10 11
200: 1 2 3 4 5 6 7
300: 1 2 3 4 5 6 7 8 9 10 11 12
So far so good. I'm trying to configure things so that any number of consumers can be spun up and consume these messages in order. By "in-order" I mean that no consumer should get message 2 for topic 100 until message 1 is COMPLETED (not just started). If message 1 for topic 100 is being worked on, consumers are free to consume from other topics that currently don't have a message being processed. If a message of a topic has been sent to a consumer, that entire topic should become "locked" until either a timeout assumes that the consumer failed or the consumer commits the message, then the topic is "unlocked" to have it's next message made available to be consumed.
My reader:
package main
import (
"fmt"
"time"
"github.com/confluentinc/confluent-kafka-go/kafka"
)
func main() {
count := 2
for i := 0; i < count; i++ {
go consumer(i + 1)
}
fmt.Println("cosuming...")
// hold this thread open indefinitely
select {}
}
func consumer(id int) {
c, err := kafka.NewConsumer(&kafka.ConfigMap{
"bootstrap.servers": "localhost",
"group.id": "0", // strconv.Itoa(id),
"enable.auto.commit": "false",
})
if err != nil {
panic(err)
}
c.SubscribeTopics([]string{`^policymanager-.+$`}, nil)
for {
msg, err := c.ReadMessage(-1)
if err != nil {
panic(err)
}
fmt.Printf("%d) Message on %s: %s\n", id, msg.TopicPartition, string(msg.Value))
time.Sleep(time.Second)
_, err = c.CommitMessage(msg)
if err != nil {
fmt.Printf("ERROR commiting: %+v\n", err)
}
}
}
From my current understanding, the way I'm likely to achieve this is by setting up my consumer properly. I've tried many different variations of this program. I've tried having all my goroutines share the same consumer. I've tried using a different group.id for each goroutine. None of these was the right configuration to get the behavior I'm after.
What the posted code does is empty out one topic at a time. Despite having multiple goroutines, the process will read all of 100 then move to 200 then 300 and only one goroutine will actually do all the reading. When I let each goroutine have a different group.id then messages get read by multiple goroutines which I would like to prevent.
My example consumer is simply breaking things up with goroutines but when I begin working this project into my use case at work, I'll need this to work across multiple kubernetes instances that won't be talking to each other so using anything that interacts between goroutines won't work as soon as there are 2 instances on 2 kubes. That's why I'm hoping to make kafka do the gatekeeping I want.
Generally speaking, you cannot. Even if you had a single consumer that consumed all the partitions for the topic, the partitions would be consumed in a non-deterministic order and your total ordering across all partitions would not be guaranteed.
Try Keyed Messages, think you may find this of good use for your use case.

Generic way of processing multiple queues with streadway library

I am looking at processing thousand of messages coming from multiple RabbitMQ queues (between 5 and 10) and push the processed messages by batch into ELK.
What would be the best generic way to process n queues with the streadway/amqp library?
What exactly should be included in each goroutine, in term of amqp.Connection, amqp.Channel and amqp.Consumer.
I mainly see 3 designs:
A) 1 Connection - 1 Channel - n Consumers
B) 1 Connection - n Channels - 1 Consumer
C) n Connection - 1 Channel - 1 Consummer
A) does not work for me:
Failed to register a consumer: Exception (504) Reason: "channel/connection is not open"
Each of those goroutine will then buffer x messages and make a BatchRequest to ELK independently from the others.
For now, starting 1 connection per queue (C) seems to work even if I have to deal with high memory consumption from the server. Is it really the most effective design or should I keep 1 connection per worker handling all 5 to 10 channels?
Here is (C) with one connection per queue.
func main() {
queues := []string{"q1", "q2", "q3", "q4"}
forever := make(chan bool)
for _, queue := range queues {
go processQueue(queue)
}
<-forever
}
func processQueue(name string) {
conn, _ := amqp.Dial("amqp://guest:guest#localhost:5672/")
defer conn.Close()
ch, _ := conn.Channel()
defer ch.Close()
msgs, _ := ch.Consume(name, "test-dev", false, false, false, false, nil)
go func() {
for d := range msgs {
log.Printf("[%s] %s", name, d.RoutingKey)
d.Ack(true)
}
}()
}

RabbitMQ multiple workers pattern

I'm trying to find a good method to consume asynchronously from an input queue, process the content using several workers and then publish to an output queue. So far I've tried a number of examples, most recently using the code from here and here as inspiration.
My current code doesn't appear to be doing what it should be however, increasing the number of workers doesn't increase performance (msg/s consumed or published) and the number of goroutines remains fairly static whilst running.
main:
func main() {
maxWorkers := 10
// channel for jobs
in := make(chan []byte)
out := make(chan []byte)
// start workers
wg := &sync.WaitGroup{}
wg.Add(maxWorkers)
for i := 1; i <= maxWorkers; i++ {
log.Println(i)
defer wg.Done()
go processor(in, out)
}
// add jobs
go collector(in)
go sender(out)
// wait for workers to complete
wg.Wait()
}
The collector is basically the example from the RabbitMQ site with a goroutine that collects messages from the queue and places them on the 'in' channel:
forever := make(chan bool)
go func() {
for d := range msgs {
in <- d.Body
d.Ack(false)
}
}()
log.Printf("[*] Waiting for messages. To exit press CTRL+C")
<-forever
The processor receives an 'in' and 'out' channel, unmarshals JSON, performs a series of regexes and then places the output into the 'out' channel:
func processor(in chan []byte, out chan []byte) {
var (
// list of regexes declared here
)
for {
body := <-in
jsonIn := &Data{}
err := json.Unmarshal(body, jsonIn)
if err != nil {
log.Fatalln("Failed to decode:", err)
}
content := jsonIn.Content
//process regexes using:
//jsonIn.a = r1.FindAllString(content, -1)
jsonOut, _ := json.Marshal(jsonIn)
out <- jsonOut
}
}
And finally the sender is simply the code from the RabbitMQ site, setting up a connection, reading from the 'out' channel and then publishing to a RMQ queue:
for {
jsonOut := <-out
err = ch.Publish(
"", // exchange
q.Name, // routing key
false, // mandatory
false,
amqp.Publishing{
DeliveryMode: amqp.Persistent,
ContentType: "text/json",
Body: []byte(jsonOut),
})
failOnError(err, "Failed to publish a message")
}
This is a pattern that I'll be using quite a lot, so I'm spending a lot of time trying to find something that works correctly (and well) - any advice or help would be appreciated (and in case it isn't obvious, I'm new to Go).
There are a couple of things that jump out:
Done within main function
wg.Add(maxWorkers)
for i := 1; i <= maxWorkers; i++ {
log.Println(i)
defer wg.Done()
go processor(in, out)
}
The defer here is executed when main returns so it's not actually indicating when processing is complete. I don't think this'll have an effect on the performance profile of your program though.
To address this you could pass in wg *sync.WaitGroup to your processor so your processor can indicate when it's done.
CPU Bound Processing
Parsing messages and performing Regex is a cpu intensive workload. How many cores is your machine? How is throughput affected if you run your program on two separate machines, does throughput 2x? What if you double your amount of cores? What about running your program with 1 worker vs 2 processor workers? does that double throughput? Are you maxing out your rabbitmq local instance? is it the bottleneck??
Setting up benchmarking and load testing harnesses should allow you to setup experiments to see where your bottle necks are :)
For queue based services it's pretty easy to setup a test harness to fill rabbitmq with a set backlog and benchmark how fast you can process those messages, or to setup a load generator to send x messages/second to rabbitmq and observe if you can keep up.
Does rabbitmq have good visibility into message processing throughput? If not I frequently add a counter to go code and then log the overall averaged throughput on an interval to get a rough idea of performance:
start := time.Now()
updateInterval := time.Tick(1 * time.Second)
numIn := 0
for {
select {
case <-updateInterval:
log.Infof("IN - Count: %d", numIn)
log.Infof("IN - Througput: %.0f events/second",
float64(numIn)/(time.Now().Sub(start)).Seconds())
case e := <-msgs:
numIn++
in <- d.Body
d.Ack(false)
}
}

Resources