Horizontally and vertically scaling RabbitMQ consumers? - ruby

Question

How do I scale RabbitMQ consumers in Ruby using the AMQP gem?
I read the documentation and came up with something that (seemingly) works
in a trivial example. The examples scale horizontally. New processes connect
to the broker and receive a subset of messages. From there each process can
spin up multiple consumer threads. It uses the low level consumer interface
described in the documentation.
Here's the code:
require 'amqp'
workers = ARGV[1] || 4
puts "Running #{workers} workers"
AMQP.start do |amqp|
channel = AMQP::Channel.new
channel.on_error do |conn, ex|
raise ex.reply_text
end
exchange = channel.fanout 'scaling.test', durable: true, prefetch: 1
queue = channel.queue("worker_queue", auto_delete: true).bind(exchange)
workers.times do |i|
consumer = AMQP::Consumer.new channel, queue, "consumer-#{i}", exclusive = false, manual_ack = false
consumer.consume.on_delivery do |meta, payload|
meta.ack
puts "Consumer #{consumer.consumer_tag} in #{Process.pid} got #{payload}"
end
end
trap('SIGTERM') do
amqp.start { EM.stop }
end
end
There are a few things I'm unsure of:
Does the exchange type matter? The documentation states a direct exchange load balances
messages between queues. I tested this example using direct and fanout exchanges and it
functioned the same way. So if I'd like to support vertical and horizontal scaling does the
exchange type matter?
What should the :prefetch option be? I thought one would be best.
How does the load balancing work specifically? The documentation states that load
balancing happens between consumers and not between queues. However when I run two
processes I can see process one print out: "1, 2, 3, 4", then process two print out
"5, 6, 7, 8". I thought they would be out of order, or is the channel itself the consumer?
This would make sense in accordance to the output but not the documentation.
Does this look correct from the EventMachine perspective? Do I need to do some sort of
thread pooling to get multiple consumers in the same process working correctly?

Answer 1

Most of this is actually covered in the documentation for brokers like RabbitMQ, but in answer to your questions:
For a worker queue you most likely want a direct exchange, that is, one which will route a message (job) to one worker exactly, and not to multiple workers at once. But this might change depending on your work. Fanout by definition should route the same message to multiple consumers.
Prefetch should be 1 for this time of setup. Generally speaking this asks the broker to fill the consumer's network buffer with 1 message until ack'd. An alternate setup would be that you have 1 consumer and n workers, in which case you would set the prefetch to n. Additionally it is worth noting that in this sort of setup you shouldn't ack until after you've done the work.
Load balancing is basically a round-robin between consumers. Which is why you're seeing everything sequentially. If each bit of work takes a different amount of time you'll see the distribution change.
I hope that helps. I haven't done anything with the Ruby AMQP library for a while — we rewrote all our workers in Go.