I want to understand how ApacheMQ's prefetch limit works. Are all the messages sent in one burst? What if there are concurrent consumers, what happens then?
What is the difference between prefetch limit of 0 and 1?
Read the link recommended by #Tim Bish -- the quotes I offer are from that page.
So ActiveMQ uses a prefetch limit on how many messages can be streamed
to a consumer at any point in time. Once the prefetch limit is
reached, no more messages are dispatched to the consumer until the
consumer starts sending back acknowledgements of messages (to indicate
that the message has been processed). The actual prefetch limit value
can be specified on a per consumer basis.
Specifically on the 0 versus 1 prefetch limit difference:
If you have very few messages and each message takes a very long time
to process you might want to set the prefetch value to 1 so that a
consumer is given one message at a time. Specifying a prefetch limit
of zero means the consumer will poll for more messages, one at a time,
instead of the message being pushed to the consumer.
Related
I'm trying to benchmark the performance of swiftMQ 5.0.0 with producer and consumer application I wrote so that I can vary the number of producer threads and consumer threads. I have added a delay on the consumer to simulate the time taken to process a message. I have run a test by setting the producer threads fixed at 2, and by varying the number of consumer threads from 20 to 92 in steps of 4.
Initially, the producer rate starts high and consumer rate is low (as expected due to the delay added and less number of consumer threads).
As the number of consumer threads increase, the producer rate drops and consumer rate increases and they become equal at around 48 consumer threads.
After that, as the number of consumer threads further increase, both producer and consumer rates keep increasing linearly. I am wandering what the reason for this behavior is?
see this image for the
result graph
Notes:
I have disabled flow control at queue level by setting flowcontrol-start-queuesize="-1" .
I also have not set a value to inbound-flow-control-enabled in routing swiftlet. (I believe it
defaults to false)
Any help on this matter is much appreciated. TIA
There is a microservice, which receives the batch of the messages from the outside and push them to kafka. Each message is sent separately, so for each batch I have around 1000 messages 100 bytes each. It seems like the messages take much more space internally, because the free space on the disk going down much faster than I expected.
I'm thinking about changing the producer logic, the way it will put all the batch in one message (the consumer then will split them by itself). But I haven't found any information about space or performance issues with many small messages, neither any guildlines about balance between size and count. And I don't know Kafka enough to have my own conclusion.
Thank you.
The producer will, by itself, batch messages that are destined to the same partition, in order to avoid unnecesary calls.
The producer makes this thanks to its background threads. In the image, you can see how it batches 3 messages before sending them to each partition.
If you also set compression in the producer-side, it will also compress (GZip, LZ4, Snappy are the valid codecs) the messages before sending it to the wire. This property can also can be set on the broker-side (so the messages are sent uncompressed by the producer, and compressed by the broker).
It depends on your network capacity to decide wether you prefer a slower producer (as the compression will slow it) or bigger load on the wire. Note that setting a big compression level on big files may affect a lot your overall performance.
Anyway, I believe the big/small msg problem hurts a lot more to the consumer side; Sending messages to Kafka is easy and fast (the default behaviour is async, so the producer won't be too busy). But on the consumer side, you'll have to look the way you are processing the messages:
One Consumer-Worker
Here you couple consuming with processing. This is the simplest way: the consumer sets its own thread, reads a kafka msg and process it. Then continues the loop.
One Consumer - Many workers
Here you decouple consuming and processing. In most cases, reading from kafka will be faster than the time you need to process the message. It is just physics. In this approach, one consumer feeds many separate worker threads that share the processing load.
More info about this here, just above the Constructors area.
Why do I explain this? Well, if your messages are too big, and you choose the first option, your consumer may not call poll() within the timeout interval, so it will rebalance continuosly. If your messages are big (and take some time to be processed), better choose to implement the second option, as the consumer will continue its own way, calling poll() without falling in rebalances.
If the messages are too big and too many, you may have to start thinking about different structures than can buffer the messages into your memory. Pools, deques, queues, for example, are different options to acomplish this.
You may also increase the poll timeout interval. This may hide you about dead consumers, so I don't really recommend it.
So my answer would be: it depends, basicallty on: your network capacity, your required latency, your processing capacity. If you are able to process big messages equally fast as smaller ones, then I wouldn't care much.
Maybe if you need to filter and reprocess older messages I'd recommend partitioning the topics and sending smaller messages, but it's only a use-case.
I have 2 ActiveMQ JMS consumer queues consumer1 and consumer2.
Dead letter queues are enabled for both queues mainly for avoiding poison messages continuous retry.
Issue is now consumer2 messages should be processed in order. Where as messages going to DLQ could be processed out of order later.
Any suggestions how to handle this scenario?
Your question is not clear; the only way to guarantee order is to process them in order (concurrency=1) and don't use any prefetch.
If you prefetch messages, rejected messages might go behind the prefetch.
Turning off prefetch will severely impact performance.
According to the SQS documentation, the maximum number of inflight messages is set to 120,000 for standard queues. However, sometimes I see my queues maxing out at lower numbers, such as here:
Does anyone know why this might be the case? I have code that dynamically changes the number of SQS listeners depending on the number of messages in the queue, but I don't want to do anything if I've hit the maximum. My problem is now that the max limit doesn't seem to be consistent. Some queues go to 120K, but this one is stuck at 100K instead, and as far as I can tell there is no setting that allows me to set this limit.
approximateNumberOfMessagesNotVisible indicates the number of messages in-flight, as you are rightly said. It depends on how many consumers you have, and what is througput of each consumer.
If the actual number is caping at 100k, then your consumers are swamped and have no more receiving capacity.
Anyways, it's better if you provide more info on the use-case as 100k in-flight messages look out of ordinary and you may be not using correct solution for your problem.
I have a simple configuration for testing: a fast C++ producer sending ~60 byte messages via Stomp to a topic, a slow consumer, and address-full-policy set to DROP.
The queue grows rapidly receiving several thousand messages per second until it reaches my max-size-bytes which amounts to about 300,000 messages. HornetQ starts dropping messages as expected, but from then on is accepting only 3-4 messages per second from the producer. What would cause that? If it's dropping messages, shouldn't it be able to accept them full speed from the producer?