Concerning ActiveMQ: I have a scenario where I have one producer which sends small (around 10KB) files to the consumers. Although the files are small, the consumers need around 10 seconds to analyze them and return the result to the producer. I've researched a lot, but I still cannot find answers to the following questions:
How do I make the broker store the files (completely) in a queue?
Should I use ObjectMessage (because the files are small) or blob messages?
Because the consumers are slow processing, should I lower their prefetchLimit or use a round-robin dispatch policy? Which one is better?
And finally, in the ActiveMQ FAQ, I read this - "If a consumer receives a message and does not acknowledge it before closing then the message will be redelivered to another consumer.". So my question here is, does ActiveMQ guarantee that only 1 consumer will process the message (and therefore there will be only 1 answer to the producer), or not? When does the consumer acknowledge a message (in the default, automatic acknowledge settings) - when receiving the message and storing it in a session, or when the onMessage handler finishes? And also, because the consumers are so slow in processing, should I change some "timeout limit" so the broker knows how much to wait before giving the work to another consumer (this is kind of related to my previous questions)?
Not sure about others, but here are some thoughts.
First: I am not sure what your exact concern is. ActiveMQ does store messages in a data store; all data need NOT reside in memory in any single place (either broker or client). So you should actually be good in that regard; earlier versions did require that all ids needed to fit in memory (not sure if that was resolved), but even that memory usage would be low enough unless you had tens of millions of in-queue messages.
As to ObjectMessage vs blob; raw byte array (blob) should be most compact representation, but since all of these get serialized for storage, it only affects memory usage on client. Pre-fetch mostly helps with access latency; but given that they are slow to process, you probably don't need any prefetching; so yes, either set it to 1 or 2 or disable altogether.
As to guarantees: best that distributed message queues can guarantee is either at-least-once (with possible duplicates), or at-most-once (no duplicates, can lose messages). It is usually better to take at-least-once, and make clients to de-duping using client-provided ids. How acknowledgement is sent is defiend by JMS specification so you can read more about JMS; this is not ActiveMQ specific.
And yes, you should set timeout high enough that worker typically can finish up work, including all network latencies. This can slow down re-transmit of dropped messages (if worked dies), but it is probably not a problem for you.
Related
Due to some reasons, we're not allowed to use more than one queue for our LOB.
So, we have set a very large prefetch number, and the consumer of that single queue will be in charge of dispatching to other in memory queues according to some message properties. Other background tasks will fetch the message from the in memory queues and handle it.
To avoid loss of messages, is it possible to manual ack the message until the background task finishes handling the message?
MassTransit handles message acknowledgement, there is no way to work around it. Once the message consumer completes, it will be acknowledged. Messages remain in the queue until they are acknowledged. There is no way messages could be lost using this approach.
Also, your approach of using a single transport queue and then having a bunch of in-memory queues is an incredibly poor (terrible, horrible, worst possible thing ever perhaps) design choice due to a really bad rule about queue usage. MassTransit can actually do all of that dispatch for you with multiple consumers on the receive endpoint for the various message types. But a single queue is still a really, really bad idea.
I want to publish 3 million lines(1 jms message = l line) to jms queue. each line is one jms message. so totally i will publish 3 million message. will JMS queue supports this much of large data in Queue?
You have not mentioned in what time frame you want to publish 3 million messages. 3 million messages to be published in a second or minute or hour or day(s). What is the size of each message? KBs or MBs ? How quickly the published messages are consumed by consumers?
So it depends on a number of such parameters to understand if a JMS provider can handle your requirement or not. It's not a good practice to use queue as database. Messages in queue must be consumed as quickly as possible so that message build up is avoided. Any message build up will affect throughput.
Absolutely.
You'll need to configure your JMS Provider appropriately, and this can often be tricky (and sometimes the defaults just work).
Typically you configure it to have disk storage for the messages that don't fit in RAM. You need to make sure you've got enough resource (RAM and disk space) to store everything.
I've had success with ActiveMQ - this page might be useful to you:
http://activemq.apache.org/how-do-i-configure-activemq-to-hold-100s-of-millions-of-queue-messages-.html
I'm looking for help regarding a strange issue where a slow consumer on a queue causes all the other consumers on the same queue to start consuming messages at 30 second intervals. That is all consumers but the slow one don't consumer messages as fast as they can, instead they wait for some magical 30s barrier before consuming.
The basic flow of my application goes like this:
a number of producers place messages onto a single queue. Messages can have different JMSXGroupIDs
a number of consumers listen to messages on that single queue
as standard practice the JMSXGroupIDs get distributed across the consumers
at some point one of the consumers becomes slow and can't process messages very quickly
the slow consumer ends up filling its prefetch buffer on the broker and AMQ recognises that it is slow (default behaviour)
at that point - or some 'random' but close time later - all consumers except the slow one start to only consume messages at the same 30s intervals
if the slow consumer becomes fast again then things very quickly return to normal operation and the 30s barrier goes away
I'm at a loss for what could be causing this issue, or how to fix it, please help.
More background and findings
I've managed to reliably reproduce this issue on AMQ 5.8.0, 5.9.0 (where the issue was originally noticed) and 5.9.1, on fresh installs and existing ops-managed installs and on different machines some vm and some not. All linux installs, different OSs and java versions.
It doesn't appear to be affected by anything prefetch related, that is: changing the prefetch value from 1 to 10 to 1000 didn't stop the issue from happening
[red herring?] Enabling debug logs on the amq instance shows logs relating to the periodic check for messages that can be expired. The queue doesn't have an expiry policy so I can only think that the scheduled expireMessagesPeriod time is just waking amq up in such a way that it then sends messages to the non-slow consumers.
If the 30s mode is entered then left then entered again the seconds-past-the-minute time is always the same, for example 14s and 44s past the minute. This is true across all consumers and all machines hosting those consumers. Those barrier points do change after restarts of amq.
While not strictly a solution to the problem, further investigation has uncovered the root cause of this issue.
TL;DR - It's known behaviour and won't be fixed before Apollo
More Details
Ultimately this is caused by the maxPageSize property and the fact that AMQ will only apply selection criteria to messages in memory. Generally these are message selectors (property = value), but in my case they are JMSXGroupID=>Consumer assignments.
As messages are received by the queue they get paged into memory and placed into a collection (named pagedInPendingDispatch in the source). To dispatch messages AMQ will scan through this list of messages and try to find a consumer that will accept it. That includes checking the group id, message selector and prefetch buffer space. For our use case we aren't using message selectors but we are using groups. If no consumer can take the message then it is left in the collection and will be checked again at the next tick.
In order to stop the pagedInPendingDispatch collection from eating up all the resources available there is a suggested limit to the size of this queue configured via the maxPageSize property. This property isn't actually a maximum, it's more a hint as to whether, under normal conditions, new message arrivals should be paged in memory or paged to disk.
With these two pieces of information and a slow consumer it turns out that eventually all the messages in the pagedInPendingDispatch collection end up only being consumable by the slow consumer, and hence the collection effectively gets blocked and no other messages get dispatched. This explains why the slow consumer wasn't affected by the 30s interval, it had maxPageSize messages waiting delivery already.
This doesn't explain why I was seeing the non-slow consumers receive messages every 30s though. As it turns out, paging messages into memory has two modes, normal and forced. Normal follows the process outlined above where the size of the collection is compared to the maxPageSize property, when forced, however, messages are always paged into memory. This mode exists to allow you to browse through messages that aren't in memory. As it happens this forced mode is also used by the expiry mechanism to allow AMQ to expire messages that aren't in memory.
So what we have now is a collection of messages in memory that are all targeted for dispatch to the same consumer, a consumer that won't accept them because it is slow or blocked. We also have a backlog of messages awaiting delivery to all consumers. Every expireMessagesPeriod milliseconds a task runs that force pages messages into memory to check if they should be expired or not. This adds those messages onto the pages in collection which now contains maxPageSize messages for the slow consumer and N more messages destined for any consumer. Those messages get delivered.
QED.
References
Ticket referring to this issue but for message selectors instead
Docs relating to the configuration properties
Somebody else with this issue but for selectors
I plan to have persistent message Queues based on some implementation of AMQP and JMS API. I would like to know whether is ok (from architectural point of view) to have messages staying in the queues for hours. A day is max.
I plan to use the message broker as another persistence layer basically. Is this viable?
The technologies that I am evaluating are ActiveMQ, RabbitMQ or qupid.
I plan to use the message broker as another persistence layer
basically. Is this viable?
The broker's persistence mechanism for message retention is usually file-based, or JDBC; either one will work. It is viable? Sure, its a feature of the broker, nothing wrong with using it for the intended purpose, assuming temporary message retention is your goal; 1 day is not a big deal.
But if you're planning to retain messages for 1 day, or more, I recommend doing some calculations based on average message size and total messages per day that may end up sitting in a queue. Queue depth, by default, is usually a low number, like 10Mb, and if exceeded, the broker will probably drop subsequent messages; you want to prevent this from happening. Vendors handle this differently, so check with RabbitMq and ActiveMQ for specifics and what configuration parameters are used to control depth. I know SonicMq has what's known as the "DeadMessage" queue, a destination for expired or undeliverable messages; other products might have something similar.
It's OK to have persistent queues, and it's OK if messages are hanging around in the queues: Clients might be disconnected because of updates, network problems etc. That's one benefit of queues to decouple sender from receiver, and the queue is the buffer. However these use cases are not the normal mode of operation, it's rather an exceptional situation.
Using a messaging broker as "another persistence layer" is technically speaking possible, but in this case a database is probably more suitable, because quick message delivery/messaging and long term storage/database are different tools/scenarios. So ask yourself the question: Is it still messaging or is it already a database?
If in your use case the normal message delay (= period between sending and reception) is always beyond an hour, a database might be better, because JMS selectors are normally slower and less comfortable than database queries using where clauses.
There is another aspect: Consider the need for an online backup of your messages in a JMS provider, especially in a HA cluster mode. It might be easier to do this using a database.
I have a bunch of servers where files are being generated constantly. These files need to be sent to a central location. The files are never larger than 50MB. I am planning to use ZeroMQ to send these files (encapsulated in messages), so that file writing on the central location does not happen concurrently (for e.g. using scp to do the transfers would start many disk write processes on the destination).
I can see a few ways to do this with ZeroMQ:
Use REQ sockets on the producers and a single REP socket on the consumer. This could work, but I think it would starve slower producers, as there is no fair queueing. Also, I am not sure if the REQ sockets would drop messages if the REP socket is not available.
Use PUSH sockets on the producers and a PULL socket on the consumer. This has fair queuing on the consumer and the docs say that PUSH sockets never discard messages. However, is it fully reliable?
My reliability requirements are:
Messages (in my case files) should not be lost. So I would like to build it in such a way that there is an acknowledgement to the producer for each message received at the consumer.
Messages from a particular producer should be received in the same order as they were produced.
Producers can come and go, and they should be resistant to the consumer being unavailable for some periods of time.
What sort of sockets are appropriate for this kind of application? Any pointers to what kind of zmq pattern I should be looking at would be great.
REQ/REP approach seems to be the best for this task, since the number of messages is low and high reliability is required.
Store the files on each of producers in a way that allows you to find out creation order (time in the filename or file index in db)
Each of producers should select the oldest file, send it to the socket and wait ACK reply. File should be deleted (or marker as delivered) upon ACK.
The consumer should read file content from the socket, flush it to disk and send ACK message afterwards.
The producer should send the next file only after receiving ack from the previous one.
This might work, however I see one major problem here: several producers will flood consumer's network Interface, even if they don't touck the disk or spawn processes on consumer.
This should be a problem in any design with producer-initiated file transfer. PUSH/PULL sockets will have the same problem.
Another point to note: ZeroMQ messages are buffered in memory until the whole message is received. So, 20 producers each sending 50MB file will require at peak 1GB RAM.
As an alternative, I would propose sending to the profucer only the names of files, and pulling files sequentially.