I have durable exchanges and queues in my application. The messages are persistent too. Using this configuration i am sure my messages gets stored in physical storage. I want to know if there is any expiry time when rabbitmq delete messages from my physical storage i mean the hard disk as it maintains the message store in it. Also in case i want to read the messages from physical storage then can i do so?
Durable queue + Persistent messages means indeed the messages will be kept.
Exceptions to this statement out of the top of my head:
you would have configured additional properties to your queues, for exemple limits in size
you reach the limit to the underlying filesystem
you delete the queues (this would delete the messages stored in it too)
As to reading the messages stored in the queues, you can typically consume them.
If you want to read them without them being deleted, you'd have few options:
trick the broker (for example by reading all of them but never acknowledging them, which would have them brought back into the queue)
republish them again to the broker for storage after reading them
But if indeed further conservation is desired, I'd seriously consider storing them somewhere else (DB of some kind) at it's clearly outside of the purpose of a message broker.
Related
When we use the javax.jms.QueueBrowser.getEnumeration() will it only browse the queue content inside the JVM? Will it significantly affect application memory usage?
Also, when we use queue browser to get the queue itself, then so much data actually takes up memory which will not be the case with getEnumeration(). Please help me understand if I am right.
A certain amount of messages are prefetched when you consume a queue with a QueueBrowser. This limit can be configured in ActiveMQPrefetchPolicy on an ActiveMQConnectionFactory. The property queueBrowserPrefetch regulates number of messages received in a batch, and buffered into memory until all are acknowledged.
I want a bunch of several hundred client apps to create and use temporary queues at one instance of the middleware.
Are there some cons regarding performance why I shouldn't use temp queues? Are there limitations, for example on how many temp. queues can be created per HornetQ instance?
On a recent project we have switched from using temporary queues to using static queues on SonicMQ. We had implemented synchronous service calls over JMS where the response of each call would be delivered on a dedicated temporary queue, created by the consumer. During stress testing we noticed that the overhead of temporary queue creation and allocated resources started to play a bigger and bigger part when pushing the maximum throughput of the solution.
We changed the solution so it would use static queues between consumer and provider and use a selector to correlate on the JMSCorrelationID. This resulted in better throughput in our case. If you are planning on each time (re)creating the temporary queues that your client applications will use, it could start to impact performance when higher throughput rates are needed.
Note that selector performance can also start to play when the number of messages in a queue increase. In our case the solution was designed to hand-off the messages as soon as possible and not play the role of a (storage) buffer in between consumer and provider. As such the number of message inside a queue would always be low.
We are using SwifMQ as the JMS infrastructure in our product. In the routerconfig.xml file there is an entry like swap path="./store/swap"/. I wanted to understand when these swap files are created in store/swap. In the customers environment we are seeing swap files under /store/swap with names like hostname-xxx.swap
My assumption is that SwiftMQ uses some datastructure to store messages to be sent. This data-structure might get filled up as it is not able to send those messages because of network issue etc. I presume that in this scenario it will write to swap files. Is my assumption correct?
Any information on this would be appreciated.
Swap is used to store non-persistent messages when the queue cache is full.
if you go to sys$queuemanager swiftlet's queues properties you can see how many messages are configured to store in cache. (default is 500)
if the producer produces over 500 non-persistent messages and the consumer has not consumed, the messages will be written to the .swp file, if the messages are persistent they will always be written to the store/db/page.db directory
I have console application which connects to activemq topics. Abount 10 messages per second are published on each topic. After some time monitored that the application memory is increasing and when all the memory is used the application crashes.
See the dump below. Why is ActiveMQTopicSubsctiber using so much heap? Also it is not visible but the ListEntries are about~14 000 (which means 14k messages).
http://imageshack.us/photo/my-images/404/amqmemoryproblem.png
A couple of things to possibly check for:
In your subscriber are you positive that the messages from the topic actually being consumed?
What is your prefetchLimit specified as?
If holding messages in memory continues to be a problem, you should consider configuring ActiveMQ to use file cursors. The use of file cursors tells ActiveMQ to spool messages to disk instead of holding them in memory.
Concerning ActiveMQ: I have a scenario where I have one producer which sends small (around 10KB) files to the consumers. Although the files are small, the consumers need around 10 seconds to analyze them and return the result to the producer. I've researched a lot, but I still cannot find answers to the following questions:
How do I make the broker store the files (completely) in a queue?
Should I use ObjectMessage (because the files are small) or blob messages?
Because the consumers are slow processing, should I lower their prefetchLimit or use a round-robin dispatch policy? Which one is better?
And finally, in the ActiveMQ FAQ, I read this - "If a consumer receives a message and does not acknowledge it before closing then the message will be redelivered to another consumer.". So my question here is, does ActiveMQ guarantee that only 1 consumer will process the message (and therefore there will be only 1 answer to the producer), or not? When does the consumer acknowledge a message (in the default, automatic acknowledge settings) - when receiving the message and storing it in a session, or when the onMessage handler finishes? And also, because the consumers are so slow in processing, should I change some "timeout limit" so the broker knows how much to wait before giving the work to another consumer (this is kind of related to my previous questions)?
Not sure about others, but here are some thoughts.
First: I am not sure what your exact concern is. ActiveMQ does store messages in a data store; all data need NOT reside in memory in any single place (either broker or client). So you should actually be good in that regard; earlier versions did require that all ids needed to fit in memory (not sure if that was resolved), but even that memory usage would be low enough unless you had tens of millions of in-queue messages.
As to ObjectMessage vs blob; raw byte array (blob) should be most compact representation, but since all of these get serialized for storage, it only affects memory usage on client. Pre-fetch mostly helps with access latency; but given that they are slow to process, you probably don't need any prefetching; so yes, either set it to 1 or 2 or disable altogether.
As to guarantees: best that distributed message queues can guarantee is either at-least-once (with possible duplicates), or at-most-once (no duplicates, can lose messages). It is usually better to take at-least-once, and make clients to de-duping using client-provided ids. How acknowledgement is sent is defiend by JMS specification so you can read more about JMS; this is not ActiveMQ specific.
And yes, you should set timeout high enough that worker typically can finish up work, including all network latencies. This can slow down re-transmit of dropped messages (if worked dies), but it is probably not a problem for you.