From the docs, I understand that WebSphere MQ can support messages up to size 100 MB. What is the size limit for the SIBus queues in WAS? Can it also support messages up to 100 MB?
Is there a limit on the file sizes that can be handled by the RFHUtil tool?
Appreciate any pointers.
Thanks.
According to the WAS Infocenter here, SIBus is designed for messages up to 40MB on 32-bit systems.
RFHUtil should handle messages up to the max size allowed. However, be aware that message properties may or may not count towards that size, depending on tuning. The Q program from SupportPac MA01 should also handle messages of any allowable size.
Related
According to the SQS documentation, the maximum number of inflight messages is set to 120,000 for standard queues. However, sometimes I see my queues maxing out at lower numbers, such as here:
Does anyone know why this might be the case? I have code that dynamically changes the number of SQS listeners depending on the number of messages in the queue, but I don't want to do anything if I've hit the maximum. My problem is now that the max limit doesn't seem to be consistent. Some queues go to 120K, but this one is stuck at 100K instead, and as far as I can tell there is no setting that allows me to set this limit.
approximateNumberOfMessagesNotVisible indicates the number of messages in-flight, as you are rightly said. It depends on how many consumers you have, and what is througput of each consumer.
If the actual number is caping at 100k, then your consumers are swamped and have no more receiving capacity.
Anyways, it's better if you provide more info on the use-case as 100k in-flight messages look out of ordinary and you may be not using correct solution for your problem.
I have a process with max flow limit enabled. The value being set at 10. Its a Asyn process and used to get thousands of messages daily. We noticed that at peak time, with the increase in messages in queue in EMS server, the performance of the tibco process decline. Is there is any dependency between slowness in Tibco with increased inflow of EMS messages. How to calculate the exact flow limit for a process ? do we have any standard procedure ?
The FlowLimit configuration setting is a BusinessWorks setting, so I am assuming that you have BusinessWorks engines that are consuming messages from an EMS queue.
The concept of flow control exists in order to ensure that the number of incoming evens to a BusinessWorks engine does not cause the JVM to exceed its available memory resources. BusinessWorks implements the flow control by temporarily disabling the process starter until the number of jobs in memory falls below a threshold. In the case of EMS-based process starters this entains closing the MessageConsumer, which causes EMS to stop delivering messages to the process. In high-volume messaging scenarios this will cause a backlog of messages on the EMS server. Additionally it will cause any message in the prefetch cache on the client-side to be re-prioritzed for re-delivery on the EMS server side. When this happens you will notice that your outbound message count is greater than you inbound message count in your EMS statistics.
You are best off avoiding getting into flow-controlled scenarios. Is your current FlowLimit parameter realistic for the heap size you are allotting your JVM and the message payload sizes you are working with? Can you increase your JVM heap size and also your FlowLimit? Are you able to run multiple instances of the BusinessWorks application dispatching off the same queue in order to increase scalability? The approaches may help you scale and avoid message backlogs.
Is there a way to find the total number of messages that pass through an IBM websphere MQ queue manager over a specific period of time?
This sounds like a perfect use of the MQ Accounting and Statistics feature. Among other things, these features record the number of messages (with a persistent and non-persistent count) and also number of bytes (not all messages are the same size).
You can turn on Accounting and Statistics for just a selection of queues and/or channels or for everything.
Further Reading:
Accounting and statistics messages
Turning on Queue Accounting
Turning on Queue Statistics
Supplied tool to view the output
How to handle messages bigger than 4MB on z/OS? I can't use segmentation, because it is not supported on z/OS.
Some OS have even bigger limits measured even in kB.
What is the common approach in this case?
You are not hitting a z/OS limit but rather the default maximum message length on WebSphere MQ. Note that the Infocenter says "On z/OS, specify a value in the range zero through 100 MB (104 857 600 bytes)".
To fix this, change the MAXMSGL on any queues and channels through which the message might pass. Don't forget to update the Dead Letter Queue's MAXMSGL as well as transmission queues.
Be aware that MAXMSGL is there to save you! Many people set the value to its highest possible size and then run out of disk space. If the application hits a soft limit such as MAXMSGL or MAXDEPTH the effect is limited and generally recoverable. If the disk space is exhausted, the entire QMgr comes to a screeching halt and all connected apps are impacted.
For more on this, please see the Mice and Elephants article on developerWorks.
UDATE:
Update based on comments asking about specifics of HP NonStop and WMQ V5.3.
Please see the WMQ V5.3 manuals available in the WMQ Documentation library. The second link is the System Administration Guide for WMQ V5.3 on HP NonStop. Message lengths are discussed on P4:
The default maximum message length is 4 MB, although you can increase
this to a maximum length of 100 MB (where 1 MB equals 1 048 576
bytes). In practice, the message length might be limited by:
The maximum message length defined for the receiving queue
The maximum message length defined for the queue manager
The maximum message length defined by the queue
The maximum message length defined by either the sending or receiving application
The amount of storage available for the message
So there's no arbitrarily small max message length on HP NonStop or associated with V5.3 of WMQ.
Maybe message grouping can help you there.
Of course the applications then have to be customized.
As the title says — is it possible to declare a maximum queue size and broker behaviour when this maximum size is reached? Or is this a broker-specific option?
I ask because I'm trying to learn about AMQP, not because I have this specific problem with any specific broker… But broker-specific answers would still be insightful.
AFAIK you can't declare maximum queue size with RabbitMQ.
Also there's no such setting in the AMQP sepc:
http://www.rabbitmq.com/amqp-0-9-1-quickref.html#queue.declare
Depending on why you're asking, you might not actually need a maximum queue size. Since version 2.0 RabbitMQ will seamlessly persist large queues to disk instead of storing all the messages in RAM. So if your concern the broker crashing because it exhausts its resources, this actually isn't much of a problem in most circumstances - assuming you aren't strapped for hard disk space.
In general this persistence actually has very little performance impact, because by definition the only "hot" parts of the queue are the head and tail, which stay in RAM; the majority of the backlog is "cold" so it makes little difference that it's sitting on disk instead.
We've recently discovered that at high throughput it isn't quite that simple - under some circumstances the throughput can deteriorate as the queue grows, which can lead to unbounded queue growth. But when that happens is a function of CPU, and we went for quite some time without hitting it.
You can read about RabbitMQ maximum queue implementation here http://www.rabbitmq.com/maxlength.html
They do not block the incoming messages addition but drop the messages from the head of the queue.
You should definitely read about Flow control here:
http://www.rabbitmq.com/memory.html
With qpid, yes
you can confire maximun queue size and politic in case raise the maximum. Ring, ignore messages,broke connection.
you also have lvq queues (las value) very configurable
There are some things that you can't do with brokers, but you can do in your app. For instance, there are two AMQP methods, basic.get and queue.declare, which return the number of messages in the queue. You can use this to periodically get a count of outstanding messages and take action (like start new consumer processes) if the message count gets too high.