How to handle messages bigger than 4MB on z/OS? I can't use segmentation, because it is not supported on z/OS.
Some OS have even bigger limits measured even in kB.
What is the common approach in this case?
You are not hitting a z/OS limit but rather the default maximum message length on WebSphere MQ. Note that the Infocenter says "On z/OS, specify a value in the range zero through 100 MB (104 857 600 bytes)".
To fix this, change the MAXMSGL on any queues and channels through which the message might pass. Don't forget to update the Dead Letter Queue's MAXMSGL as well as transmission queues.
Be aware that MAXMSGL is there to save you! Many people set the value to its highest possible size and then run out of disk space. If the application hits a soft limit such as MAXMSGL or MAXDEPTH the effect is limited and generally recoverable. If the disk space is exhausted, the entire QMgr comes to a screeching halt and all connected apps are impacted.
For more on this, please see the Mice and Elephants article on developerWorks.
UDATE:
Update based on comments asking about specifics of HP NonStop and WMQ V5.3.
Please see the WMQ V5.3 manuals available in the WMQ Documentation library. The second link is the System Administration Guide for WMQ V5.3 on HP NonStop. Message lengths are discussed on P4:
The default maximum message length is 4 MB, although you can increase
this to a maximum length of 100 MB (where 1 MB equals 1 048 576
bytes). In practice, the message length might be limited by:
The maximum message length defined for the receiving queue
The maximum message length defined for the queue manager
The maximum message length defined by the queue
The maximum message length defined by either the sending or receiving application
The amount of storage available for the message
So there's no arbitrarily small max message length on HP NonStop or associated with V5.3 of WMQ.
Maybe message grouping can help you there.
Of course the applications then have to be customized.
Related
This is a general question. Let me say I have a queue manager locally. I have a transmission queue/remote queue definition setup through which I connect to destination queue manager queue. If destination queue manager queue's maximum message length capacity is 1000 and if I put a message length more than that it automatically moves to destination queue manager dead letter queue provided that my transmission queue max message length is more than what I input. This is the expected behaviour. But is there any way on MQ world to handle this and not move it to dead queue? Or is it the sole responsibility of the application that puts this message to not put over the max length?
Thanks in advance.
Changing the default Maximum Message Length (i.e. MAXMSGL) from 4MB to a small value is a bad idea.
Myth: MQ does NOT allocate space based on the value in maximum message length field. Setting it to a very small value or to a very large value has not bearing on the disk space. MQ ONLY writes the size/amount of the real message.
Secondly, the application team should tell the MQAdmin what the largest message will be that the application will send. If they say 10MB then the MQAdmin can increase max. message length to 10MB or something a little larger i.e. 12MB.
The largest value that can be used is 100MB.
Note: The MQAdmin will need to increase the max. message length for: channel, XMITQ, local queue and the dead letter queue for any message larger than the default size of 4MB or it will not flow.
Thanks Roger and JoshMc. Infact I tried both options, client to QM and between QM and QM. Client to QM is fine as the client receives the error code and basically nothing happens. But the concern is only between sender QM and receiver QM. What I have seen mostly is that there will be only one transmission queue with maximum message length to connect to a particular queue manager. All the different remote connection/queues use that transmission queue. So if the sender commits a mistake by sending a large message than the destination queue cannot accept, it usually end up passing through the transmission queue but failing in destination and reaching the destination's dead queue. Now the destination owner is alerted/or need to take some remediation for a mistake that he didn't commit. That's the whole reason for me asking this question. Thanks a lot for you both on shedding more light and spending your time for me.
I think Morag Hughson has given out something for me to try out but still it will have its own negative impact. But I was looking for something like that where we can control at the MQ level to not allow the message to go to destination dead queue.
According to the SQS documentation, the maximum number of inflight messages is set to 120,000 for standard queues. However, sometimes I see my queues maxing out at lower numbers, such as here:
Does anyone know why this might be the case? I have code that dynamically changes the number of SQS listeners depending on the number of messages in the queue, but I don't want to do anything if I've hit the maximum. My problem is now that the max limit doesn't seem to be consistent. Some queues go to 120K, but this one is stuck at 100K instead, and as far as I can tell there is no setting that allows me to set this limit.
approximateNumberOfMessagesNotVisible indicates the number of messages in-flight, as you are rightly said. It depends on how many consumers you have, and what is througput of each consumer.
If the actual number is caping at 100k, then your consumers are swamped and have no more receiving capacity.
Anyways, it's better if you provide more info on the use-case as 100k in-flight messages look out of ordinary and you may be not using correct solution for your problem.
Is there a way to find the total number of messages that pass through an IBM websphere MQ queue manager over a specific period of time?
This sounds like a perfect use of the MQ Accounting and Statistics feature. Among other things, these features record the number of messages (with a persistent and non-persistent count) and also number of bytes (not all messages are the same size).
You can turn on Accounting and Statistics for just a selection of queues and/or channels or for everything.
Further Reading:
Accounting and statistics messages
Turning on Queue Accounting
Turning on Queue Statistics
Supplied tool to view the output
From the docs, I understand that WebSphere MQ can support messages up to size 100 MB. What is the size limit for the SIBus queues in WAS? Can it also support messages up to 100 MB?
Is there a limit on the file sizes that can be handled by the RFHUtil tool?
Appreciate any pointers.
Thanks.
According to the WAS Infocenter here, SIBus is designed for messages up to 40MB on 32-bit systems.
RFHUtil should handle messages up to the max size allowed. However, be aware that message properties may or may not count towards that size, depending on tuning. The Q program from SupportPac MA01 should also handle messages of any allowable size.
As the title says — is it possible to declare a maximum queue size and broker behaviour when this maximum size is reached? Or is this a broker-specific option?
I ask because I'm trying to learn about AMQP, not because I have this specific problem with any specific broker… But broker-specific answers would still be insightful.
AFAIK you can't declare maximum queue size with RabbitMQ.
Also there's no such setting in the AMQP sepc:
http://www.rabbitmq.com/amqp-0-9-1-quickref.html#queue.declare
Depending on why you're asking, you might not actually need a maximum queue size. Since version 2.0 RabbitMQ will seamlessly persist large queues to disk instead of storing all the messages in RAM. So if your concern the broker crashing because it exhausts its resources, this actually isn't much of a problem in most circumstances - assuming you aren't strapped for hard disk space.
In general this persistence actually has very little performance impact, because by definition the only "hot" parts of the queue are the head and tail, which stay in RAM; the majority of the backlog is "cold" so it makes little difference that it's sitting on disk instead.
We've recently discovered that at high throughput it isn't quite that simple - under some circumstances the throughput can deteriorate as the queue grows, which can lead to unbounded queue growth. But when that happens is a function of CPU, and we went for quite some time without hitting it.
You can read about RabbitMQ maximum queue implementation here http://www.rabbitmq.com/maxlength.html
They do not block the incoming messages addition but drop the messages from the head of the queue.
You should definitely read about Flow control here:
http://www.rabbitmq.com/memory.html
With qpid, yes
you can confire maximun queue size and politic in case raise the maximum. Ring, ignore messages,broke connection.
you also have lvq queues (las value) very configurable
There are some things that you can't do with brokers, but you can do in your app. For instance, there are two AMQP methods, basic.get and queue.declare, which return the number of messages in the queue. You can use this to periodically get a count of outstanding messages and take action (like start new consumer processes) if the message count gets too high.