Is it possible to check live traffic in a queue? (I mean if somebody put/get a msg in the queue , there should be an activity log, or something like that) Where it goes? where it came from?
Count the message number or any other alternate method?
For example:
Random APP1 put a message to a queue (Q.RND1)
Random APP2 get a message from (Q.RND1)
Could we live track the message (to - from) ?
Related
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
EDIT: Solved this one while I was writing it up :P -- I love those kind of solutions. I figured I'd post it anyway, maybe someone else will have the same problem and find my solution. Don't care about points/karma, etc. I just already wrote the whole thing up, so figured I'd post it and the solution.
I have an SQS FIFO queue. It is using a dead letter queue. Here is how it had been configured:
I have a single producer microservice, and I have 10 ECS images that are running as consumers.
It is important that we process the messages close to the time they are delivered in the queue for business reasons.
We're using a fairly recent version of the AWS SDK Golang client package for both producer and consumer code (if important, I can go look up the version, but it is not terribly outdated).
I capture the logs for the producer so I know exactly when messages were put in the queue and what the messages were.
I capture aggregate logs for all the consumers, so I have a full view of all 10 consumers and when messages were received and processed.
Here's what I see under normal conditions looking at the logs:
Message put in the queue at time x
Message received by one of the 10 consumers at time x
Message processed by consumer successfully
Message deleted from queue by consumer at time x + (0-2 seconds)
Repeat ad infinitum for up to about 700 messages / day at various times per day
But the problem I am seeing now is that some messages are not being processed in a timely manner. Occasionally we fail processing a message deliberately b/c of the state of the system for that message (e.g. maybe users still logged in, so it should back off and retry...which it does). The problem is if the consumer fails a message it is causing the queue to stop delivering any other messages to any other consumers.
"Failure to process a message" here just means the message was received, but the consumer declared it a failure, so we just log an error, and do not proceed to delete it from the queue. Thus, the visibility timeout (here 5m) will expire and it will be re-delivered to another consumer and retried up to 10 times, after which it will go to the dead letter queue.
After delving into the logs and analyzing it, here's what I'm seeing:
Process begins like above (message produced, consumed, deleted).
New message received at time x by consumer
Consumer fails -- logs error and just returns (does not delete)
Same message is received again at time x + 5m (visibility timeout)
Consumer fails -- logs error and just returns (does not delete)
Repeat up to 10x -- message goes to dead-letter queue
New message received but it is now 50 minutes late!
Now all messages that were put in the queue between steps 2-7 are 50 minutes late (5m visibility timeout * 10 retries)
All the docs I've read tells me the queue should not behave this way, but I've verified it several times in our logs. Sadly, we don't have a paid AWS support plan, or I'd file a ticket with them. But just consider the fact that we have 10 separate consumers all reading from the same queue. They only read from this queue. We don't have any other queues it is using.
For de-duplication we are using the automated hash of the message body. Messages are small JSON documents.
My expectation would be if we have a single bad message that causes a visibility timeout, that the queue would still happily deliver any other messages it has available while there are available consumers.
OK, so turns out I missed this little nugget of info about FIFO queues in the documentation:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html
When you receive a message with a message group ID, no more messages
for the same message group ID are returned unless you delete the
message or it becomes visible.
I was indeed using the same Message Group ID. Hadn't given it a second thought. Just be aware, if you do that and any one of your messages fails to process, it will back up all other messages in the queue, until the time that the message is finally dealt with. The solution for me was to change the message group id. There is some business logic id I can postfix on it that will work for me.
Just want to confirm the correct way the MQ delivers messages to the MQOutput node. Recently came across a situation where i a felt bit confused. Here is the scenario.
I have a local queue on Qmgr,say(A) which receives messages from applications and have a local broker associated with this qmgr(A) with a message flow deployed, which consumes messsages from this queue and drops it to another local(L.B) queue on Queue manager (B).
To successfully deliver the messages to qmgr(B) do i have to
Create a remote queue definition on Qmgr(A) with transmission queue name matching the remote queue manager name, here(B)
MQOutput node value set as, queue->remote queue definition name on (A) and queue manager value as blank
or
to create only the transmission queue that matches with the name of the remote queue manager name, here(B).
MQOutput node value set as queue-> target local queue (L.B) and Queue manager value as (B).
When i follow with the first process, noticed messages reaching the destination and when i follow with the later one, noticed messages sitting up in the local queue itself.
Is there any necessity to always create 'n' number of remote queue definitions when it needs to drop messages to 'n' number of local queues?
Kindly guide me to better understand this. Thanks in advance to each of you.
There is no necessity to create n remote queue definitions, MQ is happy to accept output marked as destined for "Queue Name" on "Queue Manager Name".
You say that when using method 2. that your messages are "sitting up in the local queue". There are a few things you must check to solve this problem.
I assume the named queue L.B is defined on QMgr B and not QMgr A?
I assume the local queue the messages are sitting on is a transmission queue?
Have you defined a channel to read messages from the transmission queue they are stuck on?
Have you started the channel which should be moving the messages off the transmission queue to QMgr B?
When a local queue manager receives the following message in it's AMQ error log:
09/13/12 08:00:19 - Process(3017.20) User(mqm) Program(amqrmppa_nd)
AMQ9544: Messages not put to destination queue.
EXPLANATION: During the processing of channel 'TO_QM_QD2T1_C1' one or
more messages could not be put to the destination queue and attempts
were made to put them to a dead-letter queue. The location of the
queue is 2, where 1 is the local dead-letter queue and 2 is the remote
dead-letter queue.
... what is the mechanism by which MQ exchanges such information? Is there a built in facility within the channel program API itself, or is the info exchanges as discrete messages placed on the SYSTEM.CLUSTER.COMMAND.QUEUE (in the case of a cluster)? Given that this could occur in a remote queue definition situation, with only simple sender/receiver channel pairs, and no corresponding COMMAND QUEUE necessarily, I could imagine that it would be handled via the channel process communications... just wondering...
The channel agents have a bi-directional communication between them, even though messages flow in only one direction. When a message fails to find the destination at the remote end there are several possibilities for what happens next. The channel will only continue to run if the message can be successfully put somewhere and the first place to try is the remote DLQ. If that fails, the local MCA must either relocate the message or stop the channel. Therefore, the two message channel agents work out between them what happens and what the status of the channel should be.
The peculiar wording of the error message reflects that the different dispositions of the message derive from within the same code path and exception handling and the optimization of WMQ. The MCA knows the message was put to a DLQ at that point and rather than having two different error messages or logic to work out the wording on which DLQ was used, it just drops a number into a template. Thus a single error message and streamlined logic are used for both possibilities.
We have quartz process that polls a ActiveMQ JMS queue.
We know that we could get several messages a minute would like to only respond to the most current message at a configured polling rate of a minute or more.
We don't need to process any of the previous messages.
Is there a way to configure the queue to get this behavior?
Its seems like a topic has the ability to do this via the subscription recovery policy using a count of 1. We would like to do this using a queue to guarantee (more or less) a single delivery of the message.
Or is there a conceptual flaw in our assumptions...
Thanks
In my opinion there is no standard operation for this, so you will have to write some code....
One possible solution would be to use a QueueBrowser together with a QueueReceiver:
Through the QueueReceiver you would get an Enumeration of the messages in the queue. For each message you can now perform a receive with a MessageSelector on the JMSMessageID as long as hasMoreElements() returns true. The last message will be the one you want to have....
When using activemq, you can use "image caching" on topics. One of the settings there is to always keep the last mesage sent..
Take a look at the Subscription recovery Policy settings:
http://activemq.apache.org/subscription-recovery-policy.html