I am using github.com/nsqio/go-nsq Go package to work with NSQ, and I've met following problem. When producer writes a message - it creates a topic, but not a channel, and it seems that NSQ server just discards a messages when there are no channels. I don't want to lose messages, also I don't want to rely on consumer and producer startup order. So a possible solution is to create a channel on producer side, so when it writes it will not discard messages. How can I create and destroy a consumer without consuming a message, just for sake of channel creation?
sequence of
nsq.NewConsumer()
consumer.AddConcurrentHandlers()
consumer.ConnectToNSQLookupd()
can consume
removing AddConcurrentHandlers leads to error
From protocol spec I cannot see how is it supposed to do, other that sending SUB followed by CLS, but because those are two commands and non "atomic" kind of op - something theoretically may happen in between..
So probably I am trying to do that wrong? RABBITMQ for example can pre-create it, is here something like that?
Related
ActiveMQ 5.15.13
Context: I have a single queue with multiple Consumers. I want to stop some consumers from processing certain messages. This has to be dynamic, I don't want to create separate queues for this. This works without any problems. e.g. Consumer1 ignores Stocks -> Consumer1 can process all invoices and Consumer2 can process all Stocks
But if there is a large number of messages already in the Queue (of one type, e.g. stocks) and I send a message of another type (e.g. invoices), Consumer1 won't process the message of type invoices. It will instead be idle until Consumer2 has processed all Stocks messages. It does not happen every time, but quite often.
Is there any option to change the order of the new messages coming into the queue, such that an idle consumer with matching selector picks up the new message?
Things I've already tried:
using a PendingMessageLimitStrategy -> it seems like it does not work for queues
increasing the maxPageSize and maxBrowsePageSize in the hope that once all Messages are in RAM, the Consumers will search for their messages.
Exclusive Consumers aren't an option since I want to be able to use more than one Consumer per message type.
Im pretty sure that there is some configuration which allows this type of usage. I'm aware that there are better solutions for this issue, but sadly I can't use them easily due to other constraints.
Thanks a lot in advance!
EDIT: I noticed that when I'm refreshing on the localhost queue browser, the stuck messages get executed immediately. It seems like this action performs some sort of queue refresh where the messages get filtered based on their selector again. So I just need this action whenever a new message enters the queue...
This is a 'window' problem where the next set of 'stocks' data needs to be processed before the 'invoicing' data can be processed.
The gotcha with window problems like this is that you need to account for the fact that some messages may never come through, or a consumer may never come back online either. Also, eventually you will be asked 'how many invoices or stocks are left to be processed'-- aka observability.
ActiveMQ has you covered-- check out wild-card destinations and consumers.
Produce 'stocks' to:
queue://data.stocks.input
Produce 'invoices' to:
queue://data.invoices.input
You then setup consumes to connect:
queue://data.*.input
note: the wildard '*'.
ActiveMQ will match queues based on the wildcard pattern, and then process data accordingly. As a bonus, you can still use a selector.
Currently, in my project, I'm using spring cloud stream with rabbitmq. Everything is okay, but the problem is when in StreamListener I set a condition for the message type. If RabbitMQ doesn't match type (Cannot find a #StreamListener matching for a message with id: ZZZZZZ) then this message disappeared from the queue. I don't want to remove this message if the message has a bad type. Is some solution for this problem?
There are two ways you can do it:
Throw an exception if the message has bad type. In this case, the message will not be removed from the queue since it wasn't processed successfully and no ACK was received.
Insert the message in the queue yourself once you have detected the type is bad.
Both approaches can suffer what is called an infinite loop. The message is processed, it is of bad type, it is re-inserted, and this repeats. To avoid this you can add some policy of reinserting, like: exponenetial delay, or a limited number of re-insertions etc.
But some doubt arises: why is your service consuming messages that it shouldn't? Perhaps you need a specific processor this messages? In this case you can route to the suitable processor.
I am putting a message containing string data to rabbitmq queue.
Message publishing is called as a part of a service and the service can be called with same data (data goes to the queue) multiple times, thus chances for having duplicated data in the queue is very likely.
We have issues with this as the consumer code is inserting this data to table where this data is primary key. Consumer will be called from 4 different nodes simultaneously thus chances for having consumers consuming same data (from different messages) can happen.
I want to know if rabbitMQ publishing has any way to avoid message duplication.
Read "define a property "x-unique-message-code" to compare them is an easy and simple way" , but don't know how to do it.
I am using spring-amqp
Any help is highly appreciated.
Thank you
There is a good article from RabbitMQ about reliability: https://www.rabbitmq.com/reliability.html
There is a note like:
In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.
For this purpose the message to produce can be supplied with a messageId property.
I am using camel to integrate with ActiveMQ JMS. I am receiving prices for products on this queue. I am using JMSXGroupID on productId to ensure ordering across a productId. Now if I fail to process this message I move it to a DeadLetterQueue. This could be because of a connection error on a dependent service or because of error with the message itself.
In case of the former I would have to manually remove it from the DLQ and put it back into the JMS queue.
Now the problem is that I dont know if any other message on that groupId has been received and processed or not. And hence unsidelining from DLQ will disrupt the order. On the other hand if I dont unsideline it and no other message has been received the product Id will not get the correct price.
1 solution that I have in mind is to use a fast key-value store(Redis) to store the last messageId or JMSTimestamp against a productId(message group). This is updated everytime I dequeue a message. Any other solution for this?
Relying on message order in JMS is a risky business - at best.
The best thing to do is to make the receiver handle messages out of sequence as a special case (but may take advantage message order during normal operation).
You may also want to distinguish between two errors: posion messages and temporary connection problems, maybe even use two different error queues for them. In the case of a posion message (invalid payload etc.) then there is nothing you can really do about it except starting a bug investigation. In such cases, you can probably send along "something else", such as dummy message to not interfere with order.
For the issues with connection problems, you can have another strategy - ActiveMQ Redelivery Policies. If there is network trouble, it's usually no use in trying to process the second message until the first has been handled. A Redelivery Policy ensures that (given you have a single consumer, that is). There is another question at SO where the poster actually has a solution to your problem and wants to avoid it. Read it. :)
I have a single AMQ queue that receives simple messages with string body. Consider I'm sending CLSIDs as message bodies. CLSIDs could be not unique, but I'd like to reject all messages with not unique bodies and keep only single instance of such messages in the queue. Is there any simple way to do it?
Currently I'm using a workaround. Messages from the queue are consumed by some processor that tries to insert bodies into a simple DB table with UNIQUE constraint applied to message_body field. If processor inserts the messages succesfuly - it's assigned to exchange.out.body and sent to other queue. If ConstraintViolationException is thrown - nothing is resent to other queue.
I would like to know does AMQ support something similar out of the box?
I believe you can write an interceptor for activemq where you can perform certain actions on messages. Check out: http://activemq.apache.org/interceptors.html
That being said, in my personal opinion this is bad practice. ActiveMQ is a messaging system which should only be responssible for transport of the message. All logic can beter be performed using your application ( either make sure the sender cannot send the same message more then once OR , create an intermediate consumer which indeed matches the received body with a database that contains already seen message bodies BEFORE, routing the message to the actual receiver queue)