I have a spring boot application which needs to consume different types of events. The events will be published by another service to an SQS queue.
My question is, do I have to create one SQS queue where all the events will be published/consumed or do I have to create one SQS queue for each and every event that will be published/consumed?
It is possible to create a queue for each message, but I don't recommend it ass you'll have to add a listener to each queue every time you add a new message.
First thing that you can do is to create a message structure that holds the topic inside it. Once your SQS handler receives a message, it will check the topic and route it to the right handler or handlers.
I still don't like this option because this means what you are implementing is a message broker and I don't think that's what you are aiming for.
The best solution would be to use SNS and publish each message to a different topic. Then configure SNS to route the message to that SQS and let your spring SQS message handler to get the right handler by the topic.
While this solution is similar to the previous one, the use of SNS here gives you the ability to publish a message to more than one queue (client) and handles, and you get the topic outside the message which is where it should be.
Related
I am learning how DLQ works in Azure service bus queue. i.e., unconsumed messages will be in DLQ. I have enabled dead lettering (deadLetteringOnMessageExpiration) on message expiration.
References:
Azure Service Bus - Subscriptions and DLQ
Azure Service Bus - *move* message from DLQ to main
https://learn.microsoft.com/en-us/azure/service-bus-messaging/enable-dead-letter
AMR template:
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-resource-manager-namespace-queue
Questions:
If deadLetteringOnMessageExpiration is enabled, would DLQ be available for that queue (like below screenshot)?
If yes, how can I process messages from DLQ? (I guess I can view such messages here but not sure what will happen next)
My goal is to create a queue with DLQ where unprocessed message can be processed at some point and what is the best way to achieve that.
If deadLetteringOnMessageExpiration is enabled, would DLQ be available for that queue?
Dead-letter queue is always there for queues and subscriptions, unrelated to how you configure your entity.
If yes, how can I process messages from DLQ?
Up to you. You can peek DLQ-ed messages, recieve and process, etc. It really depends on how you want to handle those dead-lettered messages in the context of your system.
I'm working on an node js application, called ngdf-diversion-client, that is running in AWS and is connecting to a Solace instance, that is also running in AWS.
I see that through the ngdf-diversion-client config file its receiving messages over the ngdf/diversions queue, and I can see that in the SolAdmin.
This image here shows the application ngdf-diversion-client listed in the SolAdmin tool:
This image shows ngdf-diversion-client being the owner (and in my case the recipient of data) on the ngdf/diversions queue:
But I cannot see who sends data to ngdf-diversion-client on the ngdf/diversions queue.
Does anyone know how I can see that information in Solace or SolAdmin?
I know with ActiveMQ when you select a Queue or Topic from the web console, you can see who are the consumers and producers so it was pretty easy seeing who sent and who received data over a queue or topic.
But with Solace/SolAdmin I don't see that.
With Solace, publishers and subscribers are decoupled by design so there is no way to see which publishers have sent messages to a specific queue. If it is required that the publisher of a message is identifiable then you could implement an identifier in the message as it is published in the application.
I am using Nifi to get data from IBM MQ. It is working fine. My question is once the message is read from an MQ queue, does it get deleted from the queue? How to just read messages from the queue without deleting them from the queue?
My question is once the message is read from an MQ queue, does it get
deleted from the queue?
Yes, that is the default behavior.
How to just read messages from the queue without deleting them from
the queue?
You use the option: MQGMO_BROWSE_FIRST followed by MQGMO_BROWSE_NEXT on the MQGET API calls.
You can also open the queue for browse only. i.e. MQOO_BROWSE option for MQOPEN API call.
It sounds as if you would like to use a "publish/subscribe" model rather than a "point-to-point" model.
From ActiveMQ:
Topics In JMS a Topic implements publish and subscribe semantics. When
you publish a message it goes to all the subscribers who are
interested - so zero to many subscribers will receive a copy of the
message. Only subscribers who had an active subscription at the time
the broker receives the message will get a copy of the message.
Queues A JMS Queue implements load balancer semantics. A single
message will be received by exactly one consumer. If there are no
consumers available at the time the message is sent it will be kept
until a consumer is available that can process the message. If a
consumer receives a message and does not acknowledge it before closing
then the message will be redelivered to another consumer. A queue can
have many consumers with messages load balanced across the available
consumers.
If you have a queue, when a consumer consumes that message, it is removed from the queue so that the next consumer consumes the next message. With a topic, multiple consumers can be subscribed to that topic and retrieve the same message without being exclusive.
If neither of these work for you, I'm not sure what semantics you're looking for -- a "queue" which doesn't delete the message when it is consumed will never let a consumer access any but the first message.
I am looking for a way for each consumer instance to receive a message that is published to RabbitMQ via MassTransit. The scenario would be, we have multiple microservices that need to invalidate a cache on notification. Pub-Sub won't work in this instance as there will be 5 consumers of the same type as its the same code per service instance, so only one would receive the message in a traditional PubSub.
Message observation could be an option but this means the messages would never be consumed and hang around forever on the bus.
Can anyone suggest a pattern to use in the context of MassTransit?
Thanks in advance.
You should create a management endpoint in each service, which could even be a temporary queue (just request a receive endpoint without a queue name and one will be dynamically generated). Then, put your queue invalidation consumers on that endpoint. Each service instance will receive a unique instance of the message (when Publish is called), and those queues and bindings will automatically be removed once the service exits.
This is exactly how the bus endpoint works, but in your case, you're creating a receive endpoint which can have consumer message type bindings, so that published messages are received, one copy per service.
cfg.ReceiveEndpoint(cfg => { ... });
Note that the queue name is not specified, and will be automatically generated uniquely.
Let's suppose I have several subscribers consuming from a topic. After a message has been delivered to all the subscribers I'd like to trigger a job that would use this message in input.
So the easy way to do that would be to move messages that have been succesfully delivered to all the sucscribers to a queue from which my job would consume messages.
Is it part of JMS?
Is there any message broker able to do that directly?
If not is there a simple solution to solve this problem?
You should be able to do this using activemq's advisories.
See here for more about advisory messages: http://activemq.apache.org/advisory-message.html
So what you want to do, for the topic in question, is track:
the number of consumers
when a message is dispatched to them
when the message has been ack'd by each of the consumers
to get the number of consumers, listen to the "ActiveMQ.Advisory.Consumer.Topic." advisory topic
to get when a message is dispatched, listen to the "ActiveMQ.Advisory.MessageDelivered.Topic."
to get when a message has been ack'd, listen to "ActiveMQ.Advisory.MessageConsumed.Topic."
you could easily use Apache Camel to help out with this (listening to the topics) and aggregating whether or not all consumers have processed (ack'd) the message.. then that could kick off your further processing..
You could just create another durable subscription to route the message from the topic to queue directly. From that queue your job can consume messages. This is much easier than creating a trigger to route the messages to a queue.
So the easy way to do that would be to move messages that have been
succesfully delivered to all the sucscribers to a queue from which my
job would consume messages. Is it part of JMS?
No, this is not part of JMS specification.