I have a webfarm with webgardens, I want that every process gets notified when an event is raised from other processes, this event is mostly used to update internal caches, eg if a process updates a 'customer', all other processes should get notified about that.
Each process creates a Bus with RabbitMQ, then I setup a single endpoint named randomly where I register my consumers, this seems to work, anyway at the end of the day I find lots of dead queues inside RabbitMQ, for the recycled processes.
I tried to use the temporary queue for this purpose but I do not know how to publish an event to the bus and have my consumers automatically connected to it, I have tried to use the ConnectXX methods but they do not setup all the required bindings in RabbitMQ.
Any ideas? thanks
This code creates a temporary queue:
cfg.ReceiveEndpoint(host, ec => { ... })
Related
I'm trying to configure a lambda function to consume from an SQS queue that I've been given read and delete permissions to, that I do not own/have configuration of. Is there a way to use lambda's SQS trigger functionality for a queue that doesn't exist inside my AWS account?
If not, what are some alternatives that don't include checking the queue on a scheduled event.
If the owner of the SQS queue gives you the necessary permissions (see the setup docs for what those permissions are), you can do this. But, you shouldn't.
Subscribing to someone else's SQS queue is an anti-pattern. This is because a queue represents a backlog of work, and the implicit functionality is that everything that goes in eventually comes out. All the queue does is separate input flow from output flow (data can flow in both faster and slower than they flow out).
This idea of flow, however, means that when something comes out, it's no longer in the queue. (Caveat here: there are work-arounds to this, but they're usually not preferred). A consumer, however, always has the goal of processing everything in the queue. This may be done by multiple threads under the control of one consumer, but the end result is still that everything is processed. If there are multiple consumers, then they by necessity compete with one another, and none of them get to process everything in the queue.
How do we ensure there aren't multiple consumers? Simple: the consumer owns the queue. No other consumer is granted read permissions. It might well be the case that someone other than the consumer controls the filling of the queue (receiving write permissions) - and AWS has the perfect solution for this:
SNS Topics: An SNS topic is a source of data. It is, in effect, a publisher. When someone else wants you to have access to their data, they allow you to become a subscriber to their topic. When a new message is published to the SNS topic, everyone who is subscribed to the topic gets a copy. What happens to that copy is decided by the subscriber: it may be acted upon directly, stored for later action, or acted on indirectly, e.g. by being placed in a queue. This is the Pub-Sub model. It separates the details of one entity (the publisher) creating messages and sending them out to many others, from each recipient's (subscriber's) individual decision about how to consume those messages.
TL;DR: get whoever is currently owning the queue to publish to an SNS topic instead, then set up a queue (or whatever you prefer) subscribed to that topic.
I have a Spring Boot application that is generally message driven but on special occasions, the incoming messages need to be stopped. However I cannot loose those messages, I need to buffer them and receive them in the correct order later.
There are myriads of questions asked and answered about stopping the Listener via the ListernerEndpointRegistry, such as here.
However when I stop the container the AnonymousQueue seems to disappear. I want the queue to stay on the exchange and buffer any messages, and receive them, when I restart. Is this possible or do I need to buffer them inside my application?
There are two options.
Don't use an anonymous (auto-delete) queue and use stop/start.
Don't stop the container; simply block the listener thread(s) when you want to suspend message delivery and wake them when you want to restart.
However I cannot lose those messages
If you cannot lose messages, you should NEVER use an auto-delete queue - you can lose messages at any time if you have a simple network glitch.
I am looking for a way for each consumer instance to receive a message that is published to RabbitMQ via MassTransit. The scenario would be, we have multiple microservices that need to invalidate a cache on notification. Pub-Sub won't work in this instance as there will be 5 consumers of the same type as its the same code per service instance, so only one would receive the message in a traditional PubSub.
Message observation could be an option but this means the messages would never be consumed and hang around forever on the bus.
Can anyone suggest a pattern to use in the context of MassTransit?
Thanks in advance.
You should create a management endpoint in each service, which could even be a temporary queue (just request a receive endpoint without a queue name and one will be dynamically generated). Then, put your queue invalidation consumers on that endpoint. Each service instance will receive a unique instance of the message (when Publish is called), and those queues and bindings will automatically be removed once the service exits.
This is exactly how the bus endpoint works, but in your case, you're creating a receive endpoint which can have consumer message type bindings, so that published messages are received, one copy per service.
cfg.ReceiveEndpoint(cfg => { ... });
Note that the queue name is not specified, and will be automatically generated uniquely.
I have a Mass Transit Service Bus that is listening to several queues and processing the messages. I would like to somehow pause the processing of new requests and wait for the current requests to complete so that I can run some housekeeping tasks.
A few of my own thoughts:
I have investigated the service bus BeforeConsumingMessage handler and although this would allow me to check for a 'Pause processing' flag in my database, I unsure how I would then actually pause the processing!
We are using RabbitMQ - could I use this to put the queues in a suspend state?
I have found so little on this subject that I wonder if it is an 'anti-pattern' and I should just stop my Mass Transit services if I want to run some housekeeping jobs and trust in any partially complete sagas to be picked up when the service bus starts back up. (Rather not go for this option, though).
So my question is: Is there a way to instruct the service bus to finish processing the current sagas but do not take any more messages from the queue?
Cleanly shutting down a MT service will wait for any messages in process to completely finished. Why sometimes it takes a little while for a service to shutdown. Shutting down the service is the best way to handle this, you are sure MT is not pulling any new messages.
If your sagas are serialized to a backing store, e.g. NHibernate, then the state will be saved until the service is restarted and the sagas will pick up in the state they were left after the last message was processed. You should be in good shape. We do this all the time for any maintenance periods.
If you REALLY must leave the service running, call Dispose on the IServiceBus instance. This will do the same thing, letting the current consumers finish then releasing all your resources. Once you have done maintenance you can create a new IServiceBus instance as needed.
In JMS there are Queues and Topics. As I understand it so far queues are best used for producer/consumer scenarios, where as topics can be used for publish/subscribe. However in my scenario I need a way to combine both approaches and create a producer-consumer-observer architecture.
Particularly I have producers which write to some queues and workers, which read from these queues and process the messages in those queues, then write it to a different queue (or topic). Whenever a worker has done a job my GUI should be notified and update its representation of the current system state. Since workers and GUI are different processes I cannot apply a simple observer pattern or notify the GUI directly.
What is the best way to realize this using a combination of queues and/or topics? The GUI should always be notified, but it should never consume anything from a queue?
I would like to solve this with JMS directly and not use any additional technology such as RMI to implement the observer part.
To give a more concrete example:
I have a queue with packages (PACKAGEQUEUE), produced by machine (PackageProducer)
I have a worker which takes a package from the PACKAGEQUEUE adds an address and then writes it to a MAILQUEUE (AddressWorker)
Another worker processes the MAILQUEUE and sends the packages out by mail (MailWorker).
After step 2. when a message is written to the MAILQUEUE, I want to notify the GUI and update the status of the package. Of course the GUI should not consume the messages in the MAILQUEUE, only the MailWorker must consume them.
You can use a combination of queue and topic for your solution.
Your GUI application can subscribe to a topic, say MAILQUEUE_NOTIFICATION. Every time (i.e at step 2) PackageProducer writes message to MAILQUEUE, a copy of that message should be published to MAILQUEUE_NOTIFICATION topic. Since the GUI application has subscribed to the topic, it will get that publication containing information on status of the package. GUI can be updated with the contents of that publication.
HTH