I deployed a stream with "module.*.count=0" deployment descritptor in order to force direct binding. It happens by using rabbitmq-trace that there are messages going through rabbit however I see less messages than if direct binding option was disabled. Is this the expected behavior?
I thought, by reading the docs, that nothing would go through external bus in this particular case.
You should not see any rabbit traffic for such a case.
When a producer (source, processor) is bound, the rabbit binding is converted to a direct binding if possible (this container has an appropriate matching consumer [sink, processor], which should happen with count=0).
Take a look at the logs, while deploying the producer module, successful conversions will result in this log...
logger.info("Producer bound directly: " + binding);
If the consumer doesn't exist, you'll see
logger.info("declaring queue for outbound: " + name);
Related
First, let me explain what I have tried with classic ActiveMQ which worked perfectly for my requirements:
I have few Queues with naming templates and each queue represents a Tenant(customer). The naming pattern is like queue.<<tenant-id>>.event which, here, I used test1 to test5 for simplicity.
Multiple producers are putting messages on these different queues based on which tenants are requesting it.
My ActiveMQ queues look like this in the web console:
Queues in the classic ActiveMQ
Then I started the Spring JMS listener with the wildcard to be able to read from all of these queues with one listener. the code is like this:
#JmsListener(destination = "queue.>")
public void receiveMessage(Event event) {
//Process the event message
}
What I have observed which I cannot configure Artemis to do the same is:
Listening on ActiveMQ Queues with wildcard did not create a new queue(listener Queue)
Consuming the messages with a wildcard listener would actually reduce the number of pending messages in the actual queues.
The wildcard listener would actually quite fairly read messages from all queues. It still does respect the FIFO on each queue but would not respect it cross queues. For example, when I put 100 messages in the queue.test1.event and only then add 100 messages in queue.test2.event, then if I start the wildcard listener, it starts to read messages fairly from both queues, although all the messages in queue.test2.event queue are basically added after the 100 messages in queue.test1.event.
I need features #2 and #3. The first one is just the observation which I think is the root cause of my problem in Artemis.
Now, what happened when I moved to Artemis is:
The wildcard pattern is a little different but I did the same scenario. The listener looks like:
#JmsListener(destination = "queue.#")
public void receiveMessage(Event event) {
//Process the event message
}
As you see the wildcard template is changed to queue.# to be able to read from all those queues.
My Artemis queues look like this in the web console:
Queues in the Artemis,
My observation on the web console shows I cannot achieve the same here:
As you see in the picture, the number of message count in the original queues, in which I put the messages, are still kept, despite only 44 of them are remained for processing(looks at the message count of queue.#) and the rest has been already read by the wildcard listener.
This can cause a storage issue for me since all of my messages are persisted and I can't play with the message expiry too.
As you see in the picture, the listener created another Queue named queue.# which seems Artemis is internally copying the messages from the other ones into it.
Not a problem and just an observation.
It respects the FIFO across all queues, which I guess is because Artemis is doing the copy from the original queues to the wildcard one.
This creates a huge problem for me. Although I still want it to respect the FIFO inside each queue, I also want it to start consuming messages from other queues. Because, if one customer is processing huge tasks, it should not block others to continue theirs.
PS1: I restrict the listeners in both tests to just consume one message at a time to be able to test it properly.
PS2: If you wonder why I don't use classic ActiveMQ if it does exactly what I need, The answer is: Apache will make Artemis its Major version(once it reached a certain level of maturity) in the future and I would like to be aligned with the roadmap.Quote from its website:
Once Artemis reaches a sufficient level of feature parity with the "Classic" code-base it will become the next major version of ActiveMQ
PS3: I am using spring-boot and its starter packages to connect and put/consume messages.
PS4: I am using the default configuration for both solutions and installations.
Simply put, ActiveMQ Artemis doesn't support wildcard consumers. It only supports wildcard addresses which have similar but different semantics (as explained in the answer on this question of yours).
Feel free to open an issue to request this feature be implemented.
TLDR: Can't seem to pass messages from one RabbitMQ VHost to another RabbitMQ VHost.
I'm having an issue with Spring Cloud Dataflow where it appears that despite specifying different RabbitMQ VHosts for source and sink, they don't ever get to the destination Exchange.
My dataflow stream looks like this: RabbitMQ Source | CustomProcessor | RabbitMQ Sink
RabbitMQ Source reads from a queue on vHostA and RabbitMQ Sink should output to ExchangeBlah on vHostB.
However, no messages end up on ExchangeBlah on vHostB, and I get errors in the RabbitMQ Sink log saying:
Channel shutdown: channel error; protocol method: 'method(reply-code=404, reply-text=NOT_FOUND - no exchange 'ExchangeBlah' in vhost 'vHostA', class-id=60, method-id=40)
I've got a feeling that this might be related to the Spring environment variable
spring.cloud.dataflow.applicationProperties.stream.spring.rabbitmq.virtual-host=vhostA
As Dataflow uses queues as communication between the different stages of the Stream, if I don't specify this setting, then the RabbitMQ source and sink communication queues are created on the VHosts specified in their respective configs, however, no communication queue is created for the CustomProcessor.
Therefore, data gets stuck in the Source communication queue.
Also, I know that feasibly Shovels can get around this, but it feels like if the option of outputting to a different VHost is available to you in the RabbitMQ sink then it should work.
All being said, it may well be a bug with the Rabbit Stream Source/Sink apps.
UPDATE:
Looking at the stream definition (once the stream has been deployed), the spring.rabbitmq.virtual-host switch is defined twice. Once with the vHostB which is defined against the sink and then later with the vHostA which is the Spring property.
Removing the virtual-host application property and explicitly setting spring.rabbitmq.virtual-host, host, username and password on processor (including the RabbitMQ source and sinks), and it makes it's way to the processor communication queue, but as the RabbitMQ sink is set to a different VHost, it doesn't seem to get any further.
In this scenario, the communication queues which are created between the various stages of the stream are created on the same VHost which the source is reading from (vHostA). As we can only give the spring.rabbitmq.virtual-host setting to the apps once, the sink is doesn't know to look at the communication queues to pass that data onto it's destination exchange on vHost B.
It's almost as if there are missing switches on the Source and Sink RabbitMQs, or am I missing an overall setting which defines the VHost of where the communication queues should reside, without overriding the source and destination VHosts on the RabbitMQ source and sinks?
Please note that SCDF doesn't directly communicate with RabbitMQ. SCDF attempts to automate the creation of Spring Cloud Stream "env-vars" based on well-defined naming conventions derived from the stream+app names.
It is the Apps themselves that connect to publish/subscribe to RabbitMQ exchanges independently. As far as the right "env-vars" land as properties to the apps when they bootstrap, they should be able to connect as per the configuration.
You pointed out spring.cloud.dataflow.applicationProperties.stream.spring.rabbitmq.virtual-host=vhostA property. That, if supplied, SCDF attempts to propagate that as the virtual-host to all the stream applications that it deploys to the targeted platform.
In your case, it sounds like you'd want to override the virtual-host at the source and the sink level independently, which you can accomplish as specific properties to these Apps in the stream definition, either supplied as in-line or as deployment properties.
Once when you do, you can confirm whether or not they are taking into account by accessing the App's actuator endpoint. Specifically, /configprops would be useful.
I am facing an issue when decoupling two systems by an event/message broker like Apache Kafka. The issue is related to a frontend triggering actions in a backend:
How does the producer (frontend service) know, that the published event has been properly handled by all the backend services (as consumers), if the publisher does not know neither the "identities" nor the count of consuming backends?
To be precise: Users can change for example their email address using a frontend UI. An associated service publishes that "change request" event to an appropriate topic within Kafka. The UI form is then "locked" to prevent subsequent change requests, until the change event has been fully processed by every consumer. But it's unclear how to detect this state.
You can use another topic to publish handled jobs. So your front-end publishes to one topic and your back-end publishes to another once it is done.
In Kafka terms, neither the producer nor consumer are considered backend - they're both clients connecting to a broker, which is generally considered to be the backend.
A producer will know that it has produced a message successfully, by virtue of the acks setting. A consumer will read a message, and then at a later point, its offset will be updated to a point corresponding to the last message it read. However, there is generally no interaction between a producer and a consumer, and they are generally completely unaware of one another.
I am looking for a way for each consumer instance to receive a message that is published to RabbitMQ via MassTransit. The scenario would be, we have multiple microservices that need to invalidate a cache on notification. Pub-Sub won't work in this instance as there will be 5 consumers of the same type as its the same code per service instance, so only one would receive the message in a traditional PubSub.
Message observation could be an option but this means the messages would never be consumed and hang around forever on the bus.
Can anyone suggest a pattern to use in the context of MassTransit?
Thanks in advance.
You should create a management endpoint in each service, which could even be a temporary queue (just request a receive endpoint without a queue name and one will be dynamically generated). Then, put your queue invalidation consumers on that endpoint. Each service instance will receive a unique instance of the message (when Publish is called), and those queues and bindings will automatically be removed once the service exits.
This is exactly how the bus endpoint works, but in your case, you're creating a receive endpoint which can have consumer message type bindings, so that published messages are received, one copy per service.
cfg.ReceiveEndpoint(cfg => { ... });
Note that the queue name is not specified, and will be automatically generated uniquely.
Im a web developer ended up in some j2ee development (newbie). I sincerely need this theory confirmed.
I been given the privilege to deliver a message from our system (producer) to the SOA Enterprice service bus (consumer) when the user hits the save button. The information can not be missed or not delivered and the delivery order must be kept.
Environment:
Jboss eap 5.1 as the producer.
JNDI server is the ESB (maybe standard).
Jboss ESB as the consumer.
My weapon of choice is JMS, p2p, due to the asynchronous nature.
When the producer is abut to send the message some problems can occur:
ESB is down causing JNDI exception
Queue manager is for some reason not awake or wrongly configured. This should cause some JMS exception.
Network hickup, causing a JMS error.
So Im looking for some failover pattern. Here is my suggestion:
Add a internal JMS queue to which the message is initially added.
Add a MDB that listen to the internal queue and tries to send it to the target queue (ESB).
If failing in any way log fatal and send email to cool support people.
This should generate a reliable pattern where a message remains on the internal que until processed by the MDB.
Please advice.
Best Regards
ds
Well a 'temporary' queue is not a totally bad idea, but during the time from moving data from one queue to putting it on another you'll have a potential window of risk. Even though that window is close to nothing, what would happen if you got some failure right there and then? -You'd have to put the message back on the queue (and there you'd get into the problem with getting it in the correct order - nasty stuff!) or hold on to it in some way until you put it the other queue (which in turn can be cumbersome if you'd e g get into some failure-situaton.
A more stable solution would be to put data in a db with a queue-order column. You can then select your data in the correct order, send it to the new queue, and finally flag it as 'done' or something or even (better?) remove the data in the db.