RabbitMQ Shovel basic example - jms

I am working on a basic example and am not able to work it out.
I need to forward messages from one machine (Machine1) to another (Machine2) via a queue (TestQ).
Producer is running on the Machine1 and a consumer on the Machine2.
My settings in the Machine1's rabbit broker config:
{rabbitmq_shovel, [ {shovels, [
{shovel_test, [
{sources, [{broker, "amqp://" }]},
{destinations, [{broker, "amqp://Machine2" }]},
{queue, <<"TestQ">>},
{ack_mode, on_confirm},
{reconnect_delay, 5}
]}
]} ]}
Machine2 has a default config and no shovel plugin enabled.
Producer's code running on the Machine1:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("TestQ", true, false, false, null);
channel.basicPublish("", "TestQ", null, "Hello World!".getBytes());
Consumer's code running on the Machine2:
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare("TestQ", true, false, false, null);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume("TestQ", true, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Received '" + message + "'");
}
Executing rabbitmqctl eval 'rabbit_shovel_status:status().' on the Machine1:
[{shovel_test,starting,{{2014,1,7},{9,47,38}}}]
...done.
Producer sends ok, but I never get a receive from the consumer on the Machine2.
Where is a problem? Something is missing in the conf of Machine1's broker, or Machine2's broker?
Thank you!

The status of your shovel should be running, not starting. If it stays in the starting phase, that means it can't start up correctly (e.g. can't connect to the destination broker).
One problem I spotted is that you used broker instead of brokers for specifying the list of sources. Try this:
{rabbitmq_shovel,
[{shovels, [{shovel_test,
[{sources, [{brokers, ["amqp://"]}]},
{destinations, [{broker, "amqp://Machine2"}]},
{queue, <<"TestQ">>},
{ack_mode, on_confirm},
{reconnect_delay, 5}
]}
]}
]}.

Related

Rabbitmq dead letter queue not delaying message

I have setup rabbitmq. I want to retry message after 10 second once they fail. But the way I have setup, the message is not getting delayed, it's coming back to queue immediately. I want to wait 10 second before sending message to main_queue.
Below are my code. I am using Bunny Ruby gem.
connection = Bunny.new('url_for_rabbitmq', verify_peer: true)
connection.start
channel = connection.create_channel
# Creating 2 Exchanges (One Main exchange, one retry exchange)
exchange = channel.direct('main_exchange')
retry_exchange = channel.direct('retry_exchange')
# Creating 2 Queue (One Main queue, one retry queue)
queue = channel.queue('main_queue', durable: true, arguments: { 'x-dead-letter-exchange' => retry_exchange.name })
queue.bind(exchange, routing_key: 'foo')
queue.bind(retry_exchange, routing_key: 'foo') # This one is pushing message directly to main queue without waiting for 10 second.
retry_queue = channel.queue('retry_queue', durable: true, arguments: { 'x-message-ttl' => 10_1000, 'x-dead-letter-exchange' => retry_exchange.name })
retry_queue.bind(retry_exchange, routing_key: 'foo')
If i change this line (retry_exchange to exchange)
retry_queue = channel.queue('retry_queue', durable: true, arguments: { 'x-message-ttl' => 10_1000, 'x-dead-letter-exchange' => retry_exchange.name })
to this
retry_queue = channel.queue('retry_queue', durable: true, arguments: { 'x-message-ttl' => 10_1000, 'x-dead-letter-exchange' => exchange.name })
then it works. but the message is coming from main_exchange but I want message to come from retry_exchange. How can i achieve this.
This is how I solve the problem
connection = Bunny.new('url_for_rabbitmq', verify_peer: true)
connection.start
channel = connection.create_channel
# Creating 2 Exchanges (One Main exchange, one retry exchange)
exchange = channel.direct('main_exchange')
retry_exchange = channel.direct('retry_exchange')
# Creating 2 Queue (One Main queue, one retry queue)
retry_queue = channel.queue('retry_queue', durable: true, arguments: { 'x-message-ttl' => 10_1000, 'x-dead-letter-exchange' => retry_exchange.name })
retry_queue.bind(retry_exchange, routing_key: 'foo')
retry_queue.bind(retry_exchange, routing_key: retry_queue.name)
queue = channel.queue('main_queue', durable: true, arguments: { 'x-dead-letter-exchange' => retry_exchange.name, 'x-dead-letter-routing-key' => retry_queue.name })
queue.bind(exchange, routing_key: 'foo')
queue.bind(exchange, routing_key: retry_queue.name)
basically i needed to add this to main queue 'x-dead-letter-routing-key' => retry_queue.name. and remove couple of unnecessary binding from main queue queue.bind(retry_exchange, routing_key: 'foo')
Now message come to main queue, if it fails then it goes to retry queue. Before going to retry queue, it will remove old routing key foo and replace it with new routing key retry_queue.name. It stays in retry queue for 10 seconds and then again come back to main queue for retry.

Deleting all messages in JMS queue in Widlfy 9

I am trying to remove messages from JMS queue on Wildfly 9.0.2 (JBoss) using JMX, see following code:
MBeanServer server = ManagementFactory.getPlatformMBeanServer();
try {
String name = "jboss.as.expr:subsystem=messaging,hornetq-server=default,jms-queue=MyQueue";
ObjectName objectName = new ObjectName(objectNameString);
String result = (String) server.invoke(objectName, "removeMessages", new Object[]{null},
new String[]{"java.lang.String"});
return result;
} catch (MalformedObjectNameException | InstanceNotFoundException | MBeanException | ReflectionException ignored) {
log.errorv(ignored, "Error removing messages from JMS queue [{0}]", name);
return null;
}
There is an active consumer on that queue. The code runs without exception and returns string "0", but no messages are actually removed. I tried also to use some value as message filter (*), but got failure:
javax.management.ReflectionException: "HQ119020: Invalid filter: *"
Any idea how to remove the messages?

Subscribing to a removed queue with spring-websocket and RabbitMQ broker (Queue NOT_FOUND)

I have a spring-websocket (4.1.6) application on Tomcat8 that uses a STOMP RabbitMQ (3.4.4) message broker for messaging. When a client (Chrome 47) starts the application, it subscribes to an endpoint creating a durable queue. When this client unsubscribes from the endpoint, the queue will be cleaned up by RabbitMQ after 30 seconds as defined in a custom made RabbitMQ policy. When I try to reconnect to an endpoint that has a queue that was cleaned up, I receive the following exception in the RabbitMQ logs: "NOT_FOUND - no queue 'position-updates-user9zm_szz9' in vhost '/'\n". I don't want to use an auto-delete queue since I have some reconnect logic in case the websocket connection dies.
This problem can be reproduced by adding the following code to the spring-websocket-portfolio github example.
In the container div in the index.html add:
<button class="btn" onclick="appModel.subscribe()">SUBSCRIBE</button>
<button class="btn" onclick="appModel.unsubscribe()">UNSUBSCRIBE</button>
In portfolio.js replace:
stompClient.subscribe("/user/queue/position-updates", function(message) {
with:
positionUpdates = stompClient.subscribe("/user/queue/position-updates", function(message) {
and also add the following:
self.unsubscribe = function() {
positionUpdates.unsubscribe();
}
self.subscribe = function() {
positionUpdates = stompClient.subscribe("/user/queue/position-updates", function(message) {
self.pushNotification("Position update " + message.body);
self.portfolio().updatePosition(JSON.parse(message.body));
});
}
Now you can reproduce the problem by:
Launch the application
click unsubscribe
delete the position-updates queue in the RabbitMQ console
click subscribe
Find the error message in the websocket frame via the chrome devtools and in the RabbitMQ logs.
reconnect logic in case the websocket connection dies.
and
no queue 'position-updates-user9zm_szz9' in vhost
Are fully different stories.
I'd suggest you implement "re-subscribe" logic in case of deleted queue.
Actually that is how STOMP works: it creates auto-deleted (generated) queue for the subscribe and yes, it is removed on the unsubscrire.
See more info in the RabbitMQ STOMP Adapter Manual.
From other side consider to subscribe to the existing AMQP queue:
To address existing queues created outside the STOMP adapter, destinations of the form /amq/queue/<name> can be used.
The problem is Stomp won't recreate the queue if it get's deleted by the RabbitMQ policy. I worked around it by creating the queue myself when the SessionSubscribeEvent is fired.
public void onApplicationEvent(AbstractSubProtocolEvent event) {
if (event instanceof SessionSubscribeEvent) {
MultiValueMap nativeHeaders = (MultiValueMap)event.getMessage().getHeaders().get("nativeHeaders");
List destination = (List)nativeHeaders.get("destination");
String queueName = ((String)destination.get(0)).substring("/queue/".length());
try {
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(queueName, true, false, false, null);
} catch (IOException e) {
e.printStackTrace();
}
}
}

celery task with routing key on SQS

I have a django celery set up, and I'm switching over from rabbitmq to use SQS as the broker. I'm finding that my tasks decorated with a routing_key value are not producing messages in the broker?
Set up is:
CELERY_QUEUES = {
"default": {
"exchange": "default",
"binding_key": "default"},
"sentry": {
"exchange": "default",
"binding_key": "sentry"},
}
CELERY_DEFAULT_QUEUE = "default"
CELERY_DEFAULT_EXCHANGE = "default"
CELERY_DEFAULT_EXCHANGE_TYPE = "direct"
CELERY_DEFAULT_ROUTING_KEY = "default"
task is defined as
#task(routing_key = 'sentry', ignore_result = True)
def doSomething():
print "Hello"
doSomething.delay() # No message is produced
Everything routes fine into the default queue. Unusually everything works fine with rabbit mq?
The Amazon SQS console is showing the queue 'sentry', but no messages are sent to it (I'm not sure what created the queue)
Bonus: unusually, when I first tried this out (about 4 hours ago), some messages did appear to make it to the sentry queue? What could have possibly caused this?
Thanks
I tried your example with one change. Instead of defining the routing key in the task decorator, I defined the queue in the call to the task. And I changed the call from "delay" to "apply_async"
So my code looks like
#task
def doSomething():
print "Hello"
doSomething.apply_async(queue='sentry')

Spring JMSTemplate - lost messages

We are using Spring JMSTemplate 3.0.1RELEASE to send JMS messages to an ActiveMQ cluster.
I am using useAsynSend=true to be able to send Asynchronous requests as these are all Fire and Forget. However they are still persistent and I can confirm they do get persisted on the AMQ Kaha-DB first and then forwarded on to Message Listeners. No CorelationID or JMSReplyTo as I am not listening back on any response.
<amq:connectionFactory id="amqJmsFactory"
brokerURL="failover:(tcp://MY-SERVER-01:61616,tcp://MY-SERVER-02:61616)?randomize=true&jms.useAsyncSend=true" />
<!-- JMS Producer Configuration -->
<bean id="jmsProducerTemplate" class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="amqJmsFactory" />
<bean id="activeMQBinding" class="com.my.company.activemq.ActiveMQProductBinding">
<property name="template" ref="jmsProducerTemplate" />
</bean>
In the ActiveMQProductBinding class we have the following method
public void sendActiveMQMessageAsync(final Object message, String qName) {
log.info("About to send message");
template.send(qName, new MessageCreator() {
public Message createMessage(Session session) throws JMSException {
ObjectMessage objectMessage = session.createObjectMessage((Serializable) message);
log.info("Sending message: " + objectMessage);
return objectMessage;
}
});
}
Now I can see in the logs that the logs are getting printed. No Exception is thrown. Still some messages are totally getting lost. It probably doesn't reach ActiveMQ. I have Mule JMS Listeners configured to listen to messages on the ActiveMQ queues defined by 'qName' above. Most of the messages get delivered to AMQ fine and Mule picks them up within fraction of a second. However it is only some messages which are getting lost. When I checked the Queues on ActiveMQ I can see all messages received have been dispatched fine to Mule. Hence I am suspecting the messages aren't reaching ActiveMQ at all or AMQ is rejecting. However no logs on JMSTemplate spring or ActiveMQ.
The Message created looks like this (It is printed in the method above).
2013-03-11 16:33:11,752 INFO [com.my.company.activemq.ActiveMQProductBinding$1] Sending message: ActiveMQObjectMessage {commandId = 0, responseRequired = false, messageId = null, originalDestination = null, originalTransactionId = null, producerId = null, destination = null, transactionId = null, expiration = 0, timestamp = 0, arrival = 0, brokerInTime = 0, brokerOutTime = 0, correlationId = null, replyTo = null, persistent = false, type = null, priority = 0, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence#61408493, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false}
Anything I am missing on JMSTemplate config or some behaviour on AMQ? Please help!
Any help guys?
2 Possible reasons I can think of.
1) We are using the wrong connection factory. We should use Pooled Connection Factory with JMSTemplate to send messages.
org.apache.activemq.pool.PooledConnectionFactory
2) We are using useAsyncSend=true - i.e. the sender sends the message and does not wait for acknowledgement and there are chances of message losses. This seems more probable but not sure.
Still not accepting this as Answer as someone may have a more concrete explanation. In the meantime if anyone stumbles upon this question may be benefitted by this cues.

Resources