with spring boot 1.5.9 RELEASE, code as below
#Configuration
#EnableRabbit
public class RabbitmqConfig {
#Autowired
ConnectionFactory connectionFactory;
#Bean//with or without this bean, neither works
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(connectionFactory);
}
#Bean
public Queue bbbQueue() {
return new Queue("bbb");
}
#Bean
public TopicExchange requestExchange() {
return new TopicExchange("request");
}
#Bean
public Binding bbbBinding() {
return BindingBuilder.bind(bbbQueue())
.to(requestExchange())
.with("*");
}
}
After the jar stars, there is no error message and there is no topic exchange showing in RabbitMQ managementUI(15672) exchanges page.
However, with python code, topic exchange shows and the binding can be seen on exchange detaile page. python code as below
connection = pika.BlockingConnection(pika.ConnectionParameters(host='10.189.134.47'))
channel = connection.channel()
channel.exchange_declare(exchange='request', exchange_type='topic', durable=True)
result = channel.queue_declare(queue='aaa', durable=True)
queue_name = result.method.queue
channel.queue_bind(exchange='aaa', routing_key='*',
queue=queue_name)
print(' [*] Waiting for logs. To exit press CTRL+C')
def callback(ch, method, properties, body):
print(" [x] %r" % body)
channel.basic_consume(callback, queue=queue_name, no_ack=True)
channel.start_consuming()
I just copied your code and it works fine.
NOTE The queue/binding won't be declared until a connection is opened, such as by a listener container that reads from the queue (or sending a message with a RabbitTemplate).
#RabbitListener(queues = "bbb")
public void listen(String in) {
System.out.println(in);
}
The container must have autoStartup=true (default).
Related
We have a Spring Java application using RabbitMQ, and here is the scenario:
There is a consumer receiving messages from a queue and sending them to another one. We are using "SimpleRabbitListenerContainerFactory" as the container factory, but when sending the messages to the other queue inside a "parallelStream" we've got an IllegalStateException "Cannot determine target ConnectionFactory for lookup key" Exception
When we remove the "parallelStream" it works flawlessly.
public void sendMessage(final StagingMessage stagingMessage, final Long timestamp, final String country) {
final List<TransformedMessage> messages = processMessageList(stagingMessage);
messages.parallelStream().forEach(message -> {
final TransformedMessage transformedMessage = buildMessage(timestamp, ApiConstants.POST_METHOD, country);
myMessageSender.sendQueue(country, transformedMessage);
});
}
Connectio Facotory, where the lookup key is set:
#Configuration
#EnableRabbit
public class RabbitBaseConfig {
#Autowired
private QueueProperties queueProperties;
#Bean
#Primary
public ConnectionFactory connectionFactory(final ConnectionFactory connectionFactoryA, final ConnectionFactory connectionFactoryB) {
final SimpleRoutingConnectionFactory simpleRoutingConnectionFactory = new SimpleRoutingConnectionFactory();
final Map<Object, ConnectionFactory> map = new HashMap<>();
for (final String queue : queueProperties.getAQueueMap().values()) {
map.put("[" + queue + "]", connectionFactoryA);
}
for (final String queue : queueProperties.getBQueueMap().values()) {
map.put("[" + queue + "]", connectionFactoryB);
}
simpleRoutingConnectionFactory.setTargetConnectionFactories(map);
return simpleRoutingConnectionFactory;
}
#Bean
public Jackson2JsonMessageConverter jackson2JsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
}
Welcome to stack overflow!
You should always show the pertinent code and configuration beans when asking questions like this.
I assume you are using the RoutingConnectionFactory.
It uses a ThreadLocal to store the lookup key so the send has to happen on the same thread that set the key.
You generally should never go asynchronous in a listener anyway; you risk message loss. To increase concurrency, use the concurrency properties on the container.
EDIT
One technique would be to convey the lookup key in a message header:
#Bean
public RabbitTemplate template(ConnectionFactory rcf) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(rcf);
Expression expression = new SpelExpressionParser().parseExpression("messageProperties.headers['cfSelector']");
rabbitTemplate.setSendConnectionFactorySelectorExpression(expression);
return rabbitTemplate;
}
#RabbitListener(queues = "foo")
public void listen1(String in) {
IntStream.range(0, 10)
.parallel()
.mapToObj(i -> in + i)
.forEach(val -> {
this.template.convertAndSend("bar", val.toUpperCase(), msg -> {
msg.getMessageProperties().setHeader("cfSelector", "[bar]");
return msg;
});
});
}
I am trying to figure out the best way to handle errors that might have occurred in a service that is called after a aggregate's group timeout occurred that mimics the same flow as if the releaseExpression was met.
Here is my setup:
I have a AmqpInboundChannelAdapter that takes in messages and send them to my aggregator.
When the releaseExpression has been met and before the groupTimeout has expired, if an exception gets thrown in my ServiceActivator, the messages get sent to my dead letter queue for all the messages in that MessageGroup. (10 messages in my example below, which is only used for illustrative purposes) This is what I would expect.
If my releaseExpression hasn't been met but the groupTimeout has been met and the group times out, if an exception gets throw in my ServiceActivator, then the messages do not get sent to my dead letter queue and are acked.
After reading another blog post,
link1
it mentions that this happens because the processing happens in another thread by the MessageGroupStoreReaper and not the one that the SimpleMessageListenerContainer was on. Once processing moves away from the SimpleMessageListener's thread, the messages will be auto ack.
I added the configuration mentioned in the link above and see the error messages getting sent to my error handler. My main question, is what is considered the best way to handle this scenario to minimize message getting lost.
Here are the options I was exploring:
Use a BatchRabbitTemplate in my custom error handler to publish the failed messaged to the same dead letter queue that they would have gone to if the releaseExpression was met. (This is the approach I outlined below but I am worried about messages getting lost, if an error happens during publishing)
Investigate if there is away I could let the SimpleMessageListener know about the error that occurred and have it send the batch of messages that failed to a dead letter queue? I doubt this is possible since it seems the messages are already acked.
Don't set the SimpleMessageListenerContainer to AcknowledgeMode.AUTO and manually ack the messages when they get processed via the Service when the releaseExpression being met or the groupTimeOut happening. (This seems kinda of messy, since there can be 1..N message in the MessageGroup but wanted to see what others have done)
Ideally, I want to have a flow that will that will mimic the same flow when the releaseExpression has been met, so that the messages don't get lost.
Does anyone have recommendation on the best way to handle this scenario they have used in the past?
Thanks for any help and/or advice!
Here is my current configuration using Spring Integration DSL
#Bean
public SimpleMessageListenerContainer workListenerContainer() {
SimpleMessageListenerContainer container =
new SimpleMessageListenerContainer(rabbitConnectionFactory);
container.setQueues(worksQueue());
container.setConcurrentConsumers(4);
container.setDefaultRequeueRejected(false);
container.setTransactionManager(transactionManager);
container.setChannelTransacted(true);
container.setTxSize(10);
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
return container;
}
#Bean
public AmqpInboundChannelAdapter inboundRabbitMessages() {
AmqpInboundChannelAdapter adapter = new AmqpInboundChannelAdapter(workListenerContainer());
return adapter;
}
I have defined a error channel and defined my own taskScheduler to use for the MessageStoreRepear
#Bean
public ThreadPoolTaskScheduler taskScheduler(){
ThreadPoolTaskScheduler ts = new ThreadPoolTaskScheduler();
MessagePublishingErrorHandler mpe = new MessagePublishingErrorHandler();
mpe.setDefaultErrorChannel(myErrorChannel());
ts.setErrorHandler(mpe);
return ts;
}
#Bean
public PollableChannel myErrorChannel() {
return new QueueChannel();
}
public IntegrationFlow aggregationFlow() {
return IntegrationFlows.from(inboundRabbitMessages())
.transform(Transformers.fromJson(SomeObject.class))
.aggregate(a->{
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
}
)
.handle("someService", "processMessages")
.get();
}
Here is my custom error flow
#Bean
public IntegrationFlow errorResponse() {
return IntegrationFlows.from("myErrorChannel")
.<MessagingException, Message<?>>transform(MessagingException::getFailedMessage,
e -> e.poller(p -> p.fixedDelay(100)))
.channel("myErrorChannelHandler")
.handle("myErrorHandler","handleFailedMessage")
.log()
.get();
}
Here is the custom error handler
#Component
public class MyErrorHandler {
#Autowired
BatchingRabbitTemplate batchingRabbitTemplate;
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(Message<?> message) {
ArrayList<SomeObject> payload = (ArrayList<SomeObject>)message.getPayload();
payload.forEach(m->batchingRabbitTemplate.convertAndSend("some.dlq","#", m));
}
}
Here is the BatchingRabbitTemplate bean
#Bean
public BatchingRabbitTemplate batchingRabbitTemplate() {
ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
scheduler.setPoolSize(5);
scheduler.initialize();
BatchingStrategy batchingStrategy = new SimpleBatchingStrategy(10, Integer.MAX_VALUE, 30000);
BatchingRabbitTemplate batchingRabbitTemplate = new BatchingRabbitTemplate(batchingStrategy, scheduler);
batchingRabbitTemplate.setConnectionFactory(rabbitConnectionFactory);
return batchingRabbitTemplate;
}
Update 1) to show custom MessageGroupProcessor:
public class CustomAggregtingMessageGroupProcessor extends AbstractAggregatingMessageGroupProcessor {
#Override
protected final Object aggregatePayloads(MessageGroup group, Map<String, Object> headers) {
return group;
}
}
Example Service:
#Slf4j
public class SomeService {
#ServiceActivator
public void processMessages(MessageGroup messageGroup) throws IOException {
Collection<Message<?>> messages = messageGroup.getMessages();
//Do business logic
//ack messages in the group
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug(" deliveryTag = {}",deliveryTag);
log.debug("Channel = {}",channel);
channel.basicAck(deliveryTag, false);
}
}
}
Updated integrationFlow
public IntegrationFlow aggregationFlowWithCustomMessageProcessor() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.releaseExpression("size() == 10");
a.transactional(true);
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
New ErrorHandler to do nack
public class MyErrorHandler {
#ServiceActivator(inputChannel = "myErrorChannelHandler")
public void handleFailedMessage(MessageGroup messageGroup) throws IOException {
if(messageGroup!=null) {
log.debug("Nack messages size = {}", messageGroup.getMessages().size());
Collection<Message<?>> messages = messageGroup.getMessages();
for (Message<?> m : messages) {
com.rabbitmq.client.Channel channel = (com.rabbitmq.client.Channel)
m.getHeaders().get("amqp_channel");
long deliveryTag = (long) m.getHeaders().get("amqp_deliveryTag");
log.debug("deliveryTag = {}",deliveryTag);
log.debug("channel = {}",channel);
channel.basicNack(deliveryTag, false, false);
}
}
}
}
Update 2 Added custom ReleaseStratgedy and change to aggegator
public class CustomMeasureGroupReleaseStratgedy implements ReleaseStrategy {
private static final int MAX_MESSAGE_COUNT = 10;
public boolean canRelease(MessageGroup messageGroup) {
return messageGroup.getMessages().size() >= MAX_MESSAGE_COUNT;
}
}
public IntegrationFlow aggregationFlowWithCustomMessageProcessorAndReleaseStratgedy() {
return IntegrationFlows.from(inboundRabbitMessages()).transform(Transformers.fromJson(SomeObject.class))
.aggregate(a -> {
a.sendPartialResultOnExpiry(true);
a.groupTimeout(3000);
a.expireGroupsUponCompletion(true);
a.expireGroupsUponTimeout(true);
a.correlationExpression("T(Thread).currentThread().id");
a.transactional(true);
a.releaseStrategy(new CustomMeasureGroupReleaseStratgedy());
a.outputProcessor(new CustomAggregtingMessageGroupProcessor());
}).handle("someService", "processMessages").get();
}
There are some flaws in your understanding.If you use AUTO, only the last message will be dead-lettered when an exception occurs. Messages successfully deposited in the group, before the release, will be ack'd immediately.
The only way to achieve what you want is to use MANUAL acks.
There is no way to "tell the listener container to send messages to the DLQ". The container never sends messages to the DLQ, it rejects a message and the broker sends it to the DLX/DLQ.
I have a requirement to send payload to a lot of devices whose names are picked from Database. Then, i have to send to different topics, which will be like settings/{put devicename here}.
Below is the configuration i was using which i got from spring-boot reference documents.
MQTTConfiguration.java
#Configuration
#IntegrationComponentScan
public class MQTTConfiguration {
#Autowired
private Settings settings;
#Autowired
private DevMqttMessageListener messageListener;
#Bean
MqttPahoClientFactory mqttClientFactory() {
DefaultMqttPahoClientFactory clientFactory = new DefaultMqttPahoClientFactory();
clientFactory.setServerURIs(settings.getMqttBrokerUrl());
clientFactory.setUserName(settings.getMqttBrokerUser());
clientFactory.setPassword(settings.getMqttBrokerPassword());
return clientFactory;
}
#Bean
MessageChannel mqttOutboundChannel() {
return new DirectChannel();
}
#Bean
#ServiceActivator(inputChannel = "mqttOutboundChannel")
public MessageHandler mqttOutbound() {
MqttPahoMessageHandler messageHandler = new MqttPahoMessageHandler("dev-client-outbound",
mqttClientFactory());
messageHandler.setAsync(true);
messageHandler.setDefaultTopic(settings.getMqttPublishTopic());
return messageHandler;
}
#MessagingGateway(defaultRequestChannel = "mqttOutboundChannel")
public interface DeviceGateway {
void sendToMqtt(String payload);
}
}
Here, i am sending to only 1 topic. So i added the bean like below to send to multiple number of topics;
#Bean
public MqttClient mqttClient() throws MqttException {
MqttClient mqttClient = new MqttClient(settings.getMqttBrokerUrl(), "dev-client-outbound");
MqttConnectOptions connOptions = new MqttConnectOptions();
connOptions.setUserName(settings.getMqttBrokerUser());
connOptions.setPassword(settings.getMqttBrokerPassword().toCharArray());
mqttClient.connect(connOptions);
return mqttClient;
}
and i send using,
try {
mqttClient.publish(settings.getMqttPublishTopic()+device.getName(), mqttMessage);
} catch (MqttException e) {
LOGGER.error("Error While Sending Mqtt Messages", e);
}
Which works.
But my question is, Can i achieve the same, using output channel for better performance? If yes, any help is greatly appreciated. Thank You.
MqttClient is synchronous.
The MqttPahoMessageHandler uses an MqttAsyncClient and can be configured (set async to true) to not wait for the confirmation, but publish the confirmation later as an application event.
If you are using your own code and sending multiple messages in a loop, it will probably be faster to use an async client, and wait for the IMqttDeliveryToken completions later.
When processing the reply message with AsyncRabbitTemplate.sendAndReceive() or AsyncRabbitTemplate.convertSendAndReceive() method, since the reply message is returned asynchronously with calling method, we can use message listener for reply queue to receive and process reply message, why spring-amqp framework import AsyncRabbitTemplate and RabbiteMessageFuture to process the reply message? For message listener, we can control the related consumer thread,
but for RabbitMessageFuture, the background thread can not be managed.
-------------------Added on 2017/01/06----------------------------
It's simply your choice.
Replies can come back in a different order to sends.
With the async template, the framework takes care of the correlation
for you the reply will appear in the future returned by the send
method.
When you use your own listener, you will have to take care of the
correlation yourself.
Thank you. I know this difference.But there is still a problem. If I use message listener, I can ack the reply message manually(If my message listener
implements ChannelAwareMessageListener interface and I can get the channel instance).But when I use asyncRabbitTemplate, can I ack the reply message manually? It seems that sendAndReceive method ack the reply message automatically.
I don't understand what you mean; since you can inject the listener
container into the template, you have the same "control" either way.
It seems there is some problem in this mean.
I created a rabbitTemplate instance and simple message listener container. But when I use them to construct an asyncRabbitTemplate instance as following code:
#Bean(name="rabbitTemplate")
public RabbitTemplate getRabbitTemplate()
{
RabbitTemplate rabbitTemplate = new RabbitTemplate(getConnectionFactory());
rabbitTemplate.setUseTemporaryReplyQueues(false);
rabbitTemplate.setReplyAddress("replyQueue");
rabbitTemplate.setReceiveTimeout(60000);
rabbitTemplate.setReplyTimeout(60000);
return rabbitTemplate;
}
#Bean(name="asyncRabbitTemplate")
public AsyncRabbitTemplate getAsyncRabbitTemplate()
{
AsyncRabbitTemplate asyncRabbitTemplate =
new AsyncRabbitTemplate(getRabbitTemplate(), createReplyListenerContainer());
asyncRabbitTemplate.setAutoStartup(true);
asyncRabbitTemplate.setReceiveTimeout(60000);
return asyncRabbitTemplate;
}
#Bean(name="replyMessageListenerContainer")
public SimpleMessageListenerContainer createReplyListenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(getConnectionFactory());
listenerContainer.setQueueNames("replyQueue");
listenerContainer.setMessageListener(getRabbitTemplate());
listenerContainer.setRabbitAdmin(getRabbitAdmin());
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
return listenerContainer;
}
I found I can not send message successfully. The consumer server can not receive the message.
But when I create asyncRabbitTemplate instance with following code, I found the message can be sent and received successfully.
#Bean(name="asyncRabbitTemplate")
public AsyncRabbitTemplate getAsyncRabbitTemplate()
{
AsyncRabbitTemplate asyncRabbitTemplate =
new AsyncRabbitTemplate(getConnectionFactory(),
"sendMessageExchange",
"sendMessageKey",
"replyQueue");
asyncRabbitTemplate.setReceiveTimeout(60000);
asyncRabbitTemplate.setAutoStartup(true);
return asyncRabbitTemplate;
}
If there is something wrong with my source code?
I used the spring-boot-ampq 1.4.3.RELEASE.
It's simply your choice.
Replies can come back in a different order to sends.
With the async template, the framework takes care of the correlation for you - the reply will appear in the future returned by the send method.
When you use your own listener, you will have to take care of the correlation yourself.
For message listener, we can control the related consumer thread, but for RabbitMessageFuture, the background thread can not be managed.
I don't understand what you mean; since you can inject the listener container into the template, you have the same "control" either way.
EDIT
#SpringBootApplication
public class So41481046Application {
public static void main(String[] args) throws Exception {
ConfigurableApplicationContext context = SpringApplication.run(So41481046Application.class, args);
AsyncRabbitTemplate asyncTemplate = context.getBean(AsyncRabbitTemplate.class);
RabbitConverterFuture<String> future = asyncTemplate.convertSendAndReceive("foo");
try {
String out = future.get(10, TimeUnit.SECONDS);
System.out.println(out);
}
finally {
context.close();
}
System.exit(0);
}
#Bean
public AsyncRabbitTemplate asyncTemplate(RabbitTemplate rabbitTemplate, ConnectionFactory connectionFactory) {
rabbitTemplate.setRoutingKey(queue().getName());
rabbitTemplate.setReplyAddress(replyQueue().getName());
return new AsyncRabbitTemplate(rabbitTemplate, replyContainer(connectionFactory));
}
#Bean
public Queue queue() {
return new AnonymousQueue();
}
#Bean
public Queue replyQueue() {
return new AnonymousQueue();
}
#Bean
public SimpleMessageListenerContainer replyContainer(ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueueNames(replyQueue().getName());
return container;
}
#Bean
public SimpleMessageListenerContainer remoteContainer(ConnectionFactory connectionFactory) {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
container.setQueueNames(queue().getName());
container.setMessageListener(new MessageListenerAdapter(new Object() {
#SuppressWarnings("unused")
public String handleMessage(String in) {
return in.toUpperCase();
}
}));
return container;
}
}
I am trying to implement re-routing of dead-lettered messages as described in this answer. I am using Spring config. I have no idea on how to read the headers to get the original routing key and original queue. The following is my config:
#Configuration
public class NotifEngineRabbitMQConfig {
#Bean
public MessageHandler handler(){
return new MessageHandler();
}
#Bean
public Jackson2JsonMessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public MessageListenerAdapter messageListenerAdapter(){
return new MessageListenerAdapter(handler(), messageConverter());
}
/**
* Listens for incoming messages
* Allows multiple queue to listen to
* */
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer(){
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.addQueueNames(QUEUE_TO_LISTEN_TO.split(","));
container.setMessageListener(messageListenerAdapter());
container.setConnectionFactory(rabbitConnectionFactory());
container.setDefaultRequeueRejected(false);
return container;
}
#Bean
public ConnectionFactory rabbitConnectionFactory(){
CachingConnectionFactory factory = new CachingConnectionFactory(HOST);
factory.setUsername(USERNAME);
factory.setPassword(PASSWORD);
return factory;
}
}
The headers are not available using "old" style Pojo messaging (with a MessageListenerAdapter). You need to implement MessageListener which gives you access to the headers.
However, you will need to invoke the converter yourself in that case and, if you are using request/reply messaging, you lose the reply mechanism within the adapter and you have to send the reply yourself.
Alternatively, you can use a custom message converter and "enhance" the converted object with the header after invoking the standard converter.
Consider instead using the newer style POJO messaging with #RabbitListener - it gives you access to the headers and has request/reply capability.
Here's an example:
#SpringBootApplication
public class So37581560Application {
public static void main(String[] args) {
SpringApplication.run(So37581560Application.class, args);
}
#Bean
public FooListener fooListener() {
return new FooListener();
}
public static class FooListener {
#RabbitListener(queues="foo")
public void pojoListener(String body,
#Header(required = false, name = "x-death") List<String> xDeath) {
System.out.println(body + ":" + (xDeath == null ? "" : xDeath));
}
}
}
Result:
Foo:[{reason=expired, count=1, exchange=, time=Thu Jun 02 08:44:19 EDT 2016, routing-keys=[bar], queue=bar}]
Gary's answer is the right one. Just a little detail, the type of xDeath is better to be ArrayList<HashMap<String,*>> instead List<String> xDeath. Then you can access any field by doing something like: xDeath.first().get("count")