I'm trying use spring integration to receive a big amount of MQTT messages, process them and then store in a db.
Here is the code:
#Bean
public MqttPahoClientFactory mqttClientFactory() {
DefaultMqttPahoClientFactory factory = new DefaultMqttPahoClientFactory();
factory.setServerURIs("tcp://localhost:1883");
return factory;
}
#Bean
public DefaultPahoMessageConverter messageConverter(){
DefaultPahoMessageConverter converter = new DefaultPahoMessageConverter();
converter.setPayloadAsBytes(true);
return converter;
}
#Bean
public MessageChannel mqttInputChannel() {
return new DirectChannel();
}
#Bean
public MessageProducer mqttInbound() {
MqttPahoMessageDrivenChannelAdapter adapter =
new MqttPahoMessageDrivenChannelAdapter("clientID", mqttClientFactory(), "topic1");
adapter.setCompletionTimeout(5000);
adapter.setConverter(messageConverter());
adapter.setQos(2);
adapter.setOutputChannel(mqttInputChannel());
return adapter;
}
#Bean
#ServiceActivator(inputChannel = "mqttInputChannel")
public MessageHandler handler(){
return new MessageHandler() {
#Override
public void handleMessage(Message<?> arg0) throws MessagingException {
//process messages and storing operations
}
};
}
My problem is that I'm not able to receive all messages and losing some of them, probably because I spend a lot of resources and time inside the handler method. I've tried also to use a QueueChannel instead of the DirectChannel, but when the queue is full the problem remains.
A possible solution could be this, stop the reception till the message is completely handled and then restarts it, but I don't know how. Any advice?
Thanks in advance.
Related
I have an application that should work with rabbitmq. I have RabbitMQConfig, which tries to connect with the rabbit, but if it fails to connect, the application does not start at all.
What I want to achieve is that after launching the application, it will try to connect to the rabbit and if it manages to connect, I will start functionality to create a queue and listen to it accordingly. Currently at startup if the rabbit is available and then disappears it starts throwing "java.net.ConnectException: Connection refused". Can I catch this error and how, both at startup and during the operation of the application.
This is config file:
public class RabbitMQConfig implements RabbitListenerConfigurer {
private ConnectionFactory connectionFactory;
#Autowired
public RabbitMQConfig(ConnectionFactory connectionFactory) {
this.connectionFactory = connectionFactory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(messageHandlerMethodFactory());
}
#Bean
MessageHandlerMethodFactory messageHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory messageHandlerMethodFactory = new DefaultMessageHandlerMethodFactory();
messageHandlerMethodFactory.setMessageConverter(consumerJackson2MessageConverter());
return messageHandlerMethodFactory;
}
#Bean
public MappingJackson2MessageConverter consumerJackson2MessageConverter() {
return new MappingJackson2MessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(new Jackson2JsonMessageConverter());
return rabbitTemplate;
}
#Bean
public AmqpAdmin amqpAdmin() {
return new RabbitAdmin(connectionFactory);
}
#Bean(name = "rabbitListenerContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(
SimpleRabbitListenerContainerFactoryConfigurer configurer,
ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
FixedBackOff recoveryBackOff = new FixedBackOff(10000,FixedBackOff.UNLIMITED_ATTEMPTS);
factory.setRecoveryBackOff(recoveryBackOff);
configurer.configure(factory, connectionFactory);
return factory;
}
Whit this methods i create and start listener:
#Service
public class RabbitMQService {
#RabbitListener(queues = "${queueName}", autoStartup = "false", id = "commandQueue")
public void receive(CommandDataDTO commandDataDTO) {
public void receive(Object rMessage) {
doSomething(rMessage);
}
public void createQueue() {
Queue queue = new Queue("queueName"), true, false, false);
Binding binding = new Binding("queueName"),
Binding.DestinationType.QUEUE, env.getProperty("spring.rabbitmq.exchange"),"rKey",
null);
admin.declareQueue(queue);
admin.declareBinding(binding);
startListener();
}
//When queue is active we start Listener
public void startListener() {
boolean isQueuqReady = false;
while (!isQueuqReady) {
Properties p = admin.getQueueProperties(env.getProperty("management.registry.info.device-type") + "_"
+ env.getProperty("management.registry.info.device-specific-type"));
if (p != null) {
log.info("Rabbit queue is up. Start listener.");
isQueuqReady = true;
registry.getListenerContainer("commandQueue").start();
}
}
}
Problem is that I want, regardless of whether there is a connection or not with the rabbit, the application to be able to work and, accordingly, to intercept when there is a connection and when not to do different actions.
I need to bind a queue to a topic exchange, but:
Only if the topic exists
If the topic exists, use the existing settings (e.g. durable, auto-delete, etc)
Reason is, I need a 3rd party application to create the exchange with whatever settings they want to use, I don't want to modify the topic settings.
I put the code below together by reading RabbitMQ Spring AMQP tutorial. It works, but creates an exchange if doesn't exist.
#Configuration
public class BeanConfiguration {
#Bean
public TopicExchange topic() {
return new TopicExchange("MyTopicExchange", true, false);
}
#Bean
public Queue queue() {
return QueueBuilder.durable("MyQueue").build();
}
#Bean
public Binding binding(TopicExchange topicExchange, Queue queue) {
return BindingBuilder.bind(queue).to(topicExchange).with("purchases.*");
}
}
I found a way by using superclass method setShouldDeclareFalse:
#Bean
public TopicExchange topic() {
TopicExchange topicExchange = new TopicExchange("MyTopicExchange", true, false);
topicExchange.setShouldDeclare(false);
return topicExchange;
}
Skip the exchange declaration bean and ignore the binding declaration failure.
#SpringBootApplication
public class So59994152Application {
public static void main(String[] args) {
SpringApplication.run(So59994152Application.class, args);
}
#Bean
public Queue queue() {
return QueueBuilder.durable("MyQueue").build();
}
#Bean
public Binding binding(Queue queue, AmqpAdmin admin) {
((RabbitAdmin) admin).setIgnoreDeclarationExceptions(true);
return new Binding("MyQueue", DestinationType.QUEUE, "MyTopicExchange", "purchases.*", null);
}
#Bean
public ApplicationRunner runner(CachingConnectionFactory cf) {
return args -> {
cf.createConnection();
cf.destroy();
};
}
}
If you are not using Spring Boot; set the admin property in the admin bean.
I'm new to rabbitmq and currently trying to implement a nonblocking producer with a nonblocking consumer. I've build some test producer where I played around with typereference:
#Service
public class Producer {
#Autowired
private AsyncRabbitTemplate asyncRabbitTemplate;
public <T extends RequestEvent<S>, S> RabbitConverterFuture<S> asyncSendEventAndReceive(final T event) {
return asyncRabbitTemplate.convertSendAndReceiveAsType(QueueConfig.EXCHANGE_NAME, event.getRoutingKey(), event, event.getResponseTypeReference());
}
}
And in some other place the test function that gets called in a RestController
#Autowired
Producer producer;
public void test() throws InterruptedException, ExecutionException {
TestEvent requestEvent = new TestEvent("SOMEDATA");
RabbitConverterFuture<TestResponse> reply = producer.asyncSendEventAndReceive(requestEvent);
log.info("Hello! The Reply is: {}", reply.get());
}
This so far was pretty straightforward, where I'm stuck now is how to create a consumer which is non-blocking too. My current listener:
#RabbitListener(queues = QueueConfig.QUEUENAME)
public TestResponse onReceive(TestEvent event) {
Future<TestResponse> replyLater = proccessDataLater(event.getSomeData())
return replyLater.get();
}
As far as I'm aware, when using #RabbitListener this listener runs in its own thread. And I could configure the MessageListener to use more then one thread for the active listeners. Because of that, blocking the listener thread with future.get() is not blocking the application itself. Still there might be the case where all threads are blocking now and new events are stuck in the queue, when they maybe dont need to. What I would like to do is to just receive the event without the need to instantly return the result. Which is probably not possible with #RabbitListener. Something like:
#RabbitListener(queues = QueueConfig.QUEUENAME)
public void onReceive(TestEvent event) {
/*
* Some fictional RabbitMQ API call where i get a ReplyContainer which contains
* the CorrelationID for the event. I can call replyContainer.reply(testResponse) later
* in the code without blocking the listener thread
*/
ReplyContainer replyContainer = AsyncRabbitTemplate.getReplyContainer()
// ProcessDataLater calls reply on the container when done with its action
proccessDataLater(event.getSomeData(), replyContainer);
}
What is the best way to implement such behaviour with rabbitmq in spring?
EDIT Config Class:
#Configuration
#EnableRabbit
public class RabbitMQConfig implements RabbitListenerConfigurer {
public static final String topicExchangeName = "exchange";
#Bean
TopicExchange exchange() {
return new TopicExchange(topicExchangeName);
}
#Bean
public ConnectionFactory rabbitConnectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
connectionFactory.setHost("localhost");
return connectionFactory;
}
#Bean
public MappingJackson2MessageConverter consumerJackson2MessageConverter() {
return new MappingJackson2MessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(rabbitConnectionFactory());
rabbitTemplate.setMessageConverter(producerJackson2MessageConverter());
return rabbitTemplate;
}
#Bean
public AsyncRabbitTemplate asyncRabbitTemplate() {
return new AsyncRabbitTemplate(rabbitTemplate());
}
#Bean
public Jackson2JsonMessageConverter producerJackson2MessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
Queue queue() {
return new Queue("test", false);
}
#Bean
Binding binding() {
return BindingBuilder.bind(queue()).to(exchange()).with("foo.#");
}
#Bean
public SimpleRabbitListenerContainerFactory myRabbitListenerContainerFactory() {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(rabbitConnectionFactory());
factory.setMaxConcurrentConsumers(5);
factory.setMessageConverter(producerJackson2MessageConverter());
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
return factory;
}
#Override
public void configureRabbitListeners(final RabbitListenerEndpointRegistrar registrar) {
registrar.setContainerFactory(myRabbitListenerContainerFactory());
}
}
I don't have time to test it right now, but something like this should work; presumably you don't want to lose messages so you need to set the ackMode to MANUAL and do the acks yourself (as shown).
UPDATE
#SpringBootApplication
public class So52173111Application {
private final ExecutorService exec = Executors.newCachedThreadPool();
#Autowired
private RabbitTemplate template;
#Bean
public ApplicationRunner runner(AsyncRabbitTemplate asyncTemplate) {
return args -> {
RabbitConverterFuture<Object> future = asyncTemplate.convertSendAndReceive("foo", "test");
future.addCallback(r -> {
System.out.println("Reply: " + r);
}, t -> {
t.printStackTrace();
});
};
}
#Bean
public AsyncRabbitTemplate asyncTemplate(RabbitTemplate template) {
return new AsyncRabbitTemplate(template);
}
#RabbitListener(queues = "foo")
public void listen(String in, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag,
#Header(AmqpHeaders.CORRELATION_ID) String correlationId,
#Header(AmqpHeaders.REPLY_TO) String replyTo) {
ListenableFuture<String> future = handleInput(in);
future.addCallback(result -> {
Address address = new Address(replyTo);
this.template.convertAndSend(address.getExchangeName(), address.getRoutingKey(), result, m -> {
m.getMessageProperties().setCorrelationId(correlationId);
return m;
});
try {
channel.basicAck(tag, false);
}
catch (IOException e) {
e.printStackTrace();
}
}, t -> {
t.printStackTrace();
});
}
private ListenableFuture<String> handleInput(String in) {
SettableListenableFuture<String> future = new SettableListenableFuture<String>();
exec.execute(() -> {
try {
Thread.sleep(2000);
}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
future.set(in.toUpperCase());
});
return future;
}
public static void main(String[] args) {
SpringApplication.run(So52173111Application.class, args);
}
}
I have an application (Microservice like) which should send and receives messages from other applications (Microservices). The application has several publishers with every publisher publishing to a specific queue as well as several subscriber classes with each subscriber subscribing to only one queue. Unfortunately, my subscriber classes are consuming the same messages I publish. How should I go about it?
Here is my code:
a) Publisher 1 - does not have a listener method since it only publishes to my.queues.queue1
#Configuration
public class RabbitQueue1Publisher{
private static final String QUEUE_NAME = "my.queues.queue1";
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:1675");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
}
b) Publisher 2 - also does not have a listener method since it only publishes to my.queues.queue2
#Configuration
public class RabbitQueue2Publisher{
private static final String QUEUE_NAME = "my.queues.queue2";
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:1675");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
}
c) Consumer 1 - consumes from queue3. Has a listener method
#Configuration
public class RabbitQueue3Subscriber{
private static final String QUEUE_NAME = "my.queue.queue3";
#Autowired
private Queue3Listener Queue3Listener;
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:15672");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
#Bean
public SimpleMessageListenerContainer userListenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(connectionFactory());
listenerContainer.setQueues(simpleQueue());
listenerContainer.setMessageConverter(jsonMessageConverter());
listenerContainer.setMessageListener(Queue3Listener);
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
return listenerContainer;
}
}
d) Consumer 2 - consumes from queue4. Has a listener method
#Configuration
public class RabbitQueue4Subscriber{
private static final String QUEUE_NAME = "my.queue.queue4";
#Autowired
private Queue4Listener Queue4Listener;
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory("http://127.0.0.1:15672");
connectionFactory.setUsername("guest");
connectionFactory.setPassword("guest");
return connectionFactory;
}
#Bean
public Queue simpleQueue() {
return new Queue(QUEUE_NAME);
}
#Bean
public MessageConverter jsonMessageConverter(){
return new Jackson2JsonMessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
RabbitTemplate template = new RabbitTemplate(connectionFactory());
template.setRoutingKey(QUEUE_NAME);
template.setMessageConverter(jsonMessageConverter());
return template;
}
#Bean
public SimpleMessageListenerContainer userListenerContainer() {
SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer();
listenerContainer.setConnectionFactory(connectionFactory());
listenerContainer.setQueues(simpleQueue());
listenerContainer.setMessageConverter(jsonMessageConverter());
listenerContainer.setMessageListener(Queue4Listener);
listenerContainer.setAcknowledgeMode(AcknowledgeMode.AUTO);
return listenerContainer;
}
}
Though I am publishing and consuming to/from different queues, I end up consuming the same messages I produce. Can someone point out what I am doing wrong or suggest the way to do it?
Here is how it works for me. I have publisher and a consumer of Rabbitmq. Doesn't mater if they are part of the same project or different.
Publisher:
Publisher Configuration
#Configuration
class PublisherConfig{
String queueName = "com.queueName";
String routingKey = "com.routingKey";
String exchange = "com.exchangeName";
#Bean
Queue queue() {
return new Queue(queueName, false);
}
#Bean
TopicExchange exchange() {
return new TopicExchange(exchange);
}
#Bean
Binding binding(Queue queueFoo, TopicExchange exchange) {
return BindingBuilder.bind(queueFoo).to(exchange).with(routingKey);
}
//Required only if you want to pass custom object as part of payload
#Bean
public MappingJackson2MessageConverter jackson2Converter() {
return new MappingJackson2MessageConverter();
}
}
Publish Message
#Autowired private RabbitMessagingTemplate rabbitMessagingTemplate;
#Autowired private MappingJackson2MessageConverter mappingJackson2MessageConverter;
rabbitMessagingTemplate.setMessageConverter(this.mappingJackson2MessageConverter);
rabbitMessagingTemplate.convertAndSend(exchange, routingKey, employObj)
Consumer
Consumer Configuration
#Configuration
public class RabbitMQConfiguration implements RabbitListenerConfigurer {
public MappingJackson2MessageConverter jackson2Converter() {
return new MappingJackson2MessageConverter();
}
#Bean
public DefaultMessageHandlerMethodFactory handlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setMessageConverter(jackson2Converter());
return factory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(handlerMethodFactory());
}
}
Listen a Message
#RabbitListener(queues = "com.queueName")
public void receiveMessage(Employee employee) {
// More code
}
You can encapsulate Publisher and Listener configurations in two different #configuration files.
Hope this helps you
P.S.
OP asked for explanation. Here it is:
Exchange and Routing Key
Publisher publishes a message to an exchange with a particular routing key. Routing key helps to differentiate the type of message it is.
Suppose:
Send all user logged in messages with routing key of 'user_logged_in'.
Send all email sent messages with 'email_sent'.
Queue:
Once the routing key is attached with the exchange there comes a queue.
Queue is attached a exchange and routing key and all the published messages will sit in this queue.
Now consumer explicitly, connects to such queues and listen messages.
So queue name in publisher config and consumer config has to be the same.
Once your publisher is up you can actually visit RabbitMq dashboard
and see the exchange, routing key and queue to see how it works.
I'm using spring integration to poll a database using a JdbcPollingChannelAdapter and then post the results onto an Activemq queue using JmsSendingMessageHandler. I'm serializing the jdbc results as json string using a MappingJackson2MessageConverter. When the message get's sent, it gets sent as an arraylist. Is it possible to only send a single json-serialized object with the payload of a message at a time? This would allow me to then listen onto the queue like so
#JmsListener(destination = "${activemq.queue.name}")
public void receive(DomainObj obj)
Spring Integration Configuration
#Configuration
public class SpringIntegrationConfig {
private static final Logger LOGGER = LoggerFactory.getLogger(SpringIntegrationConfig.class);
#Value("${database.polling-interval.rate-in-milliseconds}")
private Long pollingRateInMilliSeconds;
#Value("${database.max-messages-per-poll}")
private Long maxMessagesPerPoll;
#Bean
public MessageChannel helloWorldChannel() {
return new DirectChannel();
}
#Bean
public PollerMetadata poller(PlatformTransactionManager transactionManager) {
PeriodicTrigger trigger = new PeriodicTrigger(pollingRateInMilliSeconds);
trigger.setFixedRate(true);
MatchAlwaysTransactionAttributeSource attributeSource = new MatchAlwaysTransactionAttributeSource();
attributeSource.setTransactionAttribute(new DefaultTransactionAttribute());
TransactionInterceptor interceptor = new TransactionInterceptor(transactionManager, attributeSource);
PollerMetadata poller = new PollerMetadata();
poller.setTrigger(trigger);
poller.setMaxMessagesPerPoll(maxMessagesPerPoll);
poller.setAdviceChain(Collections.singletonList(interceptor));
return poller;
}
#Bean
#InboundChannelAdapter(value = "helloWorldChannel", channel = "helloWorldChannel", poller = #Poller("poller"))
public MessageSource<?> helloWorldMessageSource(DataSource dataSource) {
JdbcPollingChannelAdapter adapter = new JdbcPollingChannelAdapter(dataSource, "select * from item where type = 2");
adapter.setUpdateSql("update item set type = 10 where id in (:id)");
adapter.setRowMapper(new ItemRowMapper());
adapter.setMaxRowsPerPoll(maxMessagesPerPoll.intValue());
return adapter;
}
#Bean
#ServiceActivator(inputChannel = "helloWorldChannel")
public MessageHandler jsmOutboundAdapter(JmsTemplate template, Queue queue, MessageConverter converter) {
template.setMessageConverter(converter);
JmsSendingMessageHandler handler = new JmsSendingMessageHandler(template);
handler.setDestination(queue);
return handler;
}
#Bean // Serialize message content to json using TextMessage
public MessageConverter jsonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("_type");
return converter;
}
}
Change your select statement to use the JDBC vendor syntax to only retrieve one record - e.g. LIMIT 1.
Then, remove the setMaxRowsPerPoll() (leave it to default at 0) and you will get the single result.
#Transformer public Object transform(List<Object> list) { return list.get(0); } worked with the use of a SQL LIMIT. That's how you get it to return a json object, instead of an array.