We have an order managament system in which after every order state update we make an api call to our client to keep them updated. We do this by first sending a message to a sqs queue and inside a consumer we hit our clients api. The processing on consumer side usually takes about 300-350ms but The approximate age of oldest message in sqs dashboard is showing spikes that reach upto 50-60 secs.
Seeing this I thought that maybe one consumer is not enough for our load and I created multiple DMLC beans and multiple copies of our consumer class. I attached these consumer classes as listeners in these DMLCs. But I have not seen any improvement in approximate age of oldest message.
I am guessing that maybe only one of the DMLC is processing these messages and others are just sitting idle.
I added multiple DMLCs because there are other places in pur codebase where the same thing is used, But now I am not sure if this is the correct way to solve the problem.
My Consumer class looks like this:
#Component
#Slf4j
#RequiredArgsConstructor
public class HOAEventsOMSConsumer extends ConsumerCommon implements MessageListener {
private static final int MAX_RETRY_LIMIT = 3;
private final OMSEventsWrapper omsEventsWrapper;
#Override
public void onMessage(Message message) {
try {
TextMessage textMessage = (TextMessage) message;
String jmsMessageId = textMessage.getJMSMessageID();
ConsumerLogging.logStart(jmsMessageId);
String text = textMessage.getText();
log.info(
"Inside HOA Events consumer Request jmsMessageId:- " + jmsMessageId + " Text:- "
+ text);
processAndAcknowledge(message, text, textMessage);
} catch (JMSException e) {
log.error("JMS Exception while processing surge message", e);
}
}
private void processAndAcknowledge(Message message, String text, TextMessage textMessage) throws JMSException {
try {
TrimmedHOAEvent hoaEvent = JsonHelper.convertFromJsonPro(text, TrimmedHOAEvent.class);
if (hoaEvent == null) {
throw new OMSValidationException("Empty message in hoa events queue");
}
EventType event = EventType.fromString(textMessage.getStringProperty("eventType"));
omsEventsWrapper.handleOmsEvent(event,hoaEvent);
acknowledgeMessage(message);
} catch (Exception e) {
int retryCount = message.getIntProperty("JMSXDeliveryCount");
log.info("Retrying... retryCount: {}, HOAEventsOMSConsumer: {}", retryCount, text);
if (retryCount > MAX_RETRY_LIMIT) {
log.info("about to acknowledge the message since it has exceeded maximum retry limit");
acknowledgeMessage(message);
}
}
}
}
And my DMLC configuration class looks like this:
#Configuration
#SuppressWarnings("unused")
public class HOAEventsOMSJMSConfig extends JMSConfigCommon{
private Boolean isSQSQueueEnabled;
#Autowired
private HOAEventsOMSConsumer hoaEventsOMSConsumer;
#Autowired
private HOAEventsOMSConsumer2 hoaEventsOMSConsumer2;
#Autowired
private HOAEventsOMSConsumer3 hoaEventsOMSConsumer3;
#Autowired
private HOAEventsOMSConsumer4 hoaEventsOMSConsumer4;
#Autowired
private HOAEventsOMSConsumer5 hoaEventsOMSConsumer5;
#Autowired
private HOAEventsOMSConsumer6 hoaEventsOMSConsumer6;
#Autowired
private HOAEventsOMSConsumer7 hoaEventsOMSConsumer7;
#Autowired
private HOAEventsOMSConsumer8 hoaEventsOMSConsumer8;
#Autowired
private HOAEventsOMSConsumer9 hoaEventsOMSConsumer9;
#Autowired
private HOAEventsOMSConsumer10 hoaEventsOMSConsumer10;
public HOAEventsOMSJMSConfig(IPropertyService propertyService, Environment env) {
queueName = env.getProperty("aws.sqs.queue.oms.hoa.events.queue");
endpoint = env.getProperty("aws.sqs.queue.endpoint") + queueName;
JMSConfigCommon.accessId = env.getProperty("aws.sqs.access.id");
JMSConfigCommon.accessKey = env.getProperty("aws.sqs.access.key");
try {
ServerNameCache serverNameCache = CacheManager.getInstance().getCache(ServerNameCache.class);
if (serverNameCache == null) {
serverNameCache = new ServerNameCache();
serverNameCache.set(InetAddress.getLocalHost().getHostName());
CacheManager.getInstance().setCache(serverNameCache);
}
this.isSQSQueueEnabled = propertyService.isConsumerEnabled(serverNameCache.get(), false);
} catch (Exception e) {
this.isSQSQueueEnabled = false;
}
}
#Bean
public JmsTemplate omsHOAEventsJMSTemplate(){
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory =
SQSConnectionFactory.builder().withEndpoint(getEndpoint("sqs")).build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory(sqsConnectionFactory);
JmsTemplate jmsTemplate = new JmsTemplate(cachingConnectionFactory);
jmsTemplate.setDefaultDestinationName(queueName);
jmsTemplate.setDeliveryPersistent(false);
jmsTemplate.setSessionTransacted(false);
jmsTemplate.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return jmsTemplate;
}
#Bean
public DefaultMessageListenerContainer jmsListenerHOAEventsListenerContainer() {
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withEndpoint(getEndpoint("sqs"))
.build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(queueName);
dmlc.setAutoStartup(isSQSQueueEnabled);
dmlc.setMessageListener(hoaEventsOMSConsumer);
dmlc.setSessionTransacted(false);
dmlc.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return dmlc;
}
#Bean
public DefaultMessageListenerContainer jmsListenerHOAEventsListenerContainerNo2() {
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withEndpoint(getEndpoint("sqs"))
.build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(queueName);
dmlc.setAutoStartup(isSQSQueueEnabled);
dmlc.setMessageListener(hoaEventsOMSConsumer2);
dmlc.setSessionTransacted(false);
dmlc.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return dmlc;
}
#Bean
public DefaultMessageListenerContainer jmsListenerHOAEventsListenerContainerNo3() {
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withEndpoint(getEndpoint("sqs"))
.build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(queueName);
dmlc.setAutoStartup(isSQSQueueEnabled);
dmlc.setMessageListener(hoaEventsOMSConsumer3);
dmlc.setSessionTransacted(false);
dmlc.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return dmlc;
}
}
If this question is already answered somehwere else, then please point me towards that.
Related
I am facing an issue in Rabbit MQ regarding x-delay while connecting to spring boot. I need to schedule the messages for a variable delay according to the message type. It can be one of the units MINUTE, DAY, WEEK, MONTH, and so on…
Below is my configuration class :
private final RabbitProperties rabbitProperties;
private final Environment environment;
#Bean
public Queue rabbitMQueue() {
return new Queue(environment.getProperty(RABBITMQ_QUEUE_NAME), false);
}
#Bean
public Exchange rabbitExchange() {
String exchangeName = "test_exchange";
Map<String, Object> exchangeArgs = new HashMap<>();
exchangeArgs.put("x-delayed-type", exchangeType.toLowerCase());
exchangeArgs.put("x-delayed-message",true);
exchangeArgs.put("x-message-ttl",9922);
log.info("Loading {} exchange with name {}.", exchangeType, exchangeName);
switch (exchangeType){
default: return new CustomExchange(exchangeName, exchangeType, true, false, exchangeArgs);
case "DIRECT" : return directExchange(exchangeName, exchangeArgs);
}
}
private Exchange directExchange(String exchangeName, Map<String, Object> exchangeArgs) {
// log.info("Generating directExchange");
// DirectExchange directExchange = new DirectExchange(exchangeName,true, false, exchangeArgs);
// directExchange.setDelayed(true);
// return directExchange;
return ExchangeBuilder.directExchange(exchangeName).withArguments(exchangeArgs)
.delayed()
.build();
}
#Bean
public Binding rabbitBinding(final Queue rabbitMQueue, final Exchange rabbitExchange){
log.info("Exchange to bind : {}", rabbitExchange.getName());
return BindingBuilder
.bind(rabbitMQueue)
.to(rabbitExchange)
.with(environment.getProperty(RABBITMQ_ROUTING_KEY)).noargs();
}
#Bean
public AmqpTemplate amqpTemplate(final ConnectionFactory rabbitMQConnectionFactory,
final MessageConverter rabbitMessageConvertor,
final Exchange rabbitExchange){
RabbitTemplate rabbitTemplate = new RabbitTemplate(rabbitMQConnectionFactory);
rabbitTemplate.setMessageConverter(rabbitMessageConvertor);
rabbitTemplate.setExchange(rabbitExchange.getName());
return rabbitTemplate;
}
#Bean
public ConnectionFactory rabbitMQConnectionFactory(){
Boolean isUriBased = environment.getProperty(URI_BASED_CONNECTION_ENABLED, Boolean.class);
CachingConnectionFactory connectionFactory;
if(!Objects.isNull(isUriBased) && isUriBased){
connectionFactory = new CachingConnectionFactory();
connectionFactory.setUri(environment.getProperty(RABBITMQ_URI));
}
else{
connectionFactory = new CachingConnectionFactory(rabbitProperties.getHost(), rabbitProperties.getPort());
connectionFactory.setUsername(rabbitProperties.getUsername());
connectionFactory.setPassword(rabbitProperties.getPassword());
}
return connectionFactory;
}
#Bean
public MessageConverter rabbitMessageConvertor(){
return new Jackson2JsonMessageConverter();
}
And publisher code :
public boolean sendMessage(String tenant, T message, int delay){
MyQueueMessage<T> myQueueMessage = getQueueMessage(tenant, message);
try{
amqpTemplate.convertAndSend(exchangeName, routingKey, myQueueMessage, messagePostProcessor -> {
// MessageProperties messageProperties = messagePostProcessor.getMessageProperties();
messagePostProcessor.getMessageProperties().setHeader("x-message-ttl", 5011);
messagePostProcessor.getMessageProperties().setHeader(MessageProperties.X_DELAY, 5012);
messagePostProcessor.getMessageProperties().setDelay(5013);
messagePostProcessor.getMessageProperties().setReceivedDelay(5014);
log.info("Setting delay in properties : {}", messagePostProcessor.getMessageProperties().getHeader(MessageProperties.X_DELAY).toString());
return messagePostProcessor;
});
} catch (Exception e){
return false;
}
return true;
}
And receiver :
#RabbitListener(queues = "INVOICE")
public void receiveMessage(Message message){
log.info("Message Received : " + message.toString() + " with delay " + message.getMessageProperties().getDelay());
}
}
Issue :
The value
message.getMessageProperties().getDelay()
comes as NULL in the receiver and the message is also not delayed. It’s getting received instantly.
Did I miss anything?
Please note that I am using docker, rabbitmq-management-3, and have already installed the rabbitmq_delayed_message_exchange plugin.
I have this RabbitMQ Spring Boot Configuration:
#Configuration
public class RabbitConfiguration {
// Main queue configuration
#Value("${rabbitmq.main.messages.queue}")
private String mainQueueName;
#Value("${rabbitmq.main.exchange.queue}")
private String mainExchangeName;
#Value("${rabbitmq.main.routing.key}")
private String mainRoutingKey;
// DLQ configuration
#Value("${rabbitmq.dlq.messages.queue}")
private String dlqQueueName;
#Value("${rabbitmq.dlq.exchange.queue}")
private String dlqExchangeName;
#Value("${rabbitmq.dlq.routing.key}")
private String dlqRoutingKey;
// Connectivity
#Value("${spring.rabbitmq.host}")
private String rabbitmqHost;
#Value("${spring.rabbitmq.port}")
private int rabbitmqPort;
#Value("${spring.rabbitmq.username}")
private String rabbitmqUsername;
#Value("${spring.rabbitmq.password}")
private String rabbitmqPassword;
// Not delivered messages, will be used eventually
#Value("${rabbitmq.not.delivered.messages.queue}")
private String notDeliveredMessagesQueue;
// status with delivered messages (callback default)
#Value("${rabbitmq.delivered.messages.queue}")
private String deliveredMessagesQueue;
#Bean
DirectExchange deadLetterExchange() {
return new DirectExchange(dlqExchangeName);
}
#Bean
DirectExchange exchange() {
return new DirectExchange(mainExchangeName);
}
#Bean
Queue dlq() {
return QueueBuilder.durable(dlqQueueName).build();
}
#Bean
Queue queue() {
return QueueBuilder
.durable(mainQueueName)
.withArgument("x-dead-letter-exchange", "deadLetterExchange")
.withArgument("x-dead-letter-routing-key", dlqQueueName).build();
}
#Bean
Binding deadLetterBinding() {
return BindingBuilder.bind(dlq()).to(deadLetterExchange()).with(dlqRoutingKey);
}
#Bean
Binding binding() {
return BindingBuilder.bind(queue()).to(exchange()).with(mainQueueName);
}
#Bean
public MessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory(rabbitmqHost);
connectionFactory.setUsername(rabbitmqUsername);
connectionFactory.setPassword(rabbitmqPassword);
return connectionFactory;
}
#Bean
public RabbitAdmin rabbitAdmin() {
RabbitAdmin admin = new RabbitAdmin(connectionFactory());
admin.declareQueue(queue());
admin.declareExchange(exchange());
admin.declareBinding(binding());
return admin;
}
public AmqpTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(jsonMessageConverter());
return rabbitTemplate;
}
}
The problem is that when some exception is launched the message is not sent to DLQ queue.
Consumer:
#RabbitListener(queues = { "${rabbitmq.main.messages.queue}" })
public void recievedMessage(#Payload Mensagem item, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag) throws InvalidMessageException {
if (item.getIdCliente().equals("69")) {
logger.info("Something went wrong to: " + item);
throw new InvalidMessageException();
} else {
logger.info("==> Message consumed successfully: " + item);
}
}
This is the configuration I have on my application.properties:
spring.rabbitmq.listener.simple.retry.enabled=true
spring.rabbitmq.listener.simple.retry.initial-interval=3s
spring.rabbitmq.listener.simple.retry.max-attempts=2
spring.rabbitmq.listener.simple.retry.multiplier=2
spring.rabbitmq.listener.simple.retry.max-interval=10s
When I throw an exception on purpose just to see the message moving to DLQ nothing happens. What's wrong here? What am I forgetting here?
Try adding
spring.rabbitmq.listener.simple.default-requeue-rejected=false
to your application.properties. I think the problem is that the failed deliveries are being requeued instead of being sent to the DLQ.
I am working on homework to my studies with Java and RabbitMQ. I am not familiar much with Spring and RabbitMQ, but I can't handle this problem.
I have 2 single application.
First one, which produces the message (bolid application)
I created a producer of the message (bolid), which every 10 seconds send a message to listeners
#SpringBootApplication
public class BolidApplication implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(BolidApplication.class, args);
}
#Autowired
private RabbitTemplate rabbitTemplate;
#Override
public void run(String... args) throws Exception {
Bolid bolid = new Bolid();
int i = 10;
while (true) {
bolid.setData(new Date());
rabbitTemplate.setReplyAddress("bolidReply");
rabbitTemplate.convertAndSend("RaceExchange", "raceRouting", bolid.toString());
rabbitTemplate.convertAndSend("MonitorExchange", "raceRouting", bolid.toString());
Thread.sleep(15000);
i += 10;
}
}
}
So, I create 2 queue (RaceQueue and MonitorQueue), define exchange and bind them.
I have 2 listeners: RaceListener and MonitorListener.
There is the code of my listeners:
And the second application, which is listeners.
public class RabbitConfig {
private static final String RACE_QUEUE = "RaceQueue";
private static final String MONITOR_QUEUE = "MonitorQueue";
#Bean
Queue myQueue() {
return new Queue(RACE_QUEUE, true);
}
#Bean
Queue monitorQueue() {
return new Queue(MONITOR_QUEUE, true);
}
#Bean
Exchange myExchange() {
return ExchangeBuilder.topicExchange("RaceExchange")
.durable(true)
.build();
}
#Bean
Exchange monitorExchange() {
return ExchangeBuilder.topicExchange("MonitorExchange")
.durable(true)
.build();
}
#Bean
Binding binding() {
// return new Binding(MY_QUEUE, Binding.DestinationType.QUEUE, "MyTopicExchange", "topic", null)
return BindingBuilder
.bind(myQueue())
.to(myExchange())
.with("raceRouting")
.noargs();
}
#Bean
Binding monitorBinding() {
return BindingBuilder
.bind(monitorQueue())
.to(monitorExchange())
.with("raceRouting")
.noargs();
}
#Bean
ConnectionFactory connectionFactory() {
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory("localhost");
cachingConnectionFactory.setUsername("guest");
cachingConnectionFactory.setPassword("guest");
return cachingConnectionFactory;
}
#Bean
MessageListenerContainer rabbitRaceListener() {
SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer();
simpleMessageListenerContainer.setConnectionFactory(connectionFactory());
simpleMessageListenerContainer.setQueues(myQueue());
simpleMessageListenerContainer.setupMessageListener(new RabbitRaceListener());
return simpleMessageListenerContainer;
}
#Bean
MessageListenerContainer rabbitMonitorListener() {
SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer();
simpleMessageListenerContainer.setConnectionFactory(connectionFactory());
simpleMessageListenerContainer.setQueues(monitorQueue());
simpleMessageListenerContainer.setupMessageListener(new RabbitMonitorListener());
return simpleMessageListenerContainer;
}
}
From MonitorListener I want to use reply pattern to reply message to my first application (bolid application). So Bolid application can receive my message.
My Code for MonitorListener:
public class RabbitMonitorListener implements MessageListener {
#Autowired
private RabbitTemplate rabbitTemplate;
#Override
public void onMessage(Message message) {
String[] splitted = new String(message.getBody()).split("\\|");
int oilTemperature = Integer.parseInt(splitted[1].split(" ")[2]);
int engineTemperature = Integer.parseInt(splitted[2].split(" ")[2]);
int tirePressure = Integer.parseInt(splitted[3].split(" ")[2]);
System.out.println("message2 = [" + new String(message.getBody()) + "]");
if (oilTemperature > 120 || engineTemperature > 120 || tirePressure > 12) {
System.out.println("SEND REPLY TO BOLID!");
}
if (oilTemperature > 150 || engineTemperature > 150 || tirePressure > 17) {
System.out.println("SEND RELY TO BOLID!");
}
}
}
How can I achieve that? So here I can send the message go back to bolid and on the bolid application I can read it?
EDIT:
I did some research, I want to do it like this way:
public class RabbitMonitorListener implements MessageListener {
#Autowired
private RabbitTemplate rabbitTemplate;
#Override
public void onMessage(Message message) {
String[] splitted = new String(message.getBody()).split("\\|");
int oilTemperature = Integer.parseInt(splitted[1].split(" ")[2]);
int engineTemperature = Integer.parseInt(splitted[2].split(" ")[2]);
int tirePressure = Integer.parseInt(splitted[3].split(" ")[2]);
String response = "Hello";
MessageProperties properties = new MessageProperties();
Message responseMessage = new Message(response.getBytes(), properties);
rabbitTemplate.send(message.getMessageProperties().getReplyTo(), responseMessage);
System.out.println("message2 = [" + new String(message.getBody()) + "]");
if (oilTemperature > 120 || engineTemperature > 120 || tirePressure > 12) {
System.out.println("WARN MECHANICS");
}
if (oilTemperature > 150 || engineTemperature > 150 || tirePressure > 17) {
System.out.println("WARN MECHANICS");
}
}
}
but the rabbitTemplate is null here, so I can't #Autowired it here. How can I have access to rabbitTemplate and method send in MessageListener?
new RabbitRaceListener() - that must be a #Bean too, to get auto wiring.
However, you are over-complicating things; the framework can take care of all of this for you.
See Request/Reply Messaging for the client side - and use convertSendAndReceive() or convertSendAndReceiveAsType().
On the server side, see Annotation-driven Listener Endpoints.
#RabbitListener(queues = "request")
public String handle(String in) {
return in.toUpperCase();
}
I'm new to rabbitmq and currently trying to implement a nonblocking producer with a nonblocking consumer. I've build some test producer where I played around with typereference:
#Service
public class Producer {
#Autowired
private AsyncRabbitTemplate asyncRabbitTemplate;
public <T extends RequestEvent<S>, S> RabbitConverterFuture<S> asyncSendEventAndReceive(final T event) {
return asyncRabbitTemplate.convertSendAndReceiveAsType(QueueConfig.EXCHANGE_NAME, event.getRoutingKey(), event, event.getResponseTypeReference());
}
}
And in some other place the test function that gets called in a RestController
#Autowired
Producer producer;
public void test() throws InterruptedException, ExecutionException {
TestEvent requestEvent = new TestEvent("SOMEDATA");
RabbitConverterFuture<TestResponse> reply = producer.asyncSendEventAndReceive(requestEvent);
log.info("Hello! The Reply is: {}", reply.get());
}
This so far was pretty straightforward, where I'm stuck now is how to create a consumer which is non-blocking too. My current listener:
#RabbitListener(queues = QueueConfig.QUEUENAME)
public TestResponse onReceive(TestEvent event) {
Future<TestResponse> replyLater = proccessDataLater(event.getSomeData())
return replyLater.get();
}
As far as I'm aware, when using #RabbitListener this listener runs in its own thread. And I could configure the MessageListener to use more then one thread for the active listeners. Because of that, blocking the listener thread with future.get() is not blocking the application itself. Still there might be the case where all threads are blocking now and new events are stuck in the queue, when they maybe dont need to. What I would like to do is to just receive the event without the need to instantly return the result. Which is probably not possible with #RabbitListener. Something like:
#RabbitListener(queues = QueueConfig.QUEUENAME)
public void onReceive(TestEvent event) {
/*
* Some fictional RabbitMQ API call where i get a ReplyContainer which contains
* the CorrelationID for the event. I can call replyContainer.reply(testResponse) later
* in the code without blocking the listener thread
*/
ReplyContainer replyContainer = AsyncRabbitTemplate.getReplyContainer()
// ProcessDataLater calls reply on the container when done with its action
proccessDataLater(event.getSomeData(), replyContainer);
}
What is the best way to implement such behaviour with rabbitmq in spring?
EDIT Config Class:
#Configuration
#EnableRabbit
public class RabbitMQConfig implements RabbitListenerConfigurer {
public static final String topicExchangeName = "exchange";
#Bean
TopicExchange exchange() {
return new TopicExchange(topicExchangeName);
}
#Bean
public ConnectionFactory rabbitConnectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
connectionFactory.setHost("localhost");
return connectionFactory;
}
#Bean
public MappingJackson2MessageConverter consumerJackson2MessageConverter() {
return new MappingJackson2MessageConverter();
}
#Bean
public RabbitTemplate rabbitTemplate() {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(rabbitConnectionFactory());
rabbitTemplate.setMessageConverter(producerJackson2MessageConverter());
return rabbitTemplate;
}
#Bean
public AsyncRabbitTemplate asyncRabbitTemplate() {
return new AsyncRabbitTemplate(rabbitTemplate());
}
#Bean
public Jackson2JsonMessageConverter producerJackson2MessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
Queue queue() {
return new Queue("test", false);
}
#Bean
Binding binding() {
return BindingBuilder.bind(queue()).to(exchange()).with("foo.#");
}
#Bean
public SimpleRabbitListenerContainerFactory myRabbitListenerContainerFactory() {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(rabbitConnectionFactory());
factory.setMaxConcurrentConsumers(5);
factory.setMessageConverter(producerJackson2MessageConverter());
factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);
return factory;
}
#Override
public void configureRabbitListeners(final RabbitListenerEndpointRegistrar registrar) {
registrar.setContainerFactory(myRabbitListenerContainerFactory());
}
}
I don't have time to test it right now, but something like this should work; presumably you don't want to lose messages so you need to set the ackMode to MANUAL and do the acks yourself (as shown).
UPDATE
#SpringBootApplication
public class So52173111Application {
private final ExecutorService exec = Executors.newCachedThreadPool();
#Autowired
private RabbitTemplate template;
#Bean
public ApplicationRunner runner(AsyncRabbitTemplate asyncTemplate) {
return args -> {
RabbitConverterFuture<Object> future = asyncTemplate.convertSendAndReceive("foo", "test");
future.addCallback(r -> {
System.out.println("Reply: " + r);
}, t -> {
t.printStackTrace();
});
};
}
#Bean
public AsyncRabbitTemplate asyncTemplate(RabbitTemplate template) {
return new AsyncRabbitTemplate(template);
}
#RabbitListener(queues = "foo")
public void listen(String in, Channel channel, #Header(AmqpHeaders.DELIVERY_TAG) long tag,
#Header(AmqpHeaders.CORRELATION_ID) String correlationId,
#Header(AmqpHeaders.REPLY_TO) String replyTo) {
ListenableFuture<String> future = handleInput(in);
future.addCallback(result -> {
Address address = new Address(replyTo);
this.template.convertAndSend(address.getExchangeName(), address.getRoutingKey(), result, m -> {
m.getMessageProperties().setCorrelationId(correlationId);
return m;
});
try {
channel.basicAck(tag, false);
}
catch (IOException e) {
e.printStackTrace();
}
}, t -> {
t.printStackTrace();
});
}
private ListenableFuture<String> handleInput(String in) {
SettableListenableFuture<String> future = new SettableListenableFuture<String>();
exec.execute(() -> {
try {
Thread.sleep(2000);
}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
future.set(in.toUpperCase());
});
return future;
}
public static void main(String[] args) {
SpringApplication.run(So52173111Application.class, args);
}
}
#Configuration
#Component
public class GatewayAqrConfig {
#Autowired
ConnectorService connectorService;
#Autowired
MasterService masterService;
private HashMap<ConnectorPK, GatewayAqr> connectorMap;
#Bean
#Scope(value = "prototype")
public AbstractClientConnectionFactory clientCF(Connector connector , Master master) {
TcpNetClientConnectionFactory clientConnectionFactory = new TcpNetClientConnectionFactory(connector.getAqrIpAddr(), connector.getAqrIpPortNo());
clientConnectionFactory.setSingleUse(false);
MyByteArraySerializer obj = new MyByteArraySerializer(master.getAqrMsgHeaderLength(), master.getAqrId());
clientConnectionFactory.setSerializer(obj);
clientConnectionFactory.setDeserializer(obj);
clientConnectionFactory.setSoKeepAlive(true);
TcpMessageMapper tcpMessageMapper = new TcpMessageMapper();
tcpMessageMapper.setCharset("ISO-8859-1");
clientConnectionFactory.setMapper(tcpMessageMapper);
clientConnectionFactory.setBeanName(connector.getAqrIpAddr() + ":" + connector.getAqrIpPortNo());
clientConnectionFactory.afterPropertiesSet();
clientConnectionFactory.start();
return clientConnectionFactory;
}
#Bean
#Scope(value = "prototype")
public TcpSendingMessageHandler tcpOutGateway(AbstractClientConnectionFactory connectionFactory) {
TcpSendingMessageHandler messageHandler = new TcpSendingMessageHandler();
messageHandler.setConnectionFactory(connectionFactory);
messageHandler.setClientMode(true);
messageHandler.setTaskScheduler(getTaskScheduler());
messageHandler.setStatsEnabled(true);
messageHandler.afterPropertiesSet();
messageHandler.start();
return messageHandler;
}
#Bean
#Scope(value = "prototype")
public TcpReceivingChannelAdapter tcpInGateway(AbstractClientConnectionFactory connectionFactory) {
TcpReceivingChannelAdapter messageHandler = new TcpReceivingChannelAdapter();
messageHandler.setConnectionFactory(connectionFactory);
messageHandler.setClientMode(true);
messageHandler.setOutputChannel(receive());
messageHandler.setAutoStartup(true);
messageHandler.setTaskScheduler(getTaskScheduler());
messageHandler.afterPropertiesSet();
messageHandler.start();
return messageHandler;
}
#Bean
#Scope(value = "prototype")
public TaskScheduler getTaskScheduler() {
TaskScheduler ts = new ThreadPoolTaskScheduler();
return ts;
}
#Bean
public MessageChannel receive() {
QueueChannel channel = new QueueChannel();
return channel;
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata poller() {
return new PollerMetadata();
}
#Bean
#Transactional
public HashMap<ConnectorPK, GatewayAqr> gatewayAqr() throws Exception {
connectorMap = new HashMap();
Connector connector = null;
ConnectorPK connectorPK = null;
Master master = null;
TcpConnectionSupport connectionSupport = null;
// 1. Get List of Connections configured in Database
List<Connector> connectors = connectorService.getConnections();
if (connectors.size() > 0) {
for (int i = 0; i < connectors.size(); i++) {
// 2. Get the connection details
connector = connectors.get(i);
connectorPK = aqrConnector.getConnectorpk();
master = masterService.findById(connectorPK.getAcuirerId());
try {
// 3. Create object of TcpNetClientConnectionFactory for each Acquirer connection
AbstractClientConnectionFactory clientConnectionFactory = clientCF(aqrConnector, aqrMaster);
// 4. Create TcpSendingMessageHandler for the Connection
TcpSendingMessageHandler outHandler = tcpOutGateway(clientConnectionFactory);
// 5. Create TcpReceivingChannelAdapter object for the Connection and assign it to receive channel
TcpReceivingChannelAdapter inHandler = tcpInGateway(clientConnectionFactory);
// 6. Generate the GatewayAqr object
GatewayAqr gatewayAqr = new GatewayAqr(clientConnectionFactory, outHandler, inHandler);
// 7. Put in the MAP acuirerPK and Send MessageHandler object
connectorMap.put(aqrConnectorPK, gatewayAquirer);
} catch (Exception e) {
}
} // for
} // if
return connectorMap;
}
}
*********************************************************************************************************************************
#EnableIntegration
#IntegrationComponentScan(basePackageClasses = {GatewayEventConfig.class,GatewayAqrConfig.class })
#Configuration
#ComponentScan(basePackages = {"com.iz.zw.gateway.impl", "com.iz.zw.configuration"})
#Import({GatewayEventConfig.class,GatewayAquirerConfig.class})
public class GatewayConfig {
#Autowired
private GatewayAsyncReply<Object, Message<?>> gatewayAsyncReply;
#Autowired
private GatewayCorrelationStrategy gatewayCorrelationStrategy;
#Autowired
private HashMap<ConnectorPK, GatewayAqr> gatewayAqrs;
#Autowired
ConnectorService connectorService;
#Autowired
GatewayResponseDeserializer gatewayResponseDeserializer;
#MessagingGateway(defaultRequestChannel = "send")
public interface Gateway {
void waitForResponse(TransactionMessage transaction);
}
#Bean
public MessageChannel send() {
DirectChannel channel = new DirectChannel();
return channel;
}
#Bean
#ServiceActivator(inputChannel = "send")
public BarrierMessageHandlerWithLateGoodResponse barrier() {
BarrierMessageHandlerWithLateGoodResponse barrier = new BarrierMessageHandlerWithLateGoodResponse(25000, this.gatewayCorrelationStrategy);
barrier.setAsync(true);
barrier.setOutputChannel(out());
barrier.setDiscardChannel(lateGoodresponseChannel());
return barrier;
}
#ServiceActivator(inputChannel = "out")
public void printMessage(Message<?> message) {
System.out.println("in out channel");
}
#Transformer(inputChannel = "receive", outputChannel = "process")
public TransactionMessage convert(byte[] response) {
logger.debug("Response Received", Arrays.toString(response));
TransactionMessage transactionMessage = gatewayResponseDeserializer.deserializeResponse(response);
System.out.println("Response : " + response);
return transactionMessage;
}
#ServiceActivator(inputChannel = "process")
#Bean
public MessageHandler releaser() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
try {
gatewayAsyncReply.put(message);
barrier().trigger(message);
} catch (GatewayLateGoodMessageException exception) {
System.out.println("Late good response..!");
gatewayAsyncReply.get(message);
lateGoodresponseChannel().send(message);
}
}
};
}
#Bean
public MessageChannel process() {
QueueChannel channel = new QueueChannel();
return channel;
}
#Bean
public MessageChannel out() {
DirectChannel channel = new DirectChannel();
return channel;
}
#Bean
public MessageChannel lateGoodresponseChannel() {
QueueChannel channel = new QueueChannel();
return channel;
}
#ServiceActivator(inputChannel="lateGoodresponseChannel")
public void handleLateGoodResponse(Message<?> message) {
String strSTAN = null;
String strResponse = null;
Message<?> respMessage = null;
if(message instanceof TransactionMessage){
strSTAN = ((TransactionMessage)message).getStan();
respMessage = gatewayAsyncReply.get(strSTAN);
if (null != respMessage) {
strResponse = (String) message.getPayload();
}
}
logger.info("Late Good Response: " + strResponse);
}
}
*********************************************************************************************************************************
#Configuration
public class GatewayEventConfig {
private static final Logger logger = LoggerFactory.getLogger(GatewayEventConfig.class);
#Bean
public ApplicationEventListeningMessageProducer tcpEventListener() {
ApplicationEventListeningMessageProducer producer = new ApplicationEventListeningMessageProducer();
producer.setEventTypes(new Class[] {TcpConnectionOpenEvent.class, TcpConnectionCloseEvent.class, TcpConnectionExceptionEvent.class});
producer.setOutputChannel(tcpEventChannel());
producer.setAutoStartup(true);
producer.setTaskScheduler(getEventTaskScheduler());
producer.start();
return producer;
}
#Bean
public TaskScheduler getEventTaskScheduler() {
TaskScheduler ts = new ThreadPoolTaskScheduler();
return ts;
}
#Bean
public MessageChannel tcpEventChannel() {
return new QueueChannel();
}
#Transactional
#ServiceActivator(inputChannel = "tcpEventChannel")
public void tcpConnectionEvent(TcpConnectionEvent event) {
System.out.println("In publishing" + event.toString());
String strConnectionFactory = event.getConnectionFactoryName();
if (strConnectionFactory.equals("connection1")) {
//send some message to connector
} else {
// send message to another connector
}
}
}
this is my configuration files, my application tries to connect to 2 servers as soon as it starts.
I have made 2 configurations for 2 servers as above class
GatewayAqrConfig1 and GatewayConfig1 classes are used for first server connection
GatewayAqrConfig2 and GatewayConfig2 classes are used for second server connection
Using event I am connecting to server and sending a connection set up message, if server is already started and If I have started my application,
it gets the event, connects and sends the message but I am not getting the response instead I am getting the WARNING as below
**WARN TcpNetConnection:186 - Unexpected message - no endpoint registered with connection interceptor:**
i.e connection does not registers the listener properly
but if I am starting my application first and then servers I am getting responses perfectly, As I am connecting to servers
I could not restart it ? My application should connect to server which is already started ? what could be the problem ?
Version used:
Spring integration Version : 4.3.1
Spring version : 4.3.2
JDK 1.8 on JBOSS EAP 7
That WARN message means that, somehow, an inbound message was received without a TcpReceivingChannelAdapter having been registered with the connection factory. Client mode should make no difference.
Having looked at your code a little more, the prototype beans should be ok, as long as you use those objects (especially TcpMessageHandler directly rather than via the framework).
It's not obvious to me how that can happen, given your configuration; the listener is registered when you call setConnectionFactory on the receiving adapter.
If you can reproduce it with a trimmed-down project and post it someplace, I will take a look.