i define a queue channel
#Bean("mail-action-laundry-list-channel")
public MessageChannel mailRecipientActionMessageChannel() {
return new QueueChannel(20);
}
the flow below, i will aggrate messages from the queue channel, i tried this:
#Bean
public IntegrationFlow mailRecipientActionLaundryListMessageFlow(#Qualifier("laundryListMessageHandler") MessageHandler laundryListMessageHandler) {
return IntegrationFlows.from("mail-action-laundry-list-channel")
.log("--> laundry list messages:：")
.aggregate(aggregatorSpec -> aggregatorSpec
.correlationExpression("#this.payload.email")
.releaseExpression("#this.size() == 5")
.messageStore(new SimpleMessageStore(100))
.groupTimeout(2000))
.transform(laundryListMessageToItemProcessDtoTransformer())
.handle(laundryListMessageHandler)
.get();
}
but why it aggrate first 5 messages from the channel always, and aggrate other message no longer
You need to configure expireGroupsUponCompletion(true) on the aggregator:
When set to true (default false), completed groups are removed from the message store, allowing subsequent messages with the same correlation to form a new group. The default behavior is to send messages with the same correlation as a completed group to the discard-channel.
Looks like your subsequent messages from the queue has the same email property. Therefore an aggregator can't form a new group for the same correlation key.
https://docs.spring.io/spring-integration/docs/5.0.3.RELEASE/reference/html/messaging-routing-chapter.html#aggregator-config
Related
I am building a Springboot application using Spring Kafka where I am getting messages from a topic. I have to modify those messages and then produce them to another topic. I don't want to consume any other message till I have processed my current one. How can I achieve this?
#KafkaListener(
topics = "${event.topic.name}",
groupId = "${event.topic.group.id}",
containerFactory = "eventKafkaListenerContainerFactory"
)
public void consume(Event event) {
logger.info(String.format("Event created(from consumer)-> %s", event));
}
"event" is a json object which I am receiving as a message.
See https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_max.poll.records:
max.poll.records
The maximum number of records returned in a single call to poll().
Type: int
Default: 500
With Spring Boot you can configure it as this property:
spring.kafka.consumer.maxPollRecords
So, you set it to 1 and no more records are going to be polled from this consumer until you return from your #KafkaListener method.
I am trying to expose HTTP endpoint which will drop a message to a JMS queue I want to reply with Success if the delivery is successful and FAILURE if message can not be delivered.
#Bean
public IntegrationFlow systemTaskCall(MapToServiceTaskConfigTransformer mapTransformer, CachingConnectionFactory jmsConnectionFactory) {
return IntegrationFlows.from(
Http.inboundGateway("/spartaSystemTask")
.requestMapping(r -> r
.methods(HttpMethod.POST)
.consumes("application/json")
)
.requestPayloadType(Map.class)
.replyChannel(RESPONSE_CHANNEL)
.errorChannel("errorChannel")
)
.handle((payload, headers) -> mapTransformer.transform((Map<String, String>) payload))
.enrichHeaders(Collections.singletonMap(DESTINATION_QUEUE, "request.queue"))
.enrichHeaders(Collections.singletonMap(JMS_REPLY_TO, "response.queue"))
.transform(Transformers.toJson())
.handle(
Jms.outboundGateway(jmsConnectionFactory,)
.requestDestination(message -> message.getHeaders().get(DESTINATION_QUEUE))
)
.log(LoggingHandler.Level.ERROR)
.enrichHeaders(
c -> c.header(org.springframework.integration.http.HttpHeaders.STATUS_CODE, HttpStatus.CREATED)
)
.transform(source -> "SUCCESS")
.transform(Transformers.toJson())
.channel(RESPONSE_CHANNEL)
.get();
}
#Bean
public IntegrationFlow errorFlow(){
return IntegrationFlows.from("errorChannel")
.transform(source -> "error")
.transform(Transformers.toJson())
.channel(RESPONSE_CHANNEL)
.get();
}
When I call this URL, Message is dropped but HTTP call times out. It seems post the JMS Outbound gateway call rest of the code is not executed.
In case of Failure in message delivery I am getting correct response.
You have this configuration:
.handle(
Jms.outboundGateway(jmsConnectionFactory,)
.requestDestination(message -> message.getHeaders().get(DESTINATION_QUEUE))
)
an outboundGateway. That means you send a request and expect a response from the other side, but it looks like you only send a JMS message into a queue and no one on the other listener side answers you with something into that response.queue. That's the reason for time out your get with a normal JMS publishing.
You need to sure that logic is correct in your flow and it is indeed valid in your distributed solution that you expect some reply from the server side.
Otherwise you need to think about changing your logic into a Jms.outboundAdapter() which is really one-way sender. For HTTP reply you could use a publishSubscribeChannel() with this Jms.outboundAdapter() as the first subscriber and the rest of your flow as a second one. This way the second subscriber is not going to be called until the first one finishes its logic properly. For the error cases you can wrap that Jms.outboundAdapter() with an ExpressionEvaluatingRequestHandlerAdvice: https://docs.spring.io/spring-integration/docs/5.2.3.RELEASE/reference/html/messaging-endpoints.html#message-handler-advice-chain
We're having issues with one of the consumers, and need to debug the code. SimpleRabbitListenerContainerFactory allows to set a ConsumerTagStrategy, which should add tags during logging.
#Bean
public SimpleRabbitListenerContainerFactory analyzeTransactionListenerContainerFactory(ConnectionFactory connectionFactory, AsyncTaskExecutor asyncTaskExecutor) {
connectionFactory.getVirtualHost());
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setConcurrentConsumers(2);
factory.setMaxConcurrentConsumers(4);
factory.setTaskExecutor(asyncTaskExecutor);
ConsumerTagStrategy consumerTagStrategy = new ConsumerTagStrategy() {
#Override
public String createConsumerTag(String queue) {
return queue;
}
};
factory.setConsumerTagStrategy(consumerTagStrategy);
return factory;
}
However, the logs still do not have the tag. So there is no way to find what queue/consumer this message is for.
LogLevel=DEBUG; category=org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer; msg=Cancelling Consumer: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#10.17.1.13:5672/,47), acknowledgeMode=AUTO local queue size=0;
LogLevel=DEBUG; category=org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer; msg=Idle consumer terminating: Consumer: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#10.17.1.13:5672/,47), acknowledgeMode=AUTO local queue size=0;
How does one add some tags to SimpleMessageListenerContainer logging?
Well, looks like you misunderstood the Consumer Key a bit. From the RabbitMQ docs:
consumer-tag consumer-tag
Specifies the identifier for the consumer. The consumer tag is local to a channel, so two clients can use the same consumer tags. If this field is empty the server will generate a unique tag.
The client MUST NOT specify a tag that refers to an existing consumer. Error code: not-allowed
The consumer tag is valid only within the channel from which the consumer was created. I.e. a client MUST NOT create a consumer in one channel and then use it in another. Error code: not-allowed
The SimpleMessageListenerContainer populates its internal map for the consumerTags when it starts to listen to provided queues. If we are listening (consuming) there we can see them from the:
public String toString() {
return "Consumer: tags=[" + (this.consumerTags.toString()) + "], channel=" + channel
+ ", acknowledgeMode=" + acknowledgeMode + " local queue size=" + queue.size();
}
as you expecting, of course.
But if we don't consumer any more, in case of Channel#basicCancel, for example.
Or... looking to your second log message during idling. When you have an extra Consumer, but there is no message for it.
The code on the matter looks like:
boolean receivedOk = receiveAndExecute(this.consumer); // At least one message received
if (SimpleMessageListenerContainer.this.maxConcurrentConsumers != null) {
if (receivedOk) {
.....
}
else {
consecutiveMessages = 0;
if (consecutiveIdles++ > SimpleMessageListenerContainer.this.consecutiveIdleTrigger) {
considerStoppingAConsumer(this.consumer);
consecutiveIdles = 0;
}
}
}
So, maybe you worry for nothing. And you see the logs only for those extra consumers. For this case you have maxConcurrentConsumers > concurrentConsumers.
I want a method to browse all messages from a messsage queue and can send it to another queue using jmstemplate with using Websphere queues(NOT MQ). I have tried using receive and it is able to retrieve all the messages from the queue but it is still waiting for another message. And the messages are being lost. It must be in a transaction
The Code I have Tried:
**String message = (String) jmsTemplate.receiveAndConvert();
System.out.print(message);
while ((message = (String) jmsTemplate.receiveAndConvert()) != null) {
messages.add(message);
}
return messages;
}**
The JMStemplate should be used for only synchronous read or sending message. For asychronous read use one of the listener implementation. Read here
everyone. I have an HTTP API for posting messages in a RabbitMQ broker and I need to implement the request-response pattern in order to receive the responses from the server. So I am something like a bridge between the clients and the server. I push the messages to the broker with specific routing-key and there is a Consumer for that messages, which is publishing back massages as response and my API must consume the response for every request. So the diagram is something like this:
So what I do is the following- For every HTTP session I create a temporary responseQueue(which is bound to the default exchange, with routing key the name of that queue), after that I set the replyTo header of the message to be the name of the response queue(where I will wait for the response) and also set the template replyQueue to that queue. Here is my code:
public void sendMessage(AbstractEvent objectToSend, final String routingKey) {
final Queue responseQueue = rabbitAdmin.declareQueue();
byte[] messageAsBytes = null;
try {
messageAsBytes = new ObjectMapper().writeValueAsBytes(objectToSend);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
MessageProperties properties = new MessageProperties();
properties.setHeader("ContentType", MessageBodyFormat.JSON);
properties.setReplyTo(responseQueue.getName());
requestTemplate.setReplyQueue(responseQueue);
Message message = new Message(messageAsBytes, properties);
Message receivedMessage = (Message)requestTemplate.convertSendAndReceive(routingKey, message);
}
So what is the problem: The message is sent, after that it is consumed by the Consumer and its response is correctly sent to the right queue, but for some reason it is not taken back in the convertSendAndReceived method and after the set timeout my receivedMessage is null. So I tried to do several things- I started to inspect the spring code(by the way it's a real nightmare to do that) and saw that is I don't declare the response queue it creates a temporal for me, and the replyTo header is set to the name of the queue(the same what I do). The result was the same- the receivedMessage is still null. After that I decided to use another template which uses the default exchange, because the responseQueue is bound to that exchange:
requestTemplate.send(routingKey, message);
Message receivedMessage = receivingTemplate.receive(responseQueue.getName());
The result was the same- the responseMessage is still null.
The versions of the amqp and rabbit are respectively 1.2.1 and 1.2.0. So I am sure that I miss something, but I don't know what is it, so if someone can help me I would be extremely grateful.
1> It's strange that RabbitTemplate uses doSendAndReceiveWithFixed if you provide the requestTemplate.setReplyQueue(responseQueue). Looks like it is false in your explanation.
2> To make it worked with fixed ReplyQueue you should configure a reply ListenerContainer:
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(rabbitConnectionFactory);
container.setQueues(responseQueue);
container.setMessageListener(requestTemplate);
3> But the most important part here is around correlation. The RabbitTemplate.sendAndReceive populates correlationId message property, but the consumer side has to get deal with it, too: it's not enough just to send reply to the responseQueue, the reply message should has the same correlationId property. See here: how to send response from consumer to producer to the particular request using Spring AMQP?
BTW there is no reason to populate the Message manually: You can just simply support Jackson2JsonMessageConverter to the RabbitTemplate and it will convert your objectToSend to the JSON bytes automatically with appropriate headers.