I'm able to receive batch messages with the codes below. But now my question is, how should I manually ACK the messages. ACK all the messages in the list one by one, or ACK the last message in the list is enough? Thanks in advance!
public class MyMessageListener implements ChannelAwareBatchMessageListener {
#Override
public void onMessageBatch(List<Message> messages, Channel channel){
//Do something......
//option 1
messages.forEach(msg-> channel.basicAck(msg.getMessageProperties().getDeliveryTag(), true);
//option 2
channel.basicAck( messages.get(messages.size()-1).getMessageProperties().getDeliveryTag(), true);
}
}
This depends on your business case , generally speaking acknowledging all the messages will increase the network traffic and consequently your message throughput will go down .
The second option is a pragmatic approach provided it is OK to have the possibility of loosing some intermediate messages which may not have been delivered and a later message is delivered and acknowledged in which case the un-delivered messages will also get ackéd .
So it is a design decision which is driven by the sensitivity of payload of your messages.
Related
I have jms message endpoint like:
#Bean
public JmsMessageDrivenEndpoint fsJmsMessageDrivenEndpoint(ConnectionFactory fsConnectionFactory,
Destination fsInboundDestination,
MessageConverter fsMessageConverter) {
return Jms.messageDrivenChannelAdapter(fsConnectionFactory)
.destination(fsInboundDestination)
.jmsMessageConverter(fsMessageConverter)
.outputChannel("fsChannelRouter.input")
.errorChannel("fsErrorChannel.input")
.get();
}
So, my questions is did I get next message before current message will be processed? If it will...Did it will get all messages in mq queue until it fills up all the memory? How to avoid it?
The JmsMessageDrivenEndpoint is based on the JmsMessageListenerContainer, its threading model and MessageListener callback for pulled messages. As long as your MessageListener blocks, it doesn't go to the next message in the queue to pull. When we build an integration flow starting with JmsMessageDrivenEndpoint, it becomes as a MessageListener callback. As long as we process the message downstream in the same thread (DirectChannel by default in between endpoints), we don't pull the next message from JMS queue. If you place a QueueChannel or an ExecutorChannel in between, you shift a processing to a different thread. The current one (JMS listener) gets a control back and it is ready to pull the next message. And in this case your concern about the memory is correct. You can still use QueueChannel with limited size or your ExecutorChannel can be configured with limited thread pool.
In any way my recommendation do not do any thread shifting in the flow when you start from JMS listener container. It is better to block for the next message and let the current transaction to finish its job. So you won't lose a message when something crashes.
In a spring boot application using artemis we try to avoid queues containing too many messages. The intention is to only put in new messages if the number of messages currently in the queue falls below a certain limit, e.g. 100 messages. However, that seems not to work but we don't know why or what the "correct" method would be to implement that functionality. The number of messages as extracted by the code below is always 0 although in the gui there are messages.
To reproduce the problem I installed apache-artemis-2.13.0 locally.
We are doing something like the following
if (!jmsUtil.queueHasNotMoreElementsThan(QUEUE_ALMOST_EMPTY_MAX_AMOUNT, reprocessingMessagingProvider.getJmsTemplate())) {
log.info("Queue has too many messages. Will not send more...");
return;
}
jmsUtil is implemented like
public boolean queueHasNotMoreElementsThan(int max, JmsOperations jmsTemplate) {
return Boolean.TRUE.equals(
jmsTemplate.browse((session, queueBrowser) -> {
Enumeration enumeration = queueBrowser.getEnumeration();
return notMoreElemsThan(enumeration, max);
}));
}
private Boolean notMoreElemsThan(Enumeration enumeration, int max) {
for (int i = 0; i <= max; i++) {
if (!enumeration.hasMoreElements()) {
return true;
}
enumeration.nextElement();
}
return false;
}
As a check I used additionally the following method to give me the number of messages in the queue directly.
public int countPendingMessages(String destination, JmsOperations jmsTemplate) {
Integer totalPendingMessages = jmsTemplate.browse(destination,
(session, browser) -> Collections.list(browser.getEnumeration()).size());
int messageCount = totalPendingMessages == null ? 0 : totalPendingMessages;
log.info("Queue {} message count: {}", destination, messageCount);
return messageCount;
}
That method of extracting the queue size seems to be used as well by others and is based on the documentation of QueueBrowser: The getEnumeration method returns a java.util.Enumeration that is used to scan the queue's messages.
Would the above be the correct way on how to obtain the queue size? If so, what could be the cause of the problem? If not, how should the queue size be queried? Does spring offer any other possibility of accessing the queue?
Update: I read another post and the documentation but I wouldn't know on how to obtain the ClientSession.
There are some caveats to using a QueueBrowser to count the number of messages in the queue. The first is noted in the QueueBrowser JavaDoc:
Messages may be arriving and expiring while the scan is done. The JMS API does not require the content of an enumeration to be a static snapshot of queue content. Whether these changes are visible or not depends on the JMS provider.
So already the count may not be 100% accurate.
Then there is the fact that there may be messages still technically in the queue which have been dispatched to a consumer but have not yet been acknowledged. These messages will not be counted by the QueueBrowser even though they may be cancelled back to the queue at any point if the related consumer closes its connection.
Simply put the JMS API doesn't provide a truly reliable way to determine the number of messages in a queue. Furthermore, "Spring JMS" is tied to the JMS API. It doesn't have any other way to interact with a JMS broker. Given that, you'll need to use a provider-specific mechanism to determine the message count.
ActiveMQ Artemis has a rich management API that is accessible though, among other things, specially constructed JMS messages. You can see this in action in the "Management" example that ships with ActiveMQ Artemis in the examples/features/standard/management directory. It demonstrates how to use JMS resources and provider-specific helper classes to get the message count for a JMS queue. This is essentially the same solution as given in the other post you mentioned, but it uses the JMS API rather than the ActiveMQ Artemis "core" API.
We are using wireTap to take timestamps at different parts of the flow. When introduced to the newest flow, it started causing a timeout in the replyChannel. From what I understand from the documentation, wireTap does intercept the message and sends it to secondary channel, while not affecting the main flow - so it looks like the perfect thing to use to take snapshots of said timestamps. Are we using wrong component for the job, or is there something wrong with the configuration? And if so, how would you recommend to register such information?
The exception:
o.s.integration.core.MessagingTemplate : Failed to receive message from channel 'org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#21845b0d' within timeout: 1000
The code:
#Bean
public MarshallingWebServiceInboundGateway inboundGateway(Jaxb2Marshaller jaxb2Marshaller,
DefaultSoapHeaderMapper defaultSoapHeaderMapper) {
final MarshallingWebServiceInboundGateway inboundGateway =
new MarshallingWebServiceInboundGateway(jaxb2Marshaller);
inboundGateway.setRequestChannelName(INPUT_CHANNEL_NAME);
inboundGateway.setHeaderMapper(defaultSoapHeaderMapper);
return inboundGateway;
}
#Bean
public IntegrationFlow querySynchronous() {
return IntegrationFlows.from(INPUT_CHANNEL_NAME)
.enrichHeaders(...)
.wireTap(performanceTimestampRegistrator.registerTimestampFlow(SYNC_REQUEST_RECEIVED_TIMESTAMP_NAME))
.handle(outboundGateway)
.wireTap(performanceTimestampRegistrator.registerTimestampFlow(SYNC_RESPONSE_RECEIVED_TIMESTAMP_NAME))
//.transform( m -> m) // for tests - REMOVE
.get();
}
And the timestamp flow:
public IntegrationFlow registerTimestampFlow(String asyncRequestReceivedTimestampName) {
return channel -> channel.handle(
m -> MetadataStoreConfig.registerFlowTimestamp(m, metadataStore, asyncRequestReceivedTimestampName));
}
The notable thing here is that if I uncomment the no-operation transformer, everything suddenly works fine, but it doesn't sound right and I would like to avoid such workarounds.
Another thing is that the other, very similar flow works correctly, without any workarounds. Notable difference being it puts message in kafka using kafka adapter, instead of calling some web service with outbound gateway. It still generates response to handle (with generateResponseFlow()), so it should behave the same way. Here is the flow, which works fine:
#Bean
public MarshallingWebServiceInboundGateway workingInboundGateway(Jaxb2Marshaller jaxb2Marshaller,
DefaultSoapHeaderMapper defaultSoapHeaderMapper, #Qualifier("errorChannel") MessageChannel errorChannel) {
MarshallingWebServiceInboundGateway aeoNotificationInboundGateway =
new MarshallingWebServiceInboundGateway(jaxb2Marshaller);
aeoNotificationInboundGateway.setRequestChannelName(WORKING_INPUT_CHANNEL_NAME);
aeoNotificationInboundGateway.setHeaderMapper(defaultSoapHeaderMapper);
aeoNotificationInboundGateway.setErrorChannel(errorChannel);
return aeoNotificationInboundGateway;
}
#Bean
public IntegrationFlow workingEnqueue() {
return IntegrationFlows.from(WORKING_INPUT_CHANNEL_NAME)
.enrichHeaders(...)
.wireTap(performanceTimestampRegistrator
.registerTimestampFlow(ASYNC_REQUEST_RECEIVED_TIMESTAMP_NAME))
.filter(...)
.filter(...)
.publishSubscribeChannel(channel -> channel
.subscribe(sendToKafkaFlow())
.subscribe(generateResponseFlow()))
.wireTap(performanceTimestampRegistrator
.registerTimestampFlow(ASYNC_REQUEST_ENQUEUED_TIMESTAMP_NAME))
.get();
}
Then, there is no problem with wireTap being the last component and response is correctly received on replyChannel in time, without any workarounds.
The behavior is expected.
When the wireTap() (or log()) is used in the end of flow, there is no reply by default.
Since we can't assume what logic you try to include into the flow definition, therefore we do our best with the default behavior - the flow becomes a one-way, send-and-forget one: some people really asked to make it non replyable after log() ...
To make it still reply to the caller you need to add a bridge() in the end of flow.
See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-log
It works with your much complex scenario because one of the subscriber for your publishSubscribeChannel is that generateResponseFlow() with the reply. Honestly you need to be careful with request-reply behavior and such a publishSubscribeChannel configuration. The replyChannel can accept only one reply and if you would expect a reply from several subscribers, you would be surprised how the behavior is strange.
The wireTap in this your configuration is not a subscriber, it is an interceptor injected into that publishSubscribeChannel. So, your assumption about similarity is misleading. There is the end of the flow after that wiretap, but since one of the subscribers is replying, you get an expected behavior. Let's take a look into the publishSubscribeChannel as a parallel electrical circuit where all the connections get an electricity independently of others. And they perform they job not affecting all others. Anyway this is different story.
To conclude: to reply from the flow after wireTap(), you need to specify a bridge() and reply message will be routed properly into the replyChannel from the caller.
Dears,
I am trying to do some kind of event-driven Microservices. Currently, I was able to consume a message from Kafka and update database record when message is received using Quarkus & Smallrye-Reactive messaging extension. What I want to achieve further is to be able to send a message to other topic in case of success and send a message to error topic otherwise. I know that we can use return and #outgoing annotation for emitting new message but I don't think it will fit in my use case. I need a guidance here, if error happens while consuming a message. Should I return message to the original topic (by not acknowledging the message) or should I consume it and produce error message to different topic to rollback the original transaction.
Here is my code :
#Incoming("new-payment")
public void newMessage(String msg) {
LOG.info("New payment has been received.");
LOG.info("Payload is {}", msg);
PaymentEvent pe = jsob.fromJson(msg, PaymentEvent.class);
mysqlPool.preparedQuery("select totalBuyers from Book where isbn = ? ",
Tuple.of(pe.getIsbn()))
.thenApply(rs -> {
RowIterator<Row> iterator = rs.iterator();
if (iterator.hasNext()) {
return iterator.next().getInteger(0) + 1;
} else {
return Integer.valueOf(0);
}
})
.thenApply(totalCount -> {
return mysqlPool.preparedQuery("update Book set totalBuyers = ?",
Tuple.of(totalCount));
})
.whenComplete((rs, err) -> {
if (err != null) {
//Emit an error to error topic.
} else {
//Emit a msg to other service.
}
});
}
Also if you've better code please submit, I am still newbie in reactive programming :).
I've been doing enterprise integration for years and I think that you would want to do both.
Should I return message to the original topic (by not acknowledging
the message) or should I consume it and produce error message to
different topic to rollback the original transaction.
The event should remain on the topic for another instance to potentially pick up and process. And an error message should be logged as an event. Perhaps the same consumer could pick up and reprocess the event successfully.
An EDA (Event Driven Architecture) may offer different ways to handle this but on an ESB the message would be marked as tried. Generally three tried attempts would send it to a dead-letter queue so that it can be corrected and reprocessed later.
Our enterprise is also starting to design and build applications using EDA so I am interested to read what others have to say on this question. And KUDOS to you for focusing on Quarkus. I believe that this is one of the best technologies to come from Redhat that I have seen yet!
Another problem with this approach is that you are doing “2 writes in 1 service” e.g. one call to the db and another one to a topic. And this can become problematic when one of the 2 writes fails.
If you want to avoid this and use a pure event driven approach, then you need to reorder your events in such a way that writing to a db is the last event in the whole flow so that you can prevent 2 writes from 1 service.
Thus in your case: change the 2nd thenApply(..) method from updating the db into firing a new event to another topic. And the consumer of this new topic should do the db update. Thus the flow becomes like this:
Producer -> topic1 -> consumer (select from ...) & fire event to another topic -> topic2 -> consumer (update table).
Iam using telosB motes for implementation.
I have come across one of the way for acknowledging the packets,
task void send() {
call PacketAcknowledgements.requestAck(&myMsg);
if(call AMSend.send(1, &myMsg, 0) != SUCCESS) {
post send();
}
}
event void AMSend.sendDone(message_t *msg, error_t error) {
if(call PacketAcknowledgements.wasAcked(msg))
// do something if packet was acked
else
// do something else if packet was not acked (possibly resend)
}
Actually my doubt is, the receiving mote should have to acknowledge the packet or it should have PacketAcknowledgements interface in its application in order to send ACKs.
How this type of acknowledgement works?
And I have checked with my own type of acknowledgement, it works like after receiving the packet the mote acknowledge the packets, if source mote does not receive positive ack in certain time frame then re transmit the packet .
So which is better way of doing?
Please guide & thanks,
In TinyOS acknowledgements are implemented on the lowest communication abstraction level - active message[1]. This means that any component that operates with active messages has a built in support for synchronous acknowledgements.
Actually my doubt is, the receiving mote should have to acknowledge
the packet or it should have PacketAcknowledgements interface in its
application in order to send ACKs.
If you used PacketAcknowledgements.requestAck(&myMsg) to request acknowledgement, then you don't have to write extra code in Receive.receive event handler to process acks as this is done for you by underlying communication layer. All you need to do is wire PacketAcknowledgements interface that your component/module uses to one of the providers (AMSenderC or ActiveMessageC).
How this type of acknowledgement works?
The high level idea is following - calling PacketAcknowledgements.requestAck(&myMsg) sets a flag in a packet header and tells sender component not to signal sendDone event until ack was received (or timed out). When receiver component handles the packet on the other end it reads the flag and sends and ack if requested.
Having said all that, description of your way of acknowledging packets seems very similar to what PacketAcknowledgements offers, so personally I would avoid writing extra code for handling acknowledgements myself and stick with the tools provided.