I have an spring integration implemetation where I have two Client subscribers listening to the same JMS Topic. I am using JDBC message store( Different REGIONS) in both the implementation to save the incoming messages. While processing the data I get the exception:
org.springframework.dao.EmptyResultDataAccessException: Incorrect result size: expected 1, actual 0
I know this is Jira issue : https://jira.spring.io/browse/INT-2912
As now I cant upgrade the Spring version. I am unable to understand the workaround "The work-around is either to always use a different groupKey or to use separate tables for each Message Store. We will need to add a REGION column to the INT_GROUP_TO_MESSAGE as well."
How can i create a different groupKey?
My implementation is as below:
<bean
id="jdbcMessageStore"
class="org.springframework.integration.jdbc.JdbcMessageStore"
p:dataSource-ref="datasource"
p:region="REPORTS"/>
<si:aggregator
send-partial-result-on-expiry="false"
message-store="jdbcMessageStore"
discard-channel="discardedLogger"/>
The "groupKey" mentioned there is the correlation strategy result; by default it just uses the correlationId header.
You can use correlation-strategy-expression="'foo' + headers['correlationId']" and correlation-strategy-expression="'bar' + headers['correlationId']" to use a different group key for each app.
Related
I'd like to use Spring Kafka with Transactions but I don't really understand how it is supposed to be configured and how it works.
Here is my configuration
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.RETRIES_CONFIG, String.valueOf(Integer.MAX_VALUE));
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);
props.put(ProducerConfig.ACKS_CONFIG, "all");
This config is used in a DefaultKafkaProducerFactory with transaction id prefix:
defaultKafkaProducerFactory.setTransactionIdPrefix("my_app.");
Problem 1 :
How am I supposed to chose this transaction id prefix ?
If I understand correctly, this prefix is used by spring to generate a transactional id for each producer created.
Why couldn't we just use "UUID.randomUUID() ?
Problem 2 :
If the producer is destroyed, it will generate a new transactional id.
Therefor, if the application crash, on restart it will reuse old transactional id.
Is that normal ???
Problem 3 :
I'm using an application deployed on cloud which can be automatically scaled up/down.
This means my prefix can not be fixed since all my producers on each instances will have transactional id in conflict.
Should I add a random part in it ?
Do I need to recover the same prefix when an instance is scaled down/up or crash and restart ?
Problem 4 :
Last but not least, we are using credentials for our Kafka.
This does not seems to work :
Current ACLs for resource `TransactionalId:my_app*`:
User:CN... has Allow permission for operations: All from hosts: *
How should I set my ACLs knowing that my transactional ids are generated ?
Edit 1
After further reading, if I understand correctly.
If you have a C0(consumer) reading from P0(partition).If the broker start a consumer rebalance.
P0 could be assigned to another consumer C1.
This consumer C1 should use the same transaction id than then previous C0 to prevent duplicate (Zombies fencing) ?
How do you achieve this in spring-kafka ? The transaction id seems to have nothing to do with the consumer and thus the partition read.
Thanks
You can't use a random TID because of the zombie fencing - if the server crashes you could have a partial transaction in the topic which will never be completed and nothing more will be consumed from any partitions with a write for that transaction.
That is by design - for the above reason.
Again, you can't randomize; for the above reason.
Cloud Foundry, for example, has an environment variable that indicates the instance index. If you are using a cloud platform that doesn't include something like that, you would have to simulate it somehow. Then, use it in the transaction id:
spring.kafka.producer.transaction-id-prefix=foo-${instance.index}-
ACLs - I can't answer that; I am not familiar with kafka permissions; might be better to ask a separate question for that.
I think we need to add some logic to Spring to ensure the same transaction id is always used for a particular topic/partition.
https://github.com/spring-projects/spring-kafka/issues/800#issuecomment-419501929
EDIT
Things have changed since this answer (KIP-447); if your brokers are 2.5.0 or later - see. https://docs.spring.io/spring-kafka/docs/2.5.5.RELEASE/reference/html/#exactly-once and https://docs.spring.io/spring-kafka/docs/2.6.0-SNAPSHOT/reference/html/#exactly-once
I've been trying my hands on Spring AMQP. And I have a couple of questions:
I'd like to know what is Publisher returns and how is it different from Publisher Confirm. Of my understanding, we have a Publisher Confirm Callback that checks the status of acks. Now I looked at the documentation in Spring AMQP and Rabbit MQ. didn't really find or understand much on this.
And also why is it that if the message is tried to send to a non-existing queue, I don't get any sort of acknowledgement (ack/nack
) nor do I get any errors. Is there a way to setTimeouts for non-acknowledgements?
Short answer from the link https://www.rabbitmq.com/confirms.html :
"For unroutable messages, the broker will issue a confirm once the exchange verifies a message won't route to any queue (returns an empty list of queues). If the message is also published as mandatory, the basic.return is sent to the client before basic.ack."
In Spring AMQP if you set 'spring.rabbitmq.publisherReturns' to true this will mean messages will be 'mandatory' (unless you set mandatory to false) because of the following code:
private boolean determineMandatoryFlag() {
Boolean mandatory = this.properties.getTemplate().getMandatory();
return (mandatory != null ? mandatory : this.properties.isPublisherReturns());
}
I suggest you to read this article. There is a good description of all possible acknowledgments scenarios, including returns for the unrouted messages, like your non-existing queue.
From the Spring AMQP perspective you should bear in mind: https://docs.spring.io/spring-amqp/docs/2.0.3.RELEASE/reference/html/_reference.html#template-confirms
This feature requires a CachingConnectionFactory that has its publisherReturns property set to true.
I've been using Spring boot and getting rid of all XML files in my project.
Unfortunately it also uses Spring integration which from my experience is very heavily XML based.
I have a scenario which requires me to have an aggregator, and have that aggregator polled every x amount of seconds.
This can be done using XML like so (example taken from a previous SO question):
<!--
the poller will process 100 messages every minute
if the size of the group is 100 (the poll reached the max messages) or 60 seconds time out (poll has less than 100 messages) then the payload with the list of messages is passed to defined output channel
-->
<int:aggregator input-channel="logEntryChannel" output-channel="logEntryAggrChannel"
send-partial-result-on-expiry="true"
group-timeout="60000"
correlation-strategy-expression="T(Thread).currentThread().id"
release-strategy-expression="size() == 100">
<int:poller max-messages-per-poll="100" fixed-rate="60000"/>
</int:aggregator>
I've managed to find a class that kinda sorta does the trick and it's bean definition is:
#Bean(name = "aggregatingMessageHandler")
public AggregatingMessageHandler aggregatingMessageHandler() {
AggregatingMessageHandler aggregatingMessageHandler =
new AggregatingMessageHandler(messageGroupProcessorBean(),
new SimpleMessageStore(10));
aggregatingMessageHandler.setCorrelationStrategy(customCorrelationStrategyBean());
aggregatingMessageHandler.setReleaseStrategy(
new TimeoutCountSequenceSizeReleaseStrategy(3,
TimeoutCountSequenceSizeReleaseStrategy.DEFAULT_TIMEOUT));
aggregatingMessageHandler.setExpireGroupsUponCompletion(true);
aggregatingMessageHandler.setOutputChannel(outputAggregatedChannelBean());
return aggregatingMessageHandler;
}
However this triggers the canRelease() method of the ReleaseStrategy only when a new message is received in the inboundChannel associated with this handler, and not at a fixed time interval which is not the desired result.
I want all groups older than one minute to be redirected to the output channel.
My question is - is there a way to programmatically attach a Poller such as the one in the XML definition?
For Java & Annotation configuration take a look here and here.
The Aggregator component has AggregatorFactoryBean for easier Java Configuration.
Anyway you have to pay attention that there is a #ServiceActivator annotation together with a #Bean on that handler definition. And exactly #ServiceActivator has poller attribute.
Also pay attention that there is a Java DSL for Spring Integration.
Another part of your question is a bit confusion. The poller fully isn't related to the release strategy. Its responsibility in this case to receive messages from the PollableChannel which is that logEntryChannel. And only after that already polled messages are placed to the aggregator for it correlation and release logic.
What is done in that sample is fully different story and we can discuss it in the separate SO thread.
I am trying implement sample retry mechanism for RabbitMQ using Spring's StatefulRetryOperationsInterceptor.
As stated in documentation, I need to setup message key generator as the message id is absent. What I don't understand is the real usage of unique id generated per message. i.e. when I used below implementation I did not have any issue with retry:
StatefulRetryOperationsInterceptor interceptor =
RetryInterceptorBuilder.stateful()
.maxAttempts(3)
.backOffOptions(2000, 1, 2000)
.messageKeyGenerator(
new MessageKeyGenerator() {
#Override
public Object getKey(Message message) {
return 1;
}
);
container.setAdviceChain(new Advice[] {interceptor});
Stateful retry needs the originating message to be somehow unique - so the retry "state" for the message can be determined - the simplest way is to have the message publisher add a unique message id header.
However, it's possible something in your message body or some other header might be used to uniquely identify the message. Enter the MessageKeyGenerator which is used to determine the unique id.
Using a constant (1 in your case) won't work because every message has the same message key and will all be considered to be deliveries of the same message (from a retry perspective).
The framework does provide a MissingMessageIdAdvice which can provide limited support for stateful retry (if added to the advice chain before the retry advice). It adds a messageId to the incoming message.
"Limited" means full stateful retry support is not available - only one redelivery attempt is allowed.
If the redelivery fails, the message is rejected which causes it to be discarded or routed to the DLX/DLQ if so configured. In all cases the "temporary" state is removed from the cache.
Generally, if full retry support is needed and there is no messageId property and there is no way to generate a unique key with a MessageKeyGenerator, I would recommend using stateless retry.
I have a simple route like this
from("file:data/inbox?noop=true").transform().body().to("file:data/outbox").bean(UpdateInventory.class);
from("direct:update").to("file:data/anotherbox").to("direct:newupdate");
from("direct:newupdate").to("file:data/newbox");
And the output i am expecting is
-file://data/inbox
--transform[simple{body}] 15 ms
--file:data/outbox 5
---bean[com.classico.sample.UpdateInventory#3a469fea 19
---file:data/anotherbox 6
----direct:newupdate 5
-----file:data/newbox 4
I tried using a EventNotifier and when the ExchangeCompletedEvent is received i fetched the message History.But since the second exchange is completed first message history is showing up in the reverse order of invocation.Is it possible to store all the messgae histories in a collection and print them in reverse order or Is there any event that is suitable for this.?
if (event instanceof ExchangeCompletedEvent) {
ExchangeCompletedEvent exchangeCompletedEvent = (ExchangeCompletedEvent) event;
Exchange exchange = exchangeCompletedEvent.getExchange();
String routeId = exchange.getFromRouteId();
List<MessageHistory> list = exchange.getProperty(Exchange.MESSAGE_HISTORY, List.class);
for (MessageHistory history : list) {
String id = history.getNode().getId();
String label = URISupport.sanitizeUri(history.getNode().getLabel());
log.info(String.format(MESSAGE_HISTORY_OUTPUT, routeId, id, label, history.getElapsed()));
}
}
You can use JMX to get all that details for each processor.
There is also a dumpRouteStatsAsXml operation on each route / camelContext that can output a xml file of the route(s) with all performance stats.
We use this in the hawtio web console to list this kind of information.
http://hawt.io/
Also the Camel Karaf / Jolokia Commands uses this as well
https://github.com/apache/camel/blob/master/platforms/commands/commands-core/src/main/java/org/apache/camel/commands/ContextInfoCommand.java#L175
And in next release of Camel you can also easier get the various processor mbeans, from CamelContext if you know their id, using
https://github.com/apache/camel/blob/master/camel-core/src/main/java/org/apache/camel/CamelContext.java#L545
Then you can use the getters on the mbean to get the performance stats.
The event notifer which was suggested is also great, but the events are on a higher level, although you get an event for sending to an endpoint, such as to some external system, which often is enough to capture details about. For low level details as asked here, then you need to use the JMX stats.
Ohh I forgot to tell about the message history EIP which also has a trace of how the message was routed with time taken stats as well.
http://camel.apache.org/message-history.html
That is maybe also just what you need, then you can get that information from the exchange as shown on that link.
I would suggest using the camel EventNotifier. You can find documentation on how to use it here:
http://camel.apache.org/eventnotifier-to-log-details-about-all-sent-exchanges.html