We have a discard channel for some filters and aggregators, but we have seen that discard channel is a regular direct channel.
So, where those discarded messages go ? to DLQ ?
And also, do they live forever there ? Because we are struggling with memory consumption and we would like those messages to be deleted (garbage collected ) as soon they arrive to the DLQ.
Even if Spring Integration is based on the Messaging concepts, it is just a tool to build integration solutions. It isn't a Broker and therefore doesn't provide out-of-the-box state management like queue persistence and DLQ.
You are correct, the discardChannel is just a MessageChannel and you can inject any implementation there and do some desired logic in the discard flow on that channel. Sending to DLQ is already your particular use-case and you really should do that manually. There is nothing specific for Spring Integration on that discard channel. And if it is just regular QueueChannel without any poller on the matter, you really end up once with the OOM because messages are stored in the memory forever.
Related
I want to implement a Kafka Producer with Spring that observes a Cloud Storage and emits meta informations about newly arrived files.
Until now we did that with a Kafka Connector but for some reasons we now have to do this with a simple Kafka producer.
Now I need to persist the state of the producer (e.g. timestamp of last commited file) in a kind of Offset Topic like the Connector did, but did not find a reasonable approach to do that.
My current idea is to hold the state by committing it to a topic that the producer also consumes but just acknowledge the last consumed state when commuting a new one. So if the Kubernetes pod of the producer dies and comes up again to consume the last state (not acknowledged) and so knows where it stopped.
But this idea seems to be a bit complex to just hold a state of a Kafka app. Is there a better approach for that?
I am using spring integration with weblogic jms. my logic is to put the json object in jms queue, and consume it from queue into the channel, validate it and route it based on a particular field. if there is any error, do the fix and put it back into the queue. I have two issues. 1. when the message is consumed into the object, I still can see the object as pending in administrative console of weblogic. 2. After fixing the validation, if put the modified object in the queue, I am getting the original object from queue.
<int-jms:outbound-channel-adapter id="jmsOutbound"
channel="requestChannel" connection-factory="queueConnectionFactory"
destination="inputQueue" />
<int-jms:message-driven-channel-adapter
id="jmsInbound" connection-factory="queueConnectionFactory"
destination="inputQueue" channel="routingChannel" />
if (message.getHeaders().get("documentType").equals("sec"))
routingChannels.add(outboundSecChannel);
else if (message.getHeaders().get("documentType").equals("unds"))
routingChannels.add(outboundFChannel);
else if (message.getHeaders().get("documentType").equals("CH"))
routingChannels.add(outboundAChannel);
else{
routingChannels.add(errorChannel);
}
putting in channel using routing
thanks for your help.
I think you should distinguish the consuming part from producing to the separate threads. I don't tell that you have to switch to the transactions, but at least the simple acknowledge for the consumed message should be done.
Since you tell that you are going to putt the message back to the queue, that's definitely the fact to always acknowledge the consumed message independently of the error fact. So, what I suggest is something like to place a QueueChannel or an ExecutorChannel somewhere after <int-jms:message-driven-channel-adapter> and before <int-jms:outbound-channel-adapter> to let them do their hard work with the WebLogic JMS in their own threads.
I want my system to guarantee there is no data loss even if the system is shutting down.
What this mean is that the system must not miss the request message. So, I will change the way that accept http reqeust. Now, I am using http gateway/webservice gateway in spring integration. But, This isn't receive the message even if the system dies. So, I want to add the queue between the http client and the http receiver. So, I want to use a queue channel. Here is the question.
① I have to install other queue program such as activemq or rabbitmq and have to connect to the queue channel in spring integration?
② and which one is the best combination with spring integration? I heard that rabbit mq is the best one.
please give me a elaborate explanation. thanks.
First of all you description isn't clear...
If you don't want to lose messages from the QueueChannel use some Persistence MessageStore, like JdbcChannelMessageStore:
http://docs.spring.io/spring-integration/docs/latest-ga/reference/html/system-management-chapter.html#message-store
From other side there are channel wrappers for the AMQP as well as for JMS:
http://docs.spring.io/spring-integration/docs/latest-ga/reference/html/amqp.html#d4e5846
http://docs.spring.io/spring-integration/docs/latest-ga/reference/html/jms.html#jms-channel
Which really provide the same persistence durability, fault tollerant options for your use-case.
Re. activemq VS rabbitmq. I can say by my own expiriance that the last one is better, by configuration, usage from Spring Integration (Spring AMQP is under the shell). And its performance is really better.
All other info you can find in the Internet.
I am currently working on an application with Spring Integration. The application requires guaranteed delivery and the option that the it will be functional for a specific amount of time that the external systems are unavailable without losing messages. Channels will be JMS backed with expiration time. I would like to understand which is the best practice for redelivery with Spring Integration. We are having the following options:
The application's integration flow has a number of outbound message gateways that requires RPC calls with external system. Statefull retry advice can be used. After the max attemps is reached for specific runtime exceptions the message will be addressed to a recovery channel. The recovery channel will use a delayer and will then address the message back to the original channel. After X times that the message will reach the recovery channel it will be addressed to the error channel where it will be simply logged without further processing. The delayer component in this case should use the jdbc message store option.
Another option would be to use the standard JMS option for redelivery. In this case the redelivery policy will not be implemented on Spring Integration but on the JMS provider side.
Which is the best practice for message redelivery with Spring Integration?
I'd say like this: don't reinvent the wheel!
If there is already some similar solution on the matter, just use it as is with its specific configuration.
Right, if JMS has that solution, just go ahead.
There is need, of course, to get deal with DLQ in case of message expiration or redelivery exhausting. But the concept is here.
I have an application that consumes messages from a JMS topic. As part of the normal application flow it needs to periodically cease consumption of messages. While the application is in this state new messages are stored in the topic (note that my application is still running). Later the application resumes message consumption, also receiving those messages that were placed on the topic while the application wasn't listening.
This functionality is currently achieved by creating and disposing of connections from a ConnectionFactory. However, I now wish to migrate the application to Spring JMS. Although Spring rather neatly abstracts away much of the JMS boiler-plate - I no longer appear to have fine grained control over the underlying connection and hence cannot halt message consumption on demand.
Before I try to wade through Spring JMS internals, can anyone suggest a neat way of doing this?
Can you just avoid returning from onMessage()? How long do you want to stop consumption? Is your problem similar to https://stackoverflow.com/a/628337/20734