We have a spring integration application which will monitor on an incoming folder then process the files. I was using prevent-duplicates="true" but I have to change to false.
Here I explained the reason.
file inbound-channel-adapter performance issue
Now I am seeing multiple threads processing same file. I appreciate help!
<file:inbound-channel-adapter id="inFiles" channel="inFilesin" directory="file:${incoming.folder}"
queue-size="300" filename-regex="(?i)^(?!.*writing)" prevent-duplicates="false" auto-startup="true" auto-create-directory="false" >
<int:poller id="fw.fileInboudPoller" fixed-rate="1" receive-timeout="3" time-unit="SECONDS"
max-messages-per-poll="10" task-executor="taskExecutor" />
</file:inbound-channel-adapter>
<task:executor id="taskExecutor" pool-size="10-20" queue-capacity="20" rejection-policy="CALLER_RUNS" />
Thanks,
Mohan
As you have found, there are difficulties scaling, when a filesystem has a very large number of files.
You might want to consider alternative technologies, Rabbit, JMS, etc.
Alternatively, you can implement a custom file locker or directory scanner as described here.
The provided nio-locker won't help because it doesn't prevent multi-threaded access in this case.
Related
I am new to Spring Integration & Redis, So my apology if I am making a naive mistake.
My requirement is as below -
Need to implement a message queue. For dispatching money to the user based on some events.
The queue should not be volatile and Ensuring Atomicity. If I restarted the server or it crashed it should not lose the event message. This also includes messages which are currently in progress.
The queue should deliver the message granted once and only once. It will be a multithreaded(workers) and multi-server environment.
My progress till now is - configured Spring Integration and Spring Integration Redis in my spring project. My Spring Integration config as below -
<int-redis:queue-outbound-channel-adapter
id="event-outbound-channel-adapter"
channel="eventChannelJson"
serializer="serializer"
auto-startup="true" connection-factory="redisConnectionFactory"
queue="my-event-queue" />
<int:gateway id="eventChannelGateway"
service-interface="com.test.RedisChannelGateway"
error-channel="errorChannel" default-request-channel="eventChannel">
<int:default-header name="topic" value="queue"/>
</int:gateway>
<int:channel id="eventChannelJson"/>
<int:channel id="eventChannel">
<int:queue/>
</int:channel>
<bean id="serializer" class="org.springframework.data.redis.serializer.StringRedisSerializer"/>
<int:object-to-json-transformer input-channel="eventChannel"
output-channel="eventChannelJson"/>
<int-redis:queue-inbound-channel-adapter id="event-inbound-channel-adapter"
channel="eventChannelJson" queue="my-event-queue"
serializer="serializer" auto-startup="true"
connection-factory="redisConnectionFactory"/>
<bean id="serializer" class="org.springframework.data.redis.serializer.StringRedisSerializer"/>
<int:json-to-object-transformer input-channel="eventChannelJson"
output-channel="eventChannel"
type="com.test.PostPublishedEvent"/>
<int:service-activator input-channel="eventChannel" ref="RedisEventProcessingService"
method="process">
<int:poller fixed-delay="10" time-unit="SECONDS" max-messages-per-poll="500"/>
</int:service-activator>
I read an article on similar topic where they were using redis RPOPLPUSH for this purpose. But I am unable too figure out how to do it in Spring Integration.
Link for the article is - https://redis.io/commands/RPOPLPUSH
Please advise me regading this. I will realy appricate your help.
Spring integration does not have a component that utilizes that operation.
To use it, you should wrap a RedisTemplate in a <int:service-activator/> and call one of its rightPopAndLeftPush() methods.
I am using Tomcat and I have http:inbound-gateway as gateway and channel named request configured as it's request channel. Sometimes there are lots of requests on my service which cause occurring Too many open files Error which comes from OS.
I tried to make request channel a QueChannel and set capacity for it but it does not work. Then I tried to set request channel as a polling-consumer channel and set a poller with fixed-delay to poll and it did not work again. Is there a conventional way to limit number of input requests?
And how can I customize response in discarded requests?
<int:channel id="request">
<int:queue capacity="100"/>
</int:channel>
...
<int-http:inbound-gateway id="RESTServiceGateway"
supported-methods="GET"
request-channel="request"
error-channel="errorResolver" />
<int:chain input-channel="request" output-channel="response">
<int:poller fixed-delay="1" max-messages-per-poll=""/>
...
Or
<task:executor id="requestExecutor" pool-size="1-10" queue-capacity="10"/>
<int:channel id="request">
<int:dispatcher task-executor="requestExecutor"/>
</int:channel>
<int-http:inbound-gateway id="RESTServiceGateway"
supported-methods="GET"
request-channel="request"
error-channel="errorResolver" />
<int:chain input-channel="request" output-channel="response">
...
Too many open files is not related to your integration flow configuration. This is about opened sockets from the HTTP clients to your Tomcat. This one is already concurrent and can handle many requests in parallel. Therefore I would say that your paralleling logic in the flow doesn't bring too much value and definitely doesn't effect a number of opened sockets.
You can configure concurrency in Tomcat though: maxConnections in server.xml .
Another option is to increase an amount for those opened file on your Linux using ulimit tool: https://www.tecmint.com/increase-set-open-file-limits-in-linux/
We have an application where we need to poll a folder and process the files.
We are using clustered environment and on each server files are getting processed using multiple threads. I am using FileInBoundAdapter, poller and Task-executor. But I am seeing the same files are getting processed in different threads. After reading Spring integration documentation it seems ConcurrentMetadataStore/RedisMetadataStore will help to avoid this issue.
I am trying to find out a sample code for RedisMetadataStore API.
Can someone help me sample code or suggest different solution?
Thanks,
Mohan
The sample is pretty simple. You just need to supply the RedisConnectionFactory to the same Redis server for all your cluster nodes and inject FileSystemPersistentAcceptOnceFileListFilter to your <int-file:inbound-channel-adapter>s:
<bean id="redisMetadataStore" class="org.springframework.integration.redis.metadata.RedisMetadataStore">
<constructor-arg ref="redisConnectionFactory"/>
</bean>
<bean id="acceptOnceFilter" class="org.springframework.integration.file.filters.FileSystemPersistentAcceptOnceFileListFilter">
<constructor-arg ref="redisMetadataStore"/>
<constructor-arg value="the_key"/>
</bean>
<int-file:inbound-channel-adapter filter="acceptOnceFilter"/>
I am trying to poll local directory using ftp inbound adapter to fetch files for further processing. I want to move the file to another local directory and delete it from origination. Not getting a way to achieve it. Here's what I have so far:
<int-ftp:inbound-channel-adapter id="ftpInbound"
channel="ftpChannel" session-factory="ftpClientFactory"
filename-pattern="*.xml" auto-create-local-directory="false"
delete-remote-files="false" remote-directory="/" local-directory="//C://FBS//testmq">
<int:poller fixed-rate="20000" />
</int-ftp:inbound-channel-adapter>
<int:channel id="ftpChannel">
<int:queue />
</int:channel>
Use transaction synchronization with a pseudo transaction manager; see the file example in the documentation. Here's the configuration from that section of the docs:
<int-file:inbound-channel-adapter id="inputDirPoller"
channel="someChannel"
directory="/foo/bar"
filter="filter"
comparator="testComparator">
<int:poller fixed-rate="5000">
<int:transactional transaction-manager="transactionManager" synchronization-factory="syncFactory" />
</int:poller>
</int-file:inbound-channel-adapter>
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="payload.renameTo(new java.io.File('/success/' + payload.name))"
channel="committedChannel" />
<int:after-rollback expression="payload.renameTo(new java.io.File('/failed/' + payload.name))"
channel="rolledBackChannel" />
</int:transaction-synchronization-factory>
Continue reading into the next section...
Referring to the above section, you may be thinking it would be useful to take these 'success' or 'failure' actions when a flow completes, even if there is no 'real' transactional resources (such as JDBC) downstream of the poller. For example, consider a followed by an ftp:outbout-channel-adapter/. Neither of these components is transactional but we might want to move the input file to different directories, based on the success or failure of the ftp transfer.
To provide this functionality, the framework provides a PseudoTransactionManager, enabling the above configuration even when there is no real transactional resource involved. If the flow completes normally, the beforeCommit and afterCommit synchronizations will be called, on failure the afterRollback will be called. Of course, because it is not a real transaction there will be no actual commit or rollback. The pseudo transaction is simply a vehicle used to enable the synchronization features.
I have activemq used in my system and what i see is the following message:
TopicSubscription: consumer=...: Pending message cursor [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor#1684f89c] is full, temp usage (0%) or memory usage (100%) limit reached, blocking message add() pending the release of resources.
This is because if i understand correct my consumer is slow while my producer is fast. The result out of it is that eventually my producer is blocked untill consumer reads the message and frees some memory. What i whant is that my producer is not blocked and also when memory is full old messages are being discurded.
Given my understanding of what i have read the following configuration should do the trick (messageEvictionStrategy, pendingMessageLimitStrategy) but it is not working for me and i cannot figure out why.
I have specified low memoryusage limit low (35Mb) to make issue apear faster for testing reasons, but the case is that i need it eventually when the problem apears for activemq to just drop old messages.
I have found one non satisfactory solution of setting in ActiveMQConnectionFactory useAsyncSend=true and specifying sendTimeout. This makes producer not blocked but in this way the newest message is dropped and not the olderst one.
Finally, i am talking for non durable topics.
Any help guys would pe perfect. Below i have activemq configuration
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false" memoryLimit="35 Mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
<messageEvictionStrategy>
<oldestMessageEvictionStrategy/>
</messageEvictionStrategy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="35 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="5000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
activemq version 5.7.0
i use spring template to send messages:
<bean class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
<property name="timeToLive" value="100"/>
</bean>
I transmit javax.jms.ObjectMessage, relativelly small in size.
I found the problem in customer premisses I have many toppics in my application but managed to reproduce it loccally sending from 1 thread, non-stop messages continiusly always to the same topic. The message send was just a small string.
I have only one producer and problem seems to appear when i have 1 (or more) slow consumer(s) -but one slow consumer is enough-. if no slow consumer exists, problem does not appear.
I do not think it makes any difference but i use
<transportConnectors>
<transportConnector name="openwire" uri="nio://0.0.0.0:33029?wireFormat.maxInactivityDuration=60000&wireFormat.maxInactivityDurationInitalDelay=60000"/>
</transportConnectors>
How can I recreate this? How many producers/consumers are attached to this topic? Is it only one topic?
Your settings look okay, but you don't need to set memoryLimit=35mb on the policy. It kinda doesn't make sense to set it the same as the overall system usage. The idea is that memory limit for all topics combined would equal the system memory limit. So for example, if you have two topics, each would use 35MB (2 * 35 == 70MB) and that would exceed the overall system memory settings. I don't think this is the specific cause of what you're seeing, but something to keep in mind.
Also what version of ActiveMQ is this? If you have some tests already written that can produce this, let me know.
It turns out that when using JmsTemplate in order to have asynchronous sent and then messages that cannot be delivered we need to enable explicitQosEnabled and set deliveryMode=1 (non persistent).
Also on client side, the consumer needs to have a smaller prefetch limit
server
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="pooledJmsConnectionFactory"/>
<property name="explicitQosEnabled" value="true"/>
<property name="deliveryMode" value="1"/>
</bean>
client
<jms:listener-container .. prefetch="1000">
...
</jms:listener-container>
dont ask me why... but this seems to have solved my problem.
Basically non 100% needed but if someone can explain this to me would be perfect