Spring IMAP Poller reads emails after some hours of delay - spring

Question

I am using Spring Integration to poll emails from 3 mailboxes using 3 adapters.
I am using customSearchTermStatregy to read emails after 10 mins of receiving.
This application reads email and saving those email into local directory as .eml file.
Now the problem is, some email are read after 4-5 hours of delay by the poller.
I configured a thread pool as well, but I suspect spring poller is not getting thread allocated immediately to read email, hence the delay in reading.
Please also note, this application is running under tomcat along with other 10+ java applications running under same tomcat.
Can someone point out some probable reasons behind this 4-5 hours of delay?
Spring Integration.xml:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:mail="http://www.springframework.org/schema/integration/mail"
xmlns:int="http://www.springframework.org/schema/integration"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/integration/mail
http://www.springframework.org/schema/integration/mail/spring-integration-mail-5.2.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util-2.0.xsd
http://www.springframework.org/schema/task
http://www.springframework.org/schema/task/spring-task-4.3.xsd">
<util:properties id="javaMailProperties">
<prop key="mail.imap.socketFactory.class">javax.net.ssl.SSLSocketFactory</prop>
<prop key="mail.imap.socketFactory.fallback">false</prop>
<prop key="mail.store.protocol">imaps</prop>
<prop key="mail.debug">${imap.debug}</prop>
<prop key="mail.imaps.partialfetch">false</prop>
<prop key="mail.imaps.fetchsize">102400</prop> <!-- 100KB, default is 16KB -->
</util:properties>
<task:executor id="taskExecutor" pool-size="15-50" queue-capacity="100"/>
<mail:inbound-channel-adapter id="imapAdapter1"
store-uri="${imap.uri.mail}"
channel="recieveEmailChannel"
should-delete-messages="false"
should-mark-messages-as-read="true"
auto-startup="true"
simple-content="true"
auto-close-folder="false"
search-term-strategy="mailSearchTermStrategy"
java-mail-properties="javaMailProperties">
<!-- <int:poller fixed-delay="${imap.polling.interval}" time-unit="SECONDS"/> -->
<int:poller cron="0 0/2 1-23 * * ?" max-messages-per-poll="1" task-executor="taskExecutor"/> <!-- every 2 mins, from 1AM to 11.59PM -->
</mail:inbound-channel-adapter>
<mail:inbound-channel-adapter id="imapAdapter2"
store-uri="${imap.uri.fax}"
channel="recieveEmailChannel"
should-delete-messages="false"
should-mark-messages-as-read="true"
auto-startup="true"
simple-content="true"
auto-close-folder="false"
search-term-strategy="faxSearchTermStrategy"
java-mail-properties="javaMailProperties">
<int:poller cron="0 0/2 1-23 * * ?" max-messages-per-poll="1" task-executor="taskExecutor"/> <!-- every 2 mins, from 1AM to 11.59PM -->
</mail:inbound-channel-adapter>
<mail:inbound-channel-adapter id="imapAdapter3"
store-uri="${imap.uri.scan}"
channel="recieveEmailChannel"
should-delete-messages="false"
should-mark-messages-as-read="true"
auto-startup="true"
simple-content="true"
auto-close-folder="false"
search-term-strategy="scanSearchTermStrategy"
java-mail-properties="javaMailProperties">
<int:poller cron="0 0/2 1-23 * * ?" max-messages-per-poll="1" task-executor="taskExecutor"/> <!-- every 2 mins, from 1AM to 11.59PM -->
</mail:inbound-channel-adapter>
<int:channel id="recieveEmailChannel">
<int:interceptors>
<int:wire-tap channel="logger"/>
</int:interceptors>
</int:channel>
<int:logging-channel-adapter id="logger" level="DEBUG"/>
<int:service-activator input-channel="recieveEmailChannel" ref="emailReceiver" method="handleMessage"/>
<!-- <bean id="emailReceiver" class="com.abc.xyz.receiver.EmailReceiver">
</bean> -->
</beans>
CustomSearchTermStrategy:
#Service
public class CustomSearchTermStrategy implements SearchTermStrategy {
#Value("${email.delay.mins}")
int emailDelayMins;
Logger logger = LoggerFactory.getLogger(CustomSearchTermStrategy.class);
#Override
public SearchTerm generateSearchTerm(Flags supportedFlags, Folder folder) {
SearchTerm customSentDateTerm = new SearchTerm(){
private static final long serialVersionUID = 4583016738325920713L;
public boolean match(Message message) {
try {
ZoneId zoneId = ZoneId.of("America/Los_Angeles");
ZonedDateTime sendDtInPST = message.getSentDate().toInstant().atZone(zoneId);
ZonedDateTime currentZdt = ZonedDateTime.now(zoneId);
ZonedDateTime currentTimeMinus10Mins = currentZdt.minusMinutes(emailDelayMins);
Flags flags = message.getFlags();
if(currentTimeMinus10Mins.isAfter(sendDtInPST)) {
if(!flags.contains(Flags.Flag.SEEN)) {
logger.info("CurrentTimeMinus"+emailDelayMins+"MinsInPST is AFTER sendDtInPST, so this email picked for read, subject:{}", message.getSubject());
return true;
}
}
} catch (Exception ex) {
ex.printStackTrace();
}
return false;
}
};
return customSentDateTerm;
}
}

Answer 1

In our case, the delay in polling a large email from the IMAP server was because we used default values for properties mail.imap.fetchsize=16kb and mail.imap.partialfetch=true.
This made the email to be fetched in sizes of 16kb which resulted in long hours for polling. So we set the mail.imap.partialfetch=false to ensure that the mail gets fetched at a stretch
Refer this https://javaee.github.io/javamail/docs/api/com/sun/mail/imap/package-summary.html for further property setting in IMAP

Answer 2

At the moment you show 3 pollers which are going to be run at the same time.
By default all those pollers are triggered by the TaskScheduler (don't mix it with TaskExecutor), which is based on 10 threads pool. So, even right now I see 3 concurrent threads taken by your email channel adapters. It might not be a surprise that you have some other pollers in your your applications. And all of them share the same TaskScheduler.
You may revise all your pollers to shift the real work into particular executors, or you may consider to increase the TaskScheduler thread pool: https://docs.spring.io/spring-integration/docs/current/reference/html/configuration.html#namespace-taskscheduler
You also can do a thread dump to analyze which threads are busy and for what.