How to poll directory for a file? - spring

Question

I need to be able to poll a directory for a specific file using SCP, and once the file has been processed, it needs to keep polling.
Is this possible with Spring Batch?

Answer 1

The normal way to handle this is using Spring Integration. The way I'd address it is with a Spring Integration flow that uses a SFTP Inbound Channel Adapter to retrieve the files, then passes the transferred name to Spring Batch to launch. The flow would actually be similar to the sample in the SpringBatchIntegration in my Spring Batch Webinar here: https://github.com/mminella/SpringBatchWebinar
In that example, I use Twitter to launch the job. The only thing you'd need to change is the twitter piece for the SFTP.

Answer 2

I had to solve the same question (but just accessing to the local filesystem) and I did not find any solution in the framework, so I ended up creating my own class which polls for the file and creates a resource.I know this is just a workaround, but I haven't found a better way to do that so far.
I can't remember where (maybe in the "retry handling" part) but I read in the documentation something like "batch jobs should not try to solve issues like files not found, connections down and so, these kind of errors should make the job raise an error to be handled by operators" so I gave up...
On the other hand Spring Retry was part of Spring batch and now is a new separate library, maybe you just can assume the file is there and if the reader does not find it, let the step fail and establish a "retry policy" for that step, but for me that's overkill.
This is what I did:
<bean id="resourceFactory"
class="com.mycompany.batch.zip.ResourceFactory">
<property name="retryAttemps" value="${attemps}" />
<property name="timeBetweenAttemps" value="${timeBetweenAttemps}"/>
</bean>
<bean id="myResource"
factory-bean="resourceFactory" factory-method="create" scope="step">
<constructor-arg value="${absolutepath}" type="java.lang.String" />
</bean>
<!--step scope to avoid looking for the file when deployment-->
<bean id="myReader"
class="org.springframework.batch.item.xml.StaxEventItemReader" scope="step">
<property name="fragmentRootElementName" value="retailer" />
<property name="unmarshaller" ref="reportUnmarshaller" />
<property name="resource" ref="myResource" />
</bean>
And this is my class:
public class ResourceFactory {
public static final Logger LOG= LoggerFactory.getLogger(ResourceFactory.class);
private int retryAttemps;
private long timeBetweenAttemps;
public Resource create(String resource) throws IOException, InterruptedException {
Resource r;
File f=new File(resource);
int attemps=1;
while (!f.exists()) {
if (attemps<this.retryAttemps) {
attemps++;
LOG.warn("File "+resource+" not found, waiting "+timeBetweenAttemps+
" before retrying. Attemp: "+attemps+" of "+this.retryAttemps);
Thread.sleep(this.timeBetweenAttemps);
} else {
throw new FileNotFoundException(resource);
}
if (resource!=null && resource.endsWith(".zip")) {
ZipFile zipFile = new ZipFile(resource);
ZipEntry entry=zipFile.entries().nextElement();
if (entry==null) {
throw new FileNotFoundException("The zip file has no entries inside");
}
//TODO Test if a buffered Stream is faster than the raw InputStream
InputStream is=new BufferedInputStream(zipFile.getInputStream(entry));
r= new InputStreamResource(is);
if (LOG.isInfoEnabled()) {
int size=(int)entry.getSize();
LOG.info("Opening a compressed file of "+size+" bytes");
}
} else {
LOG.info("Opening a regular file");
r= new FileSystemResource(f);
}
}
return r;
}
}
If anyone knows a better way to do that, I'll gladly remove this answer (and implement the new solution)
PS: BTW, I've found some faults in my code when reviewing this post, so for me this is being helpful even with no other answers :)