I'm looking for a way to trigger a spring integration that does file download from a spring batch job. Currently I've seen examples of triggering batch job after the integration has received the files.
Basically a step of my batch job should be fetch file from a remote server using spring integration and the next step is process it / upload to some server.
Related
I tried to search the spring batch documentation how Spring defines the execution order of multiple batch jobs, but couldn't reach anything. I've Spring Boot app that has multiple Spring Batch jobs and I want to understand in what order job launcher is executing these multiple different jobs?
I am working on a spring batch project where use case is:
Spring batch job relies on the file present at SFTP server( remote directory)
So, Once the file gets available in SFTP server, a corresponding job (spring batch) should start. Moreover, I don't want to first start the job and then look for the file because that would be time based approach rather I am looking for notification based approach. So I thought to use spring integration (sftp inbound channel adaptor).
As part of Inbound adaptor (SFTP) once I get/receive/download new file (based on timestamp) present at remote directory of SFTP server, I want to notify spring batch application and then trigger a Job. For e.g if there are two files at remote directory say 1.txt, 2.txt. These files come on daily basis. So once my channel receives new file at remote directory, it should
download1.txt and then notify spring batch application to launch Spring batch Job1
download2.txt and then notify spring batch application to launch Spring batch Job2
Can anyone validate this approach and guide further.
See the spring-batch-integration sub-project in Spring Batch; in particular the JobLaunchingGateway and JobLaunchingMessageHandler.
We are using spring batch 2.1.x coupled with spring integration 2.0 for the poller jobs. There are scheduled jobs as well in the mix.The launching of all these poller and scheduled jobs happen by invoking the spring batch api. We have around 60-65 jobs in total. Very frequently in the production environment what we are seeing is the jobs go into "starting" and they do not recover. So once we observe such occurrences we have to keep restarting our servers which contains our spring batch application. The backend of the spring batch datasource is Oracle 11gr2.
My current project is based on Spring Batch and Spring integration.
My goal is using Spring Batch to execute job flow steps:
read file from SFTP server(step1).
decrypt the file(step2).
save the file into database(step3).
I want to divide this into steps queue(read from sftp, decrypt, write to db).
And I also need to save the file, transfer start time, transfer end time, file size into DB.
Some days ago, I used Spring integration to poll file from sftp server and then send it to Spring batch to do the extra job. Also I can not get the transfer start time.
But now my project is main based on Spring batch, not Spring Integration.
Can you give me some suggestions ?
Or, can you show me how to use Spring Batch to drive Spring Integration for reading SFTP files.
And how can I get the transfer start time ?
Thanks. :)
Add a tasklet as the first step; have the tasklet invoke a Spring Integration flow, probably using an sftp outbound gateway to GET the file.
There's an example of how to use the gateway in the sftp sample.
The 'transfer start time' would be the time you invoke the gateway.
I am new to spring batch. I want to run spring batch jobs on server a and want to launch those jobs from server b using spring batch admin.is it possible? I have searched the following two ways:
1.JMX way: i could convert spring batch beans into mbeans but i cant read them from spring batch admin.can you tell how to read mbeans from spring batch admin and launch them?
2.common repository: i think if i use the same db repository for both spring batch and spring batch admin then i can launch remote jobs from spring batch admin (from server b).but in the job xml file in spring batch admin what should be the classpath for tasklet?
can you help in the above or tell me if any new way exists?
we ended up implementing a framework using mq communication to handle this. each 'batch node' registers itself and any 'batch class' parameters such as 'nodeType=A' or 'jobSizeiCanHandle=BIG' (these are fictitious but you get the point). The client console reads this information and queries the nodes via MQ for the job list. It then submits job requests with parameters via a rudimentary text based protocol (property file format).
command=START_JOB
job=JobABC
param1=x
param2=y
One of the batch nodes will pick up the message and start the job, it will return success/fail status in the same manner with a message with the same correlation id. so the client can show response to the user.
this allows us to do what you're talking about AND spark the jobs via an external scheduler (Control-M) . The 'nodeType=A' mentioned above allows us to query individual nodes (the nodes listen where 'nodeType=A or nodeType=*'. This allows commands to be 'targeted' to specific nodes if that is necessary.
Keep in mind, this is our own console, not the spring batch admin console. So perhaps that doesn't help you, but building up a simple console doesn't take that long using the spring batch APIs (4 or 5 asps).
The batch nodes could also have started up simple services like HTTP REST services or 'whatever' but we use MQ heavily and i liked the idea of not having to preregister nodes (the framework code doesn't know/care that it's in an HTTP container, so it couldn't register the endpoint easily). With MQ, the channel is preconfigured and all apps just 'use it' so it seemed easier.
Good luck.
I am trying to do the same thing. But it seems that in order to launch job directly from Spring batch admin, all the job resource has to be added to the spring batch web app. May be try restful job submission with spring MVC
#chau
One way to use Spring batch admin as is, but "discover" and "invoke" remote jobs is to provide your own implementations for org.springframework.batch.admin.service.JobService and org.springframework.batch.core.launch.JobOperator that can query and invoke jobs from remote job registry/repository.
You can find custom implementation for JobService and JMX enabled Job administrator in : https://github.com/regunathb/Trooper/tree/master/batch-core as: org.trpr.platform.batch.impl.spring.admin.SimpleJobService and org.trpr.platform.batch.impl.spring.jmx.JobAdministrator
Spring beans XML that uses these beans are here : https://github.com/regunathb/Trooper/blob/master/batch-core/src/main/resources/packaged/common-batch-config.xml