I am using Spring Batch ItemReader to read data from a database and then using FlatFileItemWriter to write the data to a file.
Once the data is written to a file, I need this specific file to be transferred to an FTP server. Can I do this via Spring Batch or should I use Spring Integration? Could you please also provide an example?
There is nothing wrong in combining both frameworks. They really work together smoothly.
Since you say that you already have a Spring Batch ItemReader and your OK writing into a file using FlatFileItemWriter, so you are have way out for your whole solution.
In the end you can use a Providing Feedback with Informational Messages to get a file and send it to FTP using Spring Integration FTP Outbound Channel adapter.
See more info in Spring Batch Docs: https://docs.spring.io/spring-batch/4.2.x/reference/html/spring-batch-integration.html#providing-feedback-with-informational-messages
And in Spring Integration about FTP support: https://docs.spring.io/spring-integration/docs/current/reference/html/#ftp
Related
The use case is that I need to poll a folder on a remote server, and if a new file is copied to that folder, I need to call a REST get API with the file contents. The REST API will process the file contents using Spring batch.
I am trying to use Spring boot integration for that purpose but having issues finding my way. Is Spring Integration best suited for this purpose? If yes, can I have a simple example of just the Spring Integration picking up the file and calling the REST API?
Or can I simply use the Java Watcher service?
Not clear what is your remote folder, but you can't use Java WatchService for that purpose anyway. Typically the remote directory is on an (S)FTP. Spring Integration provides channel adapters to poll such a remote directory under the mentioned protocol.
See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/ftp.html#ftp-inbound
You probably don't need to have a local copy of a remote service, then you can consider to use a streaming channel adapter instead: https://docs.spring.io/spring-integration/docs/current/reference/html/ftp.html#ftp-streaming
As far as a file content is emitted into a channel configured in that channel adapter, you can use an HTTP Outbound Channel Adapter to call some REST API: https://docs.spring.io/spring-integration/docs/current/reference/html/http.html#http-outbound
You can investigate samples project for some inspiration: https://github.com/spring-projects/spring-integration-samples
I need to develop an application to perform the following activities using Spring Batch. Read data from the database --> Process data and prepare rest API request --> Write or Post into third party restful service.
I have seen a lot of examples for reading from the database to write to CSV, DB, JMS.
But I don't find any options to write into web service. Is it possible to perform this activity using spring batch or please suggest some other technology.
Spring Batch does not have a RestItemReader since the possibilities are a bit too great as to what we'd need to send. That being said, writing your own ItemWriter implementation that uses a RestTemplate to send the data would be very straight forward to do.
I am working on a spring batch project where use case is:
Spring batch job relies on the file present at SFTP server( remote directory)
So, Once the file gets available in SFTP server, a corresponding job (spring batch) should start. Moreover, I don't want to first start the job and then look for the file because that would be time based approach rather I am looking for notification based approach. So I thought to use spring integration (sftp inbound channel adaptor).
As part of Inbound adaptor (SFTP) once I get/receive/download new file (based on timestamp) present at remote directory of SFTP server, I want to notify spring batch application and then trigger a Job. For e.g if there are two files at remote directory say 1.txt, 2.txt. These files come on daily basis. So once my channel receives new file at remote directory, it should
download1.txt and then notify spring batch application to launch Spring batch Job1
download2.txt and then notify spring batch application to launch Spring batch Job2
Can anyone validate this approach and guide further.
See the spring-batch-integration sub-project in Spring Batch; in particular the JobLaunchingGateway and JobLaunchingMessageHandler.
thanks for attention,i defined a combine spring batch and spring integration project and communicate with ftp server to retrieve file and process on it and write on ftp, i am looking for a good architecture for my project, i designed an architecture with spring integration as bellow diagram:
when retrieve file from server process on it and route files based on condition to mvChannel and toGet channel, i have many process scenario on the retrieved file from server that i defined a router that router handle the scenario , and route to job channels and run spring batch
now, my is question is that are right the architecture?
Really looks good. It is is typical pattern to get gain from both Spring Integration and Spring Batch worlds integration them.
However I don't see reason in the last <aggregator>. All your jobs can send their result to the <int-ftp:outbound-gateway command="PUT">. Looks like no need to wait for all results to do some analysis on them in the single place.
My current project is based on Spring Batch and Spring integration.
My goal is using Spring Batch to execute job flow steps:
read file from SFTP server(step1).
decrypt the file(step2).
save the file into database(step3).
I want to divide this into steps queue(read from sftp, decrypt, write to db).
And I also need to save the file, transfer start time, transfer end time, file size into DB.
Some days ago, I used Spring integration to poll file from sftp server and then send it to Spring batch to do the extra job. Also I can not get the transfer start time.
But now my project is main based on Spring batch, not Spring Integration.
Can you give me some suggestions ?
Or, can you show me how to use Spring Batch to drive Spring Integration for reading SFTP files.
And how can I get the transfer start time ?
Thanks. :)
Add a tasklet as the first step; have the tasklet invoke a Spring Integration flow, probably using an sftp outbound gateway to GET the file.
There's an example of how to use the gateway in the sftp sample.
The 'transfer start time' would be the time you invoke the gateway.