I am looking for a nice way to export from a JDBC datasource in spring batch into a csv file the upload to an FTP and mark the rows as exported from the read table (a column exportdate exists).
I cannot find a transactional way to do it so that we actually mark the line as exported only if the file upload is successful.
You need a 2 step job.
First step: standard read (from JDCB), process (transform record from CSV entry) and regular write (using a FlatFileItemWriter).
Second step: upload file to FTP using Spring integration.
About column exportdate you can achive the goal to update the column from a ItemWriteListener.afterWrite because it is
called after ItemWriter.write(java.util.List). This will be called
before any transaction is committed, and before
ChunkListener.afterChunk(ChunkContext)
I hope this can help
Related
I am bit puzzled here, I need to do a task similar to the following scenario with Spring Batch
Read Person from repository ==> I can use RepositoryItemReader
(a) Generate CSV file (FlatFileItemWriter) and (b) save CSV file in DB with the generated date (I can use RepositoryItemWriter)
But here I am struggling to understand how I can give generated CSV file output of 2a to save in DB 2b.
Consider CSV File has approx 1000+ Person Data which are processed for a single day.
is it possible to merge 2a & 2b? I thought about CompositeItemWriter but as here we are combining 1000+ employee in CSV file so it won't work.
Using a CompositeItemWriter won't work as you will be trying to write an incomplete file to the database for each chunk..
I would not merge 2a and 2b. Make each step do one thing (and do it well):
Step 1 (chunk-oriented tasklet): read persons and generate the file
Step 2 (simple tasklet) : save the file in the database
Step 1 can use the job execution context to pass the name of the generated file to step 2. You can find an example in the Passing Data To Future Steps section. Moreover, with this setup, step 2 will not run if step 1 fails (which makes sense to me).
I read everywhere how to read data in spring batch itemReader and write in database using itemWriter, but I wanted to just read data using spring batch then somehow I wanted to access this list of items outside the job. I need to perform remaining processing after job finished.
The reason I wanted to do this is because I need to perform a lot of validations on every item. I have to validate each item's variable xyz if it exists in list(which is not available within job). After performing a lot of processing I have to insert information in different tables using JPA. Please help me out!
I need help, I want the code to read the data which is in a csv file and the store that data into database. I have tried reading the csv file with known rows and cols. But the challenge here is that I want to create an utility where I don't know the number of cols and rows that are in the csv file so how would I do it? Please help.
Have you explored Spring Batch? You can write your own implementation of LineTokenizer for the columns which are going to change dynamically.
I've never used Spring Batch before but it seems like a viable option for what I am attempting to accomplish. I have about 15 CSV files for 10 institutions that I need to process nightly. I am stashing the CSV into staging tables in an Oracle database.
The CSV File may look something like this.
DEPARTMENT_ID,DEPARTMENT_NAME,DEPARTMENT_CODE
100,Computer Science & Engineering,C5321
101,Math,M333
...
However when I process the row and add it to the database I need to fill in an institution id which would be determined based on the folder being processed at that time.
The database table would like like this
INSTITUTION_ID,DEPARTMENT_ID,DEPARTMENT_NAME,DEPARTMENT_CODE
1100,100,Computer Science & Engineering,C5321
There is also validation that needs to be done on each row in the CSV files as well. Is that something Spring Batch can handle as well?
I've seen reference to CustomItemReader and CustomItemWriter but not sure if that is what I need. The examples I've seen seem basic just dumping a CSV exactly as it is into a matching table.
Yes , all the task that you have reported can be done by spring batch -
For the Reader you may use - multi Resource Item Reader with your wild card name matching your - file names .
To validate the rows from file you can use item processor and handle the validation.
And for your case you need not use the custom item writer - you can configure the item writer as DB item writer in your XML file.
I suggest you to use the XML based approach for Spring batch implementation.
The XML will be used to configure all the architecture of your batch - as in
job -- step -- chunk -- reader -- processor -- writer
and to track errors and exceptions you can implement listeners at each stage.
-- step Execution Listener
-- Item Reader Listener
-- Item Processor Listener
-- Item Writer Listener
I have a requirement to write multiple files using Spring Batch. The first file will be written based on the data from the database table. The second file will contain just the number of records written to the first file. How can I create the second file? I am not sure whether org.springframework.batch.item.file.MultiResourceItemWriter is an option for me as I think it will write multiple files based on the data it will write chunks of data in the multiple files. Correct me if I am wrong here.
Please do suggest some options with sample code if possible.
You have couple of options:
You can use CompositeItemWriter which calls collection of item writers in defined order so you can define one item writer which will write records based on data from DB and second will count records and write that to another file.
You can write data to a file in first step, finish whole file and save it somewhere, you can save counter of records if that is all you need to StepContext (common batch patterns and scroll to 11.8 Passing Data to Future Steps) and read in new Taskletcounter and save to new file.
If you want to go with option 1 which I think is right choice you can check this example of batch job configuration with CompositeItemWriter