I am bit puzzled here, I need to do a task similar to the following scenario with Spring Batch
Read Person from repository ==> I can use RepositoryItemReader
(a) Generate CSV file (FlatFileItemWriter) and (b) save CSV file in DB with the generated date (I can use RepositoryItemWriter)
But here I am struggling to understand how I can give generated CSV file output of 2a to save in DB 2b.
Consider CSV File has approx 1000+ Person Data which are processed for a single day.
is it possible to merge 2a & 2b? I thought about CompositeItemWriter but as here we are combining 1000+ employee in CSV file so it won't work.
Using a CompositeItemWriter won't work as you will be trying to write an incomplete file to the database for each chunk..
I would not merge 2a and 2b. Make each step do one thing (and do it well):
Step 1 (chunk-oriented tasklet): read persons and generate the file
Step 2 (simple tasklet) : save the file in the database
Step 1 can use the job execution context to pass the name of the generated file to step 2. You can find an example in the Passing Data To Future Steps section. Moreover, with this setup, step 2 will not run if step 1 fails (which makes sense to me).
Related
I have a requirement where i will take input CSV files from one folder, process them (DB lookup and validation) one after one and generate new output file for each input file. I need to choose the input files at RUN TIME based on the DB query on user object which will tell what are the qualified files (like out of 400 files in folder - 350 may be qualified to process and I need to generate 350 output files). I want to use SpringBatch and want to create one JOB for one File. Any reference or sample code to create JOBs at RUN TIME and Execute them?
We have a constrain in our application, For test data providing in JMeter execution (using CSV Data Set Config element) we are not supposed to provide duplicate test data and it won't accept in all the fields. So we kept unique test data (upto 8K data for 8k concurrent users) for all the fields in CSV format.
Here I have a manual intervention, After each test execution (i.e) 100 users, 1000 users up to 8000 users) we need to delete each row (WRT to users in thread group) in CSV file else the duplicate data will be fetching for next execution and result will be failed.
Here my questions is,
1. How Can i avoid repeated/duplicate test data or to avoid already executed data without deleting in CSV file.
2. During JMeter test execution with CSV files, How can we specify to take the data from the respective rows. For example 101th row, 1001th row & 7999th row (which contains 8000 rows of data)?
The easiest option will be using HTTP Simple Table Server, its READ command has KEEP=FALSE attribute so you will be able to feed your test with the unique data without having to physically remove it from the original CSV file.
You can install HTTP Simple Table Server plugin using JMeter Plugins Manager:
In general if your test doesn't need to be repeatable instead of keeping the data in the CSV file you can consider generating it on the fly using such JMeter Functions as:
__Random()
__RandomString()
__RandomDate()
__UUID()
etc.
Hi I have a requirement which I need to develop in Informatica.
The requirement is
1)Determine deltas between new extract and data extracted by previous run
2) Generate three separate CSV feed files based on the deltas
Could you please let me know the process of how to do this delta thing and compare the data from previous run and the new run
And how to write these delta data into an automated .csv file which need to be created automatically by informatica for every run .
Instead of writing the data into target table,it should write the data into these automated .csv or .txt files.
Does Informatica creates .csv or .txt files automatically and saves the data in them for every informatica run?if so, could you please let me know how?
Information you are seeking is widely available on the Internet and can be found with little research. However, let me try to chip in,
If the structure of the file remains the same between two runs, create two staging tables, one for previous and one for current. Do a minus between the two to capture the delta. Move the current to previous after delta capture and truncate current every time you load into it.
Use a flat file target instead of a table.
I have a requirement to write multiple files using Spring Batch. The first file will be written based on the data from the database table. The second file will contain just the number of records written to the first file. How can I create the second file? I am not sure whether org.springframework.batch.item.file.MultiResourceItemWriter is an option for me as I think it will write multiple files based on the data it will write chunks of data in the multiple files. Correct me if I am wrong here.
Please do suggest some options with sample code if possible.
You have couple of options:
You can use CompositeItemWriter which calls collection of item writers in defined order so you can define one item writer which will write records based on data from DB and second will count records and write that to another file.
You can write data to a file in first step, finish whole file and save it somewhere, you can save counter of records if that is all you need to StepContext (common batch patterns and scroll to 11.8 Passing Data to Future Steps) and read in new Taskletcounter and save to new file.
If you want to go with option 1 which I think is right choice you can check this example of batch job configuration with CompositeItemWriter
I am looking for a nice way to export from a JDBC datasource in spring batch into a csv file the upload to an FTP and mark the rows as exported from the read table (a column exportdate exists).
I cannot find a transactional way to do it so that we actually mark the line as exported only if the file upload is successful.
You need a 2 step job.
First step: standard read (from JDCB), process (transform record from CSV entry) and regular write (using a FlatFileItemWriter).
Second step: upload file to FTP using Spring integration.
About column exportdate you can achive the goal to update the column from a ItemWriteListener.afterWrite because it is
called after ItemWriter.write(java.util.List). This will be called
before any transaction is committed, and before
ChunkListener.afterChunk(ChunkContext)
I hope this can help