I created a simple Boot/Spring Batch 3.0.8.RELEASE job. I created a simple class that implements JobParametersIncrementer to go to the database, look up how many days the query should look for and puts those into the JobParameters object.
I need that value in my JdbcCursorItemReader, as it is selecting data based upon one of the looked up JobParameters, but I cannot figure this out via Java annotations. XML examples plenty, not so much for Java.
Below is my BatchConfiguration class that runs job.
`
#Autowired
SendJobParms jobParms; // this guy queries DB and puts data into JobParameters
#Bean
public Job job(#Qualifier("step1") Step step1, #Qualifier("step2") Step step2) {
return jobs.get("DW_Send").incrementer(jobParms).start(step1).next(step2).build();
}
#Bean
protected Step step2(ItemReader<McsendRequest> reader,
ItemWriter<McsendRequest> writer) {
return steps.get("step2")
.<McsendRequest, McsendRequest> chunk(5000)
.reader(reader)
.writer(writer)
.build();
}
#Bean
public JdbcCursorItemReader reader() {
JdbcCursorItemReader<McsendRequest> itemReader = new JdbcCursorItemReader<McsendRequest>();
itemReader.setDataSource(dataSource);
// want to get access to JobParameter here so I can pull values out for my sql query.
itemReader.setSql("select xxxx where rownum <= JobParameter.getCount()");
itemReader.setRowMapper(new McsendRequestMapper());
return itemReader;
}
`
Change reader definition as follows (example for parameter of type Long and name paramCount):
#Bean
#StepScope
public JdbcCursorItemReader reader(#Value("#{jobParameters[paramCount]}") Long paramCount) {
JdbcCursorItemReader<McsendRequest> itemReader = new JdbcCursorItemReader<McsendRequest>();
itemReader.setDataSource(dataSource);
itemReader.setSql("select xxxx where rownum <= ?");
ListPreparedStatementSetter listPreparedStatementSetter = new ListPreparedStatementSetter();
listPreparedStatementSetter.setParameters(Arrays.asList(paramCount));
itemReader.setPreparedStatementSetter(listPreparedStatementSetter);
itemReader.setRowMapper(new McsendRequestMapper());
return itemReader;
}
Related
We're creating a spring batch app that reads data from a database and writes in another database. In this process, we need to dynamically set the parameter to the SQL as we have parameters that demands data accordingly.
For this, We created a JdbcCursorItemReader Reader with #StepScope as I've found in other articles and tutorials. But was not successful. The chunk reader in our Job actually uses Peekable reader which internally uses the JdbcCursorItemReader object to perform the actual read operation.
When the job is triggered, we get the error - "jobParameters cannot be found on object of type BeanExpressionContext"
Please let me know what is that I am doing wrongly in the bean configuration below.
#Bean
#StepScope
#Scope(proxyMode = ScopedProxyMode.TARGET_CLASS)
public JdbcCursorItemReader<DTO> jdbcDataReader(#Value() String param) throws Exception {
JdbcCursorItemReader<DTO> databaseReader = new JdbcCursorItemReader<DTO>();
return databaseReader;
}
// This class extends PeekableReader, and sets JdbcReader (jdbcDataReader) as delegate
#Bean
public DataPeekReader getPeekReader() {
DataPeekReader peekReader = new DataPeekReader();
return peekReader;
}
// This is the reader that uses Peekable Item Reader (getPeekReader) and also specifies chunk completion policy.
#Bean
public DataReader getDataReader() {
DataReader dataReader = new DataReader();
return dataReader;
}
// This is the step builder.
#Bean
public Step readDataStep() throws Exception {
return stepBuilderFactory.get("readDataStep")
.<DTO, DTO>chunk(getDataReader())
.reader(getDataReader())
.writer(getWriter())
.build();
}
#Bean
public Job readReconDataJob() throws Exception {
return jobBuilderFactory.get("readDataJob")
.incrementer(new RunIdIncrementer())
.flow(readDataStep())
.end()
.build();
}
Please let me know what is that I am doing wrongly in the bean configuration below.
Your jdbcDataReader(#Value() String param) is incorrect. You need to specify a Spel expression in the #Value to specify which parameter to inject. Here is an example of how to pass a job parameter to a JdbcCursorItemReader:
#Bean
#StepScope
public JdbcCursorItemReader<DTO> jdbcCursorItemReader(#Value("#{jobParameters['table']}") String table) {
return new JdbcCursorItemReaderBuilder<DTO>()
.sql("select * from " + table)
// set other properties
.build();
}
You can find more details in the late binding section of the reference documentation.
This is my first code in SpringBatch using SpringBoot, I am implementing a sample usecase.
This is the exact pseudo-code i want to implement in SpringBatch, can you please help:
It's not feasible to fetch all hotel details(>3 million records) at one shot and process it, so i decided to fetch 1 hotel(50,000 records) at a time and process it and write to DB. Want to repeat this step for each and every hotelID as described below. Does this use-case suitable for SpringBatch ?
List<Integer> allHotelIDs = execute("select distinct(hotelid) from Hotels");
List items = new Arraylist();
allHotelIDs.forEach(hotelID -> {
Object item = itemReader.jdbcReader(hotelID, dataSource);
Object processedItem = itemProcessor.process(item);
items.add(processedItem);
});
itemWriter.write(items);
I am able to pass only 1 hotelid, how can i invoke it multiple times for all list hotels ?
#Bean
Job job(JobBuilderFactory jbf, StepBuilderFactory sbf, DBReaderWriter step1) throws Exception {
Step db2db = sbf.get("db-db").<Table, List<Tendency>>chunk(1000)
.reader(step1.jdbcReader(hotelID, dataSource))
.processor(processor())
.writer(receivableWriter()).build();
return jbf.get("etl").incrementer(new RunIdIncrementer()).start(db2db).build();
}
Reader code:
#Configuration
public class DBReader {
#Bean
public ItemReader<Table> jdbcReader(Integer hotelID, DataSource dataSource) {
return new JdbcCursorItemReaderBuilder<Table>().dataSource(dataSource).name("jdbc-reader")
.sql("SELECT * FROM Hotels where hotelid ="+hotelID).rowMapper((rs, i) -> {
return read().db(rs, "Hotels");
}).build();
}
}
Thanks.
After processing some XML files with Spring Batch ItemProcessor.
The ItemProcessor returns items like this:
MetsModsDef
{
int id;
String title;
String path;
Properties identifers;
....
}
now i need to save this items into a database, so that the
(id, title, path) will go into the "Work" table
and all the Properties stored in the "identifiers" field go into a "Key/Value"-Table called "Identifier" (work, identitytype, identityValue)
how can i acheive this?
currently i am using a CompositeItemWriter to split the object and write it into two tables like this:
public ItemWriter<MetsModsDef> MultiTableJdbcWriter(#Qualifier("dataSource") DataSource dataSource) {
CompositeItemWriter<MetsModsDef> cWriter = new CompositeItemWriter<MetsModsDef>();
JdbcBatchItemWriter hsqlWorkWriter = new JdbcBatchItemWriterBuilder()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO work (id, title, path,enabled) VALUES (:id, :title,:path,1)" )
.dataSource(dataSource)
.build();
JdbcBatchItemWriter hsqlIdentifierWriter = new JdbcBatchItemWriterBuilder()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO identity (work, identitytype, identityValue) VALUES (:work, :identitytype, :identityValue)" )
.dataSource(dataSource)
.build();
List<ItemWriter<? super MetsModsDef>> mWriter = new ArrayList<ItemWriter<? super MetsModsDef>>();
mWriter.add(hsqlWorkWriter);
mWriter.add(hsqlIdentifierWriter);
cWriter.setDelegates(mWriter);
but this will not work for a propertylist since (work, identitytype, identityValue) are not part of my domain object MetModsDef which only contains one map of properties which are supposed to go into the Identifier table.
i have found advice on how to do it when writing to a file,
and even on using a splitter pattern from Spring-Integration Read one record/item and write multiple records/items using spring batch
but i am still not sure how to actually do it, when writing out via jdbc or hibernate (which i assume would be similarish )
thanks for your advice !
in case somebody is interested: after a while i have come up with own solution:
I have found one extending HibernateItemWriter (for hibernate writes) on the internet:
Spring-Batch Multi-line record Item Writer with variable number of lines per record
but i did not want to extend classes, so i had to come up with my own (based on what i could research on the internet).
I am not sure how good is, and how it will handle transactions or rollback (probably bad). but for now it is the only one i have. So if you need one too, or have comments on how to improve it. or even have a better one. You are very welcome.
i have created my own IdentifierListWriter which creates the Key/value-pairs-like-objects (here each pair is called "identitifier") for each MetsModsDef Item and writes out them all using JdbcBatchItemWriter identifierWriter wich is passed to it from the configuration
public class IdentifierListWriter implements ItemWriter<MetsModsDef>
{
private ItemWriter<Identifier> _identifierWriter;
public IdentifierListWriter ( JdbcBatchItemWriter<Identifier> identifierWriter )
{
_identifierWriter= identifierWriter;
}
#Transactional(readOnly = false, propagation = Propagation.REQUIRED)
public void write(List<? extends MetsModsDef> items) throws Exception
{
// Main Table WRITER
for ( MetsModsDef item : items )
{
ArrayList<Identifier> ids = new ArrayList<Identifier>();
for(String key : item.getAllIds().stringPropertyNames())
{
ids.add(new Identifier(item.getAllIds().getProperty(key),
key, item.getId()));
}
_identifierWriter.write(ids);
}
}
}
In the java configuration i create two jdbcBatchItemWriter Beans. One for the "Work" table and one for the "identifier" table. IdentifierListWriter bean and a CompositeItemWriter MultiTableJdbcWriter Bean which uses them all to write out the object
#Bean
#Primary
public ItemWriter<MetsModsDef> MultiTableJdbcWriter(#Qualifier("dataSource") DataSource dataSource) {
IdentifierListWriter identifierListWriter = new IdentifierListWriter(identifierWriter(dataSource) );
CompositeItemWriter cWriter = new CompositeItemWriter();
cWriter.setDelegates(Arrays.asList(hsqlWorkWriter(dataSource),identifierListWriter));
return cWriter;
}
#Bean
public JdbcBatchItemWriter<MetsModsDef> hsqlWorkWriter(#Qualifier("dataSource") DataSource dataSource) {
return new JdbcBatchItemWriterBuilder<MetsModsDef>()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO work (id, title, path,enabled) VALUES (:id, :title,:path,1)")
.dataSource(dataSource)
.build();
}
#Bean
public JdbcBatchItemWriter<Identifier> identifierWriter(#Qualifier("dataSource") DataSource dataSource) {
return new JdbcBatchItemWriterBuilder()
.itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
.sql("INSERT INTO identifier (identifier, type, work_id) VALUES ( :identifier, :type, :work)")
.dataSource(dataSource)
//.afterPropertiesSet()
.build();
}
then the multiTableJdbcWriter is called from a Step:
#Bean
public Step step1(ItemWriter<MetsModsDef> multiTableJdbcWriter) {
return stepBuilderFactory.get("step1")
.<StreamSource, MetsModsDef>chunk(1)
.reader(new MetsModsReader())
.processor(metsModsFileProcessor())
.writer(multiTableJdbcWriter)
I am reading from Oracle and writing to a CSV file. I have one step which reads and writes to the CSV file. I implemented a ChunkListener so I know how many records were written.
I want to be able to write a file trailer showing the number of records written to my file. I implemented FlatFileFooterCallback but cannot figure out how to get the data from StepExecution (the "readCount") to my FlatFileFooterCallback.
I guess I am struggling with how to get access to Job, Step scopes in my write.
Any examples, or links would be helpful. I am using [Spring Batch / Boot] so I am all annotated. I can find xml examples, so maybe this annotated stuff is more complicated.
ItemWriter<Object> databaseCsvItemWriter() {
FlatFileItemWriter<Object> csvFileWriter = new FlatFileItemWriter<>();
String exportFileHeader = "one,two,three";
StringHeaderWriter headerWriter = new StringHeaderWriter(exportFileHeader);
csvFileWriter.setHeaderCallback(headerWriter);
String exportFilePath = "/tmp/students.csv";
csvFileWriter.setResource(new FileSystemResource(exportFilePath));
LineAggregator<McsendRequest> lineAggregator = createRequestLineAggregator();
csvFileWriter.setLineAggregator(lineAggregator);
csvFileWriter.setFooterCallback(headerWriter);
return csvFileWriter;
}
You can implement CustomFooterCallback as follows:
public class CustomFooterCallback implements FlatFileFooterCallback {
#Value("#{StepExecution}")
private StepExecution stepExecution;
#Override
public void writeFooter(Writer writer) throws IOException {
writer.write("footer - number of items read: " + stepExecution.getReadCount());
writer.write("footer - number of items written: " + stepExecution.getWriteCount());
}
}
Then in a #Configuration class:
#Bean
#StepScope
public FlatFileFooterCallback customFooterCallback() {
return new CustomFooterCallback();
}
And use in the Writer:
csvFileWriter.setFooterCallback(customFooterCallback());
This way, you have access to StepExecution in order to read data as needed.
I am writing a spring batch job. I am implementing custom writer using KafkaClientWriter extends AbstractItemStreamItemWriter<ProducerMessage>
I have fields which need to be unique for each instance. But I could see this class initiated only once. Rest jobs have same instance of writer class.
Where as my custom readers and processors are getting initiated for each job.
Below is my job configurations. How can I achieve the same behavior for writer as well?
#Bean
#Scope("job")
public ZipMultiResourceItemReader reader(#Value("#{jobParameters[fileName]}") String fileName, #Value("#{jobParameters[s3SourceFolderPrefix]}") String s3SourceFolderPrefix, #Value("#{jobParameters[timeStamp]}") long timeStamp, com.fastretailing.catalogPlatformSCMProducer.service.ConfigurationService confService) {
FlatFileItemReader faltFileReader = new FlatFileItemReader();
ZipMultiResourceItemReader zipReader = new ZipMultiResourceItemReader();
Resource[] resArray = new Resource[1];
resArray[0] = new FileSystemResource(new File(fileName));
zipReader.setArchives(resArray);
DefaultLineMapper<ProducerMessage> lineMapper = new DefaultLineMapper<ProducerMessage>();
lineMapper.setLineTokenizer(new DelimitedLineTokenizer());
CSVFieldMapper csvFieldMapper = new CSVFieldMapper(fileName, s3SourceFolderPrefix, timeStamp, confService);
lineMapper.setFieldSetMapper(csvFieldMapper);
faltFileReader.setLineMapper(lineMapper);
zipReader.setDelegate(faltFileReader);
return zipReader;
}
#Bean
#Scope("job")
public ItemProcessor<ProducerMessage, ProducerMessage> processor(#Value("#{jobParameters[timeStamp]}") long timeStamp) {
ProducerProcessor processor = new ProducerProcessor();
processor.setS3FileTimeStamp(timeStamp);
return processor;
}
#Bean
#ConfigurationProperties
public ItemWriter<ProducerMessage> writer() {
return new KafkaClientWriter();
}
#Bean
public Step step1(StepBuilderFactory stepBuilderFactory,
ItemReader reader, ItemWriter writer,
ItemProcessor processor, #Value("${reader.chunkSize}")
int chunkSize) {
LOGGER.info("Step configuration loaded with chunk size {}", chunkSize);
return stepBuilderFactory.get("step1")
.chunk(chunkSize).reader(reader)
.processor(processor).writer(writer)
.build();
}
#Bean
public StepScope stepScope() {
final StepScope stepScope = new StepScope();
stepScope.setAutoProxy(true);
return stepScope;
}
#Bean
public JobScope jobScope() {
final JobScope jobScope = new JobScope();
return jobScope;
}
#Bean
public Configuration configuration() {
return new Configuration();
}
I tried making the writer with job scope. But in that case open is not getting called. This is where I am doing some initializations.
When using java based configuration and a scoped proxy what happens is that the return type of the method is detected and for that a proxy is created. So when you return ItemWriter you will get a JDK proxy only implementing ItemWriter, whereas your open method is on the ItemStream interface. Because that interface isn't included on the proxy there is no way to call the method.
Either change the return type to KafkaClientWriter or ItemStreamWriter< ProducerMessage> (assuming the KafkaCLientWriter implements that method). Next add #Scope("job") and you should have your open method called again with a properly scoped writer.