I need to create a spring batch wherein i need to fetch data from 2 different tables.
This will be the reader part.
The problem is i need to have 2 reader queries. And only one of them will be called based on the condition.
But the writer will be same.
So tasklet becomes same basically.
Can i have 2 readers inside one single step which will be called according to the condition..??
Something like inside the spring batch xml:
if (condition)
then reader1
else reader2
...
....
......
<reader1 id=".." class="..">
</reader1>
.....
........
<reader2 id=".." class="..">
</reader2>
Have you considered using the conditional control flow patterns Spring Batch offers? I think it could lead to simpler construction and adhere to some of the core patterns spring batch encourages.
You basically program the "condition" that you want a step to be called under. So define two steps, say step1 which has a reader, processor, writer for one object type and step2 which has a reader, processor, writer for the other type. Perhaps the writers are shared between types as well. You then can define control flow like so:
#Bean
public Job myJob() {
JobBuilder jobBuilder = jobs.get("Spring Batch: conditional steps");
return jobBuilder.start(determineRoute())
.on("item1").to(flow1())
.on("item2").to(flow2()).end()
.build();
}
In the example determineRoute() is a tasklet that returns custom ExitStatus values item1 or item2 & flow1 and flow2 are different flows (or steps) to handle each object.
See here in their docs: https://docs.spring.io/spring-batch/docs/current/reference/html/step.html#conditionalFlow
Edit: you can also do something similar with a JobExecutionDecider https://www.baeldung.com/spring-batch-conditional-flow#2-programmatic-branching-withjobexecutiondecider
Related
I am new to Spring Batch and I would like to seek the advice of the experienced programmer on how to implement the following:
I am writing an API that triggers a batch task to generate a report from a list of transactions from JPA repo.
In spring batch, there is ItemReader, ItemProcessor, ItemWriter for each step.
How should I implement my ItemReader such that it can execute the following:
Generate a custom query from the parameters set in the jobParameters (obtained from the API query)
Using the custom query in 2 to obtain the list of transactions from the Transactions JPA repo, which will be processed in the ItemProcessor and subsequently generate reports through ItemWriter.
Question: How should I write the ItemReader, I looking at JpaCursorItemReader (dk if it is the right one) but could not find examples of implementation online to refer to. Any help is appreciated. Thank you.
I am at the stage of trying to understand how Spring batch works, I hope to seek proper guidance from experts in this field of the direction to go to accomplished the mentioned task.
Generate a custom query from the parameters set in the jobParameters (obtained from the API query)
You can define a step-scoped bean and inject job parameters to use them in your query:
#Bean
#StepScope
public JpaCursorItemReader jpaCursorItemReader(#Value("#{jobParameters['parameter']}") String parameter) {
return new JpaCursorItemReaderBuilder()
.queryString("..") // use job parameter in your query here
// set other properties on the item reader
.build();
}
You can find more details on step-scoped components in the documentation here: Late Binding of Job and Step Attributes.
I am new to the spring batch and I have an application where for example an xml-file of flights is read and saved to a database. The whole process is already working, but due to a certain use case I also need to access the data inside the reader object (ItemReader), should it is possible.
Down there is the reader-method. It is not about this method in particular, but as mentioned it is about ItemReader.
#Bean
public StaxEventItemReader<Flight> flightReader() {
StaxEventItemReader<Flight> reader = new StaxEventItemReader<Obj>();
reader.setResource(ressource);
reader.setFragmentRootElementName("flight");
Map<String, String> aliases = new HashMap<String, String>();
aliases.put("flight", Flight.class);
XStreamMarshaller xStreamMarshaller = new XStreamMarshaller();
xStreamMarshaller.setAliases(aliases);
reader.setUnmarshaller(xStreamMarshaller);
return reader;
}
How can I access the flight objects inside the reader (StaxEventItemReader) object?
I actually tried to use the read() method (Spring doc ItemReader), but I am always getting NullPointerExceptions.
If the read() method is the correct way, how can you access the flight objects inside ItemReader correctly?
If not, are there other ways?
There is more than one way to access the items. It really depends on what you want to do with them:
If you only want to have a look without manipulating the items, you can implement an ItemReadListener with its afterRead method, and add the listener to your step.
The items are passed to the processor. So you can operate on them there.
You can extend the class StaxEventItemReader and override the read method to include additional logic.
If you prefer composition over inheritance, you can write a new reader that uses a StaxEventItemReader as a delegate.
I want to make specific part of flow as transactional. For instance, I want to make the first two transform operation in one transactional block. Here is the flow code that I use:
#Bean
public IntegrationFlow createNumberRange() {
return IntegrationFlows.from("npEventPubSubChannel")
.transform(...)
.transform(...)// should be transactional with above transform together
.transform(...) // non transactional
.handle((payload, headers) -> numbRepository.saveAll(payload))
.get();
}
I found a workaround as adding another handle and directing flow to transactional gateway like this one:
.handle("transactionalBean", "transactionalMetod") //Then implemented messagingGateway which consists of transactional method.
I also found mid flow transactional support but couldn't find an example to work on.
Is there an elegant solution rather than directing to another gateway in the middle of the flow?
If you want to wrap two transformers into the transaction, you don't have choice unless hide that call behind transactional gateway. That is fully similar when you do raw Java:
#Transactional
void myTransactionalMethod() {
transform1();
transform2();
}
I'm sure you are agree with me that we always have to do this way to have them both in the same transaction.
With Spring Integration Java DSL you can do this though:
.gateway(f -> f
.transform(...)
.transform(...),
e -> e.transactional())
Do you agree it is similar to what we have with the raw Java and not so bad from the elegance perspective?
I have a Spring Batch job that reads in a very large fixed length file and maps it just fine to an object. Validated all of the data in the associated processing task.
Being rather new to Spring and Spring Batch I am wondering if it is possible to get out of the job, a fully populated object to be used in a particular case when I am running the job as part of another process ( that I would like to have access to the data).
I realize that I could do the above without Batch, and it seems to be designed with scope limitations for its purpose.
I could serialize the objects in the processor and go that route but for my immediate satisfaction I am hoping there is a way to get around this.
Thanks
In my #configuration class for the batch processing, I created a class variable (it is a list of the object I want to get back) and instantiated with the no arg constructor.
My Step, ItemReader, LineMapper are setup to use a list for input. The custom FieldSetMapper takes that list instantiated from the constructor as a parameter and adds to the list as the file is read and mapped. Similarly my custom ItemProcessor takes the list as input and returns it.
Finally I created a ReturnObjectList bean that returns the populated list.
In my main I cast the AnnotationConfigApplicationContext getbean to the list of that object type. I am now able to use the list of objects generated from the fixed file in the scope of my main application.
Not sure if this is a healthy work around in terms of how Spring Java config is supposed to work, but it does give me what I need.
How do i pass values which are fetched from database in one job as parameters to another job in Spring framework?PLese provide a example code.
I'm guessing by jobs you mean scheduled-tasks. I was in the same situation where 2 jobs were dependent on one another eg:
<task:scheduled-tasks>
<task:scheduled ref="deleteExpiredAccountsJob" method="launch" fixed-delay="100000" />
</task:scheduled-tasks>
<task:scheduled-tasks>
<task:scheduled ref="emailDeletedAccountsConfirmationJob" method="launch" fixed-delay="100000" />
</task:scheduled-tasks>
What I did was to set a DELETE flag on the ACCOUNTS table to true and have the emailDeletedAccountsConfirmationJob read only the ACCOUNTS with DELETE = true
Another solution, if you want to share data between ItemReaders would be to be to have a Java Spring configuration class "#Configuration" and declare your ItemReaders in this class with #Bean annotation and also have a private synchronized Map or List which will be shared between all the threads/jobs or your spring batch steps and you access the synchronized list in your readers.
#Configuration
public class MySpringConfig {
private List<String> list = Collections.synchronizedList(new ArrayList<String>());
#Bean
#Scope("step")
public JdbcCursorItemReader readerOne(){
//add data to list
}
#Bean
#Scope("step")
public JdbcCursorItemReader readerTwo(){
//check for certain data in the list, if exists ... do something
}
in Spring Batch Jobs are considered distinct use cases with no conceptual relationship imposed by the framework. you could see a job as a standalone processing application.
a typical batch system consists of data sources and data sinks such as a message queue, database or file system, so one job is producing data, that is intended to be process by another part of your application.
you may rework your design to use tasks instead of jobs. this is useful if you have several interdependent process steps. there is a way to access the JobExcecution from StepExecution.
I recommend reading the excellent articles as well as "Spring Batch in Action".