I have a requirement in which a tasklet, stores all the files in the directories in an arraylist. The size of the list is stored in the job execution context. Later this count is accessed from another tasklet in another step. How do it do this. I tried to store in jobexecution context, at runtime throws unmodifiable collection exception,
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
StepContext stepContext = arg1.getStepContext();
StepExecution stepExecution = stepContext.getStepExecution();
JobExecution jobExecution = stepExecution.getJobExecution();
ExecutionContext jobContext = jobExecution.getExecutionContext();
jobContext.put("FILE_COUNT",150000);
also stored the stepexection reference in beforestep annotation .still not possioble.kindly let me know ,how to share data between two tasklets.
you have at least 4 possibilities:
use the ExecutionPromotionListener to pass data to future steps
use a (spring) bean to hold inter-step data, e.g. a ConcurrentHashMap
without further action this data won't be accessible for a re-start
access the JobExecutionContext in your tasklet, should be used with caution, will cause thread problems for parallel steps
use the new jobscope (introduced with spring batch 3)
Code Example for accessing JobExecution from Tasklet:
setting a value
public class ChangingJobExecutionContextTasklet implements Tasklet {
/** {#inheritDoc} */
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
// set variable in JobExecutionContext
chunkContext
.getStepContext()
.getStepExecution()
.getJobExecution()
.getExecutionContext()
.put("value", "foo");
// exit the step
return RepeatStatus.FINISHED;
}
}
extracting a value
public class ReadingJobExecutionContextTasklet implements Tasklet {
private static final Logger LOG = LoggerFactory.getLogger(ChangingJobExecutionContextTasklet.class);
/** {#inheritDoc} */
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
// pull variable from JobExecutionContext
String value = (String) chunkContext
.getStepContext()
.getStepExecution()
.getJobExecution()
.getExecutionContext()
.get("value");
LOG.debug("Found value in JobExecutionContext:" + value);
// exit the step
return RepeatStatus.FINISHED;
}
}
i created code examples for the first 3 solutions in my spring-batch-examples github repository, see module complex and package interstepcommunication
Another way is to use StepExecutionListener which is called after step execution.
Your tasklet can implements it and share local attribute.
public class ReadingJobExecutionContextTasklet implements Tasklet, StepExecutionListener {
private String value;
#Override
public ExitStatus afterStep(StepExecution stepExecution) {
ExecutionContext jobExecutionContext = stepExecution.getJobExecution().getExecutionContext();
jobExecutionContext.put("key", value);
//Return null to leave the old value unchanged.
return null;
}
}
So, in the step, your bean is a tasklet and a listener like bellow.
You should also configure the scope of you step to "step" :
<batch:step id="myStep" next="importFileStep">
<batch:tasklet>
<ref bean="myTasklet"/>
<batch:listeners>
<batch:listener ref="myTasklet"/>
</batch:listeners>
</batch:tasklet>
</batch:step>
<bean id="myTasklet" class="ReadingJobExecutionContextTasklet" scope="step">
Related
I have following Apache camel rest service(/sales) that internally calls another rest service(/getOrders) and get list of objects. Am able to print JSON response in the processor but getting java objects in response while trying from postman. Could anyone pls help me to resolve the issue. Attaching the response log for ref..
#Component
public class ApplicationResource extends RouteBuilder {
#Autowired
private OrderService service;
#BeanInject
private OrderProcessor processor;
#Override
public void configure() throws Exception {
restConfiguration().component("servlet").port(9090).host("localhost");
rest().get("/getOrders").produces(MediaType.APPLICATION_JSON_VALUE).route().setBody(() -> service.getOrders());
rest().get("/sales").produces(MediaType.APPLICATION_JSON_VALUE).route()
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.toD("http://localhost:9090/getOrders?bridgeEndpoint=true").convertBodyTo(String.class).marshal()
.json(JsonLibrary.Jackson, Order.class).to("log:foo?showHeaders=true");;
;
}
}
You should remove the last .endRest() on "direct:bye" route.
I think you get the rest response before calling your Processor.
This works for me.
First, I needed to set the bindingMode as RestBindingMode.json in the restConfiguration.
Secondly, instead of marshal(), you need to use unmarshal().
Third, since you are returning a list of orders, .json(JsonLibrary.Jackson, Order.class) will not be sufficient to unmarshal the list of orders. You need to use a custom format which will be able to unmarshal the list of orders into a json array. This you need to do using JacksonDataFormat format = new ListJacksonDataFormat(Order.class);
#Override
public void configure() {
JacksonDataFormat format = new ListJacksonDataFormat(Order.class);
restConfiguration().component("servlet").port(9090).host(localhost).bindingMode(RestBindingMode.json);
rest()
.get("/getOrders")
.produces(MediaType.APPLICATION_JSON_VALUE)
.route()
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
exchange.getMessage().setBody(service.getOrders());
}})
.to("log:getOrders?showHeaders=true&showBody=true");
rest()
.get("/sales")
.produces(MediaType.APPLICATION_JSON_VALUE)
.route()
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.toD("http://localhost:9090/getOrders?bridgeEndpoint=true")
.unmarshal(format)
.to("log:sales?showHeaders=true&showBody=true");
}
Solvedddd !!! i did two things as follows,May be use full for some one
1,bindingMode(RestBindingMode.auto) - RestBindingMode changes to auto
from json
2, Added this in the main
service(/getOrders).marshal().json(JsonLibrary.Jackson);
#Component
public class ApplicationResource extends RouteBuilder {
#Autowired
private OrderService service;
#BeanInject
private OrderProcessor processor;
#Override
public void configure() throws Exception {
restConfiguration().component("servlet").port(9090).host("localhost").bindingMode(RestBindingMode.auto);
rest().get("/getOrders").produces(MediaType.APPLICATION_JSON_VALUE).route().setBody(() -> service.getOrders())
.marshal().json(JsonLibrary.Jackson);
rest().get("/sales").produces(MediaType.APPLICATION_JSON_VALUE).route()
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.toD("http://localhost:9090/getOrders?bridgeEndpoint=true").convertBodyTo(String.class)
.log("body = ${body}");
;
;
}
}
When i toogle a breakpoint beside the repository.saveAndFlush and during de debug mode i see that it returne a new client objet with new Id but when i check in the data base i do not find that client. However, if i do a resume (F8 with eclipse) then i re-check the DB i find the client.
So How to force persisting data during execution of camel processors rather than the end of the route?
#Component
public class myRoute extends RouteBuilder {
#Autowired Processor validationDatasProcessor;
#Autowired Processor clientProcessor;
#Autowired Processor endCientProcessor;
#Override
public void configure() throws Exception {
from("queueIn")
.id("route_processing").messageHistory().transacted()
.log(LoggingLevel.DEBUG, log, "reception").pipeline()
.process(validationDatasProcessor)
.id(validationDatasProcessor.getClass().getSimpleName().toLowerCase())
.process(clientProcessor)
.id(clientProcessor.getClass().getSimpleName().toLowerCase())
.process(endCientProcessor).id(endCientProcessor.getClass().getSimpleName().toLowerCase())
.to("outputQueue")
.end();
}
}
Processors:
#Component
public class ValidationDatasProcessor implements Processor {
#Autowired ObjectMapper objectMapper;
#Autowired ClientRepository clientRepository;
#Override
public void process(Exchange exchange) throws Exception {
String clientString = exchange.getIn().getBody(String.class);
Client client = objectMapper.readValue(clientString, Client.class);
clientRepository.saveAndFlush(client)
exchange.setOut(generateOutMessage(client, exchange.getContext()));
}
Message generateOutMessage(Client client, CamelContext camelContext) throws JsonProcessingException {
DefaultMessage outMessage = new DefaultMessage(camelContext);
outMessage.setBody(objectMapper.writeValueAsString(client), String.class);
return outMessage;
}
}
#Component
public class ClientProcessor implements Processor {
#Autowired ObjectMapper objectMapper;
#Autowired ClientRepository clientRepository;
....
#Override
public void process(Exchange exchange) throws Exception {
String clientString = exchange.getIn().getBody(String.class);
Client client = objectMapper.readValue(clientString, Client.class);
client.setAccessDate(LocalDateTime.now);
clientRepository.saveAndFlush(client)
exchange.setOut(generateOutMessage(client, exchange.getContext()));
}
Message generateOutMessage(Client client, CamelContext camelContext) throws JsonProcessingException {
DefaultMessage outMessage = new DefaultMessage(camelContext);
outMessage.setBody(objectMapper.writeValueAsString(client), String.class);
return outMessage;
}
}
Your entire route is a transacted; which means the whole route is under a transaction scope. A commit will be done only after the whole route is executed.
if you want to execute a processor outside the transaction boundary, split the route and use a seda endpoint. seda are asynchronous and start new threads. They won't participate in the active transaction boundary.
Commiting parts in the middle of a transaction scope doesn't sound like a great idea. Perhaps your rounte needs to be split into multiple fragments.
This documentation might help you understand them better.
I have created batch application which do chunk processing. I am creating chunks using Completion Policy.
Following is my batch configuration, (keeping code minimal, please let me know if need other information)
#Bean
public Job myJob() {
ItemReader itemReader = itemReader();
return jobBuilder.get("job").start(myStep(itemReader, completionPolicyReader(itemReader), writer(), processor()));
}
#Bean
public Step myStep(ItemReader itemReader, MyCompletionPolicy completionPolicyReader, ItemWriter writer, ItemProcessor processor) {
return stepBuilder.get("step").chunk(completionPolicyReader).reader(completionPolicyReader).processor(processor).writer(writer).listener(itemReader).build(); // registered delegated itemReader to listener.
}
#Bean
public MyCompletionPolicy completionPolicyReader(ItemReader itemReader) {
MyCompletionPolicy obj = new MyCompletionPolicy();
obj.setDelegate(itemReader);
return obj;
}
#Bean
public ItemReader itemReader() {
abc === xyz ? new AReader() : new BReader();
}
// other config
Following is my MyCompletionPolicy which delegates to actual ItemReader ie either AReader or BReader depending on some condition.
class MyCompletionPolicy extends
CompletionPolicySupport implements ItemReader<MyModel>, StepExecutionListener {
public void setDelegate(ItemReader<MyModel> itemReader) {
this.itemReader = itemReader;
this.delegate = new SingleItemPeekableItemReader<MyModel>();
this.delegate.setDelegate(itemReader);
}
#Override
public MyModel read() {
currentReadItem = delegate.read(); // Here I am delegating to actual reader (ex AReader) where I cannot get `StepExecution`
return currentReadItem;
}
.... // Other overridden methods
}
Following is my AReader where I am not able to get StepExecution
class AReader implements ItemReader<MyModel>, StepExecutionListener {
#Override
public void beforeStep(StepExecution stepExecution) {
// stepExecution is NULL
}
.... // other overridden methods
}
How I can get stepExecution in my delegated ItemReader ie in AReader.
======EDIT=====
Sub question regarding best practices. If I want to increment count between chunks i.e for example between multiple calls of ItemReader and use current value of counter in ItemReader. Is it good practice to Create class field in ItemReader class or should I store it in ExecutionContext ?
Considering SingleThread App
Considering MultiThread App
By default, Spring Batch will automatically register your reader/processor/writer as listeners if they implement StepExecutionListener. In your case, the reader is MyCompletionPolicy which implements StepExecutionListener and will be registered as a listener automatically.
However, Spring Batch is not aware that your MyCompletionPolicy delegates to another reader, so you need to explicitly register your delegate as a listener in the step.
To all Spring enthusiasts, here's a good challenge. Hope someone can crack it!!!
I use Spring to batch the extraction process. I have two classes 'ExtractBatchJob' and 'TaskletImpl'
public class ExtractBatchJob {
/** Logger for current class */
private static Log log = LogFactory.getLog(Extractor.class);
public static void main(String[] args)
throws IOrganizationServiceRetrieveMultipleOrganizationServiceFaultFaultFaultMessage,
IOException {
ApplicationContext context = new ClassPathXmlApplicationContext(
"/META-INF/cxf/batch-context.xml");
SpringBusFactory factory = new SpringBusFactory(context);
Bus bus = factory.createBus();
SpringBusFactory.setDefaultBus(bus);
IOrganizationService service = (IOrganizationService) factory
.getApplicationContext().getBean("service");
JobLauncher jobLauncher = (JobLauncher)context.getBean("jobLauncher");
Job job = (Job) context.getBean("firstBatchJob");
try {
JobExecution execution = jobLauncher.run(job, new JobParameters());
}catch (Exception e){
e.printStackTrace();
}
}
The second class TaskletImpl implements the Spring Tasklet interface.
public class TaskletImpl implements Tasklet {
/** Logger for current class */
private static Log log = LogFactory.getLog(CRMExtractor.class);
/* (non-Javadoc)
* #see org.springframework.batch.core.step.tasklet.Tasklet#execute(org.springframework.batch.core.StepContribution, org.springframework.batch.core.scope.context.ChunkContext)
*/
#Overridepublic RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
// TODO Auto-generated method stub
log.info("************ CRM Extraction Batch Job is executing!!! *******");
//QUESTION: To Extract Entity from third party
// web service need object reference for
//factory and service from ExtractBatchjob class
List<Entity> orderEntities = getEntities("orderQueryImpl", factory, service);
OrderDao orderDao = (OrderDao) factory.getApplicationContext()
.getBean("orderDao");
orderDao.batchInsert(orderEntities);*/
return RepeatStatus.FINISHED;
}
public static List<Entity> getEntities(String queryImpl, SpringBusFactory factory,
IOrganizationService service)
throws IOrganizationServiceRetrieveMultipleOrganizationServiceFaultFaultFaultMessage,
IOException {
QueryBuilder queryBuilder = (QueryBuilderTemplate) factory
.getApplicationContext().getBean(queryImpl);
QueryExpression queryExpr = queryBuilder.getQuery();
EntityCollection result = service
.retrieveMultiple(queryExpr);
return result.getEntities().getEntities();
}
}
Below is the snippet of the context file
`<import resource="cxf.xml" />
<bean id="firstBatch" class="com.abc.model.TaskletImpl" />
<batch:step id="firstBatchStepOne">
<batch:tasklet ref="firstBatch" />
</batch:step>
<batch:job id="firstBatchJob">
<batch:step id="stepOne" parent="firstBatchStepOne" />
</batch:job>`
My question is quite straightforward, how do I pass the two variables/objects 'service' and 'factory' to the TaskletImpl class from the ExtractBatchJob class.
The cleanest solution is to wire service and factory using Spring injection mechanism.
You have two solution:
Create SpringBusFactory as Spring bean and wire it into tasklet
Define a ContextBean (as singleton) for you job, create SpringBusFactory and set it as property of ContextBean; wire this bean to your tasklet
If you want to use object created outside Spring context (with new I meant) must be injected into Spring context.
Is it possible to access beans defined outside of the step scope? For example, if I define a strategy "strategyA" and pass it in the job parameters I would like the #Value to resolve to the strategyA bean. Is this possible? I am currently working round the problem by getting the bean manually from the applicationContext.
#Bean
#StepScope
public Tasklet myTasklet(
#Value("#{jobParameters['strategy']}") MyCustomClass myCustomStrategy)
MyTasklet myTasklet= new yTasklet();
myTasklet.setStrategy(myCustomStrategy);
return myTasklet;
}
I would like to have the ability to add more strategies without having to modify the code.
The sort answer is yes. This is more general spring/design pattern issue rater then Spring Batch.
The Spring Batch tricky parts are the configuration and understanding scope of bean creation.
Let’s assume all your Strategies implement Strategy interface that looks like:
interface Strategy {
int execute(int a, int b);
};
Every strategy should implements Strategy and use #Component annotation to allow automatic discovery of new Strategy. Make sure all new strategy will placed under the correct package so component scan will find them.
For example:
#Component
public class StrategyA implements Strategy {
#Override
public int execute(int a, int b) {
return a+b;
}
}
The above are singletons and will be created on the application context initialization.
This stage is too early to use #Value("#{jobParameters['strategy']}") as JobParameter wasn't created yet.
So I suggest a locator bean that will be used later when myTasklet is created (Step Scope).
StrategyLocator class:
public class StrategyLocator {
private Map<String, ? extends Strategy> strategyMap;
public Strategy lookup(String strategy) {
return strategyMap.get(strategy);
}
public void setStrategyMap(Map<String, ? extends Strategy> strategyMap) {
this.strategyMap = strategyMap;
}
}
Configuration will look like:
#Bean
#StepScope
public MyTaskelt myTasklet () {
MyTaskelt myTasklet = new MyTaskelt();
//set the strategyLocator
myTasklet.setStrategyLocator(strategyLocator());
return myTasklet;
}
#Bean
protected StrategyLocator strategyLocator(){
return = new StrategyLocator();
}
To initialize StrategyLocator we need to make sure all strategy were already created. So the best approach would be to use ApplicationListener on ContextRefreshedEvent event (warning in this example strategy names start with lower case letter, changing this is easy...).
#Component
public class PlugableStrategyMapper implements ApplicationListener<ContextRefreshedEvent> {
#Autowired
private StrategyLocator strategyLocator;
#Override
public void onApplicationEvent(ContextRefreshedEvent contextRefreshedEvent) {
ApplicationContext applicationContext = contextRefreshedEvent.getApplicationContext();
Map<String, Strategy> beansOfTypeStrategy = applicationContext.getBeansOfType(Strategy.class);
strategyLocator.setStrategyMap(beansOfTypeStrategy);
}
}
The tasklet will hold a field of type String that will be injected with Strategy enum String using #Value and will be resolved using the locator using a "before step" Listener.
public class MyTaskelt implements Tasklet,StepExecutionListener {
#Value("#{jobParameters['strategy']}")
private String strategyName;
private Strategy strategy;
private StrategyLocator strategyLocator;
#BeforeStep
public void beforeStep(StepExecution stepExecution) {
strategy = strategyLocator.lookup(strategyName);
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
int executeStrategyResult = strategy.execute(1, 2);
}
public void setStrategyLocator(StrategyLocator strategyLocator) {
this.strategyLocator = strategyLocator;
}
}
To attach the listener to the taskelt you need to set it in your step configuration:
#Bean
protected Step myTaskletstep() throws MalformedURLException {
return steps.get("myTaskletstep")
.transactionManager(transactionManager())
.tasklet(deleteFileTaskelt())
.listener(deleteFileTaskelt())
.build();
}
jobParameters is holding just a String object and not the real object (and I think is not a good pratice store a bean definition into parameters).
I'll move in this way:
#Bean
#StepScope
class MyStategyHolder {
private MyCustomClass myStrategy;
// Add get/set
#BeforeJob
void beforeJob(JobExecution jobExecution) {
myStrategy = (Bind the right strategy using job parameter value);
}
}
and register MyStategyHolder as listener.
In your tasklet use #Value("#{MyStategyHolder.myStrategy}") or access MyStategyHolder instance and perform a getMyStrategy().