I originally asked a question at [Getting "Scope 'step' is not active for the current thread" while creating spring batch beans, and based on the suggestion provided at Spring batch scope issue while using spring boot, I tried to replace #StepScope annotation and instead defined StepScope bean in configuration as below
#Bean
#Qualifier("stepScope")
public org.springframework.batch.core.scope.StepScope stepScope() {
final org.springframework.batch.core.scope.StepScope stepScope = new org.springframework.batch.core.scope.StepScope();
stepScope.setAutoProxy(true);
return stepScope;
}
with this change, I'm not able to pass job parameters while creating beans as, it is throwing
'jobParameters' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext'
My configuration is like
#Configuration
#EnableBatchProcessing
public class MyConfig{
#Bean
#Qualifier("partitionJob")
public Job partitionJob() throws Exception {
return jobBuilderFactory
.get("partitionJob")
.incrementer(new RunIdIncrementer())
.start(partitionStep(null))
.build();
}
#Bean
#StepScope
public Step partitionStep(
#Value("#{jobParameters[gridSize]}") String gridSize)
throws Exception {
return stepBuilderFactory
.get("partitionStep")
.partitioner("slaveStep", partitioner())
.gridSize(gridZize)
.step(slaveStep(chunkSize))
.taskExecutor(threadPoolTaskExecutor()).build();
}
#Bean
#StepScope
public Step slaveStep(int chunkSize) throws Exception {
return stepBuilderFactory
.get("slaveStep")
......
........
}
I read that the bean should be annotated with #StepScope,if job Parameters needs to be accessed like my example. But I'm getting exceptions as explained above.
This is how you can pass job parameters
GenericApplicationContext context = new AnnotationConfigApplicationContext(MyConfiguration.class);
JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
JobParameters jobParameters = new JobParametersBuilder()
.addString("paramter1", "Test")
.toJobParameters();
Job job = (Job) context.getBean("myJob");
JobExecution execution = jobLauncher.run(job, jobParameters);
Access Job parameters
Bean
#StepScope
public Step partitionStep(
#Value("#{jobParameters['paramter1']}") String gridSize)
throws Exception {
return stepBuilderFactory
.get("partitionStep")
.partitioner("slaveStep", partitioner())
.gridSize(gridZize)
.step(slaveStep(chunkSize))
.taskExecutor(threadPoolTaskExecutor()).build();
}
Related
Hi I have been working in Spring batch recently and need some help.
1) I want to run my Job using multiple threads, hence I have used TaskExecutor as below,
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(4);
return taskExecutor;
}
#Bean
public Step myStep() {
return stepBuilderFactory.get("myStep")
.<MyEntity,AnotherEntity> chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.taskExecutor(taskExecutor())
.throttleLimit(4)
.build();
}
but, while executing in can see below line in console.
o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
What does this mean? However, while debugging I can see four SimpleAsyncExecutor threads running. Can someone shed some light on this?
2) I don't want to run my Batch application with the metadata tables that spring batch creates. I have tried adding spring.batch.initialize-schema=never. But it didn't work. I also saw some way to do this by using ResourcelessTransactionManager, MapJobRepositoryFactoryBean. But I have to make some database transactions for my job. So will it be alright if I use this?
Also I was able to do this by extending DefaultBatchConfigurer and overriding:
#Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}
Please guide me further. Thanks.
Update:
My full configuration class here.
#EnableBatchProcessing
#EnableScheduling
#Configuration
public class MyBatchConfiguration{
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Autowired
public DataSource dataSource;
/* #Override
public void setDataSource(DataSource dataSource) {
// override to do not set datasource even if a datasource exist.
// initialize will use a Map based JobRepository (instead of database)
}*/
#Bean
public Step myStep() {
return stepBuilderFactory.get("myStep")
.<MyEntity,AnotherEntity> chunk(1)
.reader(reader())
.processor(processor())
.writer(writer())
.taskExecutor(executor())
.throttleLimit(4)
.build();
}
#Bean
public Job myJob() {
return jobBuilderFactory.get("myJob")
.incrementer(new RunIdIncrementer())
.listener(listener())
.flow(myStep())
.end()
.build();
}
#Bean
public MyJobListener myJobListener()
{
return new MyJobListener();
}
#Bean
public ItemReader<MyEntity> reader()
{
return new MyReader();
}
#Bean
public ItemWriter<? super AnotherEntity> writer()
{
return new MyWriter();
}
#Bean
public ItemProcessor<MyEntity,AnotherEntity> processor()
{
return new MyProcessor();
}
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
taskExecutor.setConcurrencyLimit(4);
return taskExecutor;
}}
In the future, please break this up into two independent questions. That being said, let me shed some light on both questions.
SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
Your configuration is configuring myStep to use your TaskExecutor. What that does is it causes Spring Batch to execute each chunk in it's own thread (based on the parameters of the TaskExecutor). The log message you are seeing has nothing to do with that behavior. It has to do with launching your job. By default, the SimpleJobLauncher will launch the job on the same thread it is running on, thereby blocking that thread. You can inject a TaskExecutor into the SimpleJobLauncher which will cause the job to be executed on a different thread from the JobLauncher itself. These are two separate uses of multiple threads by the framework.
I don't want to run my Batch application with the metadata tables that spring batch creates
The short answer here is to just use an in memory database like HSQLDB or H2 for your metadata tables. This provides a production grade data store (so that concurrency is handled correctly) without actually persisting the data. If you use the ResourcelessTransactionManager, you are effectively turning transactions off (a bad idea if you're using a database in any capacity) because that TransactionManager doesn't actually do anything (it's a no-op implementation).
I'm testing Spring Batch using Spring boot. My need is to define jobs working on an Oracle Database but I don't want to save jobs and steps states inside this DB.
I've read in the documentation I can use a in-memory repository with the MapJobRepositoryFactoryBean.
Then, I've implemented this bean:
#Bean
public JobRepository jobRepository() {
MapJobRepositoryFactoryBean factoryBean = new MapJobRepositoryFactoryBean(new ResourcelessTransactionManager());
try {
JobRepository jobRepository = factoryBean.getObject();
return jobRepository;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
But when my job starts, the first thing Spring Batch does is to create the table in the Oracle DB and continues to use the Oracle datasource. It's like my JobRepository definition isn't taken account.
What did I miss ?
EDIT: I'm using Spring Boot 1.5.3 and Spring Batch 3.0.7
With SpringBoot 2.x, the solution is simpler.
You have to extend the DefaultBatchConfigurer class like this:
#Component
public class NoPersistenceBatchConfigurer extends DefaultBatchConfigurer {
#Override
public void setDataSource(DataSource dataSource) {
}
}
Without datasource, the framework automatically switches to use the MapJobRepository.
A few things here:
If you have a DataSource configured in your ApplicationContext, by default Spring Batch will try to use it.
In order to not use a DataSource when one is available within the ApplicationContext, you'll need to create your own BatchConfigurer. You can do that by extending the DefaultBatchConfigurer.
Don't use the MapJobRepository except only for testing purposes. I has a number of issues (thread safety, etc) and is not recommended for production use. Use an in memory database like HSQLDB instead (you'll still need to create your own BatchConfigurer to do so).
Thank the comment of pvpkiran I've found my problem. It's necessary to define a JobLauncher bean.
Below an example:
#Bean
public JobRepository jobRepository() {
MapJobRepositoryFactoryBean factoryBean = new MapJobRepositoryFactoryBean(new ResourcelessTransactionManager());
try {
JobRepository jobRepository = factoryBean.getObject();
return jobRepository;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
#Bean
public JobLauncher jobLauncher(JobRepository jobRepository) {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
return jobLauncher;
}
If using Spring Boot and #EnableBatchProcessing, you would extend the DefaultBatchConfigurer and override the createJobRepository method. Create a ResourcelessTransactionManager and JobRepository using MapJobRepositoryFactoryBean, the rest of the beans will be auto created by Spring Boot.
#Configuration
public class InMemoryBatchContextConfigurer extends DefaultBatchConfigurer {
#Bean
private ResourcelessTransactionManager resoucelessTransactionManager() {
return new ResourcelessTransactionManager();
}
#Override
protected JobRepository createJobRepository() throws Exception {
MapJobRepositoryFactoryBean factoryBean = new MapJobRepositoryFactoryBean();
factoryBean.setTransactionManager(resoucelessTransactionManager());
return factoryBean.getObject();
}
}`
Extend DefaultBatchConfigurer class and override createJobRepository method just like below.
#Configuration
public class InMemoryBatchConfigurer extends DefaultBatchConfigurer {
#Override
protected JobRepository createJobRepository() throws Exception {
return new MapJobRepositoryFactoryBean().getObject();
}
}
I have written a small Spring Boot service using Quartz. I am trying to schedule a job using the code below. However, I am repeatedly getting the following error:
ERROR[my-service] [2017-01-22 17:38:37,328] [] [main] [SpringApplication]: Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'schedulerFactoryBean' defined in class path resource [com/myservice/configuration/QuartzConfiguration.class]: Invocation of init method failed; nested exception is org.quartz.JobPersistenceException: Couldn't store trigger 'group1.trigger1' for 'group1.job1' job:The job (group1.job1) referenced by the trigger does not exist. [See nested exception: org.quartz.JobPersistenceException: The job (group1.job1) referenced by the trigger does not exist.]
Needless to say, the job never gets scheduled. Here is the relevant code:
#Slf4j
#NoArgsConstructor
public class TimedJob implements Job {
#Override
public void execute(JobExecutionContext context) throws JobExecutionException {
log.debug("**********timed job****************");
}
}
#Configuration
public class QuartzConfiguration {
...
#Inject
MyDao myDao;
#Bean
public SchedulerFactoryBean schedulerFactoryBean(ApplicationContext applicationContext) throws SchedulerException {
SchedulerFactoryBean factory = new SchedulerFactoryBean();
factory.setAutoStartup(true);
factory.setOverwriteExistingJobs(true);
QuartzJobFactory jobFactory = new QuartzJobFactory();
jobFactory.setApplicationContext(applicationContext);
factory.setJobFactory(jobFactory);
factory.setDataSource(dataSource());
factory.setSchedulerContextAsMap(Collections.singletonMap("my_dao", myDao));
JobDetail jobDetail = JobBuilder.newJob()
.ofType(TimedJob.class)
.storeDurably()
.withDescription("desc")
.withIdentity("job1", "group1")
.build();
Calendar calendar = Calendar.getInstance();
calendar.add(Calendar.SECOND, 10);
Trigger trigger = TriggerBuilder
.newTrigger()
.startAt(calendar.getTime())
.withSchedule(
SimpleScheduleBuilder.simpleSchedule().withIntervalInSeconds(1).repeatForever()
)
.forJob(jobDetail)
.withIdentity("trigger1", "group1")
.build();
factory.setTriggers(trigger);
Properties quartzProperties = new Properties();
quartzProperties.setProperty("org.quartz.scheduler.instanceName", instanceName);
quartzProperties.setProperty("org.quartz.scheduler.instanceId", instanceId);
quartzProperties.setProperty("org.quartz.threadPool.threadCount", threadCount);
quartzProperties.setProperty("org.quartz.jobStore.driverDelegateClass", driverDelegateClassName);
factory.setQuartzProperties(quartzProperties);
factory.start();
return factory;
}
This is almost exactly copied from the Quartz tutorials here. What am I doing wrong?
I guess you need setting your jobDetail to your factory instance.
The below URL might help you.
http://www.baeldung.com/spring-quartz-schedule
Is it possible to use both these starters in a single application?
I want to load records from a CSV file into a database table. The Spring Batch tables are stored in a different database, so I assume I need to use JTA to handle the transaction.
Whenever I add #EnableBatchProcessing to my #Configuration class it configures a PlatformTransactionManager, which stops this being auto-configured by Atomikos.
Are there any spring boot + batch + jta samples out there that show how to do this?
Many Thanks,
James
I just went through this and I found something that seems to work. As you note, #EnableBatchProcessing causes a DataSourceTransactionManager to be created, which messes up everything. I'm using modular=true in #EnableBatchProcessing, so the ModularBatchConfiguration class is activated.
What I did was to stop using #EnableBatchProcessing and instead copy the entire ModularBatchConfiguration class into my project. Then I commented out the transactionManager() method, since the Atomikos configuration creates the JtaTransactionManager. I also had to override the jobRepository() method, because that was hardcoded to use the DataSourceTransactionManager created inside DefaultBatchConfiguration.
I also had to explicitly import the JtaAutoConfiguration class. This wires everything up correctly (according to the Actuator's "beans" endpoint - thank god for that). But when you run it the transaction manager throws an exception because something somewhere sets an explicit transaction isolation level. So I also wrote a BeanPostProcessor to find the transaction manager and call txnMgr.setAllowCustomIsolationLevels(true);
Now everything works, but while the job is running, I cannot fetch the current data from batch_step_execution table using JdbcTemplate, even though I can see the data in SQLYog. This must have something to do with transaction isolation, but I haven't been able to understand it yet.
Here is what I have for my configuration class, copied from Spring and modified as noted above. PS, I have my DataSource that points to the database with the batch tables annotated as #Primary. Also, I changed my DataSource beans to be instances of org.apache.tomcat.jdbc.pool.XADataSource; I'm not sure if that's necessary.
#Configuration
#Import(ScopeConfiguration.class)
public class ModularJtaBatchConfiguration implements ImportAware
{
#Autowired(required = false)
private Collection<DataSource> dataSources;
private BatchConfigurer configurer;
#Autowired
private ApplicationContext context;
#Autowired(required = false)
private Collection<BatchConfigurer> configurers;
private AutomaticJobRegistrar registrar = new AutomaticJobRegistrar();
#Bean
public JobRepository jobRepository(DataSource batchDataSource, JtaTransactionManager jtaTransactionManager) throws Exception
{
JobRepositoryFactoryBean factory = new JobRepositoryFactoryBean();
factory.setDataSource(batchDataSource);
factory.setTransactionManager(jtaTransactionManager);
factory.afterPropertiesSet();
return factory.getObject();
}
#Bean
public JobLauncher jobLauncher() throws Exception {
return getConfigurer(configurers).getJobLauncher();
}
// #Bean
// public PlatformTransactionManager transactionManager() throws Exception {
// return getConfigurer(configurers).getTransactionManager();
// }
#Bean
public JobExplorer jobExplorer() throws Exception {
return getConfigurer(configurers).getJobExplorer();
}
#Bean
public AutomaticJobRegistrar jobRegistrar() throws Exception {
registrar.setJobLoader(new DefaultJobLoader(jobRegistry()));
for (ApplicationContextFactory factory : context.getBeansOfType(ApplicationContextFactory.class).values()) {
registrar.addApplicationContextFactory(factory);
}
return registrar;
}
#Bean
public JobBuilderFactory jobBuilders(JobRepository jobRepository) throws Exception {
return new JobBuilderFactory(jobRepository);
}
#Bean
// hopefully this will autowire the Atomikos JTA txn manager
public StepBuilderFactory stepBuilders(JobRepository jobRepository, JtaTransactionManager ptm) throws Exception {
return new StepBuilderFactory(jobRepository, ptm);
}
#Bean
public JobRegistry jobRegistry() throws Exception {
return new MapJobRegistry();
}
#Override
public void setImportMetadata(AnnotationMetadata importMetadata) {
AnnotationAttributes enabled = AnnotationAttributes.fromMap(importMetadata.getAnnotationAttributes(
EnableBatchProcessing.class.getName(), false));
Assert.notNull(enabled,
"#EnableBatchProcessing is not present on importing class " + importMetadata.getClassName());
}
protected BatchConfigurer getConfigurer(Collection<BatchConfigurer> configurers) throws Exception {
if (this.configurer != null) {
return this.configurer;
}
if (configurers == null || configurers.isEmpty()) {
if (dataSources == null || dataSources.isEmpty()) {
throw new UnsupportedOperationException("You are screwed");
} else if(dataSources != null && dataSources.size() == 1) {
DataSource dataSource = dataSources.iterator().next();
DefaultBatchConfigurer configurer = new DefaultBatchConfigurer(dataSource);
configurer.initialize();
this.configurer = configurer;
return configurer;
} else {
throw new IllegalStateException("To use the default BatchConfigurer the context must contain no more than" +
"one DataSource, found " + dataSources.size());
}
}
if (configurers.size() > 1) {
throw new IllegalStateException(
"To use a custom BatchConfigurer the context must contain precisely one, found "
+ configurers.size());
}
this.configurer = configurers.iterator().next();
return this.configurer;
}
}
#Configuration
class ScopeConfiguration {
private StepScope stepScope = new StepScope();
private JobScope jobScope = new JobScope();
#Bean
public StepScope stepScope() {
stepScope.setAutoProxy(false);
return stepScope;
}
#Bean
public JobScope jobScope() {
jobScope.setAutoProxy(false);
return jobScope;
}
}
I found a solution where I was able to keep #EnableBatchProcessing but had to implement BatchConfigurer and atomikos beans, see my full answer in this so answer.
I have a Job running on startup. I want to run this job programmatically at a particular point of my application, not when I start my app.
When running on startup I have no problem, but I got a "NoSuchJobException" (No job configuration with the name [importCityFileJob] was registered) when I try to run it programmatically.
After looking on the web, I think it's a problem related to JobRegistry, but I don't know how to solve it.
Note : my whole batch configuration is set programmatically, I don't use any XML file to configure my batch and my job. That's a big part of my problem while I lack the examples...
Here is my code to run the Job :
public String runBatch() {
try {
JobLauncher launcher = new SimpleJobLauncher();
JobLocator locator = new MapJobRegistry();
Job job = locator.getJob("importCityFileJob");
JobParameters jobParameters = new JobParameters(); // ... ?
launcher.run(job, jobParameters);
} catch (Exception e) {
e.printStackTrace();
System.out.println("Something went wrong");
}
return "Job is running";
}
My Job declaration :
#Bean
public Job importCityFileJob(JobBuilderFactory jobs, Step step) {
return jobs.get("importFileJob").incrementer(new RunIdIncrementer()).flow(step).end().build();
}
(I tried to replace importCityFileJob by importFileJob in my runBatch method, but it didn't work)
My BatchConfiguration file contains the job declaration above, a step declaration, the itemReader/itemWriter/itemProcessor, and that's all.
I use the #EnableBatchProcessing annotation.
I'm new to Spring Batch & I'm stuck on this problem. Any help would be welcome.
Thanks
Edit : I've solved my problem. I wrote my solution in the answers
Here is what I had to do to fix my problem:
Add the following Bean to the BatchConfiguration :
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor = new JobRegistryBeanPostProcessor();
jobRegistryBeanPostProcessor.setJobRegistry(jobRegistry);
return jobRegistryBeanPostProcessor;
}
Replace the JobLocator by an #Autowired JobRegistry, and use the #Autowired JobLauncher instead of creating one. My run method now have the following code :
#Autowired
private JobRegistry jobRegistry;
#Autowired
private JobLauncher launcher;
public String runBatch() {
try {
Job job = jobRegistry.getJob("importCityFileJob");
JobParameters jobParameters = new JobParameters();
launcher.run(job, jobParameters);
} catch (Exception e) {
e.printStackTrace();
System.out.println("Something went wrong");
}
return "OK";
}
I hope it will help someone.
A JobRegistry won't populate itself. In your example, you're creating a new instance, then trying to get the job from it without having registered it in the first place. Typically, the JobRegistry is configured as a bean along with an AutomaticJobRegistrar that will load all jobs into the registrar on startup. That doesn't mean they will be executed, just registered so they can be located later.
If you're using Java configuration, this should happen automatically using the #EnableBatchProcessing annotation. With that annotation, you'd just inject the provided JobRegistry and the jobs should already be there.
You can read more about the #EnableBatchProcessing in the documentation here: http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/configuration/annotation/EnableBatchProcessing.html
You can also read about the AutomaticJobRegistrar in the documentation here: http://docs.spring.io/spring-batch/apidocs/org/springframework/batch/core/configuration/support/AutomaticJobRegistrar.html
I could not find the correct answer on this page. In my case the spring batch jobs were configured in a different configuration class not annotated with #EnableBatchProcessing. In that case you need to add the Job to the JobRegistry:
import org.springframework.batch.core.Job;
import org.springframework.batch.core.configuration.DuplicateJobException;
import org.springframework.batch.core.configuration.JobRegistry;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.support.ReferenceJobFactory;
import org.springframework.batch.core.job.flow.Flow;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
#Configuration
public class MyBatchJobConfigurations {
#Bean
public Job myCountBatchJob(final JobBuilderFactory jobFactory, final JobRegistry jobRegistry, final Flow myJobFlow)
throws DuplicateJobException {
final Job countJob = jobFactory.get("myCountBatchJob")
.start(myJobFlow)
.end().build();
ReferenceJobFactory referenceJobFactory = new ReferenceJobFactory(countJob);
jobRegistry.register(referenceJobFactory);
return countJob;
}
}
Adding the following bean in the applicationContext.xml resolved the problem for me
<bean class="org.springframework.batch.core.configuration.support.JobRegistryBeanPostProcessor">
<property name="jobRegistry" ref="jobRegistry" />
</bean>
I also have this entry in the applicationContext.xml
<bean id="jobRegistry"
class="org.springframework.batch.core.configuration.support.MapJobRegistry" />
another solution:
rename the method name "importCityFileJob" to "job":
#Bean
public Job job(JobBuilderFactory jobs, Step step) {
return jobs.get("importFileJob").incrementer(new RunIdIncrementer()).flow(step).end().build();
}
#EnableBatchProcessing
#Configuration
public class SpringBatchCommon {
#Bean
public JobRegistryBeanPostProcessor jobRegistryBeanPostProcessor(JobRegistry jobRegistry) {
JobRegistryBeanPostProcessor postProcessor = new JobRegistryBeanPostProcessor();
postProcessor.setJobRegistry(jobRegistry);
return postProcessor;
}
}
Set the JobRegistry in the JobRegistryBeanPostProcessor , after that you can autowire the JobLauncher and the JobLocator
Job job = jobLocator.getJob("importFileJob");
JobParametersBuilder jobBuilder = new JobParametersBuilder();
//set any parameters if required
jobLauncher.run(job, jobBuilder.toJobParameters()