I have a simple Spring Batch application complying with JSR-352.
I need to deploy this as a managed Task on Spring Cloud Data Flow server. As far as I know - to be able to deploy this as a Task I need to convert this application as a Spring Boot app.
I have tried to add Spring Boot dependencies and Main class however it is not running the Batch job when I start the app.
Main Class
#SpringBootConfiguration
#EnableAutoConfiguration
#EnableBatchProcessing
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
Batch File created at
META-INF/batch-jobs/myjob.xml
It works when I use JobOperator in the main class to start the job (without Spring Boot).
What am I missing to run this as a Spring Boot app?
You're missing #EnableTask annotation. With that, your batch-job will be run as a short-lived application. In other words, the application will run as long as the business logic in your XML needs to run, and it will gracefully shut down and free-up resources.
Please clone and try out the Spring Cloud Task samples [see: BatchJobApplication]. All of them should work as-is in SCDF as well.
#EnableBatchProcessing
#SpringBootApplication
public class BatchApplication {
public static void main(String[] args) {
SpringApplication.run(BatchApplication.class, args);
}
#Bean
public CommandLineRunner run(JobOperator jobOperator) {
return $ -> jobOperator.start("myjob", new Properties());
}
#Bean
JobParametersConverter jobParametersConverter(DataSource dataSource) {
return new JsrJobParametersConverter(dataSource);
}
#Bean
JobOperator jsrJobOperator(ApplicationContext applicationContext, JobExplorer jobExplorer,
JobRepository jobRepository, JobParametersConverter jobParametersConverter,
PlatformTransactionManager transactionManager) {
JsrJobOperator jobOperator = new JsrJobOperator(jobExplorer, jobRepository, jobParametersConverter,
transactionManager);
jobOperator.setApplicationContext(applicationContext);
jobOperator.setTaskExecutor(new SimpleAsyncTaskExecutor());
return jobOperator;
}
}
https://gist.github.com/rixwwd/8091a717ca24fd810ff71b4fdebbf9cc
Related
I am using Spring Boot + Spring Batch(annotation), the application has 10 batch jobs configured for 10 different types of files. The file itself is pulled from AmazonS3. Can anyone guide me on how to programmatically run a batch job based on the type of file pulled from S3. Each of the 10 batch jobs runs just fine when the file is placed in the class path, not sure how to pick a job to run based on file type. Totally new to Spring boot and Spring batch, any guidance will be greatly appreciated.
#Configuration
#EnableBatchProcessing
public class FileTypeX1Configuration{
private FileTypeX1PRocessor fileTypeX1PRocessor ;
private FileTypeX1Writer fileTypeX1Writer ;
.
.
}
.
.
.
#Configuration
#EnableBatchProcessing
public class FileTypeX10Configuration{
private FileTypeX10PRocessor fileTypeX10PRocessor ;
private FileTypeX1oWriter fileTypeX10Writer ;
.
.
}
First, you only need to add #EnableBatchProcessing in one of your configuration classes. This is typically added on the main class of your Spring Boot application:
#SpringBootApplication
#EnableBatchProcessing
public class SampleApplication {
public static void main(String[] args) {
SpringApplication.run(SampleApplication.class, args);
}
}
Now regarding your conditional job launching, there are several ways to do that (like using Spring profiles, using a custom condition based on #Conditional to define your job beans, etc). But the most straight-forward way is doing in it the main method:
#SpringBootApplication
#EnableBatchProcessing
public class SampleApplication {
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(SampleApplication.class, args);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
File file = .. // get file
// based on file type, run the required job:
Job job = context.getBean("jobX", Job.class);
jobLauncher.run(job, new JobParameters());
}
}
You would need to set spring.batch.job.enabled=false to prevent Spring Boot from running all jobs at your application startup.
If you can detect the file type outside of your app (in a shell script for example) you can tell Spring Boot which job should be run from your script with the property spring.batch.job.names=jobX.
I have my standard Spring Boot application working. I have situations where I want to run a "job" which is basically some specific method normally run via a user doing something in their browser but I want to run it from command line.
I'm able to run an arbitrary class with gradlew;
./gradlew -PmainClass=kcentral.backingservices.URLMetaExtractor execute
However when run this way none of the "autowiring" works. What is a better way to execute an arbitrary class (that has a main method) such that it also works with any Autowiring?
EDIT:
I got some advice to use a CommandLineRunner and some args, which work to execute the command via:
./gradlew bootRun -Pargs=--reloadTestData
However, the Autowiring of my Repo is failing. What I have is:
#EnableAutoConfiguration
#EnableMongoAuditing
#EnableMongoRepositories(basePackageClasses=KCItemRepo.class)
#ComponentScan(basePackages = {"kcentral"})
public class ReloadTestData implements CommandLineRunner {
#Autowired
AddItemService addItemService;
#Autowired
KCItemRepo itemRepo;
#Autowired
KCItemRatingRepo itemRatingRepo;
private static final Logger log = LoggerFactory.getLogger(ReloadTestData.class);
public void reloadData(){
log.info("reloadData and called");
if (itemRepo == null){
log.error("Repo not found");
return;
}
long c = itemRepo.count();
log.warn("REMOVING ALL items "+c);
itemRepo.deleteAll();
log.warn("REMOVING ALL ratings");
itemRatingRepo.deleteAll();
}
itemRepo is always null even though I wire the same way in my 'regular' spring boot app without an issue. What do I need to do to have it wire properly?
The fact that you say you want to run a "job" suggests that you might want to use a scheduled task within your application, rather than trying to run it through the command line. e.g. Scheduling tasks in Spring
#Scheduled(fixedRate = 5000)
public void reportCurrentTime() {
log.info("The time is now {}", dateFormat.format(new Date()));
}
If you want to make a command line application work with Autowiring, you can make a command line application by making your Application class implement the CommandLineRunner interface, e.g. Spring Boot Console App
#SpringBootApplication
public class SpringBootConsoleApplication
implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(SpringBootConsoleApplication.class, args);
}
#Override
public void run(String... args) {
}
}
And add spring.main.web-application-type=NONE to the properties file.
If you want to stop the application after running you can use SpringApplication.exit(ctx). Don't know about your auto-wiring problem though, maybe try printing out the list of available beans which might give some insight. Example:
#Component
public class DoThenQuit implements CommandLineRunner {
#Autowired
private ApplicationContext ctx;
#Override
public void run(String[] args) {
// do some other stuff before quitting
String[] beanNames = ctx.getBeanDefinitionNames();
Arrays.stream(beanNames).forEach(System.out::println);
// then quit the application
SpringApplication.exit(ctx);
}
}
I have created a spring cloud task that will perform some specific task based on the requirement. I wanted to call this task from another spring boot application. Please let me know is there any way of calling the below task from an external application.
#SpringBootApplication
#EnableTask
public class FileGenerationTaskApplication {
#Autowired
private DataSource dataSource;
public class FileGeneratorTaskConfigurer extends DefaultTaskConfigurer {
public FileGeneratorTaskConfigurer(DataSource dataSource){
super(dataSource);
}
}
#Bean()
public FileGeneratorTaskConfigurer getTaskConfigurer() {
return new FileGeneratorTaskConfigurer(dataSource);
}
public static void main(String[] args) {
SpringApplication.run(FileGenerationTaskApplication.class, args);
}
#Component
public static class FileGeneratorTaskRunner implements ApplicationRunner {
#Autowired
private FulfillmentFileGenerationService service;
public void run(ApplicationArguments args) throws Exception {
System.out.println("FileGeneratorTaskRunner from Spring Cloud Task!");
service.fulFillmentFileGenerationTask();
}
}
}
Can we create a REST api to call the spring cloud task?
It would be nice to have the Task registered on Spring Cloud Dataflow.
After you have your Task registered, you can make REST calls to trigger the Task. Check this example out.
You can also use Spring Cloud Dataflow Rest Client
DataFlowOperations dataFlowOperations = new DataFlowTemplate(URI.create(springDataFlowUri));
TaskOperations operations = dataFlowOperations.taskOperations();
Then you can start launching the Tasks previously got using the API Rest.
In case you do not want to use Spring Cloud DataFlow, remember when you create a Task, this is a Spring Boot Application by itself, so you can expose end points to trigger the Task.
I have two independent spring batch jobs in the same project because I want to use the same infrastructure-related beans. Everything is configured in Java. I would like to know if there's a proper way to start the jobs independent based for example on the first java app argument in the main method for example. If I run SpringApplication.run only the second job gets executed by magic.
The main method looks like:
#ComponentScan
#EnableAutoConfiguration
public class Application {
public static void main(String[] args) {
SpringApplication app = new SpringApplication(Application.class);
app.setWebEnvironment(false);
ApplicationContext ctx= app.run(args);
}
}
and the two jobs are configured as presented in the Spring Batch Getting Started tutorial on Spring.io. Here is the configuration file of the first job, the second being configured in the same way.
#Configuration
#EnableBatchProcessing
#Import({StandaloneInfrastructureConfiguration.class, ServicesConfiguration.class})
public class AddPodcastJobConfiguration {
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory stepBuilderFactory;
//reader, writer, processor...
}
To enable modularization I created an AppConfig class, where I define factories for the two jobs:
#Configuration
#EnableBatchProcessing(modular=true)
public class AppConfig {
#Bean
public ApplicationContextFactory addNewPodcastJobs(){
return new GenericApplicationContextFactory(AddPodcastJobConfiguration.class);
}
#Bean
public ApplicationContextFactory newEpisodesNotificationJobs(){
return new GenericApplicationContextFactory(NotifySubscribersJobConfiguration.class);
}
}
P.S. I am new to Spring configuration in Java configuration Spring Boot and Spring Batch...
Just set the "spring.batch.job.names=myJob" property. You could set it as SystemProperty when you launch your application (-Dspring.batch.job.names=myjob). If you have defined this property, spring-batch-starter will only launch the jobs, that are defined by this property.
To run the jobs you like from the main method you can load the the required job configuration bean and the JobLauncher from the application context and then run it:
#ComponentScan
#EnableAutoConfiguration
public class ApplicationWithJobLauncher {
public static void main(String[] args) throws BeansException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException, InterruptedException {
Log log = LogFactory.getLog(ApplicationWithJobLauncher.class);
SpringApplication app = new SpringApplication(ApplicationWithJobLauncher.class);
app.setWebEnvironment(false);
ConfigurableApplicationContext ctx= app.run(args);
JobLauncher jobLauncher = ctx.getBean(JobLauncher.class);
JobParameters jobParameters = new JobParametersBuilder()
.addDate("date", new Date())
.toJobParameters();
if("1".equals(args[0])){
//addNewPodcastJob
Job addNewPodcastJob = ctx.getBean("addNewPodcastJob", Job.class);
JobExecution jobExecution = jobLauncher.run(addNewPodcastJob, jobParameters);
} else {
jobLauncher.run(ctx.getBean("newEpisodesNotificationJob", Job.class), jobParameters);
}
System.exit(0);
}
}
What was causing my lots of confusion was that the second job were executed, even though the first job seemed to be "picked up" by the runner... Well the problem was that in both job's configuration file I used standard method names writer(), reader(), processor() and step() and it used the ones from the second job that seemed to "overwrite" the ones from the first job without any warnings...
I used though an application config class with #EnableBatchProcessing(modular=true), that I thought would be used magically by Spring Boot :
#Configuration
#EnableBatchProcessing(modular=true)
public class AppConfig {
#Bean
public ApplicationContextFactory addNewPodcastJobs(){
return new GenericApplicationContextFactory(AddPodcastJobConfiguration.class);
}
#Bean
public ApplicationContextFactory newEpisodesNotificationJobs(){
return new GenericApplicationContextFactory(NotifySubscribersJobConfiguration.class);
}
}
I will write a blog post about it when it is ready, but until then the code is available at https://github.com/podcastpedia/podcastpedia-batch (work/learning in progress)..
There is the CommandLineJobRunner and maybe can be helpful.
From its javadoc
Basic launcher for starting jobs from the command line
Spring Batch auto configuration is enabled by adding #EnableBatchProcessing (from Spring Batch) somewhere in your context. By default it executes all Jobs in the application context on startup (see JobLauncherCommandLineRunner for details). You can narrow down to a specific job or jobs by specifying spring.batch.job.names (comma separated job name patterns).
-- Spring Boot Doc
Or disable the auto execution and run the jobs programmatically from the context using a JobLauncher based on the args passed to the main method
I have a Spring 4 application with Spring Boot -
There is no WEBINF/web.xml file, however, I'd like to initialize a Quartz 2.2.1 scheduler on application startup. However, all the examples using QuartzInitializerServlet define the settings in the web.xml file.
Can I add these configurations to my application startup configuration?
#Configuration
#ComponentScan
#EnableAutoConfiguration
public class Application {
#Bean
public DataSource dataSource() {
DriverManagerDataSource ds = new DriverManagerDataSource();
ds.setDriverClassName("org.postgresql.Driver");
ds.setUrl("jdbc:postgresql://localhost/...");
ds.setUsername("...");
ds.setPassword("...!");
return ds;
}
/** Add configuration to start Quartz here so
I can access it throughout the app? **/
#Bean
public org.springframework.scheduling.quartz.SchedulerFactoryBean SchedulerFactoryBean(){
SchedulerFactoryBean scheduler = new SchedulerFactoryBean();
scheduler.setAutoStartup(true);
scheduler.setDataSource(dataSource());
return scheduler;
}
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
Update
Figured out the spring-framework quartz bean, now I need to correctly implement the datastore to restore jobs in-between runs.
I'm using postgresql + spring-data & hibernate. This configuration reinitializes the database on each run. HSQL reinitializes some 'import.sql' data as well. Should I create a hibernate interface so that the jobs will be restored on testing?