I am trying to create a composed task with two tasks running in sequence. These two tasks internally execute spring batch jobs and the job uses job parameters for further processing.
While defining the composed task definition how can we pass the spring batch job parameters?
I am using SCDF CLI to create the composed task definition and expect start_date and end_date to be passed as job params
task create --name server-test-compoesd --definition "server-apn-psp: server --spring.batch.job.names=apnPlacementStatsProcessorJob --start_date=2022-11-20T00:00:00 --end_date=2022-11-22T00:00:00 && server-dbm-psp: server --spring.batch.job.names=dbmPlacementStatsProcessorJob --start_date=2022-11-20T00:00:00 --end_date=2022-11-22T00:00:00"
Here I want to use start_date and end_date as a JobParamter but when I try to execute this task it fails due to Null Pointer Exception because of the absence of start_date and end_date as a job parameter.
Composed task graph
This task runs fine if I manually run the task and give arguments in this way without the --
JobParams accessed in code
public class Test implements Tasklet {
#Value("#{jobParameters['start_date']}")
private String startDate;
#Value("#{jobParameters['end_date']}")
private String endDate;
Version Information SCDF
Core: 2.4.0.RELEASE (Spring Cloud Data Flow Core)
Dashboard: 2.4.0.RELEASE (Spring Cloud Dataflow UI)
Shell: 2.4.0.RELEASE (Spring Cloud Data Flow Shell)
Version Information of spring batch job
spring-boot : 2.5.9
batch-core:4.1.1.RELEASE
How can we pass the job params while defining the composed task ?
The quick answer is to use the --arguments option when launching the application for example: task launch server-test-compoesd --arguments "app.server-apn-psp.0=start_date=2022-11-20T00:00:00 app.server-apn-psp.1=end_date=2022-11-22T00:00:00 app.server-dbm-psp.0=start_date=2022-11-20T00:00:00 app.server-dbm-psp.1=end_date=2022-11-22T00:00:00"
You can read more about launching tasks using arguments here: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#spring-cloud-dataflow-task-launch
The reason the original scenario did not work for you is that when you set start_date and end_date in the task definition, these are set as properties. You can read more about setting properties in a task definition here: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_application_properties_2
Related
I am scheduling a task that runs at fixed rate in Spring boot. The function that I am using to schedule a a task is as below:
private void scheduleTask(Store store, int frequency) {
final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
Runnable task = store::scan;
scheduler.scheduleAtFixedRate(task, 0, frequency, TimeUnit.MILLISECONDS);
}
This works fine but if if there is an exception at application startup, the application should exit on exception. What is happening is that I get the exception in the log and the message "Application Failed to start" but looks like the scheduler shows as still running although it looks like only the scheduled thread is still running.
Any hints on how to properly schedule an asynchronous task in a Spring boot application? I tried the #Scheduled annotation but it does not run at all.
The #Scheduled should work. Have you added the #EnabledScheduling annotation to a #Configuration or the #SpringBootApplication? The Scheduling Getting Started explains it in detail.
Regarding the scheduleTask method: What calls that? Is it started outside the Spring context? If yes then Spring won't stop it. You have to take care of the lifecycle.
You should try to use the #Scheduled as it will manage the thread pools/executors for you and most people will find it easier to understand.
Is there any option to run Spring Batch job automatically without scheduling or trigger it from other source?
We can run spring batch job by scheduling expression or by main method (As mentioned below).
public static void runBatch() {
JobLauncher jobLauncher = (JobLauncher) getApplicationContext().getBean("jobLauncher");
Job job = get Job details here;
JobParametersBuilder jobParameters = new JobParametersBuilder();
// Setting Job Parameter and run job using below
JobExecution jobExecution = jobLauncher.run(job, jobParameters.toJobParameters());
}
public static void main(String[] args) {
runBatch();
}
Means we need to call main method manually or by some other scheduler so that it will trigger. Without this main method or scheduler can we call this batch process automatically? Or Is there any better options?
Spring batch should trigger automatically without triggering from any entry point its like daemon thread.
Suppose once batch is processing data using spring batch. How can automatically trigger spring batch job after completion of running
job?
Many options exist to trigger the batch:
To call a batch from another scheduler, issue run parameters there and trigger the Job.
If it has to be triggered automatically, say may be on certain interval, use fixedDelay scheduling
To trigger manually, you can go for MBean approach where it can be triggered from JConsole.
Or have an endpoint to call runBatch.
I learn Spring Cloud Task and I write simple application that is divided into 3 services. First is a TaskApplication that have only main() and implements CommandLineRunner, second is a TaskIntakeApplication that receives request and send them to RabbitMQ, third service is an TaskLauncherApplication that receives messages from RabbitMQ and runs the task with received parameters.
#Component
#EnableBinding(Source.class)
public class TaskProcessor {
#Autowired
private Source source;
public void publishRequest(String arguments) {
final String url = "maven://groupId:artifatcId:jar:version";
final List<String> args = Arrays.asList(arguments.split(","));
final TaskLaunchRequest request = new TaskLaunchRequest(url, args, null, null, "TaskApplication");
final GenericMessage<TaskLaunchRequest> message = new GenericMessage<>(request);
source.output().send(message);
}
}
And as you can see I call my built artifact by giving maven url but I wonder how can I call artifact from another docker container?
If you intend to launch a task application from an upstream event (e.g., a new file event; a new DB record event; a new message in Rabbit event, etc.,), you'd simply use the respective out-of-the-box applications and then launch the task via the Task Launcher.
Follow this example on how the 3-steps are orchestrated via SCDF's DSL.
Perhaps you could consider reusing the existing apps instead of reinventing them unless you have got a completely different requirement and that these apps cannot meet it. I'd suggest trying to get the example mentioned above working locally before you consider extending the behavior.
I need to create batch jobs using Spring Batch.
Job will access oracle DB then fetch records, process them in tasklet and commit results.
I am planning to use hibernate with spring to deal with data.
Jobs will be executed via AutoSys. I am using CommandLineJobRunner as entry point.
(Extra info - I am using DynamicWebProject converted to Gradle, STS, Spring 4.0, Hibernate 5.0, NO Spring Boot)
I have few queries/doubts about this entire application. They are more towards environment/deployment.
Do I need to deploy this whole app as a war in Tomcat(or any server) to instantiate all beans(spring and hibernate)?
If yes, how can I start jobs using CommandLineJobRunner ?
If no, I will have to manually instantiate beans in main method using ClassPathXmlApplicationContext. In this case how should I execute jobs ? Do I need to create jar(is this mandatory) ?
How can I test these jobs on command line ? Do I need to pass jars(spring , hibernate etc dependencies) while using CommandLineJobRunner to execute jobs ?
I am new to batch jobs and all your comments would be of great help.
Thanks
No server is needed for spring batch applications.
You can launch job using jobLauncher bean . below is sample code.
public class MyJobLauncher {
public static void main(String[] args) {
GenericApplicationContext context = new AnnotationConfigApplicationContext(MyBatchConfiguration.class);
JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
Job job = (Job) context.getBean("myJobName");//this is bean name of your job
JobExecution execution = jobLauncher.run(job, jobParameters);
}
}
You will need to create jar. Also all other jar that are needed are also required. You can use maven maven assembly plugin for this.
Currently I'm moving from Spring XD as my workflow and runtime environment to Spring Cloud DataFlow and Apache Airflow. I want to create workflows in Airflow and use custom Airflow operator to run Spring Cloud Tasks on Spring Cloud DataFlow server by REST-API.
It's possible using:
curl -X GET http://SERVER:9393/tasks/deployments/...
Unfortunately DataFlow doesn't return job execution ID in this request to create simple way for monitoring of app. Is there a way to get this id in synchronic way? Because getting the last execution of specific job can lead to mistakes eg. missing job execution if I ran many the same jobs at the same time.
On Spring DataFlow I am running Spring Batch jobs so maybe better way is too somehow set execution job id and pass it as input parameter?
Try to use the following annotations to collect the task information from your bean:
public class MyBean {
#BeforeTask
public void methodA(TaskExecution taskExecution) {
}
#AfterTask
public void methodB(TaskExecution taskExecution) {
}
#FailedTask
public void methodC(TaskExecution taskExecution, Throwable throwable) {
}
}
https://docs.spring.io/spring-cloud-task/docs/current-SNAPSHOT/reference/htmlsingle/#features-task-execution-listener