I've encountered an issue on Spring Cloud Dataflow when running multiple composed tasks at once.
Hikari DataSource takes 10 connections from the connection pool by default. When running for example 10 composed tasks at once, this means 100 connections + connections required for every task on each composed task.
I tried running the Composed Task Runner locally with spring.datasource.hikari.maximum-pool-size=1 and it worked.
Is there any way how to set this property to every Composed Task Runner by default ? I did not find any documentation related to modifying things like this for composed tasks.
Related
We have a spring boot application and scheduled tasks.
We want to deploy our application on multiple servers , so multiple instances of applications.
How to configure spring to run scheduled tasks on multiple instances run at same time?
For eg : An application deployed in 1st server instance at 12am and task running scheduled at 12am.
The same application deployed in 2nd server instance at 12.03 am and since there is a difference in deployment task is also started at 12.03am with same cron expression ,and third one at deployed at 12.05am at task starting at 12.05am.
How to make tasks run at same time (all three tasks running in multiple instances need to run at same time 12.05am) ,so we get same results for all three tasks .
The cron reference is with respect to server time not deployed time of application..so all will run at exact 12.05 even they deployed in different minutes
I have a spring boot batch application that has 5 unique jobs that execute by console using the command:
java -jar artifactName jobName param1
but now this project will be move to cloud, so I need to use spring cloud task. So far so good.
I know that I have to define in the main class the #enableTask and also in the application.properties define the properties:
spring.application.name=cloudTask
So reading the Spring documentation understand that for triggering my jobs using spring cloud dataflow server, can define a task that in this case i should use as cloudTask. But does not make sense because how will tigger it, because my application has 5 different jobs, so the question is:
how do i connect this task name with my jobs define in the application?
The logic tell me that I need to define also 5 task name, then how do I bind this task name with the respective job.
With #EnableTask annotation, you should be able to register your batch as Task application in SCDF - Under 'Apps'
Once your batch appears in Apps,
If all jobs 5 jobs are independent, you should be able to create 5 different Composed Tasks with same App name but with different parameters,
OR
If those are interlinked, then linked jobs can be combined together in 1 composed task, by providing alias and passing corresponding set of parameters, in DSL.
Once the composed task is launched, task execution status can be viewed under 'Task -> Executions' and corresponding to Jobs status can be viewed under 'Jobs'
To pass custom parameters to tasks, #EnableConfigurationProperties #ConfigurationProperties can be leveraged.
We have String Batch applications with triggers defined in each app.
Each Batch application runs tens of similar jobs with different parameters and is able to do that with 1400 MiB per app.
We use Spring Batch Admin, which is deprecated years ago, to launch individual job and to get brief overview what is going in jobs. Migration guide recommends to replace Spring Batch Admin with Spring Cloud DataFlow.
Spring Cloud DataFlow docs says about grabbing jar from Maven repo and running it with some parameters. I don't like idea to wait 20 sec for application downloading, 2 min to application launching and all that security/certificates/firewall issues (how can I download proprietary jar across intranets?).
I'd like to register existing applications in Spring Cloud DataFlow via IP/port and pass job definitions to Spring Batch applications and monitor executions (including ability to stop job). Is Spring Cloud DataFlow usable for that?
Few things to unpack here. Here's an attempt at it.
Spring Cloud DataFlow docs says about grabbing jar from Maven repo and running it with some parameters. I don't like idea to wait 20 sec for application downloading, 2 min to application launching and all that security/certificates/firewall issues
Yes, there's an App resolution process. However, once downloaded, we would reuse the App from Maven cache.
As for the 2mins bootstrapping window, it is up to Boot and the number of configuration objects, and of course, your business logic. Maybe all that in your case is 2mins.
how can I download proprietary jar across intranets?
There's an option to resolve artifacts from a Maven artifactory hosted behind the firewall through proxies - we have users on this model for proprietary JARs.
Each Batch application runs tens of similar jobs with different parameters and is able to do that with 1400 MiB per app.
You may want to consider the Composed Task feature. It not only provides the ability to launch child Tasks as Direct Acyclic Graphs, but it also allows transitions based on exit-codes at each node, to further split and branch to launch more Tasks. All this, of course, is automatically recorded at each execution level for further tracking and monitoring from the SCDF Dashboard.
I'd like to register existing applications in Spring Cloud DataFlow via IP/port and pass job definitions to Spring Batch applications and monitor executions (including ability to stop job).
As far as the batch-jobs are wrapped into Spring Cloud Task Apps, yes, you'd be able to register them in SCDF and use it in the DSL or drag & drop them into the visual canvas, to create coherent data pipelines. We have a few "batch-job as task" samples here and here.
I have a Spring Boot web application project using an embedded Activiti engine (using the activiti-spring-boot-starter-basic Maven dependency) with a simple workflow processing a business request. There are service tasks implemented with JavaDelegate objects.
We may have multiple parallel requests that must be processed at the same time. However, the Activiti engine is waiting for a given process instance to reach a pause state (waiting for messages) before beginning a new process instance.
We are starting a process instance using this command :
runtimeService.startProcessInstanceByKey("process_simple", variables);
I have added to application.properties the following to no use :
# Activiti config
spring.activiti.jobExecutorActivate = true
spring.activiti.asyncExecutorEnabled = true
spring.activiti.asyncExecutorActivate = true
I also have define all service tasks to Asynchronous, and to Multi-instance type : parallel but that does not work either.
Have I missed a setting ?
Thanks for any help.
Have one service named AccountService.5 instances are needed to support the traffic.all these instances run in docker container.Now I need to run a schedule task.this task should only run on a single AccountService instance.but not all the five instances.which one is not important
My question is how to configure to achieve this.Can eureka do this?and zookeeper seems have the ability to manage the cluster.Do I need to register the AccountService into Zookeeper?
Hope someone can share experience with me
Consider using a shared data store like Redis or, if you're already using a DB, a table in the DB, to have a task lock. The first instance to come up can grab the lock, run the task, and release the lock.
Include spring-cloud-task in your dependency (which is suitable for scheduled tasks).
Then enable this property - spring.cloud.task.single-instance-enabled=true
Add Spring Integration dependencies. Copy/paste from here - https://docs.spring.io/spring-cloud-task/docs/current/reference/#features-single-instance-enabled
Note:
Locks are created and stored under TASK_LOCK table. Make sure its clean, otherwise you will have problems restarting.
Use Spring Based task scheduling #Schedule
For more click here