I created a small camunda application with spring boot. all tasks are external.The probliem,Is I the taks do not run n parallel . just one task at a time .
And one of the tasks doenot complete until ,I close the spring boot application.
Please help
Adding the following to application.yml did not solve the problem
spring.task.execution.pool:
core-size: 10
max-size: 20
spring.task.scheduling:
pool.size: 20
#Component
#ExternalTaskSubscription("getCustomers") // create a subscription for this topic name
public class GetCustomers implements ExternalTaskHandler {
#Resource(name = "amlDataSourceJdbcTemplate")
private JdbcTemplate jdbc;
public void execute(ExternalTask externalTask, ExternalTaskService externalTaskService) {
Logger.getLogger("GetCustomers").log(Level.INFO, "worker: GetCustomers started" );
String sql = "SELECT cust_i_id FROM CUSTOMERS";
List<String> customersList = jdbc.query(
sql,
(rs, rowNum) ->
new String(
rs.getString("cust_i_id")
)
);
// we could call an external service to create the loan documents here
// complete the external task
VariableMap variables = Variables.createVariables();
variables.put(Common.VAR_NAME_CUSTOMERS_LIST, customersList);
Logger.getLogger("GetCustomers").log(Level.INFO, "worker: GetCustomers finished" );
// complete the external task
externalTaskService.complete(externalTask, variables);
// externalTaskService.complete(externalTask);
Logger.getLogger("GetCustomers").log(Level.INFO, "worker: GetCustomers Task completed" );
}
}
Related
i am working in the spring-batch where i am having one situation
we have 2 set of schedulers
public class SchedulerA {
#Autowired
private Job a;
#Autowired
private Job b;
#Autowired
private Job c;
#Autowired
private Job d;
#Autowired
private SpringBatchJobHandler springBatchJobHandler;
#Autowired
private JobHandler jobHandler;
private List<String> jobName = new ArrayList<>();
#Bean
public void SchedulerLoad() {
jobName.add(a.getName());
jobName.add(b.getName());
jobName.add(c.getName());
jobName.add(d.getName());
}
#Scheduled(fixedDelay = 300000)
private void jobScheduler() throws Exception {
for (String job : jobName) {
if (!jobHandler.isJobForceStopped()) {
springBatchJobHandler.runJob(job);
}
}
}
and this is the second set of scheduler
public class SchedulerB {
#Autowired
private Job q;
#Autowired
private Job w;
#Autowired
private Job e;
#Autowired
private Job r;
#Autowired
private SpringBatchJobHandler springBatchJobHandler;
#Autowired
private JobHandler jobHandler;
private List<String> jobName = new ArrayList<>();
#Bean
public void SchedulerLoad() {
jobName.add(q.getName());
jobName.add(w.getName());
jobName.add(e.getName());
jobName.add(r.getName());
}
#Scheduled(fixedDelay = 300000)
private void jobScheduler() throws Exception {
for (String job : jobName) {
if (!jobHandler.isJobForceStopped()) {
springBatchJobHandler.runJob(job);
}
}
}
Now what we are trying is jobs within the each scheduler will run in sequential mode but both scheduler class will parallely.
Ex: Both schedulerA and schedulerB has to run at same time parallely, But the jobs within the respective classes has to be run in sequential mode only.
is that possible to achieve the above scenario?
i know above question will be so much confusing but we dont have any other choice..!!
please share your feedback for this situation..
There are two TaskExecutors involved in your scenario:
The one used by Spring Batch to run jobs when launched via a JobLauncher. By default, this uses a SyncTaskExecutor which runs submitted jobs in the calling thread. If you use the default JobLauncher in your SpringBatchJobHandler, then your jobs will be run in sequence since you submit them in a for loop.
The one used by Spring Boot to run scheduled tasks. By default, this one has a pool size of 1, see Spring Boot documentation: The thread pool uses one thread by default and those settings can be fine-tuned using the spring.task.scheduling namespace. This means if you schedule two tasks to run at the same time as in your use case, they will be run in sequence by the same thread.
Now If you want to run those scheduled tasks in parallels using different threads, you need to increase the pool size of the TaskExecutor configured by Spring Boot to 2 or more. You can do that by setting the following property:
spring.task.scheduling.pool.size=2
need to fetch the job parameters in ItemReades from jobLancher.run(job, parameters); in to ItemReader<>
Spring Boot------
i did like below.
i created listener where i can monitor after/before batch job running.
in listener i created static method which is return JobExecution entity where as we can find job related parameters like below.
public class MyListener extends Job***Listener**{
private static Jobexceution jobExecution1;
#Override
p v beforeJob(Jobexceution jobExecution){
this.jobExecution1=jobExecution;
}
p s JobExecution getJobExecution (){
return jobExecution1;
}
}
I have a spring batch job that I'd like to do the following...
Step 1 -
Tasklet - Create a list of dates, store the list of dates in the job execution context.
Step 2 -
JDBC Item Reader - Get list of dates from job execution context.
Get element(0) in dates list. Use is as input for jdbc query.
Store element(0) date is job execution context
Remove element(0) date from list of dates
Store element(0) date in job execution context
Flat File Item Writer - Get element(0) date from job execution context and use for file name.
Then using a job listener repeat step 2 until no remaining dates in the list of dates.
I've created the job and it works okay for the first execution of step 2. But step 2 is not repeating as I want it to. I know this because when I debug through my code it only breaks for the initial run of step 2.
It does however continue to give me messages like below as if it is running step 2 even when I know it is not.
2016-08-10 22:20:57.842 INFO 11784 --- [ main] o.s.batch.core.job.SimpleStepHandler : Duplicate step [readStgDbAndExportMasterListStep] detected in execution of job=[exportMasterListCsv]. If either step fails, both will be executed again on restart.
2016-08-10 22:20:57.846 INFO 11784 --- [ main] o.s.batch.core.job.SimpleStepHandler : Executing step: [readStgDbAndExportMasterListStep]
This ends up in a never ending loop.
Could someone help me figure out or give a suggestion as to why my stpe 2 is only running once?
thanks in advance
I've added two links to PasteBin for my code so as not to pollute this post.
http://pastebin.com/QhExNikm (Job Config)
http://pastebin.com/sscKKWRk (Common Job Config)
http://pastebin.com/Nn74zTpS (Step execution listener)
From your question and your code I deduct that based on the amount of dates that you retrieve (this happens before the actual job starts), you will execute a step for the amount of times you have dates.
I suggest a design change. Create a java class that will get you the dates as a list and based on that list you will dynamically create your steps. Something like this:
#EnableBatchProcessing
public class JobConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private JobDatesCreator jobDatesCreator;
#Bean
public Job executeMyJob() {
List<Step> steps = new ArrayList<Step>();
for (String date : jobDatesCreator.getDates()) {
steps.add(createStep(date));
}
return jobBuilderFactory.get("executeMyJob")
.start(createParallelFlow(steps))
.end()
.build();
}
private Step createStep(String date){
return stepBuilderFactory.get("readStgDbAndExportMasterListStep" + date)
.chunk(your_chunksize)
.reader(your_reader)
.processor(your_processor)
.writer(your_writer)
.build();
}
private Flow createParallelFlow(List<Step> steps) {
SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
// max multithreading = -1, no multithreading = 1, smart size = steps.size()
taskExecutor.setConcurrencyLimit(1);
List<Flow> flows = steps.stream()
.map(step -> new FlowBuilder<Flow>("flow_" + step.getName()).start(step).build())
.collect(Collectors.toList());
return new FlowBuilder<SimpleFlow>("parallelStepsFlow")
.split(taskExecutor)
.add(flows.toArray(new Flow[flows.size()]))
.build();
}
}
EDIT: added "jobParameter" input (slightly different approach also)
Somewhere on your classpath add the following example .properties file:
sql.statement="select * from awesome"
and add the following annotation to your JobDatesCreator class
#PropertySource("classpath:example.properties")
You can provide specific sql statements as a command line argument as well. From the spring documentation:
you can launch with a specific command line switch (e.g. java -jar
app.jar --name="Spring").
For more info on that see http://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html
The class that gets your dates (why use a tasklet for this?):
#PropertySource("classpath:example.properties")
public class JobDatesCreator {
#Value("${sql.statement}")
private String sqlStatement;
#Autowired
private CommonExportFromStagingDbJobConfig commonJobConfig;
private List<String> dates;
#PostConstruct
private void init(){
// Execute your logic here for getting the data you need.
JdbcTemplate jdbcTemplate = new JdbcTemplate(commonJobConfig.onlineStagingDb);
// acces to your sql statement provided in a property file or as a command line argument
System.out.println("This is the sql statement I provided in my external property: " + sqlStatement);
// for now..
dates = new ArrayList<>();
dates.add("date 1");
dates.add("date 2");
}
public List<String> getDates() {
return dates;
}
public void setDates(List<String> dates) {
this.dates = dates;
}
}
I also noticed that you have alot of duplicate code that you can quite easily refactor. Now for each writer you have something like this:
#Bean
public FlatFileItemWriter<MasterList> division10MasterListFileWriter() {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(new File(outDir, MerchHierarchyConstants.DIVISION_NO_10 )));
writer.setHeaderCallback(masterListFlatFileHeaderCallback());
writer.setLineAggregator(masterListFormatterLineAggregator());
return writer;
}
Consider using something like this instead:
public FlatFileItemWriter<MasterList> divisionMasterListFileWriter(String divisionNumber) {
FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
writer.setResource(new FileSystemResource(new File(outDir, divisionNumber )));
writer.setHeaderCallback(masterListFlatFileHeaderCallback());
writer.setLineAggregator(masterListFormatterLineAggregator());
return writer;
}
As not all code is available to correctly replicate your issue, this answer is a suggestion/indication to solve your problem.
Based on our discussion on Spring batch execute dynamically generated steps in a tasklet I'm trying to answer the questions on how to access jobParameter before the job is actually being executed.
I assume that there is restcall which will execute the batch. In general, this will require the following steps to be taken.
1. a piece of code that receives the rest call with its parameters
2. creation of a new springcontext (there are ways to reuse an existing context and launch the job again but there are some issues when it comes to reuse of steps, readers and writers)
3. launch the job
The simplest solution would be to store the jobparameter received from the service as an system-property and then access this property when you build up the job in step 3. But this could lead to a problem if more than one user starts the job at the same moment.
There are other ways to pass parameters into the springcontext, when it is loaded. But that depends on the way you setup your context.
For instance, if you are using SpringBoot directly for step 2, you could write a method like:
private int startJob(Properties jobParamsAsProps) {
SpringApplication springApp = new SpringApplication(.. my config classes ..);
springApp.setDefaultProperties(jobParamsAsProps);
ConfigurableApplicationContext context = springApp.run();
ExitCodeGenerator exitCodeGen = context.getBean(ExitCodeGenerator.class);
int code = exitCodeGen.getExitCode();
context.close();
return cod;
}
This way, you could access the properties as normal with standard Value- or ConfigurationProperties Annotations.
Brief:
Is there a way to ensure that a connection to the database is returned to the pool?
Not-brief:
Data flow:
I have some long running tasks that could be sent to the server in large volume bursts.
Each of the requests is recorded in the DB that the submission was started. Then send that request off for processing.
If failure or success the request is recorded after the task is completed.
The issue is that after the submission is recorded all the way through the long running task, the connection pool uses an "active" connection. This could potential use up any size pool I have if the burst was large enough.
I am using spring boot with the following structure:
Controller - responds at "/" and has the "service" autowired.
Service - Contains all the JPA repositories and #Transactional methods to interact with the database.
When every the first service method call is made from the controller it opens an active connection and doesn't release it until the controller method returns.
So, Is there a way to return the connection to the pool after each service method?
Here is the service class in total:
#Service
#Slf4j
class SubmissionService {
#Autowired
CompanyRepository companyRepository;
#Autowired
SubmissionRepository submissionRepository;
#Autowired
FailureRepository failureRepository;
#Autowired
DataSource dataSource
#Transactional(readOnly = true)
public Long getCompany(String apiToken){
if(!apiToken){
return null
}
return companyRepository.findByApiToken(apiToken)?.id
}
#Transactional
public void successSubmission(Long id) {
log.debug("updating submission ${id} to success")
def submissionInstance = submissionRepository.findOne(id)
submissionInstance.message = "successfully analyzed."
submissionInstance.success = true
submissionRepository.save(submissionInstance)
}
#Transactional
public long createSubmission(Map properties) {
log.debug("creating submission ${properties}")
dataSource.pool.logPoolState()
def submissionInstance = new Submission()
for (key in properties.keySet()) {
if(submissionInstance.hasProperty(key)){
submissionInstance."${key}" = properties.get(key)
}
}
submissionInstance.company = companyRepository.findOne(properties.companyId)
submissionRepository.save(submissionInstance)
return submissionInstance.id
}
#Transactional
public Long failureSubmission(Exception e, Object analysis, Long submissionId){
//Track the failures
log.debug("updating submission ${submissionId} to failure")
def submissionInstance
if (submissionId) {
submissionInstance = submissionRepository.findOne(submissionId)
submissionRepository.save(submissionInstance)
}
def failureInstance = new Failure(submission: submissionInstance, submittedJson: JsonOutput.toJson(analysis), errorMessage: e.message)
failureRepository.save(failureInstance)
return failureInstance.id
}
}
It turns out that #M.Deinum was onto the right track. Spring Boot JPA automatically turns on the "OpenEntityManagerInViewFilter" if the application property spring.jpa.open_in_view is set to true, which it is by default. I found this in the JPA Configuration Source.
After setting this to false, the database session wasn't held onto, and my problems went away.
I'm developing a simple Spring MVC application to download tweets from the streaming API and show them in a webpage. Users of the application can submit a Task with the keywords of the tweets that they want to download. This tasks are shared so everyone can start, stop, modify, change or cancel a task.
TwitterFetcher is the class responsible of download tweets. This class receives a Task and persists all tweets downloaded in a database.
#Service
public class TwitterFetcher {
#Autowired
private OAuthService oAuthService;
#Autowired
private TweetService tweetService;
private Task task;
private TwitterStream twitterStream;
public void start(Task task) {
/* Stop previous stream */
stop();
/* Get OAuth credentials */
OAuth oAuth = oAuthService.findOneEnabled();
if (oAuth == null) {
} else {
this.task = task;
Configuration oAuthConfiguration = getOAuthConfiguration(oAuth);
twitterStream = new TwitterStreamFactory(oAuthConfiguration).getInstance();
twitterStream.addListener(new TwitterListener());
String keywords = task.getBaseKeywords() + ", " + task.getExpandedKeywords();
FilterQuery filterQuery = new FilterQuery();
filterQuery.track(keywords.split(", "));
twitterStream.filter(filterQuery);
}
}
public void stop() {
if (twitterStream != null) {
twitterStream.shutdown();
}
}
private Configuration getOAuthConfiguration(OAuth oAuth) {
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(false);
cb.setJSONStoreEnabled(true);
cb.setOAuthAccessToken(oAuth.getAccessToken());
cb.setOAuthAccessTokenSecret(oAuth.getAccessTokenSecret());
cb.setOAuthConsumerKey(oAuth.getConsumerKey());
cb.setOAuthConsumerSecret(oAuth.getConsumerSecret());
return cb.build();
}
private class TwitterListener implements StatusListener {
#Override
public void onStatus(Status status) {
/* Persist new tweet */
Tweet tweet = new Tweet();
tweet.setJson(DataObjectFactory.getRawJSON(status));
tweetService.save(tweet);
}
[Omitted code]
}
}
The basic functionality would be the next one:
A user start the fetcher from the website.
The fetcher receives a new tweet and it's saved in the DB
The fetcher keeps receiving tweets until a user stop it.
The application has a dashboard to control the fetchers and the tasks and the users must be able to interact with it while the fetcher is downloading.
My question is, Would the fetcher block the app or will be executed in a different thread? In the worst case, what I have to change to solve this? I'm still far from an usable app so I can't test it. Even so, I want to fix it right now if possible.
You can use ExecutorService to run the fetcher in a separate thread. I'd recommend using ThreadPool so you don't blow performance if too many users running the fetcher:
ExecutorService executor = Executors.newFixedThreadPool(maxThreads)
When a task is submitted through the executor it will return a Future object from which you can check for job completion
Future f = executor.submit(myTask);
boolean isDone = f.isDone();
Please read more about Java concurrency if you're not familiar: http://docs.oracle.com/javase/tutorial/essential/concurrency/index.html
Annotate your start() method with #Async.
#Async
public void start(Task task)
This will make the start method asynchronous and will not block the application.
You can check out a simple example here.