I have create some Spring Batch Project and deploy these Job by using Spring Cloud Data Flow (SCDF).
After Launching task(job) in SCDF, it create JVM to execute the task(job).
However when the task is finished, that JVM is not end. It still existed.
When I launch my job 20 times, it announced that
Cannot launch task A. The maximum concurrent task executions is at its limit [20]
And there are some information about my Job, the first log of the jobs is end with:
HikariPool-1 - Shutting down...
But after I used a below property for the Spring Batch Project:
spring.cloud.task.singleInstanceEnabled=true
and used afterJob method by using JobExecutionListenerSupport
The log of the task(job) end with:
o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=remindJob]] completed with the following parameters: [{run.id=3}] and the following status: [COMPLETED] in 7s636ms
o.s.integration.leader.DefaultCandidate : DefaultCandidate{role=EngineProcess, id=126} leadership has been revoked: LockContext{role=Process, id=126, isLeader=false}
Are these any problems with my Spring Batch Job?
And my major question is that which one is responsible for completely stopping JVM(task)? Spring Cloud Data Flow part or Spring Batch part and how?
I think that when task are finished, it should be destroyed (JVM is stopped) and the number of concurrent task executions could not be at the limit.
I got the solutions from github that setting the property spring.cloud.task.closecontext_enabled = true. However I would like to understand deeply the reason why the context is not closed completely without spring.cloud.task.closecontext_enabled. After setting that property to true. My Spring Batch project's log showed the WARN:
main] o.s.b.f.support.DisposableBeanAdapter : Destroy method 'close' on bean with name 'getStudent' threw an exception: org.springframework.batch.item.ItemStreamException: Error while closing item reader
And there is the ItemReader code:
#Bean
public JdbcCursorItemReader<Student> getStudent() {
JdbcCursorItemReader <Student> reader = new JdbcCursorItemReader<>();
reader.setDataSource(dataSource);
reader.setSql(QueryConstants.getStudent);
reader.setRowMapper(new BeanPropertyRowMapper<>(Student.class));
return reader;
}
Related
I have a spring boot app which uses spring data and hikaricp for db connection pooling. I noticed the following behaviour that looks strange to me:
I have one method which is not transactional and in that method several db queries are executed using spring data repositories
public void testMethod(final Long firstRepositoryId, final Long secondRepositoryId) {
final DomainObject result = firstRepository.findById(firstRepositoryId);
// here there's some code that is processing the result without db queries
secondRepository.findById(secondRepositoryId);
// some more logic here without additional db queries
}
So as expected when there's no transaction on the method then the spring data methods opens a transaction for executing the query and complete it after the methods returns. I have enabled transaction logging so there's the following log output:
2021-06-03 15:34:30.961 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Getting transaction for [com.test.FirstRepository.findById]
2021-06-03 15:34:30.966 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Completing transaction for [com.test.FirstRepository.findById]
2021-06-03 15:34:30.967 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Getting transaction for [com.test.SecondRepository.findById]
2021-06-03 15:34:30.972 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Completing transaction for [com.test.SecondRepository.findById]
Everything seems to be exactly how I expects to be. The thing I can't understand is the hikari behaviour. This method is invoked within a http request. A connection is taken from hikari cp right after the execution of the firstRepository.findById but this connection is returned in the pool only after the http controller returns response. What I expect is that a connection is taken after a transaction is opened and returned back after the transaction is completed. Is there something that I miss or maybe I have some wrong configuration?
P.S. I'm monitoring the active hikari connections through the spring boot actuator prometheus data. And to be able to reproduce the behavior I explained above I'm suspending the connection thread with several debug breakpoints.
I found out what causes this behaviour- it's Spring functionality to maintain the hibernate session in view in order to be able to retrieve lazy loaded data in the view. In order to disable this you need the following property:
spring.jpa.open-in-view=false
Here's another SO post where is explained what it does:
What is this spring.jpa.open-in-view=true property in Spring Boot?
I am scheduling a task that runs at fixed rate in Spring boot. The function that I am using to schedule a a task is as below:
private void scheduleTask(Store store, int frequency) {
final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
Runnable task = store::scan;
scheduler.scheduleAtFixedRate(task, 0, frequency, TimeUnit.MILLISECONDS);
}
This works fine but if if there is an exception at application startup, the application should exit on exception. What is happening is that I get the exception in the log and the message "Application Failed to start" but looks like the scheduler shows as still running although it looks like only the scheduled thread is still running.
Any hints on how to properly schedule an asynchronous task in a Spring boot application? I tried the #Scheduled annotation but it does not run at all.
The #Scheduled should work. Have you added the #EnabledScheduling annotation to a #Configuration or the #SpringBootApplication? The Scheduling Getting Started explains it in detail.
Regarding the scheduleTask method: What calls that? Is it started outside the Spring context? If yes then Spring won't stop it. You have to take care of the lifecycle.
You should try to use the #Scheduled as it will manage the thread pools/executors for you and most people will find it easier to understand.
It appears that Spring Cloud Task lifecycle is incorrectly managed when spring boot application has hierarchical application contexts.
When i add #EnableTask annotation to the parent ApplicationContext, it registers the task, but records execution time from the parent context, failing to record accurate execution time and exit code (always success as parent context closes successfuly).
On the other hand, if i add annotation to the child context (which actually runs CommandlineRunner), it fails to start the task at all with below exception:
o.s.c.t.listener.TaskLifecycleListener : [] [] An event to end a task has been received for a task that has not yet started.
s.c.a.AnnotationConfigApplicationContext : [] [] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.IllegalArgumentException: Invalid TaskExecution, ID 132515 not found
Looking at the TaskLifecycleListener source, it appears that it reacts to ApplicationEvents from parent context and catches ApplicationReadyEvent from parent context before the task is started.
Spring Boot 2.2.6 / Spring Cloud Task 2.2.2
Any thoughts?
This is by design. The "task" is registered as a single execution of a Boot application, not the execution of an ApplicationContext. In a microservices world, you would want to break up your tasks into independent artifacts, therefore running them as independent Spring Boot applications. There is no support for Spring Cloud Task to support multiple "task" executions within a single Spring Boot application as that goes against what its intent is. If you feel this is something that should be added, feel free to open up an issue on Github where we can explore your use case deeper.
Actually, I realized what happens is that Spring auto configures task in my main parent context which closes the task. And same appears to be happening in my child context. Since executionId is passed as parameter, it is already updated by the parent as executed, so it fails in my child context.
Solution I found to this was to exclude autoconfiguration of SimpleTaskAutoConfiguration from my parent context.
#SpringBootApplication(exclude = {SimpleTaskAutoConfiguration.class})
Instead I manually import configuration class in my child context which executes the task:
#EnableTask
#Import(SimpleTaskAutoConfiguration.class)
That now tracks actual time it took to execute the command line runner as a task within child context, although it doesn't account for time of running whole app, at least it reflects more accurate picture.
I need to create batch jobs using Spring Batch.
Job will access oracle DB then fetch records, process them in tasklet and commit results.
I am planning to use hibernate with spring to deal with data.
Jobs will be executed via AutoSys. I am using CommandLineJobRunner as entry point.
(Extra info - I am using DynamicWebProject converted to Gradle, STS, Spring 4.0, Hibernate 5.0, NO Spring Boot)
I have few queries/doubts about this entire application. They are more towards environment/deployment.
Do I need to deploy this whole app as a war in Tomcat(or any server) to instantiate all beans(spring and hibernate)?
If yes, how can I start jobs using CommandLineJobRunner ?
If no, I will have to manually instantiate beans in main method using ClassPathXmlApplicationContext. In this case how should I execute jobs ? Do I need to create jar(is this mandatory) ?
How can I test these jobs on command line ? Do I need to pass jars(spring , hibernate etc dependencies) while using CommandLineJobRunner to execute jobs ?
I am new to batch jobs and all your comments would be of great help.
Thanks
No server is needed for spring batch applications.
You can launch job using jobLauncher bean . below is sample code.
public class MyJobLauncher {
public static void main(String[] args) {
GenericApplicationContext context = new AnnotationConfigApplicationContext(MyBatchConfiguration.class);
JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
Job job = (Job) context.getBean("myJobName");//this is bean name of your job
JobExecution execution = jobLauncher.run(job, jobParameters);
}
}
You will need to create jar. Also all other jar that are needed are also required. You can use maven maven assembly plugin for this.
There is two way to close the Spring ApplicationContext:
Solution 1:
ApplicationContext context = new ClassPathXmlApplicationContext(
"spring/application-context.xml");
// Application core ...
((AbstractApplicationContext) context).close();
Solution 2:
ApplicationContext context = new ClassPathXmlApplicationContext(
"spring/application-context.xml");
((AbstractApplicationContext) context).registerShutdownHook();
// Application core ...
What's the difference between this 2 solutions and what's the best in terms of performance ?
Solution 1 shuts down the application context
Solution 2 registers a callback, so spring will shut down, if the JVM is shut down, from javadoc :
Register a shutdown hook with the JVM runtime, closing this context on JVM shutdown unless it has already been closed at that time.
So both are two diffrent things, normally you will call registerShutdownHook() directly after you created the appication context. So when your user terminates the JVM, spring will be called and shuts itself down.
You should call close() when your application ends, to allow spring to destroy its beans.