In the Spring Boot Document, they said that 'Each SpringApplication will register a shutdown hook with the JVM to ensure that the ApplicationContext is closed gracefully on exit.'
When I click ctrl+c on the shell command, the application can be shutdown gracefully. If I run the application in a production machine, I have to use the command
java -jar ProApplicaton.jar. But I can't close the shell terminal, otherwise it will close the process.
If I run command like nohup java -jar ProApplicaton.jar &, I can't use ctrl+c to shutdown it gracefully.
What is the correct way to start and stop a Spring Boot Application in the production environment?
If you are using the actuator module, you can shutdown the application via JMX or HTTP if the endpoint is enabled.
add to application.properties:
Spring Boot 2.0 and newer:
management.endpoints.shutdown.enabled=true
Following URL will be available:
/actuator/shutdown - Allows the application to be gracefully shutdown (not enabled by default).
Depending on how an endpoint is exposed, the sensitive parameter may be used as a security hint.
For example, sensitive endpoints will require a username/password when they are accessed over HTTP (or simply disabled if web security is not enabled).
From the Spring boot documentation
Here is another option that does not require you to change the code or exposing a shut-down endpoint. Create the following scripts and use them to start and stop your app.
start.sh
#!/bin/bash
java -jar myapp.jar & echo $! > ./pid.file &
Starts your app and saves the process id in a file
stop.sh
#!/bin/bash
kill $(cat ./pid.file)
Stops your app using the saved process id
start_silent.sh
#!/bin/bash
nohup ./start.sh > foo.out 2> foo.err < /dev/null &
If you need to start the app using ssh from a remote machine or a CI pipeline then use this script instead to start your app. Using start.sh directly can leave the shell to hang.
After eg. re/deploying your app you can restart it using:
sshpass -p password ssh -oStrictHostKeyChecking=no userName#www.domain.com 'cd /home/user/pathToApp; ./stop.sh; ./start_silent.sh'
As to #Jean-Philippe Bond 's answer ,
here is a maven quick example for maven user to configure HTTP endpoint to shutdown a spring boot web app using spring-boot-starter-actuator so that you can copy and paste:
1.Maven pom.xml:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
2.application.properties:
#No auth protected
endpoints.shutdown.sensitive=false
#Enable shutdown endpoint
endpoints.shutdown.enabled=true
All endpoints are listed here:
3.Send a post method to shutdown the app:
curl -X POST localhost:port/shutdown
Security Note:
if you need the shutdown method auth protected, you may also need
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
configure details:
You can make the springboot application to write the PID into file and you can use the pid file to stop or restart or get the status using a bash script. To write the PID to a file, register a listener to SpringApplication using ApplicationPidFileWriter as shown below :
SpringApplication application = new SpringApplication(Application.class);
application.addListeners(new ApplicationPidFileWriter("./bin/app.pid"));
application.run();
Then write a bash script to run the spring boot application . Reference.
Now you can use the script to start,stop or restart.
All of the answers seem to be missing the fact that you may need to complete some portion of work in coordinated fashion during graceful shutdown (for example, in an enterprise application).
#PreDestroy allows you to execute shutdown code in the individual beans. Something more sophisticated would look like this:
#Component
public class ApplicationShutdown implements ApplicationListener<ContextClosedEvent> {
#Autowired ... //various components and services
#Override
public void onApplicationEvent(ContextClosedEvent event) {
service1.changeHeartBeatMessage(); // allows loadbalancers & clusters to prepare for the impending shutdown
service2.deregisterQueueListeners();
service3.finishProcessingTasksAtHand();
service2.reportFailedTasks();
service4.gracefullyShutdownNativeSystemProcessesThatMayHaveBeenLaunched();
service1.eventLogGracefulShutdownComplete();
}
}
Use the static exit() method in the SpringApplication class for closing your spring boot application gracefully.
public class SomeClass {
#Autowired
private ApplicationContext context
public void close() {
SpringApplication.exit(context);
}
}
As of Spring Boot 2.3 and later, there's a built-in graceful shutdown mechanism.
Pre-Spring Boot 2.3, there is no out-of-the box graceful shutdown mechanism.
Some spring-boot starters provide this functionality:
https://github.com/jihor/hiatus-spring-boot
https://github.com/gesellix/graceful-shutdown-spring-boot
https://github.com/corentin59/spring-boot-graceful-shutdown
I am the author of nr. 1. The starter is named "Hiatus for Spring Boot". It works on the load balancer level, i.e. simply marks the service as OUT_OF_SERVICE, not interfering with application context in any way. This allows to do a graceful shutdown and means that, if required, the service can be taken out of service for some time and then brought back to life. The downside is that it doesn't stop the JVM, you will have to do it with kill command. As I run everything in containers, this was no big deal for me, because I will have to stop and remove the container anyway.
Nos. 2 and 3 are more or less based on this post by Andy Wilkinson. They work one-way - once triggered, they eventually close the context.
I don't expose any endpoints and start (with nohup in background and without out files created through nohup) and stop with shell script(with KILL PID gracefully and force kill if app is still running after 3 mins). I just create executable jar and use PID file writer to write PID file and store Jar and Pid in folder with same name as of application name and shell scripts also have same name with start and stop in the end. I call these stop script and start script via jenkins pipeline also. No issues so far. Perfectly working for 8 applications(Very generic scripts and easy to apply for any app).
Main Class
#SpringBootApplication
public class MyApplication {
public static final void main(String[] args) {
SpringApplicationBuilder app = new SpringApplicationBuilder(MyApplication.class);
app.build().addListeners(new ApplicationPidFileWriter());
app.run();
}
}
YML FILE
spring.pid.fail-on-write-error: true
spring.pid.file: /server-path-with-folder-as-app-name-for-ID/appName/appName.pid
Here is the start script(start-appname.sh):
#Active Profile(YAML)
ACTIVE_PROFILE="preprod"
# JVM Parameters and Spring boot initialization parameters
JVM_PARAM="-Xms512m -Xmx1024m -Dspring.profiles.active=${ACTIVE_PROFILE} -Dcom.webmethods.jms.clientIDSharing=true"
# Base Folder Path like "/folder/packages"
CURRENT_DIR=$(readlink -f "$0")
BASE_PACKAGE="${CURRENT_DIR%/bin/*}"
# Shell Script file name after removing path like "start-yaml-validator.sh"
SHELL_SCRIPT_FILE_NAME=$(basename -- "$0")
# Shell Script file name after removing extension like "start-yaml-validator"
SHELL_SCRIPT_FILE_NAME_WITHOUT_EXT="${SHELL_SCRIPT_FILE_NAME%.sh}"
# App name after removing start/stop strings like "yaml-validator"
APP_NAME=${SHELL_SCRIPT_FILE_NAME_WITHOUT_EXT#start-}
PIDS=`ps aux |grep [j]ava.*-Dspring.profiles.active=$ACTIVE_PROFILE.*$APP_NAME.*jar | awk {'print $2'}`
if [ -z "$PIDS" ]; then
echo "No instances of $APP_NAME with profile:$ACTIVE_PROFILE is running..." 1>&2
else
for PROCESS_ID in $PIDS; do
echo "Please stop the process($PROCESS_ID) using the shell script: stop-$APP_NAME.sh"
done
exit 1
fi
# Preparing the java home path for execution
JAVA_EXEC='/usr/bin/java'
# Java Executable - Jar Path Obtained from latest file in directory
JAVA_APP=$(ls -t $BASE_PACKAGE/apps/$APP_NAME/$APP_NAME*.jar | head -n1)
# To execute the application.
FINAL_EXEC="$JAVA_EXEC $JVM_PARAM -jar $JAVA_APP"
# Making executable command using tilde symbol and running completely detached from terminal
`nohup $FINAL_EXEC </dev/null >/dev/null 2>&1 &`
echo "$APP_NAME start script is completed."
Here is the stop script(stop-appname.sh):
#Active Profile(YAML)
ACTIVE_PROFILE="preprod"
#Base Folder Path like "/folder/packages"
CURRENT_DIR=$(readlink -f "$0")
BASE_PACKAGE="${CURRENT_DIR%/bin/*}"
# Shell Script file name after removing path like "start-yaml-validator.sh"
SHELL_SCRIPT_FILE_NAME=$(basename -- "$0")
# Shell Script file name after removing extension like "start-yaml-validator"
SHELL_SCRIPT_FILE_NAME_WITHOUT_EXT="${SHELL_SCRIPT_FILE_NAME%.*}"
# App name after removing start/stop strings like "yaml-validator"
APP_NAME=${SHELL_SCRIPT_FILE_NAME_WITHOUT_EXT:5}
# Script to stop the application
PID_PATH="$BASE_PACKAGE/config/$APP_NAME/$APP_NAME.pid"
if [ ! -f "$PID_PATH" ]; then
echo "Process Id FilePath($PID_PATH) Not found"
else
PROCESS_ID=`cat $PID_PATH`
if [ ! -e /proc/$PROCESS_ID -a /proc/$PROCESS_ID/exe ]; then
echo "$APP_NAME was not running with PROCESS_ID:$PROCESS_ID.";
else
kill $PROCESS_ID;
echo "Gracefully stopping $APP_NAME with PROCESS_ID:$PROCESS_ID..."
sleep 5s
fi
fi
PIDS=`/bin/ps aux |/bin/grep [j]ava.*-Dspring.profiles.active=$ACTIVE_PROFILE.*$APP_NAME.*jar | /bin/awk {'print $2'}`
if [ -z "$PIDS" ]; then
echo "All instances of $APP_NAME with profile:$ACTIVE_PROFILE has has been successfully stopped now..." 1>&2
else
for PROCESS_ID in $PIDS; do
counter=1
until [ $counter -gt 150 ]
do
if ps -p $PROCESS_ID > /dev/null; then
echo "Waiting for the process($PROCESS_ID) to finish on it's own for $(( 300 - $(( $counter*5)) ))seconds..."
sleep 2s
((counter++))
else
echo "$APP_NAME with PROCESS_ID:$PROCESS_ID is stopped now.."
exit 0;
fi
done
echo "Forcefully Killing $APP_NAME with PROCESS_ID:$PROCESS_ID."
kill -9 $PROCESS_ID
done
fi
Spring Boot provided several application listener while try to create application context one of them is ApplicationFailedEvent. We can use to know weather the application context initialized or not.
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.context.event.ApplicationFailedEvent;
import org.springframework.context.ApplicationListener;
public class ApplicationErrorListener implements
ApplicationListener<ApplicationFailedEvent> {
private static final Logger LOGGER =
LoggerFactory.getLogger(ApplicationErrorListener.class);
#Override
public void onApplicationEvent(ApplicationFailedEvent event) {
if (event.getException() != null) {
LOGGER.info("!!!!!!Looks like something not working as
expected so stoping application.!!!!!!");
event.getApplicationContext().close();
System.exit(-1);
}
}
}
Add to above listener class to SpringApplication.
new SpringApplicationBuilder(Application.class)
.listeners(new ApplicationErrorListener())
.run(args);
SpringApplication implicitly registers a shutdown hook with the JVM to ensure that ApplicationContext is closed gracefully on exit. That will also call all bean methods annotated with #PreDestroy. That means we don't have to explicitly use the registerShutdownHook() method of a ConfigurableApplicationContext in a boot application, like we have to do in spring core application.
#SpringBootConfiguration
public class ExampleMain {
#Bean
MyBean myBean() {
return new MyBean();
}
public static void main(String[] args) {
ApplicationContext context = SpringApplication.run(ExampleMain.class, args);
MyBean myBean = context.getBean(MyBean.class);
myBean.doSomething();
//no need to call context.registerShutdownHook();
}
private static class MyBean {
#PostConstruct
public void init() {
System.out.println("init");
}
public void doSomething() {
System.out.println("in doSomething()");
}
#PreDestroy
public void destroy() {
System.out.println("destroy");
}
}
}
Spring Boot now supports graceful shut down (currently in pre-release versions, 2.3.0.BUILD-SNAPSHOT)
When enabled, shutdown of the application will include a grace period
of configurable duration. During this grace period, existing requests
will be allowed to complete but no new requests will be permitted
You can enable it with:
server.shutdown.grace-period=30s
https://docs.spring.io/spring-boot/docs/2.3.0.BUILD-SNAPSHOT/reference/html/spring-boot-features.html#boot-features-graceful-shutdown
They are many ways to shutdown a spring application. One is to call close() on the ApplicationContext:
ApplicationContext ctx =
SpringApplication.run(HelloWorldApplication.class, args);
// ...
ctx.close()
Your question suggest you want to close your application by doing Ctrl+C, that is frequently used to terminate a command. In this case...
Use endpoints.shutdown.enabled=true is not the best recipe. It means you expose an end-point to terminate your application. So, depending on your use case and your environment, you will have to secure it...
A Spring Application Context may have register a shutdown hook with the JVM runtime. See ApplicationContext documentation.
Spring Boot configure this shutdown hook automatically since version 2.3 (see jihor's answer). You may need to register some #PreDestroy methods that will be executed during the graceful shutdown (see Michal's answer).
Ctrl+C should work very well in your case. I assume your issue is caused by the ampersand (&) More explanation:
On Ctrl+C, your shell sends an INT signal to the foreground application. It means "please interrupt your execution". The application can trap this signal and do cleanup before its termination (the hook registered by Spring), or simply ignore it (bad).
nohup is command that execute the following program with a trap to ignore the HUP signal. HUP is used to terminate program when you hang up (close your ssh connexion for example). Moreover it redirects outputs to avoid that your program blocks on a vanished TTY. nohupdoes NOT ignore INT signal. So it does NOT prevent Ctrl+C to work.
I assume your issue is caused by the ampersand (&), not by nohup. Ctrl+C sends a signal to the foreground processes. The ampersand causes your application to be run in background. One solution: do
kill -INT pid
Use kill -9 or kill -KILL is bad because the application (here the JVM) cannot trap it to terminate gracefully.
Another solution is to bring back your application in foreground. Then Ctrl+C will work. Have a look on Bash Job control, more precisely on fg.
I am able to do it on Spring Boot Version >=2.5.3 using these steps.
1. Add following actuator dependency
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
</dependencies>
2. Add these properties in application.properties to do a graceful shutdown
management.endpoint.shutdown.enabled=true
management.endpoints.web.exposure.include=shutdown
server.shutdown=GRACEFUL
3. When you start the application, you should see this in the console
(based on number of endpoints you have exposed)
Exposing 1 endpoint(s) beneath base path '/actuator'
4. To shutdown the application do:
POST: http://localhost:8080/<context-path-if-any>/actuator/shutdown
A lot of the actuator answers are mostly correct. Unfortunately, the configuration and endpoint information has changed so they aren't 100% correct. To enable the actuator, add for Maven
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
</dependencies>
or for Gradle
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
}
For configuration, add the following to the application.properties. This will expose all endpoints in the actuator:
management.endpoints.web.exposure.include=*
management.endpoint.shutdown.enabled=true
To expose just the shutdown endpoint, change to:
management.endpoints.web.exposure.include=shutdown
management.endpoint.shutdown.enabled=true
Finally the shutdown endpoint is not available using GET - only POST. So you have to use something like:
curl -X POST localhost:8080/actuator/shutdown
If you are using maven you could use the Maven App assembler plugin.
The daemon mojo (which embed JSW) will output a shell script with start/stop argument. The stop will shutdown/kill gracefully your Spring application.
The same script can be used to use your maven application as a linux service.
If you are in a linux environment all you have to do is to create a symlink to your .jar file from inside /etc/init.d/
sudo ln -s /path/to/your/myboot-app.jar /etc/init.d/myboot-app
Then you can start the application like any other service
sudo /etc/init.d/myboot-app start
To close the application
sudo /etc/init.d/myboot-app stop
This way, application will not terminate when you exit the terminal. And application will shutdown gracefully with stop command.
For Spring boot web apps, Spring boot provides the out-of-box solution for graceful shutdown from version 2.3.0.RELEASE.
An excerpt from Spring doc
Refer this answer for the Code Snippet
If you are using spring boot version 2.3 n up , There is an in-build way to shutdown app gracefully.Add below in application.properties
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=20s
If you are using lower spring boot version, You can write a custom shutdown hook and handle different beans, how they should shutdown or in which order they should shutdown. Example code below.
#Component
public class AppShutdownHook implements ApplicationListener<ContextClosedEvent> {
private static final Logger logger = LoggerFactory.getLogger(AppShutdownHook.class);
#Override
public void onApplicationEvent(ContextClosedEvent event) {
logger.info("shutdown requested !!!");
try {
//TODO Add logic to shutdown, diff elements of your application
} catch (Exception e) {
logger.error("Exception occcured while shutting down Application:", e);
}
}
}
try to use following command under the server running cmd or bash terminal.
kill $(jobs -p)
Recommendation get from Microservices With Spring Boot And Spring Cloud - Build Resilient And Scalable Microservices book.
Try this : Press ctrl+C
- [INFO]
------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO]
------------------------------------------------------------------------ [INFO] Total time: 04:48 min [INFO] Finished at:
2022-09-07T18:17:35+05:30 [INFO]
------------------------------------------------------------------------
Terminate batch job (Y/N)?
Type Y to terminate
I'm using SpringBoot 2.4.x app with SpringBatch 4.3.x. I've created a simple job.
Where I've FlatFileItemReader which reads from CSV file. I've ImportKafkaItemWriter which writes to Kafka topic. One step where I combines these. I'm using SimpleJobLauncher and I've set ThreadPoolTaskExecutor as TasKExecutor of the JobLauncher. It is working fine as I've expected. But one resilience use case I've which is if I kill the app and then restart the app and trigger the job then it would carry on and finish the remaining job. Unfortunately it is not happening. I did further investigate and found that when I forcibly close the app SpringBatch job repository key tables look like this:
job_execution_id
version
job_instance_id
create_time
start_time
end_time
status
exit_code
exit_message
last_updated
job_configuration_location
1
1
1
2021-06-16 09:32:43
2021-06-16 09:32:43
STARTED
UNKNOWN
2021-06-16 09:32:43
and
step_execution_id
version
step_name
job_execution_id
start_time
end_time
status
commit_count
read_count
filter_count
write_count
read_skip_count
write_skip_count
process_skip_count
rollback_count
exit_code
exit_message
last_updated
1
4
productImportStep
1
2021-06-16 09:32:43
STARTED
3
6
0
6
0
0
0
0
EXECUTING
2021-06-16 09:32:50
If I manually update these tables where I set a valid end_time and status to FAILED then I can restart the job and works absolutely fine. May I know what I need to do so that Spring Batch can update those relevant repositories appropriately and I can avoid this manual steps. I can provide more information about code if needed.
If I manually update these tables where I set a valid end_time and status to FAILED then I can restart the job and works absolutely fine. May I know what I need to do so that Spring Batch can update those relevant repositories appropriately and I can avoid this manual steps
When a job is killed abruptly, Spring Batch won't have a chance to update its status in the Job repository, so the status is stuck at STARTED. Now when the job is restarted, the only information that Spring Batch has is the status in the job repository. By just looking at the status in the database, Spring Batch cannot distinguish between a job that is effectively running and a job that has been killed abruptly (in both cases, the status is STARTED).
The way to go in indeed manually updating the tables to either mark the status as FAILED to be able to restart the job or ABANDONED to abandon it. This is a business decision that you have to make and there is no way to automate it on the framework side. For more details, please refer to the reference documentation here: Aborting a Job.
You can add a faked parameter example Version a counter to increment for every new job execution so you don't have to check for the table database job.
What I mean mvn clean package
Then you try to launch the program like this :
java my-jarfile.jar dest=/tmp/foo Version="0"
java my-jarfile.jar dest=/tmp/foo Version="1"
java my-jarfile.jar dest=/tmp/foo Version="2"
etc ... Or
You Can use jobParameters to launch thé job programatically via jobLauncher and use date paramèter date = new Date().toString() which gives date with New stamp on every New job execution
You can use "JVM Shutdown Hook":
Something like this:
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
if (jobExecution.isRunning()) {
jobExecution.setEndTime(new Date());
jobExecution.setStatus(BatchStatus.FAILED);
jobExecution.setExitStatus(ExitStatus.FAILED);
jobRepository.update(jobExecution);
}
}));
SCDF Composed Task Runner gives us the option to turn on the --increment-instance-enabled. This option creates an artificial run.id parameter, which increments for every run. Therefore the task is unique for Spring Batch and will restart.
The problem with the IdIncrementer is when I mix it with execution without the IdIncrementer. In the event when a task does not finish, I want to resume the Task. The problem I encountered was when the task finishes without the IdIncrementer, I could not start the task again with the IdIncrementer.
I was wondering what would be the best way to restart with the option to resume?
My idea would be to create a new IdResumer, which uses the same run.id as the last execution.
We are run SCDF 2.2.1 on Openshift v3.11.98 and we use CTR 2.1.1.
The steps to reproduce this:
Create a new SCDF Task Definition with the following definition: dummy1:dummy && dummy2: dummy && dummy3: dummy. The dummy app is a docker container, that fails randomly with 50% chance.
Execute the SCDF Task with the --increment-instance-enabled=true and wait for one of the dummy task to fail (restart if needed).
To resume the same failed execution, execute the SCDF Task now --increment-instance-enabled=false. And let it finish successfully (Redo if needed).
Start the SCDF Task again with --increment-instance-enabled=true.
At step 4 the composed task throws the JobInstanceAlreadyCompleteException, even though the --increment-instance-enabled is enabled again.
Caused by:
org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException:
A job instance already exists and is complete for
parameters={-spring.cloud.data.flow.taskappname=composed-task-runner,
-spring.cloud.task.executionid=3190, -spring.datasource.username=testuser, -graph=aaa-stackoverflow-dummy2 && aaa-stackoverflow-dummy3, -spring.cloud.data.flow.platformname=default, -spring.datasource.url=jdbc:postgresql://10.10.10.10:5432/tms_efa?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory¤tSchema=dev,
-spring.datasource.driverClassName=org.postgresql.Driver, -spring.datasource.password=pass1234, -spring.cloud.task.name=aaa-stackoverflow, -dataflowServerUri=https://scdf-dev.company.com:443/ , -increment-instance-enabled=true}. If you want to run this job again, change the parameters.
Is there a better way to resume and restart the task?
I am getting following error while running a workflow in informatica.
Session task instance [worklet.session] : [TM_6775 The master DTM process was unable to connect to the master service process to update the session status with the following message: error message [ERROR: The session run for [Session task instance [worklet.session]] and [ folder id = 206, workflow id = 16042, workflow run id = 65095209, worklet run id = 65095337, task instance id = 13272 ] is not yet registered with this service process.] and error code [1417].]
This error comes randomly for many other sessions, when they are ran through workflow as a whole. However if I "start task" that failed task next time, it runs successfully.
Any help is much appreciated.
Just an idea to try if you use versioning. Check that everthing is checked in correctly. If the mapping, worflow or worklet is checked out then you and informatica will run different versions wich may cause the behaivour to differ when you start it manually.
Infromatica will allways use the checked in version and you will allways use the checked out version.