how can I kill hudson job from bash script when the log file doesn't change? (hudson is freezed).
Context: I have a bash script that check if a log file had change after X seconds and I want to modified it to check that if the timeout raises, and there's no error in console, this means that hudson job is freezed, so I want to be notified about this.
It might be easier to use the Build Timeout plugin.
Finally the solution was to use the following command:
#!/bin/bash
#if the log file does not change
if [ "something" ]; then
kill -9 $(pidof eclipse)
fi
This kills the eclipse instance (who's calls hudson), and continues with the build of the others elements and that it's Ok for my task.
Related
When i try to execute the jobs from terminal it works for one hour without any issue. when i try to execute shell from Jenkins it works for just one minute and stops the execution. The output from Jenkins console output as follows :
Creating folder path in /jenkins/workspace/load_test/scripts/loadtest/loadtest1
PWD is : /jenkins/workspace/load_test/scripts/loadtest
Running /jenkins/workspace/load_test/scripts/loadtest/loadtest1/testRestApi.sh
1495126268
1495129868
3600
Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information
Finished: SUCCESS
Any ideas/ suggestion to make the script run for one hour from Jenkins job ?
Have you tried with BUILD_ID=dontKillMe it is commonly used for daemons. https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller however this should let you run your script
Im working with the LSF, running bsub commands.
I'm implementing the -Ep switch to run a post exec script. This works great until the Job is killed or hits a memory limit, run limit etc.
Is there any way for the job to detect its running out of resource and then run the script? or to force it to run the script even if its been killed?
I guess my other option is running job with a dependency on that job which will run the "post exec" script when it finishes.
Any thoughts?
Kind Regards,
TheBigPeeler
From the documentation, you should be seeing the behaviour that you want.
A post-execution command runs after the job finishes, regardless of
the exit state of the job. Once a post-execution command is associated
with a job, that command runs even if the job fails. You cannot
configure the post-execution command to run only under certain
conditions.
I thought that maybe the interaction with JOB_INCLUDE_POSTEXEC (lsb.params) could account for the difference, but from my test the post-exec still runs in both cases. I used runlimit (bsub -W) to trigger the job kill.
Is it possible that the post exec is running, but exits early?
What version of LSF are you using? (What's the output of mbatchd -V and sbatchd -V)
I would like to find a way to find the process id of a Jenkins job, so I can kill the process if the job gets hung. The Jenkins instance is on Ubuntu. Sometimes, we are unable to stop a job via the Jenkins interface. I am able to stop a job by killing the process id if I run a Jenkins job that contains a simple shell script where I manually collect the process id such as:
#!/bin/bash
echo "Process ID: $$"
for i in {1..10000}
do
sleep 10;
echo "Welcome $i times"
done
In the command shell, I can run sudo kill -9 [process id]and it successfully kills the job.
The problem is, most of our jobs have multiple build steps and we have multiple projects running on this server. Many of our build steps are shell scripts, windows batch files, and a few of them are ant scripts. I'm wondering how to find the process id of the Jenkins job which is the parent process of all of the build steps. As of now, I have to wait until all other builds have completed and restart the server. Thanks for any help!
On *nix OS you can review environment variables of a running process by investigating a /proc/$pid/environ and look for Jenkins specific variables like BUILD_ID, BUILD_URL, etc.
cat /proc/'$pid'/environ | grep BUILD_URL
You can do it know you $pid or go through of running processes.
This is an update to my question. For killing hung (zombie) jobs, I believe that this will only work for cases where Jenkins is running from the same server as its jobs. I doubt this would work if you are trying to kill a hung process running on a Jenkins slave.
#FIND THE PROCESS ID BASED ON JENKINS JOB
user#ubuntu01x64:~$ sudo egrep -l -i 'BUILD_TAG=jenkins-Wait_Job-11' /proc/*/environ
/proc/5222/environ
/proc/6173/environ
/proc/self/environ
# ONE OF THE PROCESSES LISTED FROM THE EGREP OUTPUT IS THE 'EGREP'COMMAND ITSELF,
# ENSURE THAT (LOOP THROUGH) THE PROCESS ID'S TO DETERMINE WHICH IS
# STILL RUNNING
user#ubuntu01x64:~$ if [[ -e /proc/6173 ]]; then echo "yes"; fi
user#ubuntu01x64:~$ if [[ -e /proc/5222 ]]; then echo "yes"; fi
yes
# KILL THE PROCESS
sudo kill -9 5222
I am building an executable jar file using jenkins and copying into a remote server. After the copying, I need to run the jar file in the remote server. I am using SSH Plugin for executing the remote script.
The remote script looks like this:
startServer.sh
pkill -f MyExecutable
nohup java -jar /home/administrator/app/MyExecutable.jar &
Jenkins is able to execute the script file, but it is not stopping the job after the execution. It is still continuing the process and showing the log in jenkins console. This is creating problems since these continuing jobs blocks other jobs to execute.
How can I stop the job once the script is executed.
Finally, I was able to fix the problem. I am posting it for others sake.
I used ssh -f user#server ....
this solved my problem.
ssh -f root#${server} sh /home/administrator/bin/startServer.sh
I ran into a similar issue using the Publish Over SSH Plugin. For some reason Jenkins wasn't stopping after executing the remote script. Ticking the below configuration fixed the problem.
SSH Publishers > Transfers > Advanced > Exec in pty
Hope it helps someone else.
I got the solution for you my friend.
Make sure to add usePty: true in the pipeline that you are using which will enable the execution of sudo commands that require a tty (and possibly help in other scenarios too.)
sshTransfer(
sourceFiles: "target/*.zip",
removePrefix: "target",
remoteDirectory: "'/root/'yyyy-MM-dd",
execTimeout: 300000,
usePty: true,
verbose: true,
execCommand: '''
pkill -f MyExecutable
nohup java -jar /home/administrator/app/MyExecutable.jar &
echo $! >> /tmp/jenkins/jenkins.pid
sleep 1
'''
)
I am on shared hosting and I'm trying to schedule cronjob to run every now and then. Via cPanel I scheduled to execute my script but even though that according to my host support the cronjob runs, the script doesn't seem as doing anything. The cron job command I set via cPanel is:
/bin/sh /home1/myusername/public_html/somefolder/cronjob2.sh
and the cronjob2.sh
#!/bin/bash
/home1/myusername/public_html/somefolder/node_modules/forever/bin/forever stop 0
when via SSH I execute:
/home1/myusername/public_html/somefolder/cronjob2.sh
it stops forever process as needed. From cronjob doesn't do anything.
How can I get this working?
EDIT:
So I've tried:
/bin/sh /home1/username/public_html/somefolder/cronjob2.sh >> /tmp/mylog 2>&1
and mylog entries say:
/usr/bin/env: node: No such file or directory
It seems that forever needs to run node and this cannot be found. How would I possibly fix this?
EDIT2:
Accepted answer at superuser.com. Thank you all for help
https://superuser.com/questions/763261/simple-script-run-via-cronjob-doesnt-work-but-works-from-shell/763288#763288
For cron job lines in a crontab it's not required to specify kind of shell or e.g. of perl.
It's enough, that your script contains
shebang
line.
Therefore you should remove /bin/sh from your cron job line.
Another aspect, that might cause a different behavior of your script by interactive start and by cron daemon start is possible different environment, first of all the PATH variable. Therefore check, if you script is able to be executed in very restricted environment, that is provided by cron daemon. You can determine your cron job environment experimentally by start of temporary cron job, that executes "env" command and writes its output to a file.
Once more aspect: Have you redirected STDOUT and STDERR of the cron job to a log file and read its content to analyze the issue? You can do it as follows:
your_cron_job >/tmp/any_name.log 2>&1
According to what you wrote, when you run your script via SSH, you are using bash, because this line is the first of your script:
#!/bin/bash
However, in the crontab, you are forcing the use of sh instead of bash. Are you sure your script is fully compatible with sh? Otherwise, simply replace /bin/sh with /bin/bash in your cron command and test again.