I would like to find a way to find the process id of a Jenkins job, so I can kill the process if the job gets hung. The Jenkins instance is on Ubuntu. Sometimes, we are unable to stop a job via the Jenkins interface. I am able to stop a job by killing the process id if I run a Jenkins job that contains a simple shell script where I manually collect the process id such as:
#!/bin/bash
echo "Process ID: $$"
for i in {1..10000}
do
sleep 10;
echo "Welcome $i times"
done
In the command shell, I can run sudo kill -9 [process id]and it successfully kills the job.
The problem is, most of our jobs have multiple build steps and we have multiple projects running on this server. Many of our build steps are shell scripts, windows batch files, and a few of them are ant scripts. I'm wondering how to find the process id of the Jenkins job which is the parent process of all of the build steps. As of now, I have to wait until all other builds have completed and restart the server. Thanks for any help!
On *nix OS you can review environment variables of a running process by investigating a /proc/$pid/environ and look for Jenkins specific variables like BUILD_ID, BUILD_URL, etc.
cat /proc/'$pid'/environ | grep BUILD_URL
You can do it know you $pid or go through of running processes.
This is an update to my question. For killing hung (zombie) jobs, I believe that this will only work for cases where Jenkins is running from the same server as its jobs. I doubt this would work if you are trying to kill a hung process running on a Jenkins slave.
#FIND THE PROCESS ID BASED ON JENKINS JOB
user#ubuntu01x64:~$ sudo egrep -l -i 'BUILD_TAG=jenkins-Wait_Job-11' /proc/*/environ
/proc/5222/environ
/proc/6173/environ
/proc/self/environ
# ONE OF THE PROCESSES LISTED FROM THE EGREP OUTPUT IS THE 'EGREP'COMMAND ITSELF,
# ENSURE THAT (LOOP THROUGH) THE PROCESS ID'S TO DETERMINE WHICH IS
# STILL RUNNING
user#ubuntu01x64:~$ if [[ -e /proc/6173 ]]; then echo "yes"; fi
user#ubuntu01x64:~$ if [[ -e /proc/5222 ]]; then echo "yes"; fi
yes
# KILL THE PROCESS
sudo kill -9 5222
Related
I need to run rsync in background through shell script but once it has started, I need to monitor the status of that jobs through shell.
jobs command return empty when its run in shell after the script exits. ps -ef | grep rsync shows that the rsync is still running.
I can check the status through script but I need to run the script multiple times so it uses a different ip.txt file to push. So I can't have the script running to check jobs status.
Here is the script:
for i in `cat $ip.txt`; do
rsync -avzh $directory/ user#"$i":/cygdrive/c/test/$directory 2>&1 > /dev/null &
done
jobs; #shows the jobs status while in the shell script.
exit 1
Output of jobs command is empty after the shell script exits:
root#host001:~# jobs
root#host001:~#
What could be the reason and how could I get the status of jobs while the rsync is running in background? I can't find an article online related to this.
Since your shell (the one from which you execute jobs) did not start rsync, it doesn't know anything about it. There are different approaches to fixing that, but it boils down to starting the background process from your shell. For example, you can start the script you have using the source BASH command instead of executing it in a separate process. Of course, you'd have to remove the exit 1 at the end, because that exits your shell otherwise.
I'd like to have a Jenkins job which kills all processes on port 5000 (bash).
The easy solution
fuser -k 5000/tcp
works fine when I execute this command in the terminal, but on Jenkins ("execute shell") marks build as failure.
I have tried also
kill $(lsof -i -t:5000)
but again, as it works on regular terminal, on Jenkins I get
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Build step 'Execute shell' marked build as failure
Any ideas how to fix this?
I hadd the same problem. It did not work when the process was not running. bash just did it, but jenkins failed.
You can add an || true to your jenkins job to indicate jenkins to proceed with the job if the bash command fails.
So its:
fuser -k 5000/tcp || true
see also don't fail jenkins build if execute shell fails
Try put the command with the path
/usr/bin/kill $(/usr/sbin/lsof -i -t:5000)
If the user running the jenkins service is not the same as the user with the process on port 5000 you won't be able to kill the process. Maybe you will need to run this with sudo.
Try this
su -s jenkins #Or the user who run jenkins
/usr/bin/kill $(/usr/sbin/lsof -i -t:5000)
maybe jenkins user can't see the processes because of privileges so the expansion of $(lsof ..) is empty.
the error output may not be complete because if lsof call fails there will be a message on stderr.
The problem is that $ is a special char in jenkins commands. It means you are referring to an ENV VAR.
You should try writing the command wrapped with single quotes.
I was facing the same issue and in many cases standard input/output is disabled (specially when you have ssh to the target machine). What you can do is create a executable shell file in the target server and execute that file.
So, the step would look something like below :
Step 1 . -> create the shell file
cat > kill_process.sh << EOF
target_port_num=\`lsof -i:\${1} -t\`;
echo "Kill process at port is :: \${target_port_num}"
kill -9 \${target_port_num}
EOF
Step 2 . -> make it executable
chmod +x process_killer.sh
Step 3 . -> execute the shell and pass the port number
./process_killer.sh 3005
Hope this help.
In Start Script when Gnome Starts Up it was asked how to automatically start a script on gnome login. But how to automatically stop a long running script on logout, that was started on login? In my case there are two processes when I login twice. Interestingly the process started first does not reside under gnome-session anymore.
I would wrap the binary that gets executed in a simple bash script that saves the pid of the started process in a temporary file. If this file already exists it skips the start of the application. Since the file is saved in the /tmp directory everything gets deleted once you restart your computer.
#!/bin/bash
binary="git-cola"
temp_file="/tmp/my_${binary}_instance.pid"
if [[ -f ${temp_file} ]]
then
echo "PID exists"
else
exec ${binary} &
echo $! > ${temp_file}
fi
With a little more effort you can check if the pid of the process is still running and restart it on the login again (for example if the process crashed or the other user closed it).
I actually don't use Gnome, so I can't tell you if there is a more elegant way to kill the process. Like a logout hook. But once you got the pid of the process saved you can kill it with kill -9 PID. (See man kill for more gentle ways to end the process).
This might not be the solution to stop the process. But to prevent it starting twice.
how can I kill hudson job from bash script when the log file doesn't change? (hudson is freezed).
Context: I have a bash script that check if a log file had change after X seconds and I want to modified it to check that if the timeout raises, and there's no error in console, this means that hudson job is freezed, so I want to be notified about this.
It might be easier to use the Build Timeout plugin.
Finally the solution was to use the following command:
#!/bin/bash
#if the log file does not change
if [ "something" ]; then
kill -9 $(pidof eclipse)
fi
This kills the eclipse instance (who's calls hudson), and continues with the build of the others elements and that it's Ok for my task.
I want a way to write a daemon in a shell script, which runs another application in a loop, restarting it if it dies.
When run using ./myscript.sh from an SSH session, it shall launch a new instance of the daemon, except if the daemon is already running.
When the SSH session ends, the daemon shall persist.
There shall be a parameter (./myscript -stop) that kills any existing daemon.
(Notes on edit - The original question specified that nohup and similar tools may not be used. This artificial requirement was an "XY question", and the accepted answer in fact uses all the tools the OP claimed were not possible to use.)
Based on clarifications in comments, what you actually want is a daemon process that keeps a child running, relaunching it whenever it exits. You want a way to type "./myscript.sh" in an ssh session and have the daemon started.
#!/usr/bin/env bash
PIDFILE=~/.mydaemon.pid
if [ x"$1" = x-daemon ]; then
if test -f "$PIDFILE"; then exit; fi
echo $$ > "$PIDFILE"
trap "rm '$PIDFILE'" EXIT SIGTERM
while true; do
#launch your app here
/usr/bin/server-or-whatever &
wait # needed for trap to work
done
elif [ x"$1" = x-stop ]; then
kill `cat "$PIDFILE"`
else
nohup "$0" -daemon
fi
Run the script: it will launch the daemon process for you with nohup. The daemon process is a loop that watches for the child to exit, and relaunches it when it does.
To control the daemon, there's a -stop argument the script can take that will kill the daemon. Look at examples in your system's init scripts for more complete examples with better error checking.
The pid of the most recently "backgrounded" process is stored in $!
$ cat &
[1] 7057
$ echo $!
7057
I am unaware of a fork command in bash. Are you sure bash is the right tool for this job?