I wrote a script to run constantly on startup. If for whatever reason the script were to fail, I wrote a second script to check if it has failed, and if so, run the first script again. I then set this second script as a cronjob to run every minute so that it is constantly checking if the first script is alive.
So to test this, I reboot my system. I can see in htop that the first script is running from start up as expected. Good. I kill the process to test the second script. Sure enough, the second script starts the first script again. Still good. I then kill this process, but the second script won't run again now. It still updates a txt file when I manually start the first script, but the second script just doesn't start the first script like it's supposed to. Is it because I killed the cronjob? Restarting the cron service doesn't fix anything though, so I don't know why my second script isn't running again at all.
First script:
#!/bin/bash
stamp=$(date +%Y%m%d-%H%M)
timeout 10d tcpdump -i eth0 -s 96 -z gzip -C 10 -w /home/user/Documents/${stamp}
Second script:
#!/bin/bash
echo "not running" > /home/working.txt
if (( $(ps -ef | grep -v grep | grep tcpdump.sh | wc -l) > 0 ))
then
echo "tcpdump is running!!!" > /home/working.txt
else
/usr/local/bin/tcpdump.sh start
fi
Any help?
You would probably be better off running a simple for loop as the main script, and that kicks off the tcpdump script in the background, so something like:
#!/bin/bash
while true; do
if ps -ef | grep -v grep | grep -q tcpdump; then
: tcpdump running OK
else
# tcpdump not running - start it off
nohup /usr/local/bin/firstscript.sh start &
fi
sleep 30
done
This checks that "tcpdump.sh" is in the output of the "ps -ef" command - if it is, then do nothing (note that you must have an actual command between the "then" and "else" - the ":" command, which just takes it s arguments and ignores them, is sufficient). If it isn't running, start the first script in the background. Then sleep 30 seconds and check again. (Yes, I could have inverted the test so that I didn't need an empty "then" arm, but it would have made the code less obvious)
You put this script as the one which starts at boot time.
Edit: Do you really want to check for "tcpdump.sh"? Is that what the first script is actually called? Assuming that you actually want to check for the tcpdump program, you could use:
if pgrep tcpdump; then
I want to check if there are any errors with the last command, hence redirecting stderr to a file and checking the file for "error" string.(Only one possible error in this case.)
My script looks like below:
#aquire lock
rm -f /some/path/err.out
MyProgramme 2>/some/path/err.out &
if grep -i "error" /some/path/err.out ; then
echo "ERROR while running MyProgramme, check /some/path/err.out for error(s)"
#release lock
exit 1
fi
'if' condition is giving error 'No such file or directory' on err.out, however I can see the file exists.
Did I miss anything ?.. Any help is appreciated. Thanks!
PS: I couldn't check the exit code using $? as it is running in background.
In addition to the file possibly not existing when you call grep, you only call grep once, and it only sees whatever data is currently in the file. grep will not continue reading from the file when it reaches the end, waiting for MyProgramme to complete. Instead, I would recommend using a named pipe as the input to grep. This will cause grep to continue reading from the pipe until MyProgramme does, in fact, complete.
#aquire lock
rm -f /some/path/err.out
p=/some/path/err.out
mkfifo "$p"
MyProgramme 2> "$p" &
if grep -i "error" "$p" ; then
echo "ERROR while running MyProgramme, check /some/path/err.out for error(s)"
#release lock
exit
fi
When you start MyProgramme in the background, it's possible that grep executes before MyProgramme could write (and thus create) to the file /some/path/err.out. That's why even though the file exists later when you check it yourself, grep couldn't find it.
You can wait until the background job completes using wait before inspecting the file using grep.
I'm currently creating a lock folder which is created when my script runs, I also move files into sub folders here for processing. When the script ends a TRAP is called which removes the lock folder and contents, all of which is working fine. We had an issue the other day when someone pulled the power from one of the servers so my TRAP was never called so when re-booted the lock folder was still there which meant my scripts couldn't re-start until they were manually removed. What's the best way of checking if the script is already running ? I currently have this approach using process id's:
if ! mkdir $LOCK_DIR 2>/dev/null; then # Try to create the lock dir. This should pass successfully first run.
# If the lock dir exists
pid=$(cat $LOCK_DIR/pid.txt)
if [[ $(ps -ef | awk '{print $2}' | grep $pid | grep -v grep | wc -l) == 1 ]]; then
echo "Script is already running"
exit 1
else
echo "It looks like the previous script was killed. Restarting process."
# Do some cleanup here before removing dir and re-starting process.
fi
fi
# Create a file in the lock dir containing the pid. Echo the current process id into the file.
touch $LOCK_DIR/pid.txt
echo $$ > $LOCK_DIR/pid.txt
# Rest of script below
Checking /proc/ and cmdline is a good call - especially as at the moment you are simply checking that there isn't a process with the process id and not if the process is actually your script.
You could still do this with your ps command - which would offer some form of platform agnosticism.
COMMAND=$(ps -o comm= -p $pid)
if [[ $COMMAND == my_process ]]
then
.....
Note the command line arguments to ps limit it to command only with no header.
Many systems nowadays use tmpfs for directories like /tmp. These directories will therefore always be cleared after a reboot.
If using your pid file, note you can easily see the command
running under that pid in /proc/$pid/cmdline and /proc/$pid/exe.
I have a bash script I want to run every 5 minutes from cron... but there's a chance the previous run of the script isn't done yet... in this case, i want the new run to just exit. I don't want to rely on just a lock file in /tmp.. I want to make sure sure the process is actually running before i honor the lock file (or whatever)...
Here is what I have stolen from the internet so far... how do i smarten it up a bit? or is there a completely different way that's better?
if [ -f /tmp/mylockFile ] ; then
echo 'Script is still running'
else
echo 1 > /tmp/mylockFile
/* Do some stuff */
rm -f /tmp/mylockFile
fi
# Use a lockfile containing the pid of the running process
# If script crashes and leaves lockfile around, it will have a different pid so
# will not prevent script running again.
#
lf=/tmp/pidLockFile
# create empty lock file if none exists
cat /dev/null >> $lf
read lastPID < $lf
# if lastPID is not null and a process with that pid exists , exit
[ ! -z "$lastPID" -a -d /proc/$lastPID ] && exit
echo not running
# save my pid in the lock file
echo $$ > $lf
# sleep just to make testing easier
sleep 5
There is at least one race condition in this script. Don't use it for a life support system, lol. But it should work fine for your example, because your environment doesn't start two scripts simultaneously. There are lots of ways to use more atomic locks, but they generally depend on having a particular thing optionally installed, or work differently on NFS, etc...
You might want to have a look at the man page for the flock command, if you're lucky enough to get it on your distribution.
NAME
flock - Manage locks from shell scripts
SYNOPSIS
flock [-sxon] [-w timeout] lockfile [-c] command...
Never use a lock file always use a lock directory.
In your specific case, it's not so important because the start of the script is scheduled in 5min intervals. But if you ever reuse this code for a webserver cgi-script you are toast.
if mkdir /tmp/my_lock_dir 2>/dev/null
then
echo "running now the script"
sleep 10
rmdir /tmp/my_lock_dir
fi
This has a problem if you have a stale lock, means the lock is there but no associated process. Your cron will never run.
Why use a directory? Because mkdir is an atomic operation. Only one process at a time can create a directory, all other processes get an error. This even works across shared filesystems and probably even between different OS types.
Store your pid in mylockFile. When you need to check, look up ps for the process with the pid you read from file. If it exists, your script is running.
If you want to check the process's existence, just look at the output of
ps aux | grep your_script_name
If it's there, it's not dead...
As pointed out in the comments and other answers, using the PID stored in the lockfile is much safer and is the standard approach most apps take. I just do this because it's convenient and I almost never see the corner cases (e.g. editing the file when the cron executes) in practice.
If you use a lockfile, you should make sure that the lockfile is always removed. You can do this with 'trap':
if ( set -o noclobber; echo "locked" > "$lockfile") 2> /dev/null; then
trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
echo "Locking succeeded" >&2
rm -f "$lockfile"
else
echo "Lock failed - exit" >&2
exit 1
fi
The noclobber option makes the creation of lockfile atomic, like using a directory.
As a one-liner and if you do not want to use a lockfile (e.g. b/c/ of a read only filesystem, etc)
test "$(pidof -x $(basename $0))" != $$ && exit
It checks that the full list of PID that bear the name of your script is equal to the current PID. The "-x" also checks for the name of shell scripts.
Bash makes it even shorter and faster:
[[ "$(pidof -x $(basename $0))" != $$ ]] && exit
In some cases, you might want to be able to distinguish between who is running the script and allow some concurrency but not all. In that case, you can use per-user, per-tty or cron-specific locks.
You can use environment variables such as $USER or the output of a program such as tty to create the filename. For cron, you can set a variable in the crontab file and test for it in your script.
you can use this one:
pgrep -f "/bin/\w*sh .*scriptname" | grep -vq $$ && exit
I was trying to solve this problem today and I came up with the below:
COMMAND_LINE="$0 $*"
JOBS=$(SUBSHELL_PID=$BASHPID; ps axo pid,command | grep "${COMMAND_LINE}" | grep -v $$ | g rep -v ${SUBSHELL_PID} | grep -v grep)
if [[ -z "${JOBS}" ]]
then
# not already running
else
# already running
fi
This relies on $BASHPID which contains the PID inside a subshell ($$ in the subshell is the parent pid). However, this relies on Bash v4 and I needed to run this on OSX which has Bash v3.2.48. I ultimately came up with another solution and it is cleaner:
JOBS=$(sh -c "ps axo pid,command | grep \"${COMMAND_LINE}\" | grep -v grep | grep -v $$")
You can always just:
if ps -e -o cmd | grep scriptname > /dev/null; then
exit
fi
But I like the lockfile myself, so I wouldn't do this without the lock file as well.
Since a socket solution has not yet been mentioned it is worth pointing out that sockets can be used as effective mutexes. Socket creation is an atomic operation, like mkdir is as Gunstick pointed out, so a socket is suitable to use as a lock or mutex.
Tim Kay's Perl script 'Solo' is a very small and effective script to make sure only one copy of a script can be run at any one time. It was designed specifically for use with cron jobs, although it works perfectly for other tasks as well and I've used it for non-crob jobs very effectively.
Solo has one advantage over the other techniques mentioned so far in that the check is done outside of the script you only want to run one copy of. If the script is already running then a second instance of that script will never even be started. This is as opposed to isolating a block of code inside the script which is protected by a lock. EDIT: If flock is used in a cron job, rather than from inside a script, then you can also use that to prevent a second instance of the script from starting - see example below.
Here's an example of how you might use it with cron:
*/5 * * * * solo -port=3801 /path/to/script.sh args args args
# "/path/to/script.sh args args args" is only called if no other instance of
# "/path/to/script.sh" is running, or more accurately if the socket on port 3801
# is not open. Distinct port numbers can be used for different programs so that
# if script_1.sh is running it does not prevent script_2.sh from starting, I've
# used the port range 3801 to 3810 without conflicts. For Linux non-root users
# the valid port range is 1024 to 65535 (0 to 1023 are reserved for root).
* * * * * solo -port=3802 /path/to/script_1.sh
* * * * * solo -port=3803 /path/to/script_2.sh
# Flock can also be used in cron jobs with a distinct lock path for different
# programs, in the example below script_3.sh will only be started if the one
# started a minute earlier has already finished.
* * * * * flock -n /tmp/path.to.lock -c /path/to/script_3.sh
Links:
Solo web page: http://timkay.com/solo/
Solo script: http://timkay.com/solo/solo
Hope this helps.
You can use this.
I'll just shamelessly copy-paste the solution here, as it is an answer for both questions (I would argue that it's actually a better fit for this question).
Usage
include sh_lock_functions.sh
init using sh_lock_init
lock using sh_acquire_lock
check lock using sh_check_lock
unlock using sh_remove_lock
Script File
sh_lock_functions.sh
#!/bin/bash
function sh_lock_init {
sh_lock_scriptName=$(basename $0)
sh_lock_dir="/tmp/${sh_lock_scriptName}.lock" #lock directory
sh_lock_file="${sh_lock_dir}/lockPid.txt" #lock file
}
function sh_acquire_lock {
if mkdir $sh_lock_dir 2>/dev/null; then #check for lock
echo "$sh_lock_scriptName lock acquired successfully.">&2
touch $sh_lock_file
echo $$ > $sh_lock_file # set current pid in lockFile
return 0
else
touch $sh_lock_file
read sh_lock_lastPID < $sh_lock_file
if [ ! -z "$sh_lock_lastPID" -a -d /proc/$sh_lock_lastPID ]; then # if lastPID is not null and a process with that pid exists
echo "$sh_lock_scriptName is already running.">&2
return 1
else
echo "$sh_lock_scriptName stopped during execution, reacquiring lock.">&2
echo $$ > $sh_lock_file # set current pid in lockFile
return 2
fi
fi
return 0
}
function sh_check_lock {
[[ ! -f $sh_lock_file ]] && echo "$sh_lock_scriptName lock file removed.">&2 && return 1
read sh_lock_lastPID < $sh_lock_file
[[ $sh_lock_lastPID -ne $$ ]] && echo "$sh_lock_scriptName lock file pid has changed.">&2 && return 2
echo "$sh_lock_scriptName lock still in place.">&2
return 0
}
function sh_remove_lock {
rm -r $sh_lock_dir
}
Usage example
sh_lock_usage_example.sh
#!/bin/bash
. /path/to/sh_lock_functions.sh # load sh lock functions
sh_lock_init || exit $?
sh_acquire_lock
lockStatus=$?
[[ $lockStatus -eq 1 ]] && exit $lockStatus
[[ $lockStatus -eq 2 ]] && echo "lock is set, do some resume from crash procedures";
#monitoring example
cnt=0
while sh_check_lock # loop while lock is in place
do
echo "$sh_scriptName running (pid $$)"
sleep 1
let cnt++
[[ $cnt -gt 5 ]] && break
done
#remove lock when process finished
sh_remove_lock || exit $?
exit 0
Features
Uses a combination of file, directory and process id to lock to make sure that the process is not already running
You can detect if the script stopped before lock removal (eg. process kill, shutdown, error etc.)
You can check the lock file, and use it to trigger a process shutdown when the lock is missing
Verbose, outputs error messages for easier debug
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Quick-and-dirty way to ensure only one instance of a shell script is running at a time
At a previous workplace we used to have a highly-refined bash function called run-only-once that we could write into any long-running shell script, and then call it at the start of the script, which would check to see whether the script was already running as another process, and if so exit with a notification to STDOUT.
Does anyone have a function/fragment like this they could share?
Our old function (which I no longer have) would check for a PID file (in scriptname.$$ format) in /var/run, and use then either exit or simply continue. In the case where a PID file existed, it would do some checks to make sure that the process was still active. It also had a few options for controlling whether a notification was output at all.
From memory, our function only worked in bash. Bonus points for a /bin/sh version.
Put this at the start of the script
SCRIPTNAME=`basename $0`
PIDFILE=/var/run/${SCRIPTNAME}.pid
if [ -f ${PIDFILE} ]; then
#verify if the process is actually still running under this pid
OLDPID=`cat ${PIDFILE}`
RESULT=`ps -ef | grep ${OLDPID} | grep ${SCRIPTNAME}`
if [ -n "${RESULT}" ]; then
echo "Script already running! Exiting"
exit 255
fi
fi
#grab pid of this process and update the pid file with it
PID=`ps -ef | grep ${SCRIPTNAME} | head -n1 | awk ' {print $2;} '`
echo ${PID} > ${PIDFILE}
and at the end
if [ -f ${PIDFILE} ]; then
rm ${PIDFILE}
fi
This first of all checks for the existence of the pid file and exits if it's present. If so then it confirms that a process under this script name with the old pid is running and exits if so. If not then it carries on and updates the script with the new pid file. The bit at the end checks for the existence of the pid file and deletes it, so the script can run next time.
Check permissions on /var/run are OK for your script though, otherwise create the PID file in another directory. Same directory as the script runs in would be fine.
There are two ways to do atomic locks from the shell. The simplest and most portable is mkdir. Creation of a directory will always fail if a file/directory by that name already exists, so simply use a "lock directory" instead of "lock file".
mkdir "$LOCK" || { echo "Script already running" ; exit 1 ; }
The other method is the noclobber option, set using set -C. This option forces the shell to open files for output redirection with the O_EXCL flag. Use it like this:
set -C
> "$LOCK" || { echo "Script already running" ; exit 1 ; }
set +C
Rather than redirecting a null command, you might prefer to echo the current PID or something else useful to be stored in the file.
It's worth keeping in mind that some ancient historic shells have broken implementations of the noclobber option -C which do not use O_EXCL but instead buggy non-atomic checks for file existence. Probably not an issue in the 21st century, but you should be aware just in case.
It is more reliable to use the lockfile utility in Linux.
From the man page:
...
lockfile important.lock
...
access_"important"_to_your_hearts_content
...
rm -f important.lock
...
You can specify retries and wait times. This was specifically created for this purpose so it takes care of race conditions