Related
What's a quick-and-dirty way to make sure that only one instance of a shell script is running at a given time?
Use flock(1) to make an exclusive scoped lock a on file descriptor. This way you can even synchronize different parts of the script.
#!/bin/bash
(
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200 || exit 1
# Do stuff
) 200>/var/lock/.myscript.exclusivelock
This ensures that code between ( and ) is run only by one process at a time and that the process doesn’t wait too long for a lock.
Caveat: this particular command is a part of util-linux. If you run an operating system other than Linux, it may or may not be available.
Naive approaches that test the existence of "lock files" are flawed.
Why? Because they don't check whether the file exists and create it in a single atomic action. Because of this; there is a race condition that WILL make your attempts at mutual exclusion break.
Instead, you can use mkdir. mkdir creates a directory if it doesn't exist yet, and if it does, it sets an exit code. More importantly, it does all this in a single atomic action making it perfect for this scenario.
if ! mkdir /tmp/myscript.lock 2>/dev/null; then
echo "Myscript is already running." >&2
exit 1
fi
For all details, see the excellent BashFAQ: http://mywiki.wooledge.org/BashFAQ/045
If you want to take care of stale locks, fuser(1) comes in handy. The only downside here is that the operation takes about a second, so it isn't instant.
Here's a function I wrote once that solves the problem using fuser:
# mutex file
#
# Open a mutual exclusion lock on the file, unless another process already owns one.
#
# If the file is already locked by another process, the operation fails.
# This function defines a lock on a file as having a file descriptor open to the file.
# This function uses FD 9 to open a lock on the file. To release the lock, close FD 9:
# exec 9>&-
#
mutex() {
local file=$1 pid pids
exec 9>>"$file"
{ pids=$(fuser -f "$file"); } 2>&- 9>&-
for pid in $pids; do
[[ $pid = $$ ]] && continue
exec 9>&-
return 1 # Locked by a pid.
done
}
You can use it in a script like so:
mutex /var/run/myscript.lock || { echo "Already running." >&2; exit 1; }
If you don't care about portability (these solutions should work on pretty much any UNIX box), Linux' fuser(1) offers some additional options and there is also flock(1).
Here's an implementation that uses a lockfile and echoes a PID into it. This serves as a protection if the process is killed before removing the pidfile:
LOCKFILE=/tmp/lock.txt
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
echo "already running"
exit
fi
# make sure the lockfile is removed when we exit and then claim it
trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT
echo $$ > ${LOCKFILE}
# do stuff
sleep 1000
rm -f ${LOCKFILE}
The trick here is the kill -0 which doesn't deliver any signal but just checks if a process with the given PID exists. Also the call to trap will ensure that the lockfile is removed even when your process is killed (except kill -9).
There's a wrapper around the flock(2) system call called, unimaginatively, flock(1). This makes it relatively easy to reliably obtain exclusive locks without worrying about cleanup etc. There are examples on the man page as to how to use it in a shell script.
To make locking reliable you need an atomic operation. Many of the above proposals
are not atomic. The proposed lockfile(1) utility looks promising as the man-page
mentioned, that its "NFS-resistant". If your OS does not support lockfile(1) and
your solution has to work on NFS, you have not many options....
NFSv2 has two atomic operations:
symlink
rename
With NFSv3 the create call is also atomic.
Directory operations are NOT atomic under NFSv2 and NFSv3 (please refer to the book 'NFS Illustrated' by Brent Callaghan, ISBN 0-201-32570-5; Brent is a NFS-veteran at Sun).
Knowing this, you can implement spin-locks for files and directories (in shell, not PHP):
lock current dir:
while ! ln -s . lock; do :; done
lock a file:
while ! ln -s ${f} ${f}.lock; do :; done
unlock current dir (assumption, the running process really acquired the lock):
mv lock deleteme && rm deleteme
unlock a file (assumption, the running process really acquired the lock):
mv ${f}.lock ${f}.deleteme && rm ${f}.deleteme
Remove is also not atomic, therefore first the rename (which is atomic) and then the remove.
For the symlink and rename calls, both filenames have to reside on the same filesystem. My proposal: use only simple filenames (no paths) and put file and lock into the same directory.
You need an atomic operation, like flock, else this will eventually fail.
But what to do if flock is not available. Well there is mkdir. That's an atomic operation too. Only one process will result in a successful mkdir, all others will fail.
So the code is:
if mkdir /var/lock/.myscript.exclusivelock
then
# do stuff
:
rmdir /var/lock/.myscript.exclusivelock
fi
You need to take care of stale locks else aftr a crash your script will never run again.
Another option is to use shell's noclobber option by running set -C. Then > will fail if the file already exists.
In brief:
set -C
lockfile="/tmp/locktest.lock"
if echo "$$" > "$lockfile"; then
echo "Successfully acquired lock"
# do work
rm "$lockfile" # XXX or via trap - see below
else
echo "Cannot acquire lock - already locked by $(cat "$lockfile")"
fi
This causes the shell to call:
open(pathname, O_CREAT|O_EXCL)
which atomically creates the file or fails if the file already exists.
According to a comment on BashFAQ 045, this may fail in ksh88, but it works in all my shells:
$ strace -e trace=creat,open -f /bin/bash /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/zsh /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/pdksh /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/dash /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 3
Interesting that pdksh adds the O_TRUNC flag, but obviously it's redundant:
either you're creating an empty file, or you're not doing anything.
How you do the rm depends on how you want unclean exits to be handled.
Delete on clean exit
New runs fail until the issue that caused the unclean exit to be resolved and the lockfile is manually removed.
# acquire lock
# do work (code here may call exit, etc.)
rm "$lockfile"
Delete on any exit
New runs succeed provided the script is not already running.
trap 'rm "$lockfile"' EXIT
You can use GNU Parallel for this as it works as a mutex when called as sem. So, in concrete terms, you can use:
sem --id SCRIPTSINGLETON yourScript
If you want a timeout too, use:
sem --id SCRIPTSINGLETON --semaphoretimeout -10 yourScript
Timeout of <0 means exit without running script if semaphore is not released within the timeout, timeout of >0 mean run the script anyway.
Note that you should give it a name (with --id) else it defaults to the controlling terminal.
GNU Parallel is a very simple install on most Linux/OSX/Unix platforms - it is just a Perl script.
For shell scripts, I tend to go with the mkdir over flock as it makes the locks more portable.
Either way, using set -e isn't enough. That only exits the script if any command fails. Your locks will still be left behind.
For proper lock cleanup, you really should set your traps to something like this psuedo code (lifted, simplified and untested but from actively used scripts) :
#=======================================================================
# Predefined Global Variables
#=======================================================================
TMPDIR=/tmp/myapp
[[ ! -d $TMP_DIR ]] \
&& mkdir -p $TMP_DIR \
&& chmod 700 $TMPDIR
LOCK_DIR=$TMP_DIR/lock
#=======================================================================
# Functions
#=======================================================================
function mklock {
__lockdir="$LOCK_DIR/$(date +%s.%N).$$" # Private Global. Use Epoch.Nano.PID
# If it can create $LOCK_DIR then no other instance is running
if $(mkdir $LOCK_DIR)
then
mkdir $__lockdir # create this instance's specific lock in queue
LOCK_EXISTS=true # Global
else
echo "FATAL: Lock already exists. Another copy is running or manually lock clean up required."
exit 1001 # Or work out some sleep_while_execution_lock elsewhere
fi
}
function rmlock {
[[ ! -d $__lockdir ]] \
&& echo "WARNING: Lock is missing. $__lockdir does not exist" \
|| rmdir $__lockdir
}
#-----------------------------------------------------------------------
# Private Signal Traps Functions {{{2
#
# DANGER: SIGKILL cannot be trapped. So, try not to `kill -9 PID` or
# there will be *NO CLEAN UP*. You'll have to manually remove
# any locks in place.
#-----------------------------------------------------------------------
function __sig_exit {
# Place your clean up logic here
# Remove the LOCK
[[ -n $LOCK_EXISTS ]] && rmlock
}
function __sig_int {
echo "WARNING: SIGINT caught"
exit 1002
}
function __sig_quit {
echo "SIGQUIT caught"
exit 1003
}
function __sig_term {
echo "WARNING: SIGTERM caught"
exit 1015
}
#=======================================================================
# Main
#=======================================================================
# Set TRAPs
trap __sig_exit EXIT # SIGEXIT
trap __sig_int INT # SIGINT
trap __sig_quit QUIT # SIGQUIT
trap __sig_term TERM # SIGTERM
mklock
# CODE
exit # No need for cleanup code here being in the __sig_exit trap function
Here's what will happen. All traps will produce an exit so the function __sig_exit will always happen (barring a SIGKILL) which cleans up your locks.
Note: my exit values are not low values. Why? Various batch processing systems make or have expectations of the numbers 0 through 31. Setting them to something else, I can have my scripts and batch streams react accordingly to the previous batch job or script.
Really quick and really dirty? This one-liner on the top of your script will work:
[[ $(pgrep -c "`basename \"$0\"`") -gt 1 ]] && exit
Of course, just make sure that your script name is unique. :)
Here's an approach that combines atomic directory locking with a check for stale lock via PID and restart if stale. Also, this does not rely on any bashisms.
#!/bin/dash
SCRIPTNAME=$(basename $0)
LOCKDIR="/var/lock/${SCRIPTNAME}"
PIDFILE="${LOCKDIR}/pid"
if ! mkdir $LOCKDIR 2>/dev/null
then
# lock failed, but check for stale one by checking if the PID is really existing
PID=$(cat $PIDFILE)
if ! kill -0 $PID 2>/dev/null
then
echo "Removing stale lock of nonexistent PID ${PID}" >&2
rm -rf $LOCKDIR
echo "Restarting myself (${SCRIPTNAME})" >&2
exec "$0" "$#"
fi
echo "$SCRIPTNAME is already running, bailing out" >&2
exit 1
else
# lock successfully acquired, save PID
echo $$ > $PIDFILE
fi
trap "rm -rf ${LOCKDIR}" QUIT INT TERM EXIT
echo hello
sleep 30s
echo bye
Create a lock file in a known location and check for existence on script start? Putting the PID in the file might be helpful if someone's attempting to track down an errant instance that's preventing execution of the script.
This example is explained in the man flock, but it needs some impovements, because we should manage bugs and exit codes:
#!/bin/bash
#set -e this is useful only for very stupid scripts because script fails when anything command exits with status more than 0 !! without possibility for capture exit codes. not all commands exits >0 are failed.
( #start subprocess
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200
if [ "$?" != "0" ]; then echo Cannot lock!; exit 1; fi
echo $$>>/var/lock/.myscript.exclusivelock #for backward lockdir compatibility, notice this command is executed AFTER command bottom ) 200>/var/lock/.myscript.exclusivelock.
# Do stuff
# you can properly manage exit codes with multiple command and process algorithm.
# I suggest throw this all to external procedure than can properly handle exit X commands
) 200>/var/lock/.myscript.exclusivelock #exit subprocess
FLOCKEXIT=$? #save exitcode status
#do some finish commands
exit $FLOCKEXIT #return properly exitcode, may be usefull inside external scripts
You can use another method, list processes that I used in the past. But this is more complicated that method above. You should list processes by ps, filter by its name, additional filter grep -v grep for remove parasite nad finally count it by grep -c . and compare with number. Its complicated and uncertain
The existing answers posted either rely on the CLI utility flock or do not properly secure the lock file. The flock utility is not available on all non-Linux systems (i.e. FreeBSD), and does not work properly on NFS.
In my early days of system administration and system development, I was told that a safe and relatively portable method of creating a lock file was to create a temp file using mkemp(3) or mkemp(1), write identifying information to the temp file (i.e. PID), then hard link the temp file to the lock file. If the link was successful, then you have successfully obtained the lock.
When using locks in shell scripts, I typically place an obtain_lock() function in a shared profile and then source it from the scripts. Below is an example of my lock function:
obtain_lock()
{
LOCK="${1}"
LOCKDIR="$(dirname "${LOCK}")"
LOCKFILE="$(basename "${LOCK}")"
# create temp lock file
TMPLOCK=$(mktemp -p "${LOCKDIR}" "${LOCKFILE}XXXXXX" 2> /dev/null)
if test "x${TMPLOCK}" == "x";then
echo "unable to create temporary file with mktemp" 1>&2
return 1
fi
echo "$$" > "${TMPLOCK}"
# attempt to obtain lock file
ln "${TMPLOCK}" "${LOCK}" 2> /dev/null
if test $? -ne 0;then
rm -f "${TMPLOCK}"
echo "unable to obtain lockfile" 1>&2
if test -f "${LOCK}";then
echo "current lock information held by: $(cat "${LOCK}")" 1>&2
fi
return 2
fi
rm -f "${TMPLOCK}"
return 0;
};
The following is an example of how to use the lock function:
#!/bin/sh
. /path/to/locking/profile.sh
PROG_LOCKFILE="/tmp/myprog.lock"
clean_up()
{
rm -f "${PROG_LOCKFILE}"
}
obtain_lock "${PROG_LOCKFILE}"
if test $? -ne 0;then
exit 1
fi
trap clean_up SIGHUP SIGINT SIGTERM
# bulk of script
clean_up
exit 0
# end of script
Remember to call clean_up at any exit points in your script.
I've used the above in both Linux and FreeBSD environments.
Add this line at the beginning of your script
[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$#" || :
It's a boilerplate code from man flock.
If you want more logging, use this one
[ "${FLOCKER}" != "$0" ] && { echo "Trying to start build from queue... "; exec bash -c "FLOCKER='$0' flock -E $E_LOCKED -en '$0' '$0' '$#' || if [ \"\$?\" -eq $E_LOCKED ]; then echo 'Locked.'; fi"; } || echo "Lock is free. Completing."
This sets and checks locks using flock utility.
This code detects if it was run first time by checking FLOCKER variable, if it is not set to script name, then it tries to start script again recursively using flock and with FLOCKER variable initialized, if FLOCKER is set correctly, then flock on previous iteration succeeded and it is OK to proceed. If lock is busy, it fails with configurable exit code.
It seems to not work on Debian 7, but seems to work back again with experimental util-linux 2.25 package. It writes "flock: ... Text file busy". It could be overridden by disabling write permission on your script.
When targeting a Debian machine I find the lockfile-progs package to be a good solution. procmail also comes with a lockfile tool. However sometimes I am stuck with neither of these.
Here's my solution which uses mkdir for atomic-ness and a PID file to detect stale locks. This code is currently in production on a Cygwin setup and works well.
To use it simply call exclusive_lock_require when you need get exclusive access to something. An optional lock name parameter lets you share locks between different scripts. There's also two lower level functions (exclusive_lock_try and exclusive_lock_retry) should you need something more complex.
function exclusive_lock_try() # [lockname]
{
local LOCK_NAME="${1:-`basename $0`}"
LOCK_DIR="/tmp/.${LOCK_NAME}.lock"
local LOCK_PID_FILE="${LOCK_DIR}/${LOCK_NAME}.pid"
if [ -e "$LOCK_DIR" ]
then
local LOCK_PID="`cat "$LOCK_PID_FILE" 2> /dev/null`"
if [ ! -z "$LOCK_PID" ] && kill -0 "$LOCK_PID" 2> /dev/null
then
# locked by non-dead process
echo "\"$LOCK_NAME\" lock currently held by PID $LOCK_PID"
return 1
else
# orphaned lock, take it over
( echo $$ > "$LOCK_PID_FILE" ) 2> /dev/null && local LOCK_PID="$$"
fi
fi
if [ "`trap -p EXIT`" != "" ]
then
# already have an EXIT trap
echo "Cannot get lock, already have an EXIT trap"
return 1
fi
if [ "$LOCK_PID" != "$$" ] &&
! ( umask 077 && mkdir "$LOCK_DIR" && umask 177 && echo $$ > "$LOCK_PID_FILE" ) 2> /dev/null
then
local LOCK_PID="`cat "$LOCK_PID_FILE" 2> /dev/null`"
# unable to acquire lock, new process got in first
echo "\"$LOCK_NAME\" lock currently held by PID $LOCK_PID"
return 1
fi
trap "/bin/rm -rf \"$LOCK_DIR\"; exit;" EXIT
return 0 # got lock
}
function exclusive_lock_retry() # [lockname] [retries] [delay]
{
local LOCK_NAME="$1"
local MAX_TRIES="${2:-5}"
local DELAY="${3:-2}"
local TRIES=0
local LOCK_RETVAL
while [ "$TRIES" -lt "$MAX_TRIES" ]
do
if [ "$TRIES" -gt 0 ]
then
sleep "$DELAY"
fi
local TRIES=$(( $TRIES + 1 ))
if [ "$TRIES" -lt "$MAX_TRIES" ]
then
exclusive_lock_try "$LOCK_NAME" > /dev/null
else
exclusive_lock_try "$LOCK_NAME"
fi
LOCK_RETVAL="${PIPESTATUS[0]}"
if [ "$LOCK_RETVAL" -eq 0 ]
then
return 0
fi
done
return "$LOCK_RETVAL"
}
function exclusive_lock_require() # [lockname] [retries] [delay]
{
if ! exclusive_lock_retry "$#"
then
exit 1
fi
}
If flock's limitations, which have already been described elsewhere on this thread, aren't an issue for you, then this should work:
#!/bin/bash
{
# exit if we are unable to obtain a lock; this would happen if
# the script is already running elsewhere
# note: -x (exclusive) is the default
flock -n 100 || exit
# put commands to run here
sleep 100
} 100>/tmp/myjob.lock
Some unixes have lockfile which is very similar to the already mentioned flock.
From the manpage:
lockfile can be used to create one
or more semaphore files. If lock-
file can't create all the specified
files (in the specified order), it
waits sleeptime (defaults to 8)
seconds and retries the last file that
didn't succeed. You can specify the
number of retries to do until
failure is returned. If the number
of retries is -1 (default, i.e.,
-r-1) lockfile will retry forever.
I use a simple approach that handles stale lock files.
Note that some of the above solutions that store the pid, ignore the fact that the pid can wrap around. So - just checking if there is a valid process with the stored pid is not enough, especially for long running scripts.
I use noclobber to make sure only one script can open and write to the lock file at one time. Further, I store enough information to uniquely identify a process in the lockfile. I define the set of data to uniquely identify a process to be pid,ppid,lstart.
When a new script starts up, if it fails to create the lock file, it then verifies that the process that created the lock file is still around. If not, we assume the original process died an ungraceful death, and left a stale lock file. The new script then takes ownership of the lock file, and all is well the world, again.
Should work with multiple shells across multiple platforms. Fast, portable and simple.
#!/usr/bin/env sh
# Author: rouble
LOCKFILE=/var/tmp/lockfile #customize this line
trap release INT TERM EXIT
# Creates a lockfile. Sets global variable $ACQUIRED to true on success.
#
# Returns 0 if it is successfully able to create lockfile.
acquire () {
set -C #Shell noclobber option. If file exists, > will fail.
UUID=`ps -eo pid,ppid,lstart $$ | tail -1`
if (echo "$UUID" > "$LOCKFILE") 2>/dev/null; then
ACQUIRED="TRUE"
return 0
else
if [ -e $LOCKFILE ]; then
# We may be dealing with a stale lock file.
# Bring out the magnifying glass.
CURRENT_UUID_FROM_LOCKFILE=`cat $LOCKFILE`
CURRENT_PID_FROM_LOCKFILE=`cat $LOCKFILE | cut -f 1 -d " "`
CURRENT_UUID_FROM_PS=`ps -eo pid,ppid,lstart $CURRENT_PID_FROM_LOCKFILE | tail -1`
if [ "$CURRENT_UUID_FROM_LOCKFILE" == "$CURRENT_UUID_FROM_PS" ]; then
echo "Script already running with following identification: $CURRENT_UUID_FROM_LOCKFILE" >&2
return 1
else
# The process that created this lock file died an ungraceful death.
# Take ownership of the lock file.
echo "The process $CURRENT_UUID_FROM_LOCKFILE is no longer around. Taking ownership of $LOCKFILE"
release "FORCE"
if (echo "$UUID" > "$LOCKFILE") 2>/dev/null; then
ACQUIRED="TRUE"
return 0
else
echo "Cannot write to $LOCKFILE. Error." >&2
return 1
fi
fi
else
echo "Do you have write permissons to $LOCKFILE ?" >&2
return 1
fi
fi
}
# Removes the lock file only if this script created it ($ACQUIRED is set),
# OR, if we are removing a stale lock file (first parameter is "FORCE")
release () {
#Destroy lock file. Take no prisoners.
if [ "$ACQUIRED" ] || [ "$1" == "FORCE" ]; then
rm -f $LOCKFILE
fi
}
# Test code
# int main( int argc, const char* argv[] )
echo "Acquring lock."
acquire
if [ $? -eq 0 ]; then
echo "Acquired lock."
read -p "Press [Enter] key to release lock..."
release
echo "Released lock."
else
echo "Unable to acquire lock."
fi
I wanted to do away with lockfiles, lockdirs, special locking programs and even pidof since it isn't found on all Linux installations. Also wanted to have the simplest code possible (or at least as few lines as possible). Simplest if statement, in one line:
if [[ $(ps axf | awk -v pid=$$ '$1!=pid && $6~/'$(basename $0)'/{print $1}') ]]; then echo "Already running"; exit; fi
Actually although the answer of bmdhacks is almost good, there is a slight chance the second script to run after first checked the lockfile and before it wrote it. So they both will write the lock file and they will both be running. Here is how to make it work for sure:
lockfile=/var/lock/myscript.lock
if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null ; then
trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
else
# or you can decide to skip the "else" part if you want
echo "Another instance is already running!"
fi
The noclobber option will make sure that redirect command will fail if file already exists. So the redirect command is actually atomic - you write and check the file with one command. You don't need to remove the lockfile at the end of file - it'll be removed by the trap. I hope this helps to people that will read it later.
P.S. I didn't see that Mikel already answered the question correctly, although he didn't include the trap command to reduce the chance the lock file will be left over after stopping the script with Ctrl-C for example. So this is the complete solution
An example with flock(1) but without subshell. flock()ed file /tmp/foo is never removed, but that doesn't matter as it gets flock() and un-flock()ed.
#!/bin/bash
exec 9<> /tmp/foo
flock -n 9
RET=$?
if [[ $RET -ne 0 ]] ; then
echo "lock failed, exiting"
exit
fi
#Now we are inside the "critical section"
echo "inside lock"
sleep 5
exec 9>&- #close fd 9, and release lock
#The part below is outside the critical section (the lock)
echo "lock released"
sleep 5
This one line answer comes from someone related Ask Ubuntu Q&A:
[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$#" || :
# This is useful boilerplate code for shell scripts. Put it at the top of
# the shell script you want to lock and it'll automatically lock itself on
# the first run. If the env var $FLOCKER is not set to the shell script
# that is being run, then execute flock and grab an exclusive non-blocking
# lock (using the script itself as the lock file) before re-execing itself
# with the right arguments. It also sets the FLOCKER env var to the right
# value so it doesn't run again.
PID and lockfiles are definitely the most reliable. When you attempt to run the program, it can check for the lockfile which and if it exists, it can use ps to see if the process is still running. If it's not, the script can start, updating the PID in the lockfile to its own.
I find that bmdhack's solution is the most practical, at least for my use case. Using flock and lockfile rely on removing the lockfile using rm when the script terminates, which can't always be guaranteed (e.g., kill -9).
I would change one minor thing about bmdhack's solution: It makes a point of removing the lock file, without stating that this is unnecessary for the safe working of this semaphore. His use of kill -0 ensures that an old lockfile for a dead process will simply be ignored/over-written.
My simplified solution is therefore to simply add the following to the top of your singleton:
## Test the lock
LOCKFILE=/tmp/singleton.lock
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
echo "Script already running. bye!"
exit
fi
## Set the lock
echo $$ > ${LOCKFILE}
Of course, this script still has the flaw that processes that are likely to start at the same time have a race hazard, as the lock test and set operations are not a single atomic action. But the proposed solution for this by lhunath to use mkdir has the flaw that a killed script may leave behind the directory, thus preventing other instances from running.
The semaphoric utility uses flock (as discussed above, e.g. by presto8) to implement a counting semaphore. It enables any specific number of concurrent processes you want. We use it to limit the level of concurrency of various queue worker processes.
It's like sem but much lighter-weight. (Full disclosure: I wrote it after finding the sem was way too heavy for our needs and there wasn't a simple counting semaphore utility available.)
Answered a million times already, but another way, without the need for external dependencies:
LOCK_FILE="/var/lock/$(basename "$0").pid"
trap "rm -f ${LOCK_FILE}; exit" INT TERM EXIT
if [[ -f $LOCK_FILE && -d /proc/`cat $LOCK_FILE` ]]; then
// Process already exists
exit 1
fi
echo $$ > $LOCK_FILE
Each time it writes the current PID ($$) into the lockfile and on script startup checks if a process is running with the latest PID.
Using the process's lock is much stronger and takes care of the ungraceful exits also.
lock_file is kept open as long as the process is running. It will be closed (by shell) once the process exists (even if it gets killed).
I found this to be very efficient:
lock_file=/tmp/`basename $0`.lock
if fuser $lock_file > /dev/null 2>&1; then
echo "WARNING: Other instance of $(basename $0) running."
exit 1
fi
exec 3> $lock_file
I use oneliner # the very beginning of script:
#!/bin/bash
if [[ $(pgrep -afc "$(basename "$0")") -gt "1" ]]; then echo "Another instance of "$0" has already been started!" && exit; fi
.
the_beginning_of_actual_script
It is good to see the presence of process in the memory (no matter what the status of process is); but it does the job for me.
If you do not want to or cannot use flock (e.g. you are not using a shared file system), consider using an external service like lockable.
It exposes advisory lock primitives, much like flock would. In particular, you can acquire a lock via:
https://lockable.dev/api/acquire/my-lock-name
and release it via
https://lockable.dev/api/release/my-lock-name
By wrapping script execution with lock acquisition and release, you can make sure only a single instance of the process is running at any given time.
What am I looking for is a replacement for all those things you’ve probably seen by hitting Mod+R in various tiling WM or Alt+F2 in Gnome DE — a small window to run things from there. I feel very sad without having my bash aliases and other stuff missing because those shells (at least those ones I can use now) are non-interactive and their interactiveness can’t be set as an option at run time.
This is why I decided to use URxvt window as a ‘shell for one command’. In my WM I have Mod+R shortcut bound to execute
/bin/bash -c 'export ONE_COMMAND_SHELL=t && urxvt -name one_command_shell'
And put
[ -v ONE_COMMAND_SHELL ] && bind -x '"\C-m":"exit"'
in my ~/.bashrc
This way I can distinguish an URxvt instance which will become ‘shell for one command’, and set C-m combination (yes, it also works for Enter) to exit the shell, and, therefore, URxvt. The problem is how to execute string entered so far before exit?
I found two potential clues:
a) Making use of BASH_COMMAND
BASH_COMMAND
The command currently being executed or about to be executed, unless the
shell is executing a command as the result of a trap, in which case it is
the command executing at the time of the trap.
But I didn’t get it working because
[ -v ONE_COMMAND_SHELL ] && bind -x '"\C-m":"exec $BASH_COMMAND exit"'
will cause exec exec exec.
b) A trap on SIGCHLD!
It may work for you, if you place
[ -v ONE_COMMAND_SHELL ] && {
PS1=
eaoc() { exit; }
trap eaoc SIGCHLD
}
At the end of your ~/.bashrc. PS1 may be non-empty, but it may not have subshell calls, or the trap will execute and shell will exit before the prompt is to be generated. This holds URxvt open till the forked process exit, so may be considered a half-way to solution.
c) A trap to DEBUG.
DEBUG trap executes before any simple command to be executed and before the first command in each function. If we provide a number via OC_SHELL and will count the number of executed commands, like…
/bin/bash -c 'export OC_SHELL=0 && urxvt -name one_command_shell'
# note a zero ----------------^
And at the end of ~/.bashrc
[ -v OC_SHELL ] && {
export PS1=
eaoc() { echo $OC_SHELL && [ $(( OC_SHELL++ )) -ge 1 ] && wait $! && exit; }
trap eaoc DEBUG
}
We get a fail again. Because our process, forked like
$ gimp &
Dies with its parent. How to deal with it?
Interesting thing is that DEBUG trap executes right after the command was entered, this trap does not wait for a process to return its status code or pid of background process in case of using &.
Ok, that’s how create process independent of bash Need to somehow wrap entered string into (nohup … &)
This works for me:
one_command_execute() {
eval $(echo "$READLINE_LINE") &
exit
}
[ -v ONE_COMMAND_SHELL ] && bind -x '"\C-m":"one_command_execute"'
Another version is without eval:
one_command_checkexit() {
[ -v ONE_COMMAND_DONE ] && exit
ONE_COMMAND_DONE=1
}
[ -v ONE_COMMAND_SHELL ] && PROMPT_COMMAND=one_command_checkexit
This doesn't close the window until the command exits. To execute everything in background automatically, add:
[ -v ONE_COMMAND_SHELL ] && bind '"\C-m":" & \n"'
I have a bash script I want to run every 5 minutes from cron... but there's a chance the previous run of the script isn't done yet... in this case, i want the new run to just exit. I don't want to rely on just a lock file in /tmp.. I want to make sure sure the process is actually running before i honor the lock file (or whatever)...
Here is what I have stolen from the internet so far... how do i smarten it up a bit? or is there a completely different way that's better?
if [ -f /tmp/mylockFile ] ; then
echo 'Script is still running'
else
echo 1 > /tmp/mylockFile
/* Do some stuff */
rm -f /tmp/mylockFile
fi
# Use a lockfile containing the pid of the running process
# If script crashes and leaves lockfile around, it will have a different pid so
# will not prevent script running again.
#
lf=/tmp/pidLockFile
# create empty lock file if none exists
cat /dev/null >> $lf
read lastPID < $lf
# if lastPID is not null and a process with that pid exists , exit
[ ! -z "$lastPID" -a -d /proc/$lastPID ] && exit
echo not running
# save my pid in the lock file
echo $$ > $lf
# sleep just to make testing easier
sleep 5
There is at least one race condition in this script. Don't use it for a life support system, lol. But it should work fine for your example, because your environment doesn't start two scripts simultaneously. There are lots of ways to use more atomic locks, but they generally depend on having a particular thing optionally installed, or work differently on NFS, etc...
You might want to have a look at the man page for the flock command, if you're lucky enough to get it on your distribution.
NAME
flock - Manage locks from shell scripts
SYNOPSIS
flock [-sxon] [-w timeout] lockfile [-c] command...
Never use a lock file always use a lock directory.
In your specific case, it's not so important because the start of the script is scheduled in 5min intervals. But if you ever reuse this code for a webserver cgi-script you are toast.
if mkdir /tmp/my_lock_dir 2>/dev/null
then
echo "running now the script"
sleep 10
rmdir /tmp/my_lock_dir
fi
This has a problem if you have a stale lock, means the lock is there but no associated process. Your cron will never run.
Why use a directory? Because mkdir is an atomic operation. Only one process at a time can create a directory, all other processes get an error. This even works across shared filesystems and probably even between different OS types.
Store your pid in mylockFile. When you need to check, look up ps for the process with the pid you read from file. If it exists, your script is running.
If you want to check the process's existence, just look at the output of
ps aux | grep your_script_name
If it's there, it's not dead...
As pointed out in the comments and other answers, using the PID stored in the lockfile is much safer and is the standard approach most apps take. I just do this because it's convenient and I almost never see the corner cases (e.g. editing the file when the cron executes) in practice.
If you use a lockfile, you should make sure that the lockfile is always removed. You can do this with 'trap':
if ( set -o noclobber; echo "locked" > "$lockfile") 2> /dev/null; then
trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
echo "Locking succeeded" >&2
rm -f "$lockfile"
else
echo "Lock failed - exit" >&2
exit 1
fi
The noclobber option makes the creation of lockfile atomic, like using a directory.
As a one-liner and if you do not want to use a lockfile (e.g. b/c/ of a read only filesystem, etc)
test "$(pidof -x $(basename $0))" != $$ && exit
It checks that the full list of PID that bear the name of your script is equal to the current PID. The "-x" also checks for the name of shell scripts.
Bash makes it even shorter and faster:
[[ "$(pidof -x $(basename $0))" != $$ ]] && exit
In some cases, you might want to be able to distinguish between who is running the script and allow some concurrency but not all. In that case, you can use per-user, per-tty or cron-specific locks.
You can use environment variables such as $USER or the output of a program such as tty to create the filename. For cron, you can set a variable in the crontab file and test for it in your script.
you can use this one:
pgrep -f "/bin/\w*sh .*scriptname" | grep -vq $$ && exit
I was trying to solve this problem today and I came up with the below:
COMMAND_LINE="$0 $*"
JOBS=$(SUBSHELL_PID=$BASHPID; ps axo pid,command | grep "${COMMAND_LINE}" | grep -v $$ | g rep -v ${SUBSHELL_PID} | grep -v grep)
if [[ -z "${JOBS}" ]]
then
# not already running
else
# already running
fi
This relies on $BASHPID which contains the PID inside a subshell ($$ in the subshell is the parent pid). However, this relies on Bash v4 and I needed to run this on OSX which has Bash v3.2.48. I ultimately came up with another solution and it is cleaner:
JOBS=$(sh -c "ps axo pid,command | grep \"${COMMAND_LINE}\" | grep -v grep | grep -v $$")
You can always just:
if ps -e -o cmd | grep scriptname > /dev/null; then
exit
fi
But I like the lockfile myself, so I wouldn't do this without the lock file as well.
Since a socket solution has not yet been mentioned it is worth pointing out that sockets can be used as effective mutexes. Socket creation is an atomic operation, like mkdir is as Gunstick pointed out, so a socket is suitable to use as a lock or mutex.
Tim Kay's Perl script 'Solo' is a very small and effective script to make sure only one copy of a script can be run at any one time. It was designed specifically for use with cron jobs, although it works perfectly for other tasks as well and I've used it for non-crob jobs very effectively.
Solo has one advantage over the other techniques mentioned so far in that the check is done outside of the script you only want to run one copy of. If the script is already running then a second instance of that script will never even be started. This is as opposed to isolating a block of code inside the script which is protected by a lock. EDIT: If flock is used in a cron job, rather than from inside a script, then you can also use that to prevent a second instance of the script from starting - see example below.
Here's an example of how you might use it with cron:
*/5 * * * * solo -port=3801 /path/to/script.sh args args args
# "/path/to/script.sh args args args" is only called if no other instance of
# "/path/to/script.sh" is running, or more accurately if the socket on port 3801
# is not open. Distinct port numbers can be used for different programs so that
# if script_1.sh is running it does not prevent script_2.sh from starting, I've
# used the port range 3801 to 3810 without conflicts. For Linux non-root users
# the valid port range is 1024 to 65535 (0 to 1023 are reserved for root).
* * * * * solo -port=3802 /path/to/script_1.sh
* * * * * solo -port=3803 /path/to/script_2.sh
# Flock can also be used in cron jobs with a distinct lock path for different
# programs, in the example below script_3.sh will only be started if the one
# started a minute earlier has already finished.
* * * * * flock -n /tmp/path.to.lock -c /path/to/script_3.sh
Links:
Solo web page: http://timkay.com/solo/
Solo script: http://timkay.com/solo/solo
Hope this helps.
You can use this.
I'll just shamelessly copy-paste the solution here, as it is an answer for both questions (I would argue that it's actually a better fit for this question).
Usage
include sh_lock_functions.sh
init using sh_lock_init
lock using sh_acquire_lock
check lock using sh_check_lock
unlock using sh_remove_lock
Script File
sh_lock_functions.sh
#!/bin/bash
function sh_lock_init {
sh_lock_scriptName=$(basename $0)
sh_lock_dir="/tmp/${sh_lock_scriptName}.lock" #lock directory
sh_lock_file="${sh_lock_dir}/lockPid.txt" #lock file
}
function sh_acquire_lock {
if mkdir $sh_lock_dir 2>/dev/null; then #check for lock
echo "$sh_lock_scriptName lock acquired successfully.">&2
touch $sh_lock_file
echo $$ > $sh_lock_file # set current pid in lockFile
return 0
else
touch $sh_lock_file
read sh_lock_lastPID < $sh_lock_file
if [ ! -z "$sh_lock_lastPID" -a -d /proc/$sh_lock_lastPID ]; then # if lastPID is not null and a process with that pid exists
echo "$sh_lock_scriptName is already running.">&2
return 1
else
echo "$sh_lock_scriptName stopped during execution, reacquiring lock.">&2
echo $$ > $sh_lock_file # set current pid in lockFile
return 2
fi
fi
return 0
}
function sh_check_lock {
[[ ! -f $sh_lock_file ]] && echo "$sh_lock_scriptName lock file removed.">&2 && return 1
read sh_lock_lastPID < $sh_lock_file
[[ $sh_lock_lastPID -ne $$ ]] && echo "$sh_lock_scriptName lock file pid has changed.">&2 && return 2
echo "$sh_lock_scriptName lock still in place.">&2
return 0
}
function sh_remove_lock {
rm -r $sh_lock_dir
}
Usage example
sh_lock_usage_example.sh
#!/bin/bash
. /path/to/sh_lock_functions.sh # load sh lock functions
sh_lock_init || exit $?
sh_acquire_lock
lockStatus=$?
[[ $lockStatus -eq 1 ]] && exit $lockStatus
[[ $lockStatus -eq 2 ]] && echo "lock is set, do some resume from crash procedures";
#monitoring example
cnt=0
while sh_check_lock # loop while lock is in place
do
echo "$sh_scriptName running (pid $$)"
sleep 1
let cnt++
[[ $cnt -gt 5 ]] && break
done
#remove lock when process finished
sh_remove_lock || exit $?
exit 0
Features
Uses a combination of file, directory and process id to lock to make sure that the process is not already running
You can detect if the script stopped before lock removal (eg. process kill, shutdown, error etc.)
You can check the lock file, and use it to trigger a process shutdown when the lock is missing
Verbose, outputs error messages for easier debug
What's a quick-and-dirty way to make sure that only one instance of a shell script is running at a given time?
Use flock(1) to make an exclusive scoped lock a on file descriptor. This way you can even synchronize different parts of the script.
#!/bin/bash
(
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200 || exit 1
# Do stuff
) 200>/var/lock/.myscript.exclusivelock
This ensures that code between ( and ) is run only by one process at a time and that the process doesn’t wait too long for a lock.
Caveat: this particular command is a part of util-linux. If you run an operating system other than Linux, it may or may not be available.
Naive approaches that test the existence of "lock files" are flawed.
Why? Because they don't check whether the file exists and create it in a single atomic action. Because of this; there is a race condition that WILL make your attempts at mutual exclusion break.
Instead, you can use mkdir. mkdir creates a directory if it doesn't exist yet, and if it does, it sets an exit code. More importantly, it does all this in a single atomic action making it perfect for this scenario.
if ! mkdir /tmp/myscript.lock 2>/dev/null; then
echo "Myscript is already running." >&2
exit 1
fi
For all details, see the excellent BashFAQ: http://mywiki.wooledge.org/BashFAQ/045
If you want to take care of stale locks, fuser(1) comes in handy. The only downside here is that the operation takes about a second, so it isn't instant.
Here's a function I wrote once that solves the problem using fuser:
# mutex file
#
# Open a mutual exclusion lock on the file, unless another process already owns one.
#
# If the file is already locked by another process, the operation fails.
# This function defines a lock on a file as having a file descriptor open to the file.
# This function uses FD 9 to open a lock on the file. To release the lock, close FD 9:
# exec 9>&-
#
mutex() {
local file=$1 pid pids
exec 9>>"$file"
{ pids=$(fuser -f "$file"); } 2>&- 9>&-
for pid in $pids; do
[[ $pid = $$ ]] && continue
exec 9>&-
return 1 # Locked by a pid.
done
}
You can use it in a script like so:
mutex /var/run/myscript.lock || { echo "Already running." >&2; exit 1; }
If you don't care about portability (these solutions should work on pretty much any UNIX box), Linux' fuser(1) offers some additional options and there is also flock(1).
Here's an implementation that uses a lockfile and echoes a PID into it. This serves as a protection if the process is killed before removing the pidfile:
LOCKFILE=/tmp/lock.txt
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
echo "already running"
exit
fi
# make sure the lockfile is removed when we exit and then claim it
trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT
echo $$ > ${LOCKFILE}
# do stuff
sleep 1000
rm -f ${LOCKFILE}
The trick here is the kill -0 which doesn't deliver any signal but just checks if a process with the given PID exists. Also the call to trap will ensure that the lockfile is removed even when your process is killed (except kill -9).
There's a wrapper around the flock(2) system call called, unimaginatively, flock(1). This makes it relatively easy to reliably obtain exclusive locks without worrying about cleanup etc. There are examples on the man page as to how to use it in a shell script.
To make locking reliable you need an atomic operation. Many of the above proposals
are not atomic. The proposed lockfile(1) utility looks promising as the man-page
mentioned, that its "NFS-resistant". If your OS does not support lockfile(1) and
your solution has to work on NFS, you have not many options....
NFSv2 has two atomic operations:
symlink
rename
With NFSv3 the create call is also atomic.
Directory operations are NOT atomic under NFSv2 and NFSv3 (please refer to the book 'NFS Illustrated' by Brent Callaghan, ISBN 0-201-32570-5; Brent is a NFS-veteran at Sun).
Knowing this, you can implement spin-locks for files and directories (in shell, not PHP):
lock current dir:
while ! ln -s . lock; do :; done
lock a file:
while ! ln -s ${f} ${f}.lock; do :; done
unlock current dir (assumption, the running process really acquired the lock):
mv lock deleteme && rm deleteme
unlock a file (assumption, the running process really acquired the lock):
mv ${f}.lock ${f}.deleteme && rm ${f}.deleteme
Remove is also not atomic, therefore first the rename (which is atomic) and then the remove.
For the symlink and rename calls, both filenames have to reside on the same filesystem. My proposal: use only simple filenames (no paths) and put file and lock into the same directory.
You need an atomic operation, like flock, else this will eventually fail.
But what to do if flock is not available. Well there is mkdir. That's an atomic operation too. Only one process will result in a successful mkdir, all others will fail.
So the code is:
if mkdir /var/lock/.myscript.exclusivelock
then
# do stuff
:
rmdir /var/lock/.myscript.exclusivelock
fi
You need to take care of stale locks else aftr a crash your script will never run again.
Another option is to use shell's noclobber option by running set -C. Then > will fail if the file already exists.
In brief:
set -C
lockfile="/tmp/locktest.lock"
if echo "$$" > "$lockfile"; then
echo "Successfully acquired lock"
# do work
rm "$lockfile" # XXX or via trap - see below
else
echo "Cannot acquire lock - already locked by $(cat "$lockfile")"
fi
This causes the shell to call:
open(pathname, O_CREAT|O_EXCL)
which atomically creates the file or fails if the file already exists.
According to a comment on BashFAQ 045, this may fail in ksh88, but it works in all my shells:
$ strace -e trace=creat,open -f /bin/bash /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/zsh /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/pdksh /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/dash /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 3
Interesting that pdksh adds the O_TRUNC flag, but obviously it's redundant:
either you're creating an empty file, or you're not doing anything.
How you do the rm depends on how you want unclean exits to be handled.
Delete on clean exit
New runs fail until the issue that caused the unclean exit to be resolved and the lockfile is manually removed.
# acquire lock
# do work (code here may call exit, etc.)
rm "$lockfile"
Delete on any exit
New runs succeed provided the script is not already running.
trap 'rm "$lockfile"' EXIT
You can use GNU Parallel for this as it works as a mutex when called as sem. So, in concrete terms, you can use:
sem --id SCRIPTSINGLETON yourScript
If you want a timeout too, use:
sem --id SCRIPTSINGLETON --semaphoretimeout -10 yourScript
Timeout of <0 means exit without running script if semaphore is not released within the timeout, timeout of >0 mean run the script anyway.
Note that you should give it a name (with --id) else it defaults to the controlling terminal.
GNU Parallel is a very simple install on most Linux/OSX/Unix platforms - it is just a Perl script.
For shell scripts, I tend to go with the mkdir over flock as it makes the locks more portable.
Either way, using set -e isn't enough. That only exits the script if any command fails. Your locks will still be left behind.
For proper lock cleanup, you really should set your traps to something like this psuedo code (lifted, simplified and untested but from actively used scripts) :
#=======================================================================
# Predefined Global Variables
#=======================================================================
TMPDIR=/tmp/myapp
[[ ! -d $TMP_DIR ]] \
&& mkdir -p $TMP_DIR \
&& chmod 700 $TMPDIR
LOCK_DIR=$TMP_DIR/lock
#=======================================================================
# Functions
#=======================================================================
function mklock {
__lockdir="$LOCK_DIR/$(date +%s.%N).$$" # Private Global. Use Epoch.Nano.PID
# If it can create $LOCK_DIR then no other instance is running
if $(mkdir $LOCK_DIR)
then
mkdir $__lockdir # create this instance's specific lock in queue
LOCK_EXISTS=true # Global
else
echo "FATAL: Lock already exists. Another copy is running or manually lock clean up required."
exit 1001 # Or work out some sleep_while_execution_lock elsewhere
fi
}
function rmlock {
[[ ! -d $__lockdir ]] \
&& echo "WARNING: Lock is missing. $__lockdir does not exist" \
|| rmdir $__lockdir
}
#-----------------------------------------------------------------------
# Private Signal Traps Functions {{{2
#
# DANGER: SIGKILL cannot be trapped. So, try not to `kill -9 PID` or
# there will be *NO CLEAN UP*. You'll have to manually remove
# any locks in place.
#-----------------------------------------------------------------------
function __sig_exit {
# Place your clean up logic here
# Remove the LOCK
[[ -n $LOCK_EXISTS ]] && rmlock
}
function __sig_int {
echo "WARNING: SIGINT caught"
exit 1002
}
function __sig_quit {
echo "SIGQUIT caught"
exit 1003
}
function __sig_term {
echo "WARNING: SIGTERM caught"
exit 1015
}
#=======================================================================
# Main
#=======================================================================
# Set TRAPs
trap __sig_exit EXIT # SIGEXIT
trap __sig_int INT # SIGINT
trap __sig_quit QUIT # SIGQUIT
trap __sig_term TERM # SIGTERM
mklock
# CODE
exit # No need for cleanup code here being in the __sig_exit trap function
Here's what will happen. All traps will produce an exit so the function __sig_exit will always happen (barring a SIGKILL) which cleans up your locks.
Note: my exit values are not low values. Why? Various batch processing systems make or have expectations of the numbers 0 through 31. Setting them to something else, I can have my scripts and batch streams react accordingly to the previous batch job or script.
Really quick and really dirty? This one-liner on the top of your script will work:
[[ $(pgrep -c "`basename \"$0\"`") -gt 1 ]] && exit
Of course, just make sure that your script name is unique. :)
Here's an approach that combines atomic directory locking with a check for stale lock via PID and restart if stale. Also, this does not rely on any bashisms.
#!/bin/dash
SCRIPTNAME=$(basename $0)
LOCKDIR="/var/lock/${SCRIPTNAME}"
PIDFILE="${LOCKDIR}/pid"
if ! mkdir $LOCKDIR 2>/dev/null
then
# lock failed, but check for stale one by checking if the PID is really existing
PID=$(cat $PIDFILE)
if ! kill -0 $PID 2>/dev/null
then
echo "Removing stale lock of nonexistent PID ${PID}" >&2
rm -rf $LOCKDIR
echo "Restarting myself (${SCRIPTNAME})" >&2
exec "$0" "$#"
fi
echo "$SCRIPTNAME is already running, bailing out" >&2
exit 1
else
# lock successfully acquired, save PID
echo $$ > $PIDFILE
fi
trap "rm -rf ${LOCKDIR}" QUIT INT TERM EXIT
echo hello
sleep 30s
echo bye
Create a lock file in a known location and check for existence on script start? Putting the PID in the file might be helpful if someone's attempting to track down an errant instance that's preventing execution of the script.
This example is explained in the man flock, but it needs some impovements, because we should manage bugs and exit codes:
#!/bin/bash
#set -e this is useful only for very stupid scripts because script fails when anything command exits with status more than 0 !! without possibility for capture exit codes. not all commands exits >0 are failed.
( #start subprocess
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200
if [ "$?" != "0" ]; then echo Cannot lock!; exit 1; fi
echo $$>>/var/lock/.myscript.exclusivelock #for backward lockdir compatibility, notice this command is executed AFTER command bottom ) 200>/var/lock/.myscript.exclusivelock.
# Do stuff
# you can properly manage exit codes with multiple command and process algorithm.
# I suggest throw this all to external procedure than can properly handle exit X commands
) 200>/var/lock/.myscript.exclusivelock #exit subprocess
FLOCKEXIT=$? #save exitcode status
#do some finish commands
exit $FLOCKEXIT #return properly exitcode, may be usefull inside external scripts
You can use another method, list processes that I used in the past. But this is more complicated that method above. You should list processes by ps, filter by its name, additional filter grep -v grep for remove parasite nad finally count it by grep -c . and compare with number. Its complicated and uncertain
The existing answers posted either rely on the CLI utility flock or do not properly secure the lock file. The flock utility is not available on all non-Linux systems (i.e. FreeBSD), and does not work properly on NFS.
In my early days of system administration and system development, I was told that a safe and relatively portable method of creating a lock file was to create a temp file using mkemp(3) or mkemp(1), write identifying information to the temp file (i.e. PID), then hard link the temp file to the lock file. If the link was successful, then you have successfully obtained the lock.
When using locks in shell scripts, I typically place an obtain_lock() function in a shared profile and then source it from the scripts. Below is an example of my lock function:
obtain_lock()
{
LOCK="${1}"
LOCKDIR="$(dirname "${LOCK}")"
LOCKFILE="$(basename "${LOCK}")"
# create temp lock file
TMPLOCK=$(mktemp -p "${LOCKDIR}" "${LOCKFILE}XXXXXX" 2> /dev/null)
if test "x${TMPLOCK}" == "x";then
echo "unable to create temporary file with mktemp" 1>&2
return 1
fi
echo "$$" > "${TMPLOCK}"
# attempt to obtain lock file
ln "${TMPLOCK}" "${LOCK}" 2> /dev/null
if test $? -ne 0;then
rm -f "${TMPLOCK}"
echo "unable to obtain lockfile" 1>&2
if test -f "${LOCK}";then
echo "current lock information held by: $(cat "${LOCK}")" 1>&2
fi
return 2
fi
rm -f "${TMPLOCK}"
return 0;
};
The following is an example of how to use the lock function:
#!/bin/sh
. /path/to/locking/profile.sh
PROG_LOCKFILE="/tmp/myprog.lock"
clean_up()
{
rm -f "${PROG_LOCKFILE}"
}
obtain_lock "${PROG_LOCKFILE}"
if test $? -ne 0;then
exit 1
fi
trap clean_up SIGHUP SIGINT SIGTERM
# bulk of script
clean_up
exit 0
# end of script
Remember to call clean_up at any exit points in your script.
I've used the above in both Linux and FreeBSD environments.
Add this line at the beginning of your script
[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$#" || :
It's a boilerplate code from man flock.
If you want more logging, use this one
[ "${FLOCKER}" != "$0" ] && { echo "Trying to start build from queue... "; exec bash -c "FLOCKER='$0' flock -E $E_LOCKED -en '$0' '$0' '$#' || if [ \"\$?\" -eq $E_LOCKED ]; then echo 'Locked.'; fi"; } || echo "Lock is free. Completing."
This sets and checks locks using flock utility.
This code detects if it was run first time by checking FLOCKER variable, if it is not set to script name, then it tries to start script again recursively using flock and with FLOCKER variable initialized, if FLOCKER is set correctly, then flock on previous iteration succeeded and it is OK to proceed. If lock is busy, it fails with configurable exit code.
It seems to not work on Debian 7, but seems to work back again with experimental util-linux 2.25 package. It writes "flock: ... Text file busy". It could be overridden by disabling write permission on your script.
When targeting a Debian machine I find the lockfile-progs package to be a good solution. procmail also comes with a lockfile tool. However sometimes I am stuck with neither of these.
Here's my solution which uses mkdir for atomic-ness and a PID file to detect stale locks. This code is currently in production on a Cygwin setup and works well.
To use it simply call exclusive_lock_require when you need get exclusive access to something. An optional lock name parameter lets you share locks between different scripts. There's also two lower level functions (exclusive_lock_try and exclusive_lock_retry) should you need something more complex.
function exclusive_lock_try() # [lockname]
{
local LOCK_NAME="${1:-`basename $0`}"
LOCK_DIR="/tmp/.${LOCK_NAME}.lock"
local LOCK_PID_FILE="${LOCK_DIR}/${LOCK_NAME}.pid"
if [ -e "$LOCK_DIR" ]
then
local LOCK_PID="`cat "$LOCK_PID_FILE" 2> /dev/null`"
if [ ! -z "$LOCK_PID" ] && kill -0 "$LOCK_PID" 2> /dev/null
then
# locked by non-dead process
echo "\"$LOCK_NAME\" lock currently held by PID $LOCK_PID"
return 1
else
# orphaned lock, take it over
( echo $$ > "$LOCK_PID_FILE" ) 2> /dev/null && local LOCK_PID="$$"
fi
fi
if [ "`trap -p EXIT`" != "" ]
then
# already have an EXIT trap
echo "Cannot get lock, already have an EXIT trap"
return 1
fi
if [ "$LOCK_PID" != "$$" ] &&
! ( umask 077 && mkdir "$LOCK_DIR" && umask 177 && echo $$ > "$LOCK_PID_FILE" ) 2> /dev/null
then
local LOCK_PID="`cat "$LOCK_PID_FILE" 2> /dev/null`"
# unable to acquire lock, new process got in first
echo "\"$LOCK_NAME\" lock currently held by PID $LOCK_PID"
return 1
fi
trap "/bin/rm -rf \"$LOCK_DIR\"; exit;" EXIT
return 0 # got lock
}
function exclusive_lock_retry() # [lockname] [retries] [delay]
{
local LOCK_NAME="$1"
local MAX_TRIES="${2:-5}"
local DELAY="${3:-2}"
local TRIES=0
local LOCK_RETVAL
while [ "$TRIES" -lt "$MAX_TRIES" ]
do
if [ "$TRIES" -gt 0 ]
then
sleep "$DELAY"
fi
local TRIES=$(( $TRIES + 1 ))
if [ "$TRIES" -lt "$MAX_TRIES" ]
then
exclusive_lock_try "$LOCK_NAME" > /dev/null
else
exclusive_lock_try "$LOCK_NAME"
fi
LOCK_RETVAL="${PIPESTATUS[0]}"
if [ "$LOCK_RETVAL" -eq 0 ]
then
return 0
fi
done
return "$LOCK_RETVAL"
}
function exclusive_lock_require() # [lockname] [retries] [delay]
{
if ! exclusive_lock_retry "$#"
then
exit 1
fi
}
If flock's limitations, which have already been described elsewhere on this thread, aren't an issue for you, then this should work:
#!/bin/bash
{
# exit if we are unable to obtain a lock; this would happen if
# the script is already running elsewhere
# note: -x (exclusive) is the default
flock -n 100 || exit
# put commands to run here
sleep 100
} 100>/tmp/myjob.lock
Some unixes have lockfile which is very similar to the already mentioned flock.
From the manpage:
lockfile can be used to create one
or more semaphore files. If lock-
file can't create all the specified
files (in the specified order), it
waits sleeptime (defaults to 8)
seconds and retries the last file that
didn't succeed. You can specify the
number of retries to do until
failure is returned. If the number
of retries is -1 (default, i.e.,
-r-1) lockfile will retry forever.
I use a simple approach that handles stale lock files.
Note that some of the above solutions that store the pid, ignore the fact that the pid can wrap around. So - just checking if there is a valid process with the stored pid is not enough, especially for long running scripts.
I use noclobber to make sure only one script can open and write to the lock file at one time. Further, I store enough information to uniquely identify a process in the lockfile. I define the set of data to uniquely identify a process to be pid,ppid,lstart.
When a new script starts up, if it fails to create the lock file, it then verifies that the process that created the lock file is still around. If not, we assume the original process died an ungraceful death, and left a stale lock file. The new script then takes ownership of the lock file, and all is well the world, again.
Should work with multiple shells across multiple platforms. Fast, portable and simple.
#!/usr/bin/env sh
# Author: rouble
LOCKFILE=/var/tmp/lockfile #customize this line
trap release INT TERM EXIT
# Creates a lockfile. Sets global variable $ACQUIRED to true on success.
#
# Returns 0 if it is successfully able to create lockfile.
acquire () {
set -C #Shell noclobber option. If file exists, > will fail.
UUID=`ps -eo pid,ppid,lstart $$ | tail -1`
if (echo "$UUID" > "$LOCKFILE") 2>/dev/null; then
ACQUIRED="TRUE"
return 0
else
if [ -e $LOCKFILE ]; then
# We may be dealing with a stale lock file.
# Bring out the magnifying glass.
CURRENT_UUID_FROM_LOCKFILE=`cat $LOCKFILE`
CURRENT_PID_FROM_LOCKFILE=`cat $LOCKFILE | cut -f 1 -d " "`
CURRENT_UUID_FROM_PS=`ps -eo pid,ppid,lstart $CURRENT_PID_FROM_LOCKFILE | tail -1`
if [ "$CURRENT_UUID_FROM_LOCKFILE" == "$CURRENT_UUID_FROM_PS" ]; then
echo "Script already running with following identification: $CURRENT_UUID_FROM_LOCKFILE" >&2
return 1
else
# The process that created this lock file died an ungraceful death.
# Take ownership of the lock file.
echo "The process $CURRENT_UUID_FROM_LOCKFILE is no longer around. Taking ownership of $LOCKFILE"
release "FORCE"
if (echo "$UUID" > "$LOCKFILE") 2>/dev/null; then
ACQUIRED="TRUE"
return 0
else
echo "Cannot write to $LOCKFILE. Error." >&2
return 1
fi
fi
else
echo "Do you have write permissons to $LOCKFILE ?" >&2
return 1
fi
fi
}
# Removes the lock file only if this script created it ($ACQUIRED is set),
# OR, if we are removing a stale lock file (first parameter is "FORCE")
release () {
#Destroy lock file. Take no prisoners.
if [ "$ACQUIRED" ] || [ "$1" == "FORCE" ]; then
rm -f $LOCKFILE
fi
}
# Test code
# int main( int argc, const char* argv[] )
echo "Acquring lock."
acquire
if [ $? -eq 0 ]; then
echo "Acquired lock."
read -p "Press [Enter] key to release lock..."
release
echo "Released lock."
else
echo "Unable to acquire lock."
fi
I wanted to do away with lockfiles, lockdirs, special locking programs and even pidof since it isn't found on all Linux installations. Also wanted to have the simplest code possible (or at least as few lines as possible). Simplest if statement, in one line:
if [[ $(ps axf | awk -v pid=$$ '$1!=pid && $6~/'$(basename $0)'/{print $1}') ]]; then echo "Already running"; exit; fi
Actually although the answer of bmdhacks is almost good, there is a slight chance the second script to run after first checked the lockfile and before it wrote it. So they both will write the lock file and they will both be running. Here is how to make it work for sure:
lockfile=/var/lock/myscript.lock
if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null ; then
trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
else
# or you can decide to skip the "else" part if you want
echo "Another instance is already running!"
fi
The noclobber option will make sure that redirect command will fail if file already exists. So the redirect command is actually atomic - you write and check the file with one command. You don't need to remove the lockfile at the end of file - it'll be removed by the trap. I hope this helps to people that will read it later.
P.S. I didn't see that Mikel already answered the question correctly, although he didn't include the trap command to reduce the chance the lock file will be left over after stopping the script with Ctrl-C for example. So this is the complete solution
An example with flock(1) but without subshell. flock()ed file /tmp/foo is never removed, but that doesn't matter as it gets flock() and un-flock()ed.
#!/bin/bash
exec 9<> /tmp/foo
flock -n 9
RET=$?
if [[ $RET -ne 0 ]] ; then
echo "lock failed, exiting"
exit
fi
#Now we are inside the "critical section"
echo "inside lock"
sleep 5
exec 9>&- #close fd 9, and release lock
#The part below is outside the critical section (the lock)
echo "lock released"
sleep 5
This one line answer comes from someone related Ask Ubuntu Q&A:
[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$#" || :
# This is useful boilerplate code for shell scripts. Put it at the top of
# the shell script you want to lock and it'll automatically lock itself on
# the first run. If the env var $FLOCKER is not set to the shell script
# that is being run, then execute flock and grab an exclusive non-blocking
# lock (using the script itself as the lock file) before re-execing itself
# with the right arguments. It also sets the FLOCKER env var to the right
# value so it doesn't run again.
PID and lockfiles are definitely the most reliable. When you attempt to run the program, it can check for the lockfile which and if it exists, it can use ps to see if the process is still running. If it's not, the script can start, updating the PID in the lockfile to its own.
I find that bmdhack's solution is the most practical, at least for my use case. Using flock and lockfile rely on removing the lockfile using rm when the script terminates, which can't always be guaranteed (e.g., kill -9).
I would change one minor thing about bmdhack's solution: It makes a point of removing the lock file, without stating that this is unnecessary for the safe working of this semaphore. His use of kill -0 ensures that an old lockfile for a dead process will simply be ignored/over-written.
My simplified solution is therefore to simply add the following to the top of your singleton:
## Test the lock
LOCKFILE=/tmp/singleton.lock
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
echo "Script already running. bye!"
exit
fi
## Set the lock
echo $$ > ${LOCKFILE}
Of course, this script still has the flaw that processes that are likely to start at the same time have a race hazard, as the lock test and set operations are not a single atomic action. But the proposed solution for this by lhunath to use mkdir has the flaw that a killed script may leave behind the directory, thus preventing other instances from running.
The semaphoric utility uses flock (as discussed above, e.g. by presto8) to implement a counting semaphore. It enables any specific number of concurrent processes you want. We use it to limit the level of concurrency of various queue worker processes.
It's like sem but much lighter-weight. (Full disclosure: I wrote it after finding the sem was way too heavy for our needs and there wasn't a simple counting semaphore utility available.)
Answered a million times already, but another way, without the need for external dependencies:
LOCK_FILE="/var/lock/$(basename "$0").pid"
trap "rm -f ${LOCK_FILE}; exit" INT TERM EXIT
if [[ -f $LOCK_FILE && -d /proc/`cat $LOCK_FILE` ]]; then
// Process already exists
exit 1
fi
echo $$ > $LOCK_FILE
Each time it writes the current PID ($$) into the lockfile and on script startup checks if a process is running with the latest PID.
Using the process's lock is much stronger and takes care of the ungraceful exits also.
lock_file is kept open as long as the process is running. It will be closed (by shell) once the process exists (even if it gets killed).
I found this to be very efficient:
lock_file=/tmp/`basename $0`.lock
if fuser $lock_file > /dev/null 2>&1; then
echo "WARNING: Other instance of $(basename $0) running."
exit 1
fi
exec 3> $lock_file
I use oneliner # the very beginning of script:
#!/bin/bash
if [[ $(pgrep -afc "$(basename "$0")") -gt "1" ]]; then echo "Another instance of "$0" has already been started!" && exit; fi
.
the_beginning_of_actual_script
It is good to see the presence of process in the memory (no matter what the status of process is); but it does the job for me.
If you do not want to or cannot use flock (e.g. you are not using a shared file system), consider using an external service like lockable.
It exposes advisory lock primitives, much like flock would. In particular, you can acquire a lock via:
https://lockable.dev/api/acquire/my-lock-name
and release it via
https://lockable.dev/api/release/my-lock-name
By wrapping script execution with lock acquisition and release, you can make sure only a single instance of the process is running at any given time.
What's a quick-and-dirty way to make sure that only one instance of a shell script is running at a given time?
Use flock(1) to make an exclusive scoped lock a on file descriptor. This way you can even synchronize different parts of the script.
#!/bin/bash
(
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200 || exit 1
# Do stuff
) 200>/var/lock/.myscript.exclusivelock
This ensures that code between ( and ) is run only by one process at a time and that the process doesn’t wait too long for a lock.
Caveat: this particular command is a part of util-linux. If you run an operating system other than Linux, it may or may not be available.
Naive approaches that test the existence of "lock files" are flawed.
Why? Because they don't check whether the file exists and create it in a single atomic action. Because of this; there is a race condition that WILL make your attempts at mutual exclusion break.
Instead, you can use mkdir. mkdir creates a directory if it doesn't exist yet, and if it does, it sets an exit code. More importantly, it does all this in a single atomic action making it perfect for this scenario.
if ! mkdir /tmp/myscript.lock 2>/dev/null; then
echo "Myscript is already running." >&2
exit 1
fi
For all details, see the excellent BashFAQ: http://mywiki.wooledge.org/BashFAQ/045
If you want to take care of stale locks, fuser(1) comes in handy. The only downside here is that the operation takes about a second, so it isn't instant.
Here's a function I wrote once that solves the problem using fuser:
# mutex file
#
# Open a mutual exclusion lock on the file, unless another process already owns one.
#
# If the file is already locked by another process, the operation fails.
# This function defines a lock on a file as having a file descriptor open to the file.
# This function uses FD 9 to open a lock on the file. To release the lock, close FD 9:
# exec 9>&-
#
mutex() {
local file=$1 pid pids
exec 9>>"$file"
{ pids=$(fuser -f "$file"); } 2>&- 9>&-
for pid in $pids; do
[[ $pid = $$ ]] && continue
exec 9>&-
return 1 # Locked by a pid.
done
}
You can use it in a script like so:
mutex /var/run/myscript.lock || { echo "Already running." >&2; exit 1; }
If you don't care about portability (these solutions should work on pretty much any UNIX box), Linux' fuser(1) offers some additional options and there is also flock(1).
Here's an implementation that uses a lockfile and echoes a PID into it. This serves as a protection if the process is killed before removing the pidfile:
LOCKFILE=/tmp/lock.txt
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
echo "already running"
exit
fi
# make sure the lockfile is removed when we exit and then claim it
trap "rm -f ${LOCKFILE}; exit" INT TERM EXIT
echo $$ > ${LOCKFILE}
# do stuff
sleep 1000
rm -f ${LOCKFILE}
The trick here is the kill -0 which doesn't deliver any signal but just checks if a process with the given PID exists. Also the call to trap will ensure that the lockfile is removed even when your process is killed (except kill -9).
There's a wrapper around the flock(2) system call called, unimaginatively, flock(1). This makes it relatively easy to reliably obtain exclusive locks without worrying about cleanup etc. There are examples on the man page as to how to use it in a shell script.
To make locking reliable you need an atomic operation. Many of the above proposals
are not atomic. The proposed lockfile(1) utility looks promising as the man-page
mentioned, that its "NFS-resistant". If your OS does not support lockfile(1) and
your solution has to work on NFS, you have not many options....
NFSv2 has two atomic operations:
symlink
rename
With NFSv3 the create call is also atomic.
Directory operations are NOT atomic under NFSv2 and NFSv3 (please refer to the book 'NFS Illustrated' by Brent Callaghan, ISBN 0-201-32570-5; Brent is a NFS-veteran at Sun).
Knowing this, you can implement spin-locks for files and directories (in shell, not PHP):
lock current dir:
while ! ln -s . lock; do :; done
lock a file:
while ! ln -s ${f} ${f}.lock; do :; done
unlock current dir (assumption, the running process really acquired the lock):
mv lock deleteme && rm deleteme
unlock a file (assumption, the running process really acquired the lock):
mv ${f}.lock ${f}.deleteme && rm ${f}.deleteme
Remove is also not atomic, therefore first the rename (which is atomic) and then the remove.
For the symlink and rename calls, both filenames have to reside on the same filesystem. My proposal: use only simple filenames (no paths) and put file and lock into the same directory.
You need an atomic operation, like flock, else this will eventually fail.
But what to do if flock is not available. Well there is mkdir. That's an atomic operation too. Only one process will result in a successful mkdir, all others will fail.
So the code is:
if mkdir /var/lock/.myscript.exclusivelock
then
# do stuff
:
rmdir /var/lock/.myscript.exclusivelock
fi
You need to take care of stale locks else aftr a crash your script will never run again.
Another option is to use shell's noclobber option by running set -C. Then > will fail if the file already exists.
In brief:
set -C
lockfile="/tmp/locktest.lock"
if echo "$$" > "$lockfile"; then
echo "Successfully acquired lock"
# do work
rm "$lockfile" # XXX or via trap - see below
else
echo "Cannot acquire lock - already locked by $(cat "$lockfile")"
fi
This causes the shell to call:
open(pathname, O_CREAT|O_EXCL)
which atomically creates the file or fails if the file already exists.
According to a comment on BashFAQ 045, this may fail in ksh88, but it works in all my shells:
$ strace -e trace=creat,open -f /bin/bash /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/zsh /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/pdksh /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_LARGEFILE, 0666) = 3
$ strace -e trace=creat,open -f /bin/dash /home/mikel/bin/testopen 2>&1 | grep -F testopen.lock
open("/tmp/testopen.lock", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0666) = 3
Interesting that pdksh adds the O_TRUNC flag, but obviously it's redundant:
either you're creating an empty file, or you're not doing anything.
How you do the rm depends on how you want unclean exits to be handled.
Delete on clean exit
New runs fail until the issue that caused the unclean exit to be resolved and the lockfile is manually removed.
# acquire lock
# do work (code here may call exit, etc.)
rm "$lockfile"
Delete on any exit
New runs succeed provided the script is not already running.
trap 'rm "$lockfile"' EXIT
You can use GNU Parallel for this as it works as a mutex when called as sem. So, in concrete terms, you can use:
sem --id SCRIPTSINGLETON yourScript
If you want a timeout too, use:
sem --id SCRIPTSINGLETON --semaphoretimeout -10 yourScript
Timeout of <0 means exit without running script if semaphore is not released within the timeout, timeout of >0 mean run the script anyway.
Note that you should give it a name (with --id) else it defaults to the controlling terminal.
GNU Parallel is a very simple install on most Linux/OSX/Unix platforms - it is just a Perl script.
For shell scripts, I tend to go with the mkdir over flock as it makes the locks more portable.
Either way, using set -e isn't enough. That only exits the script if any command fails. Your locks will still be left behind.
For proper lock cleanup, you really should set your traps to something like this psuedo code (lifted, simplified and untested but from actively used scripts) :
#=======================================================================
# Predefined Global Variables
#=======================================================================
TMPDIR=/tmp/myapp
[[ ! -d $TMP_DIR ]] \
&& mkdir -p $TMP_DIR \
&& chmod 700 $TMPDIR
LOCK_DIR=$TMP_DIR/lock
#=======================================================================
# Functions
#=======================================================================
function mklock {
__lockdir="$LOCK_DIR/$(date +%s.%N).$$" # Private Global. Use Epoch.Nano.PID
# If it can create $LOCK_DIR then no other instance is running
if $(mkdir $LOCK_DIR)
then
mkdir $__lockdir # create this instance's specific lock in queue
LOCK_EXISTS=true # Global
else
echo "FATAL: Lock already exists. Another copy is running or manually lock clean up required."
exit 1001 # Or work out some sleep_while_execution_lock elsewhere
fi
}
function rmlock {
[[ ! -d $__lockdir ]] \
&& echo "WARNING: Lock is missing. $__lockdir does not exist" \
|| rmdir $__lockdir
}
#-----------------------------------------------------------------------
# Private Signal Traps Functions {{{2
#
# DANGER: SIGKILL cannot be trapped. So, try not to `kill -9 PID` or
# there will be *NO CLEAN UP*. You'll have to manually remove
# any locks in place.
#-----------------------------------------------------------------------
function __sig_exit {
# Place your clean up logic here
# Remove the LOCK
[[ -n $LOCK_EXISTS ]] && rmlock
}
function __sig_int {
echo "WARNING: SIGINT caught"
exit 1002
}
function __sig_quit {
echo "SIGQUIT caught"
exit 1003
}
function __sig_term {
echo "WARNING: SIGTERM caught"
exit 1015
}
#=======================================================================
# Main
#=======================================================================
# Set TRAPs
trap __sig_exit EXIT # SIGEXIT
trap __sig_int INT # SIGINT
trap __sig_quit QUIT # SIGQUIT
trap __sig_term TERM # SIGTERM
mklock
# CODE
exit # No need for cleanup code here being in the __sig_exit trap function
Here's what will happen. All traps will produce an exit so the function __sig_exit will always happen (barring a SIGKILL) which cleans up your locks.
Note: my exit values are not low values. Why? Various batch processing systems make or have expectations of the numbers 0 through 31. Setting them to something else, I can have my scripts and batch streams react accordingly to the previous batch job or script.
Really quick and really dirty? This one-liner on the top of your script will work:
[[ $(pgrep -c "`basename \"$0\"`") -gt 1 ]] && exit
Of course, just make sure that your script name is unique. :)
Here's an approach that combines atomic directory locking with a check for stale lock via PID and restart if stale. Also, this does not rely on any bashisms.
#!/bin/dash
SCRIPTNAME=$(basename $0)
LOCKDIR="/var/lock/${SCRIPTNAME}"
PIDFILE="${LOCKDIR}/pid"
if ! mkdir $LOCKDIR 2>/dev/null
then
# lock failed, but check for stale one by checking if the PID is really existing
PID=$(cat $PIDFILE)
if ! kill -0 $PID 2>/dev/null
then
echo "Removing stale lock of nonexistent PID ${PID}" >&2
rm -rf $LOCKDIR
echo "Restarting myself (${SCRIPTNAME})" >&2
exec "$0" "$#"
fi
echo "$SCRIPTNAME is already running, bailing out" >&2
exit 1
else
# lock successfully acquired, save PID
echo $$ > $PIDFILE
fi
trap "rm -rf ${LOCKDIR}" QUIT INT TERM EXIT
echo hello
sleep 30s
echo bye
Create a lock file in a known location and check for existence on script start? Putting the PID in the file might be helpful if someone's attempting to track down an errant instance that's preventing execution of the script.
This example is explained in the man flock, but it needs some impovements, because we should manage bugs and exit codes:
#!/bin/bash
#set -e this is useful only for very stupid scripts because script fails when anything command exits with status more than 0 !! without possibility for capture exit codes. not all commands exits >0 are failed.
( #start subprocess
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200
if [ "$?" != "0" ]; then echo Cannot lock!; exit 1; fi
echo $$>>/var/lock/.myscript.exclusivelock #for backward lockdir compatibility, notice this command is executed AFTER command bottom ) 200>/var/lock/.myscript.exclusivelock.
# Do stuff
# you can properly manage exit codes with multiple command and process algorithm.
# I suggest throw this all to external procedure than can properly handle exit X commands
) 200>/var/lock/.myscript.exclusivelock #exit subprocess
FLOCKEXIT=$? #save exitcode status
#do some finish commands
exit $FLOCKEXIT #return properly exitcode, may be usefull inside external scripts
You can use another method, list processes that I used in the past. But this is more complicated that method above. You should list processes by ps, filter by its name, additional filter grep -v grep for remove parasite nad finally count it by grep -c . and compare with number. Its complicated and uncertain
The existing answers posted either rely on the CLI utility flock or do not properly secure the lock file. The flock utility is not available on all non-Linux systems (i.e. FreeBSD), and does not work properly on NFS.
In my early days of system administration and system development, I was told that a safe and relatively portable method of creating a lock file was to create a temp file using mkemp(3) or mkemp(1), write identifying information to the temp file (i.e. PID), then hard link the temp file to the lock file. If the link was successful, then you have successfully obtained the lock.
When using locks in shell scripts, I typically place an obtain_lock() function in a shared profile and then source it from the scripts. Below is an example of my lock function:
obtain_lock()
{
LOCK="${1}"
LOCKDIR="$(dirname "${LOCK}")"
LOCKFILE="$(basename "${LOCK}")"
# create temp lock file
TMPLOCK=$(mktemp -p "${LOCKDIR}" "${LOCKFILE}XXXXXX" 2> /dev/null)
if test "x${TMPLOCK}" == "x";then
echo "unable to create temporary file with mktemp" 1>&2
return 1
fi
echo "$$" > "${TMPLOCK}"
# attempt to obtain lock file
ln "${TMPLOCK}" "${LOCK}" 2> /dev/null
if test $? -ne 0;then
rm -f "${TMPLOCK}"
echo "unable to obtain lockfile" 1>&2
if test -f "${LOCK}";then
echo "current lock information held by: $(cat "${LOCK}")" 1>&2
fi
return 2
fi
rm -f "${TMPLOCK}"
return 0;
};
The following is an example of how to use the lock function:
#!/bin/sh
. /path/to/locking/profile.sh
PROG_LOCKFILE="/tmp/myprog.lock"
clean_up()
{
rm -f "${PROG_LOCKFILE}"
}
obtain_lock "${PROG_LOCKFILE}"
if test $? -ne 0;then
exit 1
fi
trap clean_up SIGHUP SIGINT SIGTERM
# bulk of script
clean_up
exit 0
# end of script
Remember to call clean_up at any exit points in your script.
I've used the above in both Linux and FreeBSD environments.
Add this line at the beginning of your script
[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$#" || :
It's a boilerplate code from man flock.
If you want more logging, use this one
[ "${FLOCKER}" != "$0" ] && { echo "Trying to start build from queue... "; exec bash -c "FLOCKER='$0' flock -E $E_LOCKED -en '$0' '$0' '$#' || if [ \"\$?\" -eq $E_LOCKED ]; then echo 'Locked.'; fi"; } || echo "Lock is free. Completing."
This sets and checks locks using flock utility.
This code detects if it was run first time by checking FLOCKER variable, if it is not set to script name, then it tries to start script again recursively using flock and with FLOCKER variable initialized, if FLOCKER is set correctly, then flock on previous iteration succeeded and it is OK to proceed. If lock is busy, it fails with configurable exit code.
It seems to not work on Debian 7, but seems to work back again with experimental util-linux 2.25 package. It writes "flock: ... Text file busy". It could be overridden by disabling write permission on your script.
When targeting a Debian machine I find the lockfile-progs package to be a good solution. procmail also comes with a lockfile tool. However sometimes I am stuck with neither of these.
Here's my solution which uses mkdir for atomic-ness and a PID file to detect stale locks. This code is currently in production on a Cygwin setup and works well.
To use it simply call exclusive_lock_require when you need get exclusive access to something. An optional lock name parameter lets you share locks between different scripts. There's also two lower level functions (exclusive_lock_try and exclusive_lock_retry) should you need something more complex.
function exclusive_lock_try() # [lockname]
{
local LOCK_NAME="${1:-`basename $0`}"
LOCK_DIR="/tmp/.${LOCK_NAME}.lock"
local LOCK_PID_FILE="${LOCK_DIR}/${LOCK_NAME}.pid"
if [ -e "$LOCK_DIR" ]
then
local LOCK_PID="`cat "$LOCK_PID_FILE" 2> /dev/null`"
if [ ! -z "$LOCK_PID" ] && kill -0 "$LOCK_PID" 2> /dev/null
then
# locked by non-dead process
echo "\"$LOCK_NAME\" lock currently held by PID $LOCK_PID"
return 1
else
# orphaned lock, take it over
( echo $$ > "$LOCK_PID_FILE" ) 2> /dev/null && local LOCK_PID="$$"
fi
fi
if [ "`trap -p EXIT`" != "" ]
then
# already have an EXIT trap
echo "Cannot get lock, already have an EXIT trap"
return 1
fi
if [ "$LOCK_PID" != "$$" ] &&
! ( umask 077 && mkdir "$LOCK_DIR" && umask 177 && echo $$ > "$LOCK_PID_FILE" ) 2> /dev/null
then
local LOCK_PID="`cat "$LOCK_PID_FILE" 2> /dev/null`"
# unable to acquire lock, new process got in first
echo "\"$LOCK_NAME\" lock currently held by PID $LOCK_PID"
return 1
fi
trap "/bin/rm -rf \"$LOCK_DIR\"; exit;" EXIT
return 0 # got lock
}
function exclusive_lock_retry() # [lockname] [retries] [delay]
{
local LOCK_NAME="$1"
local MAX_TRIES="${2:-5}"
local DELAY="${3:-2}"
local TRIES=0
local LOCK_RETVAL
while [ "$TRIES" -lt "$MAX_TRIES" ]
do
if [ "$TRIES" -gt 0 ]
then
sleep "$DELAY"
fi
local TRIES=$(( $TRIES + 1 ))
if [ "$TRIES" -lt "$MAX_TRIES" ]
then
exclusive_lock_try "$LOCK_NAME" > /dev/null
else
exclusive_lock_try "$LOCK_NAME"
fi
LOCK_RETVAL="${PIPESTATUS[0]}"
if [ "$LOCK_RETVAL" -eq 0 ]
then
return 0
fi
done
return "$LOCK_RETVAL"
}
function exclusive_lock_require() # [lockname] [retries] [delay]
{
if ! exclusive_lock_retry "$#"
then
exit 1
fi
}
If flock's limitations, which have already been described elsewhere on this thread, aren't an issue for you, then this should work:
#!/bin/bash
{
# exit if we are unable to obtain a lock; this would happen if
# the script is already running elsewhere
# note: -x (exclusive) is the default
flock -n 100 || exit
# put commands to run here
sleep 100
} 100>/tmp/myjob.lock
Some unixes have lockfile which is very similar to the already mentioned flock.
From the manpage:
lockfile can be used to create one
or more semaphore files. If lock-
file can't create all the specified
files (in the specified order), it
waits sleeptime (defaults to 8)
seconds and retries the last file that
didn't succeed. You can specify the
number of retries to do until
failure is returned. If the number
of retries is -1 (default, i.e.,
-r-1) lockfile will retry forever.
I use a simple approach that handles stale lock files.
Note that some of the above solutions that store the pid, ignore the fact that the pid can wrap around. So - just checking if there is a valid process with the stored pid is not enough, especially for long running scripts.
I use noclobber to make sure only one script can open and write to the lock file at one time. Further, I store enough information to uniquely identify a process in the lockfile. I define the set of data to uniquely identify a process to be pid,ppid,lstart.
When a new script starts up, if it fails to create the lock file, it then verifies that the process that created the lock file is still around. If not, we assume the original process died an ungraceful death, and left a stale lock file. The new script then takes ownership of the lock file, and all is well the world, again.
Should work with multiple shells across multiple platforms. Fast, portable and simple.
#!/usr/bin/env sh
# Author: rouble
LOCKFILE=/var/tmp/lockfile #customize this line
trap release INT TERM EXIT
# Creates a lockfile. Sets global variable $ACQUIRED to true on success.
#
# Returns 0 if it is successfully able to create lockfile.
acquire () {
set -C #Shell noclobber option. If file exists, > will fail.
UUID=`ps -eo pid,ppid,lstart $$ | tail -1`
if (echo "$UUID" > "$LOCKFILE") 2>/dev/null; then
ACQUIRED="TRUE"
return 0
else
if [ -e $LOCKFILE ]; then
# We may be dealing with a stale lock file.
# Bring out the magnifying glass.
CURRENT_UUID_FROM_LOCKFILE=`cat $LOCKFILE`
CURRENT_PID_FROM_LOCKFILE=`cat $LOCKFILE | cut -f 1 -d " "`
CURRENT_UUID_FROM_PS=`ps -eo pid,ppid,lstart $CURRENT_PID_FROM_LOCKFILE | tail -1`
if [ "$CURRENT_UUID_FROM_LOCKFILE" == "$CURRENT_UUID_FROM_PS" ]; then
echo "Script already running with following identification: $CURRENT_UUID_FROM_LOCKFILE" >&2
return 1
else
# The process that created this lock file died an ungraceful death.
# Take ownership of the lock file.
echo "The process $CURRENT_UUID_FROM_LOCKFILE is no longer around. Taking ownership of $LOCKFILE"
release "FORCE"
if (echo "$UUID" > "$LOCKFILE") 2>/dev/null; then
ACQUIRED="TRUE"
return 0
else
echo "Cannot write to $LOCKFILE. Error." >&2
return 1
fi
fi
else
echo "Do you have write permissons to $LOCKFILE ?" >&2
return 1
fi
fi
}
# Removes the lock file only if this script created it ($ACQUIRED is set),
# OR, if we are removing a stale lock file (first parameter is "FORCE")
release () {
#Destroy lock file. Take no prisoners.
if [ "$ACQUIRED" ] || [ "$1" == "FORCE" ]; then
rm -f $LOCKFILE
fi
}
# Test code
# int main( int argc, const char* argv[] )
echo "Acquring lock."
acquire
if [ $? -eq 0 ]; then
echo "Acquired lock."
read -p "Press [Enter] key to release lock..."
release
echo "Released lock."
else
echo "Unable to acquire lock."
fi
I wanted to do away with lockfiles, lockdirs, special locking programs and even pidof since it isn't found on all Linux installations. Also wanted to have the simplest code possible (or at least as few lines as possible). Simplest if statement, in one line:
if [[ $(ps axf | awk -v pid=$$ '$1!=pid && $6~/'$(basename $0)'/{print $1}') ]]; then echo "Already running"; exit; fi
Actually although the answer of bmdhacks is almost good, there is a slight chance the second script to run after first checked the lockfile and before it wrote it. So they both will write the lock file and they will both be running. Here is how to make it work for sure:
lockfile=/var/lock/myscript.lock
if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null ; then
trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
else
# or you can decide to skip the "else" part if you want
echo "Another instance is already running!"
fi
The noclobber option will make sure that redirect command will fail if file already exists. So the redirect command is actually atomic - you write and check the file with one command. You don't need to remove the lockfile at the end of file - it'll be removed by the trap. I hope this helps to people that will read it later.
P.S. I didn't see that Mikel already answered the question correctly, although he didn't include the trap command to reduce the chance the lock file will be left over after stopping the script with Ctrl-C for example. So this is the complete solution
An example with flock(1) but without subshell. flock()ed file /tmp/foo is never removed, but that doesn't matter as it gets flock() and un-flock()ed.
#!/bin/bash
exec 9<> /tmp/foo
flock -n 9
RET=$?
if [[ $RET -ne 0 ]] ; then
echo "lock failed, exiting"
exit
fi
#Now we are inside the "critical section"
echo "inside lock"
sleep 5
exec 9>&- #close fd 9, and release lock
#The part below is outside the critical section (the lock)
echo "lock released"
sleep 5
This one line answer comes from someone related Ask Ubuntu Q&A:
[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -en "$0" "$0" "$#" || :
# This is useful boilerplate code for shell scripts. Put it at the top of
# the shell script you want to lock and it'll automatically lock itself on
# the first run. If the env var $FLOCKER is not set to the shell script
# that is being run, then execute flock and grab an exclusive non-blocking
# lock (using the script itself as the lock file) before re-execing itself
# with the right arguments. It also sets the FLOCKER env var to the right
# value so it doesn't run again.
PID and lockfiles are definitely the most reliable. When you attempt to run the program, it can check for the lockfile which and if it exists, it can use ps to see if the process is still running. If it's not, the script can start, updating the PID in the lockfile to its own.
I find that bmdhack's solution is the most practical, at least for my use case. Using flock and lockfile rely on removing the lockfile using rm when the script terminates, which can't always be guaranteed (e.g., kill -9).
I would change one minor thing about bmdhack's solution: It makes a point of removing the lock file, without stating that this is unnecessary for the safe working of this semaphore. His use of kill -0 ensures that an old lockfile for a dead process will simply be ignored/over-written.
My simplified solution is therefore to simply add the following to the top of your singleton:
## Test the lock
LOCKFILE=/tmp/singleton.lock
if [ -e ${LOCKFILE} ] && kill -0 `cat ${LOCKFILE}`; then
echo "Script already running. bye!"
exit
fi
## Set the lock
echo $$ > ${LOCKFILE}
Of course, this script still has the flaw that processes that are likely to start at the same time have a race hazard, as the lock test and set operations are not a single atomic action. But the proposed solution for this by lhunath to use mkdir has the flaw that a killed script may leave behind the directory, thus preventing other instances from running.
The semaphoric utility uses flock (as discussed above, e.g. by presto8) to implement a counting semaphore. It enables any specific number of concurrent processes you want. We use it to limit the level of concurrency of various queue worker processes.
It's like sem but much lighter-weight. (Full disclosure: I wrote it after finding the sem was way too heavy for our needs and there wasn't a simple counting semaphore utility available.)
Answered a million times already, but another way, without the need for external dependencies:
LOCK_FILE="/var/lock/$(basename "$0").pid"
trap "rm -f ${LOCK_FILE}; exit" INT TERM EXIT
if [[ -f $LOCK_FILE && -d /proc/`cat $LOCK_FILE` ]]; then
// Process already exists
exit 1
fi
echo $$ > $LOCK_FILE
Each time it writes the current PID ($$) into the lockfile and on script startup checks if a process is running with the latest PID.
Using the process's lock is much stronger and takes care of the ungraceful exits also.
lock_file is kept open as long as the process is running. It will be closed (by shell) once the process exists (even if it gets killed).
I found this to be very efficient:
lock_file=/tmp/`basename $0`.lock
if fuser $lock_file > /dev/null 2>&1; then
echo "WARNING: Other instance of $(basename $0) running."
exit 1
fi
exec 3> $lock_file
I use oneliner # the very beginning of script:
#!/bin/bash
if [[ $(pgrep -afc "$(basename "$0")") -gt "1" ]]; then echo "Another instance of "$0" has already been started!" && exit; fi
.
the_beginning_of_actual_script
It is good to see the presence of process in the memory (no matter what the status of process is); but it does the job for me.
If you do not want to or cannot use flock (e.g. you are not using a shared file system), consider using an external service like lockable.
It exposes advisory lock primitives, much like flock would. In particular, you can acquire a lock via:
https://lockable.dev/api/acquire/my-lock-name
and release it via
https://lockable.dev/api/release/my-lock-name
By wrapping script execution with lock acquisition and release, you can make sure only a single instance of the process is running at any given time.