I would like to lock a directory while a Bash script is running and make sure it's not locked anymore when the script dies.
My script creates a directory, and I want to try deleting it, if it I can't delete it then it means it's locked. If it's not locked it should create the directory.
rm "$dir_path" > /dev/null 2>&1
if [ -d "$dir_path" ]; then
exit 0
fi
cp -r "$template_dir" "$dir_path"
# Lock directory
#LOCK "$dir_path"
# flock --exclusive --nonblock "$app_apex_path" # flock: bad file descriptor
# When script ends the lock is automatically removed without need to do any cleanup
# this is necessary because if for example in case of power failure the dir would still
# be locked on next boot.
I have looked into flock but it doesn't seem to work like this.
Here’s an example with advisory locking, which works fine as long as all participating scripts follow the same protocol.
set -e
if ! mkdir '/tmp/my-magic-lock'; then
exit 1 # Report an error, maybe?
fi
trap "rmdir '/tmp/my-magic-lock'" EXIT
# We hold the advisory lock now.
rm -Rf "$dir_path"
cp -a "$template_dir" "$dir_path"
As a side note, if I were to tackle this situation, I would simply make $template_dir and $dir_path Btrfs subvolumes and use snapshots instead of copies:
set -e
btrfs subvolume delete "$dir_path"
btrfs subvolume snapshot "$template_dir" "$dir_path"
This^^^ is way more efficient, “atomic” (in a number of beneficial ways), copy-on-write and also resilient towards multiple concurrent replacement attempts of the same kind, yielding a correct state once all attempts finish.
Related
I've got this:
#!/bin/bash
while :
do
SUM=$(tree | md5sum)
if [ "$SUMOLD" != "$SUM" ]; then
# do something here
SUMOLD=$SUM
sleep 1
fi
done
Which works just fine. But, the problem is that it consumes 50% of the CPU, a Core2 Duo T8300. Why is that? How to improve the efficiency?
This is a job for inotifywait. inotify is Linux's event-based system for monitoring files and directories for changes. Goodbye polling loops!
NAME
inotifywait - wait for changes to files using inotify
SYNOPSIS
inotifywait [-hcmrq] [-e <event> ] [-t <seconds> ] [--format <fmt> ]
[--timefmt <fmt> ] <file> [ ... ]
DESCRIPTION
inotifywait efficiently waits for changes to files using Linux's inotify(7) interface. It is suitable for waiting for changes to
files from shell scripts. It can either exit once an event occurs, or continually execute and output events as they occur.
Here's how you could write a simple loop which detects whenever files are added, modified, or deleted from a directory:
inotifywait -mq /dir | while read event; do
echo "something happened in /dir: $event"
done
Take a look at the man page for more options. If you only care about modifications and want to ignore files simply being read, you could use -e to limit the types of events.
While a OS specific solution like inotify would be better, you can dramatically improve your script by moving the sleep out of the if statement:
#!/bin/bash
while :
do
SUM=$(tree | md5sum)
if [ "$SUMOLD" != "$SUM" ]; then
# do something here
SUMOLD=$SUM
# move sleep from here
fi
sleep 1 # to here
done
CPU usage should drop dramatically now that you're always checking once per second, instead of as often as possible when there are no changes. You can also replace tree with find, and sleep for longer between each check.
The reason is that the script is continuously running even when the command sleep is called. My recommendation is to launch your script using the "inotifywait" or the "watch" (Alternative solution) command and avoid to use the while loop.
See: http://linux.die.net/man/1/watch
An example taken from the MAN pages:
To watch the contents of a directory change, you could use
watch -d ls -l
Watch is going to launch your script periodically but without continuously execute your script.
In kornshell, `basename $0` gives me the name of the current script.
How would I exploit $$ or $PPID to implement the singleton pattern of only having one script named `basename $0` executed on this server by any user?
ps -ef|grep `basename $0`
This will show me all processes which are running that have the name of the currently running script.
I need a script which can abort when a thread which is not $$ is running the script named `basename $0`.
To provide a race-free mutex, flock is your friend. If you aren't on Linux -- where it's provided by util-linux -- a portable version is available.
If you truly want it to apply to the entire system -- crossing user accounts -- you'll need a directory for your locks to live where all users can create files, and you'll need to ensure that all users can write to your lockfiles.
Assuming you have the flock utility, each program which wants to participate in this protocol can behave as follows:
#!/bin/ksh
umask 000 # allow all users to access the file we're about to create
exec 9>"/tmp/${0##*/}.lck" # open lockfile on FD 9, based on basename of argv[0]
umask 022 # move back to more restrictive file permissions
flock -x -n 9 || exit # grab that lock, or exit the script early
# continue
One key note: Do not try to delete lockfiles when your script exits. If you're in a condition where someone else is actively trying to grab a lock, they'll already have a file descriptor on that existing file; if you delete the file while they have a handle on it, you just ensured a race wherein that program can think it holds the lock while someone else creates a new file under the same name and locks it.
I have a shell script which usually runs nearly 10 mins for a single run,but i need to know if another request for running the script comes while a instance of the script is running already, whether new request need to wait for existing instance to compplete or a new instance will be started.
I need a new instance must be started whenever a request is available for the same script.
How to do it...
The shell script is a polling script which looks for a file in a directory and execute the file.The execution of the file takes nearly 10 min or more.But during execution if a new file arrives, it also has to be executed simultaneously.
the shell script is below, and how to modify it to execute multiple requests..
#!/bin/bash
while [ 1 ]; do
newfiles=`find /afs/rch/usr8/fsptools/WWW/cgi-bin/upload/ -newer /afs/rch/usr$
touch /afs/rch/usr8/fsptools/WWW/cgi-bin/upload/.my_marker
if [ -n "$newfiles" ]; then
echo "found files $newfiles"
name2=`ls /afs/rch/usr8/fsptools/WWW/cgi-bin/upload/ -Art |tail -n 2 |head $
echo " $name2 "
mkdir -p -m 0755 /afs/rch/usr8/fsptools/WWW/dumpspace/$name2
name1="/afs/rch/usr8/fsptools/WWW/dumpspace/fipsdumputils/fipsdumputil -e -$
$name1
touch /afs/rch/usr8/fsptools/WWW/dumpspace/tempfiles/$name2
fi
sleep 5
done
When writing scripts like the one you describe, I take one of two approaches.
First, you can use a pid file to indicate that a second copy should not run. For example:
#!/bin/sh
pidfile=/var/run/$(0##*/).pid
# remove pid if we exit normally or are terminated
trap "rm -f $pidfile" 0 1 3 15
# Write the pid as a symlink
if ! ln -s "pid=$$" "$pidfile"; then
echo "Already running. Exiting." >&2
exit 0
fi
# Do your stuff
I like using symlinks to store pid because writing a symlink is an atomic operation; two processes can't conflict with each other. You don't even need to check for the existence of the pid symlink, because a failure of ln clearly indicates that a pid cannot be set. That's either a permission or path problem, or it's due to the symlink already being there.
Second option is to make it possible .. nay, preferable .. not to block additional instances, and instead configure whatever it is that this script does to permit multiple servers to run at the same time on different queue entries. "Single-queue-single-server" is never as good as "single-queue-multi-server". Since you haven't included code in your question, I have no way to know whether this approach would be useful for you, but here's some explanatory meta bash:
#!/usr/bin/env bash
workdir=/var/tmp # Set a better $workdir than this.
a=( $(get_list_of_queue_ids) ) # A command? A function? Up to you.
for qid in "${a[#]}"; do
# Set a "lock" for this item .. or don't, and move on.
if ! ln -s "pid=$$" $workdir/$qid.working; then
continue
fi
# Do your stuff with just this $qid.
...
# And finally, clean up after ourselves
remove_qid_from_queue $qid
rm $workdir/$qid.working
done
The effect of this is to transfer the idea of "one at a time" from the handler to the data. If you have a multi-CPU system, you probably have enough capacity to handle multiple queue entries at the same time.
ghoti's answer shows some helpful techniques, if modifying the script is an option.
Generally speaking, for an existing script:
Unless you know with certainty that:
the script has no side effects other than to output to the terminal or to write to files with shell-instance specific names (such as incorporating $$, the current shell's PID, into filenames) or some other instance-specific location,
OR that the script was explicitly designed for parallel execution,
I would assume that you cannot safely run multiple copies of the script simultaneously.
It is not reasonable to expect the average shell script to be designed for concurrent use.
From the viewpoint of the operating system, several processes may of course execute the same program in parallel. No need to worry about this.
However, it is conceivable, that a (careless) programmer wrote the program in such a way that it produces incorrect results, when two copies are executed in parallel.
How do I avoid cronjob from executing multiple times on the same command? I had tried to look around and try to check and kill in processes but it doesn't work with the below code. With the below code it keeps entering into else condition where it suppose to be "running". Any idea which part I did it wrongly?
#!/bin/sh
devPath=`ps aux | grep "[i]mport_shell_script"` | xargs
if [ ! -z "$devPath" -a "$devPath" != " " ]; then
echo "running"
exit
else
while true
do
sudo /usr/bin/php /var/www/html/xxx/import_from_datafile.php /dev/null 2>&1
sleep 5
done
fi
exit
cronjob:
*/2 * * * * root /bin/sh /var/www/html/xxx/import_shell_script.sh /dev/null 2>&1
I don't see the point to add a cron job which then starts a loop that runs a job. Either use cron to run the job every minute or use a daemon script to make sure your service is started and is kept running.
To check whether your script is already running, you can use a lock directory (unless your daemon framework already does that for you):
LOCK=/tmp/script.lock # You may want a better name here
mkdir $LOCK || exit 1 # Exit with error if script is already running
trap "rmdir $LOCK" EXIT # Remove the lock when the script terminates
...normal code...
If your OS supports it, then /var/lock/script might be a better path.
Your next question is probably how to write a daemon. To answer that, I need to know what kind of Linux you're using and whether you have things like systemd, daemonize, etc.
check the presence of a file at the beginning of your script ( for example /tmp/runonce-import_shell_script ). If it exists, that means the same script is already running (or the previous one halted with an error).
You can also add a timestamp in that file so you can check since when the script was running (and maybe decide to run it again after 24h even if the file is present)
I wrote an alias in my .bashrc file that open a txt file every time I start bash shell.
The problem is that I would like to open such a file only once, that is the first time I open the shell.
Is there any way to do that?
The general solution to this problem is to have a session lock of some kind. You could create a file /tmp/secret with the pid and/or tty of the process which is editing the other file, and remove the lock file when done. Now, your other sessions should be set up to not create this file if it already exists.
Proper locking is a complex topic, but for the simple cases, this might be good enough. If not, google for "mutual exclusion". Do note that there may be security implications if you get it wrong.
Why are you using an alias for this? Sounds like the code should be directly in your .bashrc, not in an alias definition.
So if, say, what you have now in your .bashrc is something like
alias start_editing_my_project_work_hour_report='emacs ~/prj.txt &̈́'
start_editing_my_project_work_hour_report
unalias start_editing_my_project_work_hour_report
... then with the locking, and without the alias, you might end up with something like
# Obtain my UID on this host, and construct directory name and lock file name
uid=$(id -u)
dir=/tmp/prj-$uid
lock=$dir/pid.lock
# The loop will execute at most twice,
# but we don't know yet whether once is enough
while true; do
if mkdir -p "$dir"; then
# Yay, we have the lock!
( echo $$ >"$lock" ; emacs ~/prj.txt; rm -f "$lock" ) &
break
else
other=$(cat "$lock")
# If the process which created the UID is still live, do nothing
if kill -0 $other; then
break
else
echo "removing stale lock file dir (dead PID $other) and retrying" >&2
rm -rf "$dir"
continue
fi
fi
done