Log files is written line by line by underwater drones on a server. TWhen at surface, the drones speak slowly to the server (say ~200o/s on a phone line which is not stable) and only from time to time (say every ~6h). Depending on the messages, I have to execute commands on the server while the drones are online and when they hang up other commands. Other processes may be looking at the same files with similar tasks.
A lot can be found on this website on somewhat similar problems but the solution I have built on is still unsatisfactory. Presently I'm doing this with bash
while logfile_drone=`inotifywait -e create --format '%f' log_directory`; do
logfile=log_directory/${logfile_drone}
while action=`inotifywait -q -t 120 -e modify -e close --format '%e' ${logfile} ` ; do
exidCode=$?
lastLine=$( tail -n2 ${logFile} | head -n1 ) # because with tail -n1 I can got only part of the line. this happens quite often
match =$( # match set to true if lastLine matches some pattern )
if [[ $action == 'MODIFY' ]] && $match ; then # do something ; fi
if [[ $( echo $action | cut -c1-5 ) == 'CLOSE' ]] ; then
# do something
break
fi
if [[ $exitCode -eq 2 ]] ; then break ; fi
done
# do something after the drone has hang up
done # wait for a new call from the same or another drone
The main problems are :
the second inotify misses lines, may be because of the other processes looking at the same file.
the way I catch the time out doesn't seem to work.
I can't monitor 2 drones simultaneously.
Basically the code works more or less but isn't very robust. I wonder if problem 3 can be managed by putting the second while loop in a function which is put in background when called. Finally, I wonder if a higher level language (I'm familiar with php which has a PECL extension for inotify) would not do this much better. However, I imagine that php will not solve problem 3 better than than bash.
Here is the code where I'm facing the problem of abrupt exit from the while loop, implemented according to Philippe's answer, which works fine otherwise:
while read -r action ; do
...
resume=$( grep -e 'RESUMING MISSION' <<< $lastLine )
if [ -n "$resume" ] ; then
ssh user#another_server "/usr/bin/php /path_to_matlab_command/matlabCmd.php --drone=${vehicle}" &
fi
if [ $( echo $action | cut -c1-5 ) == 'CLOSE' ] ; then ... ; sigKill=true ; fi
...
if $sigKill ; then break; fi
done < <(inotifywait -q -m -e modify -e close_write --format '%e' ${logFile})
When I comment the line with ssh the script can exit properly with a break triggered by CLOSE, otherwise the while loop finishes abruptly after the ssh command. The ssh is put in background because the matlab code runs for long time.
monitor mode (-m) of inotifywait may serve better here :
inotifywait -m -q -e create -e modify -e close log_directory |\
while read -r dir action file; do
...
done
monitor mode (-m) does not buffer, it just print all events to standard output.
To preserve the variables :
while read -r dir action file; do
echo $dir $action $file
done < <(inotifywait -m -q -e create -e modify -e close log_directory)
echo "End of script"
So I've looked up other questions and answers for this and as you can imagine, there are lots of ways to find this. However, my situation is kind of different.
I'm able to check whether a bash script is already running or not and I want to kill the script if it's already running.
The problem is that with the below code, -since I'm running this within the same script- the script kills itself too because it sees a script already running.
result=`ps aux | grep -i "myscript.sh" | grep -v "grep" | wc -l`
if [ $result -ge 1 ]
then
echo "script is running"
else
echo "script is not running"
fi
So how can I check if a script is already running besides it's own self and kill itself if there's another instance of the same script is running, else, continue without killing itself.
I thought I could combine the above code with $$ command to find the script's own PID and differentiate them this way but I'm not sure how to do that.
Also a side note, my script can be run multiple times at the same time within the same machine but with different arguments and that's fine. I only need to identify if script is already running with the same arguments.
pid=$(pgrep myscript.sh | grep -x -v $$)
# filter non-existent pids
pid=$(<<<"$pid" xargs -n1 sh -c 'kill -0 "$1" 2>/dev/null && echo "$1"' --)
if [ -n "$pid" ]; then
echo "Other script is running with pid $pid"
echo "Killing him!"
kill $pid
fi
pgrep lists the pids that match the name myscript.sh. From the list we filter current $$ shell with grep -v. It the result is non-empty, then you could kill the other pid.
Without the xargs, it would work, but the pgrep myscript.sh will pick up the temporary pid created for command substitution or the pipe. So the pid will never be empty and the kill will always execute complaining about the non-existent process. To do that, for each pid in pids, I check if the pid exists with kill -0. If it does, then it is outputted, effectively filtering all nonexistent pids.
You could also use a normal for loop to filter the pids:
# filter non-existent pids
pid=$(
for i in $pid; do
if kill -0 "$i" 2>/dev/null; then
echo "$i"
fi
done
)
Alternatively, you could use flock to lock the file and use lsof to list current open files with filtering the current one. As it is now, I think it will kill also editors that are editing the file and such. I believe the lsof output could be better filtered to accommodate this.
if [ "${FLOCKER}" != "$0" ]; then
pids=$(lsof -p "^$$" -- ./myscript.sh | awk 'NR>1{print $2}')
if [ -n "$pids" ]; then
echo "Other processes with $(echo $pids) found. Killing them"
kill $pids
fi
exec env FLOCKER="$0" flock -en "$0" "$0" "$#"
fi
I would go with either of 2 ways to solve this problem.
1st solution: Create a watchdog file lets say a .lck file kind of on a location before starting the script's execution(Make sure we use trap etc commands in case script is aborted so that .lck file should be removed) AND remove it once execution of script is completed successfully.
Example script for 1st solution: This is just an example a test one. We need to take care of interruptions in the script, lets say script got interrupted by a command or etc then we could use trap in it too, since at that time it would have not been completed but you may need to kick it off again(since last time it was not completed).
cat file.ksh
#!/bin/bash
PWD=`pwd`
watchdog_file="$PWD/script.lck"
if [[ -f "$watchdog_file" ]]
then
echo "Please wait script is still running, exiting from script now.."
exit 1;
else
touch $watchdog_file
fi
while true
do
echo "singh" > test1
done
if [[ -f "$watchdog_file" ]]
then
rm "$watchdog_file"
fi
2nd solution: Take pid of current running shell using $$ save it in a file. Then check if that process is still running come out of script if NOT running then move on to run statements in script.
So I'm trying to check for the output of a command, but I also want to be able display the output directly in the terminal.
#!/bin/bash
while :
do
OUT=$(streamlink -o "$NAME" "$STREAM" best)
echo "$OUT"
if [[ $OUT == *"No playable streams"* ]]; then
echo "Delaying!"
sleep 15s
fi
done
This is what I tried to do.
The code checks if the output of a command contains that error substring, if so it'd add a delay. It works well on that part.
But it doesn't work well when the command is actually successfully downloading a file as it won't perform that echo until it is finished with the download (which would take hours). So until then I have no way of personally checking the output of the command
Plus the output of this particular command displays and updates the speed and filesize in real-time, something echo wouldn't be able to replicate.
So is there a way to be able to display the output of a command in real-time, while also command substituting them in order to check the output for substrings after the command is finished?
Use a temporary file:
TEMP=$(mktemp) || exit 1
while true
do
streamlink -o "$NAME" "$STREAM" best |& tee "$TEMP"
OUT=$( cat "$TEMP" )
#echo "$OUT" # not longer needed
if [[ $OUT == *"No playable streams"* ]]; then
echo "Delaying!"
sleep 15s
fi
done
# not really needed here because of endless loop
rm -f "$TEMP"
I need a solution in creating a script to tail a log file that is recreated (with the same name) after it reaches a certain size.
Using "tail -f" causes the tailing to stop when the file is recreated/rotated.
What I would like to do is create a script that would tail the file and after it reaches 100 lines for example, then restart the command... Or even better to restart the command when the file is recreated?
Is it possible?
Yes! Use this (the retry will make tail retry when the file doesn't exist or is otherwise inaccessible rather than just failing - such as potentially when you are changing files):
tail -f --retry <filename>
OR
tail --follow=name --retry
OR
tail -F <filename>
try running
watch "tail -f" yourfile.log
If tail -F is not available, and you are trying to recover from logrotate, you may add the copytruncate option to your logrotate.d/ spec file so instead of creating a new file each time after rotation, the file is kept and truncated, while a copy is rotated out.
This way the old file handle continues to point to the new (truncated) log file where new logs are appended.
Note that there may be some loss of data during this copy-truncate process.
Since you dont have a tail that support all the features and because you dont have watch you could use a simple script that loop indefinitely to execute the tail.
#!/bin/bash
PID=`mktemp`
while true;
do
[ -e "$1" ] && IO=`stat -c %i "$1"`
[ -e "$1" ] && echo "restarting tail" && { tail -f "$1" 2> /dev/null & echo $! > $PID; }
# as long as the file exists and the inode number did not change
while [[ -e "$1" ]] && [[ $IO = `stat -c %i "$1"` ]]
do
sleep 0.5
done
[ ! -z $PID ] && kill `cat $PID` 2> /dev/null && echo > $PID
sleep 0.5
done 2> /dev/null
rm -rf $PID
You might want to use trap to cleanly exit this script. This is up to you.
Basicaly, this script check if the inode number (using stat -c %i "$1") change to kill the tail command and start a new one when the file is recreated.
Note: you might get rid of the echo "restarting tail" which will pollute your output. It was only useful for testing. Also problems might occur if the file is replaced after we check the inode number and before we start the tail process.
How do you check if a process on Mac OS X is running using the process's name in a Bash script?
I am trying to write a Bash script that will restart a process if it has stopped but do nothing if it is still running.
Parsing this:
ps aux | grep -v grep | grep -c [-i] $ProcessName
...is probably your best bet.
ps aux lists all the currently running processes including the Bash script itself which is parsed out by grep -v grep with advice from Jacob (in comments) and grep -c [-i] $ProcessName returns the optionally case-insensitive integer number of processes with integer return suggested by Sebastian.
Here's a short script that does what you're after:
#!/bin/bash
PROCESS=myapp
number=$(ps aux | grep -v grep | grep -ci $PROCESS)
if [ $number -gt 0 ]
then
echo Running;
fi
EDIT: I initially included a -i flag to grep to make it case insensitive; I did this because the example program I tried was python, which on Mac OS X runs as Python -- if you know your application's case exactly, the -i is not necessary.
The advantage of this approach is that it scales with you -- in the future, if you need to make sure, say, five instances of your application are running, you're already counting. The only caveat is if another application has your program's name in its command line, it might come up -- regular expressions to grep will resolve that issue, if you're crafty (and run into this).
Research the Darwin man pages for ps, grep, and wc.
A shorter solution:
if pgrep $PROCESS_NAME; then
echo 'Running';
fi
Explanation:
pgrep exits with 0 if there is a process matching $PROCESS_NAME running, otherwise it exist with 1.
if checks the exit code of pgrep, and, as far as exit codes go, 0 is success.
Another way is to use (abuse?) the -d option of the killall command. The -d options won't actually kill the process, but instead print what will be done. It will also exit with status 0 if it finds a matching process, or 1 if it does not. Putting this together:
#!/bin/bash
`/usr/bin/killall -d "$1" &> /dev/null`
let "RUNNING = ! $?" # this simply does a boolean 'not' on the return code
echo $RUNNING
To give credit where its due, I originally pulled this technique from a script in the iTunes installer.
This simple command will do the trick. The brackets around the process name prevent the grep command from showing in the process list. Note there is no space after the comma. There may be some portability issues as ps on some unix systems may require a dash before the options:
ps axo pid,command | grep "[S]kype"
The advantage is that you can use the results in an if statement like this:'
if [[ ! $(ps axo pid,command | grep "[i]Tunes.app") ]]; then
open -a iTunes
fi
Or if you prefer this style:
[[ ! $(ps axo pid,command | grep "[S]kype") ]] && open -a Skype || echo "Skype is up"
Another advantage is that you can get the pid by adding a pipe to awk '{print $1}'.
echo "iTunes pid: $(ps axo pid,command | grep "[i]Tunes.app" | awk '{print $1}')"
You can use either killall or kill, depending on if you are trying to find the task by PID or by name.
By Name:
if ! killall -s -0 $PROCESS_NAME >/dev/null 2>&1; then
# Restart failed app, or do whatever you need to prepare for starting the app.
else
at -f $0 +30seconds # If you don't have this on cron, you can use /usr/bin/at
fi
By PID:
if ! kill -0 $PID 2>/dev/null; then
# Restart app, do the needful.
else
at -f $0 +30seconds
fi
If you look at the OSX Manual you will see a different set of process management commands; since it's not the linux kernel, it makes sense that they would manage processes differently.
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/killall.1.html
A sample output from my terminal (striking out the user and hostname, of course):
user#localhost:~$ kill -0 782 # This was my old, stale SSH Agent.
bash: kill: (782) - No such process
user#localhost:~$ echo $?
1
user#localhost:~$ kill -0 813 # This is my new SSH agent, I only just created.
user#localhost:~$ echo $?
0
The return code from a kill -0 will always result in a safe way to check if the process is running, because -0 sends no signal that will ever be handled by an application. It won't kill the application, and "kill" is only called "kill" because it's usually used to stop an application.
When you look at the interfaces it uses in the source, you'll see that it's actually interacting with the process table directly (and not grepping a potentially loaded output from ps), and just sending a signal to an application. Some signals indicate the application should shutdown or stop, while other signals tell it to restart services, or re-read configuration, or re-open file descriptors to log files that have been recently rotated. There are a plethora of things that "kill" and "killall" can do that doesn't terminate the application, and it's used regularly to simply send a signal to the application.
I lack the reputation to comment on the killall answer above, but there's killall -s for doing it without sending any signals:
killall -s "$PROCESSNAME" &> /dev/null
if [ $? -eq 0 ]; then
echo "$PROCESSNAME is running"
# if you also need the PID:
PID=`killall -s "$PROCESSNAME" | awk '{print $3}'`
echo "it's PID is $PID"
fi
It has for sure!
pgrep, pkill and pfind for OpenBSD and Darwin (Mac OS X)
http://proctools.sourceforge.net
(also available via MacPorts: port info proctools )
pidof by nightproductions.net
I've extended a pidof script found on the net to use regular expressions (usually substrings) and be case insensitive
#!/bin/sh
ps axc |awk "BEGIN{ n=tolower(\"$1\")}\
tolower(\$5) ~n {print \$1}";
just create a script named "pidof" with this content, and put it in you path, i.e. in one of the dirs in
echo $PATH
and make it executable (maybe using sudo)
chmod 755 /usr/local/bin/pidof
and use it like this, of course
kill -9 `pidof pyth`
does Mac have pidof? ...
if pidof $processname >/dev/null ; then echo $processname is running ; fi
Perhaps too late for the OP but this may help others who find this thread.
The following modification of the amrox theme above works well for restarting applications on my OS X:
killall -d TextEdit &> /dev/null && killall TextEdit &> /dev/null; open -a TextEdit
I use the following AppleScript to update and restart daemons:
tell application "System Events" to set pwd to POSIX path of container of (path to me)
do shell script "launchctl unload -w /Library/LaunchDaemons/time-test.plist; cp -f " & quoted form of pwd & "/time-test.plist /Library/LaunchDaemons; launchctl load -w /Library/LaunchDaemons/time-test.plist" with administrator privileges
It assumes the original or updated plist file is in the same directory as the AppleScript.