I'm writing a barrier to stall the execution of a script until a certain keyword is logged. The script is pretty simple:
tail -F -n0 logfile.log | while read LINE; do
[[ "$LINE" == *'STOP'* ]] && echo ${LINE} && break;
done
or
tail -F -n0 logfile.log | grep -m1 STOP
The thing is it doesn't quit as soon as the keyword is detected, but only after the next line is written. I.e:
printf "foo\n" >> logfile.log # keeps reading
printf "foo\n" >> logfile.log # keeps reading
printf "STOP\n" >> logfile.log # STOP printed
printf "foo\n" >> logfile.log # code exits at last
Unfortunately I can't rely on the fact that another line will be logged after the "STOP" (not within an interval useful for my purposes at least).
The workaround found so far is to tail also another file I know for sure gets updated quite frequently, but what is the "clean" solution so that the code will exit right after it logs STOP?
In bash, when executing a command of the form
command1 | command2
and command2 dies or terminates, the pipe which receives /dev/stdout from command1 becomes broken. This, however, does not terminate command1 instantly.
So to achieve what you want is to use process substitution and not a pipe
awk '/STOP/{exit}1' < <(tail -f logfile)
When you use awk, you can see the behaviour in a bit more detail:
$ touch logfile
$ tail -f logfile | awk '/STOP/{exit}1;END{print "end"}'
This awk program will check if "STOP" is seen, and if not print the line again. If "STOP" is seen it will print "end"
When you do in another terminal
$ echo "a" >> logfile
$ echo "STOP" >> logfile
$ echo "b" >> logfile
You see that awk prints the following output:
a # result of print
end # awk test STOP, exits and executes END statement
Furthermore, if you look more closely, you see that awk is at this point already terminated.
ps before sending "STOP":
13625 pts/39 SN 0:00 | \_ bash
32151 pts/39 SN+ 0:00 | \_ tail -f foo
32152 pts/39 SN+ 0:00 | \_ awk 1;/STOP/{exit}1;END{print "end"}
ps after sending "STOP":
13625 pts/39 SN 0:00 | \_ bash
32151 pts/39 SN+ 0:00 | \_ tail -f foo
So the awk program terminated, but tail did not crash because it is not yet aware the pipe is broken as it did not attempt to write to it.
When you do the following in the terminal with the pipeline, you see the exit status of tail:
$ echo "${PIPESTATUS[0]} ${PIPESTATUS[1]}"
$ 141 0
Which states that awk terminated nicely, but tail terminated with exit code 141 which means SIGPIPE.
Related
Using bash process substitution, I want to run two different commands on a file simultaneously. In this example it is not necessary but imagine that "cat /usr/share/dict/words" was a very expensive operation such as uncompressing a 50gb file.
cat /usr/share/dict/words | tee >(head -1 > h.txt) >(tail -1 > t.txt) > /dev/null
After this command I would expect h.txt to contain the first line of the words file "A", and t.txt to contain the last line of the file "Zyzzogeton".
However what actually happens is that h.txt contains "A" but t.txt contains "argillaceo" which is about 5% into the file.
Why does this happen? It seems like either the "tail" process is terminating early or the streams are getting mixed up.
Running another similar command like this behaves as expected:
cat /usr/share/dict/words | tee >(grep ^a > a.txt) >(grep ^z > z.txt) > /dev/null
After this command I'd expect a.txt to contain all the words that begin with "a", while z.txt contains all of the words that begin with "z", which is exactly what happened.
So why doesn't this work with "tail", and with what other commands will this not work?
Ok, what seems to happen is that once the head -1 command finishes it exits and that causes tee to get a SIGPIPE it tries to write to the named pipe that the process substitution setup which generates an EPIPE and according to man 2 write will also generate SIGPIPE in the writing process, which causes tee to exit and that forces the tail -1 to exit immediately, and the cat on the left gets a SIGPIPE as well.
We can see this a little better if we add a bit more to the process with head and make the output both more predictable and also written to stderr without relying on the tee:
for i in {1..30}; do echo "$i"; echo "$i" >&2; sleep 1; done | tee >(head -1 > h.txt; echo "Head done") >(tail -1 > t.txt) >/dev/null
which when I run it gave me the output:
1
Head done
2
so it got just 1 more iteration of the loop before everything exited (though t.txt still only has 1 in it). If we then did
echo "${PIPESTATUS[#]}"
we see
141 141
which this question ties to SIGPIPE in a very similar fashion to what we're seeing here.
The coreutils maintainers have added this as an example to their tee "gotchas" for future posterity.
For a discussion with the devs about how this fits into POSIX compliance you can see the (closed notabug) report at http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22195
If you have access to GNU version 8.24 they have added some options (not in POSIX) that can help like -p or --output-error=warn. Without that you can take a bit of a risk but get the desired functionality in the question by trapping and ignoring SIGPIPE:
trap '' PIPE
for i in {1..30}; do echo "$i"; echo "$i" >&2; sleep 1; done | tee >(head -1 > h.txt; echo "Head done") >(tail -1 > t.txt) >/dev/null
trap - PIPE
will have the expected results in both h.txt and t.txt, but if something else happened that wanted SIGPIPE to be handled correctly you'd be out of luck with this approach.
Another hacky option would be to zero out t.txt before starting then not let the head process list finish until it is non-zero length:
> t.txt; for i in {1..10}; do echo "$i"; echo "$i" >&2; sleep 1; done | tee >(head -1 > h.txt; echo "Head done"; while [ ! -s t.txt ]; do sleep 1; done) >(tail -1 > t.txt; date) >/dev/null
How can I break a tail -f in bash? Since this question is related to this question
tail -f | awk and end tail once data is found
I tried the following:
#! /bin/bash
tvar="testing"
(set -o pipefail && tail -f <<< "$tvar" | awk '{print; exit} END{ exit 1}' )
But the script is still hanging on to tail -f
Well, the problem is not the tail -f but the awk which hangs. It is meant to terminate when EOF is found (with exit 1). But there is no EOF found; the tail -f does not terminate, so there comes no EOF.
Would the awk terminate, then this would also break the pipe and the tail would receive a SIGPIPE (which would terminate it).
You must find a different condition on which to terminate.
EDIT:
To achieve what you want you can start the tail -f in the background, remember its PID and kill it as soon as you do not need it anymore. Running in the background and using a pipe at the same time is tricky. The easiest way to do it would be to use a named pipe (FIFO):
mkfifo log.pipe
tail -f log > log.pipe & tail_pid=$!
awk ... < log.pipe
kill $tail_pid
rm log.pipe
It seems that switching from using <<< to echo "$tvar" | tail -f does what you want instead?
$> cat test.sh
#! /bin/bash
tvar="testing"
(set -o pipefail && echo "$tvar" | tail -f | awk '{print} END{ exit 1}' )
$> ./test.sh
testing
$>
Although the awk doesn't print anything out afterwards.
Here is my shell script:
#!/bin/bash
PIDS=$(ps -e | grep $1 |grep -v grep| awk '{print $1}')
kill -s SIGINT $PIDS
echo "Done sendings signal"
I am passing the name of the process as command line argument.
The echo command is not getting executed, although the target processes are actually receiving the SIGINT signal and exited.
Any suggestions?
Update:
I changed the code to:
#!/bin/bash
PIDS=$(ps -e |grep $1 | grep -v grep | awk '{print $1}'|grep -v $$)
echo $PIDS
kill -s SIGINT $PIDS
echo "Done sendings signal"
echo "The current process is $$"
Now I am noticing a strange thing:
The script is working but not as expected. Executing following command in command line outside the script
ps -e|grep process-name|grep -v grep|awk '{print $1}'|grep -v $$
gives pid of the process-name but when I execute the same command inside shell script, assign it to variable PIDS and then echo PIDS then it shows one more pid in addition to the pid of process-name. Therefore when the kill command executes it gives an error that the process with second pid doesn't exist. It does echo the remaining sentences in the terminal. Any clue ?
There really are only a couple of possibilities. Assuming you're just running this from the command line, you should see the message ... unless, of course, what you're doing puts the PID of your shell process in PIDS, in which case the kill would kill the (sub) shell running your command before you hit the echo.
Suggestion: echo $PIDS before you call kill and see what's there. In fact, I'd be tempted to comment out the kill and try the command, just to see what happens.
#!/bin/bash
PIDS=$(ps -e | grep $1 |grep -v grep| awk '{print $1}')
echo $PIDS
# kill -s SIGINT $PIDS
echo "Done sendings signal"
Of course, you can always run the script with bash -x to see everything.
Your script works. The only reason I can see for the echo not being executed is that some value of $1 and the script file name combine so that your script PID is also gathered, thereby making the script suicide.
The PIDS line spawns a process running ps, grep, another grep -- so you won't find in PIDS the processes running grep, but what about the parent process itself?
Try:
#!/bin/bash
PIDS=$(ps -e | grep $1 |grep -v grep | awk '{print $1}' | grep -v "^$$\$" )
kill -s SIGINT $PIDS
echo "Done sendings signal"
or run the pipes one after the other with suitable safety greps.
Edit: it is evident that the "$1" selection is selecting too much. So I'd rewrite the script like this:
#!/bin/bash
# Gather the output of "ps -e". This will also gather the PIDs of this
# process and of ps process and its subshell.
PSS=$( ps -e )
# Extract PIDs, excluding this one PID and excluding a process called "ps".
# Don't need to expunge 'grep' since no grep was running when getting PSS.
PIDS=$( echo "$PSS" | grep -v "\<ps\>" | grep "$1" | awk '{print $1}' | grep -v "^$$\$" )
if [ -n "$PIDS" ]; then
kill -s SIGINT $PIDS
else
echo "No process found matching $1"
fi
echo "Done sending signal."
ps -e is identical to ps -A and selects all processes ( cf. http://linux.die.net/man/1/ps ), i. e. ps -e displays "information about other users' processes, including those without controlling terminals" (Mac OS X man page of ps). This means you will also kill the PID ($$) of your shell process, as already pointed out by Charlie Martin, because you will also grep a line of output of the ps -e command that looks like so:
67988 ttys000 0:00.00 /bin/bash ./killpids sleep
Just log the output of ps -e to a file to see that your script commits suicide:
./killpids sleep 2>err.log
#!/bin/bash
# cat killpids
echo $$
for n in {1..10}; do
sleep 5000 &
done
sleep 1
unset PIDS
PIDS="$(ps -e | tee /dev/stderr | grep "$1" | grep -v grep | awk '{print $1}')"
#PIDS="$(ps -www -U $USER -o pid,uid,comm | tee /dev/stderr | grep "$1" | grep -v grep | awk '{print $1}')"
wc -l <<<"$PIDS"
#kill -s SIGINT $PIDS
echo kill -s TERM $PIDS
kill -s TERM $PIDS
echo "Done sendings signal"
I want to do a tail -F on a file until matching a pattern. I found a way using awk, but IMHO my command is not really clean. The problem is that I need to do it in only one line, because of some limitations.
tail -n +0 -F /tmp/foo | \
awk -W interactive '{if ($1 == "EOF") exit; print} END {system("echo EOF >> /tmp/foo")}'
The tail will block until EOF appears in the file. It works pretty well. The END block is mandatory because awk's exit does not exit right away. It makes awk to eval the END block before quitting. The END block hangs on a read call (because of tail), so the last thing I need to do, is to write another line in the file to force tail to exit.
Does someone know a better way to do that?
Use tail's --pid option and tail will stop when the shell dies. No need to add extra to the tailed file.
sh -c 'tail -n +0 --pid=$$ -f /tmp/foo | { sed "/EOF/ q" && kill $$ ;}'
Try this:
sh -c 'tail -n +0 -f /tmp/foo | { sed "/EOF/ q" && kill $$ ;}'
The whole command-line will exit as soon as the "EOF" string is seen in /tmp/foo.
There is one side-effect: the tail process will be left running (in the background) until anything is written to /tmp/foo.
I've not results with the solution:
sh -c 'tail -n +0 -f /tmp/foo | { sed "/EOF/ q" && kill $$ ;}'
There is some issue related with the buffer because if there aren't more lines appended to the file, then sed will not read the input. So, with a little more research i came up with this:
sed '/EOF/q' <(tail -n 0 -f /tmp/foo)
The script is in https://gist.github.com/2377029
This is something Tcl is quite good at. If the following is "tail_until.tcl",
#!/usr/bin/env tclsh
proc main {filename pattern} {
set pipe [open "| tail -n +0 -F $filename"]
set pid [pid $pipe]
fileevent $pipe readable [list handler $pipe $pattern]
vwait ::until_found
catch {exec kill $pid}
}
proc handler {pipe pattern} {
if {[gets $pipe line] == -1} {
if {[eof $pipe]} {
set ::until_found 1
}
} else {
puts $line
if {[string first $pattern $line] != -1} {
set ::until_found 1
}
}
}
main {*}$argv
Then you'd do tail_until.tcl /tmp/foo EOF
Does this work for you?
tail -n +0 -F /tmp/foo | sed '/EOF/q'
I'm assuming that 'EOF' is the pattern you're looking for. The sed command quits when it finds it, which means that the tail should quit the next time it writes.
I suppose that there is an outside chance that tail would hang around if the pattern is found at about the end of the file, waiting for more output to appear in the file which will never appear. If that's really a concern, you could probably arrange to kill it - the pipeline as a whole will terminate when sed terminates (unless you're using a funny shell that decides that isn't the correct behaviour).
Grump about Bash
As feared, bash (on MacOS X, at least, but probably everywhere) is a shell that thinks it needs to hang around waiting for tail to finish even though sed quit. Sometimes - more often than I like - I prefer the behaviour of good old Bourne shell which wasn't so clever and therefore guessed wrong less often than Bash does. dribbler is a program which dribbles out messages one per second ('1: Hello' etc in the example), with the output going to standard output. In Bash, this command sequence hangs until I did 'echo pqr >>/tmp/foo' in a separate window.
date
{ timeout -t 2m dribbler -t -m Hello; echo EOF; } >/tmp/foo &
echo Hi
sleep 1 # Ensure /tmp/foo is created
tail -n +0 -F /tmp/foo | sed '/EOF/q'
date
Sadly, I don't immediately see an option to control this behaviour. I did find shopt lithist, but that's unrelated to this problem.
Hooray for Korn Shell
I note that when I run that script using Korn shell, it works as I'd expect - leaving a tail lurking around to be killed somehow. What works there is 'echo pqr >> /tmp/foo' after the second date command completes.
Here's an extended version of Jon's solution which uses sed instead of grep so that the output of tail goes to stdout:
sed -r '/EOF/q' <( exec tail -n +0 -f /tmp/foo ); kill $! 2> /dev/null
This works because sed gets created before tail so $! holds the PID of tail
The main advantage of this over the sh -c solutions is that killing a sh seems to print something to the output such as 'Terminated' which is unwelcome
sh -c 'tail -n +0 --pid=$$ -f /tmp/foo | { sed "/EOF/ q" && kill $$ ;}'
Here the main problem is with $$.
If you run command as is, $$ is set not to sh but to the PID of the current shell where command is run.
To make kill work you need to change kill $$ to kill \$$
After that you can safely get rid of --pid=$$ passed to tail command.
Summarising, following will work just fine:
/bin/sh -c 'tail -n 0 -f /tmp/foo | { sed "/EOF/ q" && kill \$$ ;}
Optionally you can pass -n to sed to keep it quiet :)
To kill the dangling tail process as well you may execute the tail command in a (Bash) process substituion context which can later be killed as if it had been a backgrounded process. (Code taken from How to read one line from 'tail -f' through a pipeline, and then terminate?).
: > /tmp/foo
grep -m 1 EOF <( exec tail -f /tmp/foo ); kill $! 2> /dev/null
echo EOF > /tmp/foo # terminal window 2
As an alternative you could use a named pipe.
(
: > /tmp/foo
rm -f pidfifo
mkfifo pidfifo
sh -c '(tail -n +0 -f /tmp/foo & echo $! > pidfifo) |
{ sed "/EOF/ q" && kill $(cat pidfifo) && kill $$ ;}'
)
echo EOF > /tmp/foo # terminal window 2
ready to use for tomcat =
sh -c 'tail -f --pid=$$ catalina.out | { grep -i -m 1 "Server startup in" && kill $$ ;}'
for above scenario :
sh -c 'tail -f --pid=$$ /tmp/foo | { grep -i -m 1 EOF && kill $$ ;}'
tail -f <filename> | grep -q "<pattern>"
I have a couple of scripts to control some applications (start/stop/list/etc). Currently my "stop" script just sends an interrupt signal to an application, but I'd like to have more feedback about what application does when it is shutting down. Ideally, I'd like to start tailing its log, then send an interrupt signal and then keep tailing that log until the application stops.
How to do this with a shell script?
For just tailing a log file until a certain process stops (using tail from GNU coreutils):
do_something > logfile &
tail --pid $! -f logfile
UPDATE The above contains a race condition: In case do_something spews many lines into logfile, the tail will skip all of them but the last few. To avoid that and always have tail print the complete logfile, add a -n +1 parameter to the tail call (that is even POSIX tail(1)):
do_something > logfile &
tail --pid $! -n +1 -f logfile
Here's a Bash script that works without --pid. Change $log_file and $p_name to suit your need:
#!/bin/bash
log_file="/var/log/messages"
p_name="firefox"
tail -n10 $log_file
curr_line="$(tail -n1 $log_file)"
last_line="$(tail -n1 $log_file)"
while [ $(ps aux | grep $p_name | grep -v grep | wc -l) -gt 0 ]
do
curr_line="$(tail -n1 $log_file)"
if [ "$curr_line" != "$last_line" ]
then
echo $curr_line
last_line=$curr_line
fi
done
echo "[*] $p_name exited !!"
If you need to tail log until process exited, but watch stdout / sdterr at the same time, try this:
# Run some process in bg (background):
some_process &
# Get process id:
pid=$!
# Tail the log once it is created, but watch process stdout/stderr at the same time:
tail --pid=$pid -f --retry log_file_path &
# Since tail is running in bg also - wait until the process has completed:
tail --pid=$pid -f /dev/null