I've created a named pipe for some other process to write to and want to check that the other process started correctly, but don't know its PID. The context is running a command in screen, making sure the command started correctly. I was hoping this might work:
mkfifo /tmp/foo
echo hello > /tmp/foo &
lsof /tmp/foo
Sadly, lsof does not report echo. inotifywait might be another option, but isn't always installed and I really want to poll just once, rather than block until some event.
Is there any way to check if a named pipe is open for writing? Even open in general?
UPDATE:
Once both ends are connected lsof seems to work. This actually solves my problem, but for the sake of the question I'd be interested to know if it's possible to detect the initial redirection to the named pipe without a reader.
> mkfifo /tmp/foo
> yes > /tmp/foo &
> lsof /tmp/foo
> cat /tmp/foo > /dev/null &
> lsof /tmp/foo
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
yes 16915 user 1w FIFO 8,18 0t0 16660270 /tmp/foo
cat 16950 user 3r FIFO 8,18 0t0 16660270 /tmp/foo
Update 2: After playing with inotify-tools, there doesn't seem to be a way to get a notification that a named pipe has been opened for writing and is blocking. This is probably why lsof doesn't show the pipe until it has a reader and a writer.
Update: After researching named pipes, I don't believe that there is any method that will work with named pipes by themselves.
Reasoning:
there is no way to limit the number of writers to a named pipe (without resorting to locking)
all writers block if there is no reader
no writers block if there is a reader (presumably as long as the kernel buffers aren't full)
You could try writing nothing to the pipe with a short timeout. If the timeout expires, then the write blocked indicating that someone has already opened the pipe for writing.
Note: As pointed out in the comments, if a reader exists and presumably is fast enough, our test write will not block and the test essentially fails. Comment out the cat line below to test this.
#!/bin/bash
is_named_pipe_already_opened_for_writing() {
local named_pipe="$1"
# Make sure it's a named pipe
if ! [ -p "$named_pipe" ]; then
return 1
fi
# Try to write zero bytes in the background
echo -n > "$named_pipe" &
pid=$!
# Wait a short amount of time
sleep 0.1
# Kill the background process. If kill succeeds, then
# the write was blocked indicating that someone
# else is already writing to the named pipe.
kill $pid 2>/dev/null
}
PIPE=/tmp/foo
# Ignore any bash messages from killing below
trap : TERM
mkfifo $PIPE
# a writer
yes > $PIPE &
# a reader
cat $PIPE >/dev/null &
if is_named_pipe_already_opened_for_writing "$PIPE"; then
echo "$PIPE is already being written to by another process"
else
echo "$PIPE is NOT being written to by another process"
fi
jobs -pr | kill 2>/dev/null
rm -f $PIPE
you need two pipes one for each directions:
one is use to wait for the ready for new data signal, another just for the data:
in my case process to files, line by line:
mkfifo r w;
cat file1 | while read l; do echo "$l" >w; read <r; done &
cat file2 | while read ln; do if read l <w; then echo "$ln"; echo "$l"; fi; echo 1>r; done
Related
I want to kill a bash command when I found some string in the output.
To clarify, I want the solution to be similar to a timeout command:
timeout 10s looping_program.sh
Which will execute the script: looping_program.sh and kill the script after 10 seconds of execute.
Instead I want something like:
regexout "^Success$" looping_program.sh
Which will execute the script until it matches a line that just says Success in the stdout of the program.
Note that I'm assuming that this looping_program.sh does not exit at the same time it outputs Success for whatever reason, so simply waiting for the program to exit would waste time if I don't care about what happens after that.
So something like:
bash -e looping_program.sh > /tmp/output &
PID="$(ps aux | grep looping_program.sh | head -1 | tr -s ' ' | cut -f 2 -d ' ')"
echo $PID
while :; do
echo "$(tail -1 /tmp/output)"
if [[ "$(tail -1 /tmp/output)" == "Success" ]]; then
kill $PID
exit 0
fi
sleep 1
done
Where looping_program.sh is something like:
echo "Fail"
sleep 1;
echo "Fail"
sleep 1;
echo "Fail"
sleep 1;
echo "Success"
sleep 1;
echo "Fail"
sleep 1;
echo "Fail"
sleep 1;
echo "Fail"
sleep 1;
But that is not very robust (uses a single tmp file... might kill other programs...) and I want it to just be one command. Does something like this exist? I may just write a c program to do it if not.
P.S.: I provided my code as an example of what I wanted the program to do. It does not use good programming practices. Notes from other commenters:
#KamilCuk Do not use temporary file. Use a fifo.
#pjh Note that any approach that involves using kill with a PID in shell code runs the risk of killing the wrong process. Use kill in shell programs only when it is absolutely necessary.
There are more suggestions below from other users, I just wanted to make sure no one came across this and thought it would be good to model their code after.
looping_program() {
for i in 1 2 3; do echo $i; sleep 1; done
echo Success
yes
}
coproc looping_program
while IFS= read -r line; do
if [[ "$line" =~ Success ]]; then
break
fi
done <&${COPROC[0]}
exec {COPROC[0]}>&- {COPROC[1]}>&-
kill ${COPROC_PID}
wait ${COPROC_PID}
Notes:
Do not use temporary file. Use a fifo.
Do not use tail -n1 to read last line. Read from the stream in a loop.
Do not repeat tail -1 twice. Cache the result.
Wait for pid after killing to synchronize.
When you're using a coprocess, use COPROC_PID to get the PID
When you're not using a coprocess, use $! to get the PID of a background process started from the current shell.
When you can't use $! (because the process you're trying to get a PID of was not spawned in the background as a direct child of the current shell), do not use ps aux | grep to get the pid. Use pgrep.
Do not use echo $(stuff). Just run the stuff, no echo.
With expect
#!/usr/bin/env -S expect -f
set timeout -1
spawn ./looping_program.sh
expect "Success"
send -- "\x03"
expect eof
Call it looping_killer:
$ ./looping_killer
spawn ./looping_program.sh
Fail
Fail
Fail
Success
^C
To pass the program and pattern:
./looping_killer some_program "some pattern"
You'd change the expect script to
#!/usr/bin/env -S expect -f
set timeout -1
spawn [lindex $argv 0]
expect -- [lindex $argv 1]
send -- "\x03"
expect eof
Assuming that your looping program exists when it tries to write to a broken pipe, this will print all output up to and including the 'Success' line and then exit:
./looping_program | sed '/^Success$/q'
You may need to disable buffering of the looping program output. See Force line-buffering of stdout in a pipeline and How to make output of any shell command unbuffered? for ways to do it.
See Should I save my scripts with the .sh extension? and Erlkonig: Commandname Extensions Considered Harmful for reasons why I dropped the '.sh' suffix.
Note that any approach that involves using kill with a PID in shell code runs the risk of killing the wrong process. Use kill in shell programs only when it is absolutely necessary.
I know how to redirect stdout to a file:
exec > foo.log
echo test
this will put the 'test' into the foo.log file.
Now I want to redirect the output into the log file AND keep it on stdout
i.e. it can be done trivially from outside the script:
script | tee foo.log
but I want to do declare it within the script itself
I tried
exec | tee foo.log
but it didn't work.
#!/usr/bin/env bash
# Redirect stdout ( > ) into a named pipe ( >() ) running "tee"
exec > >(tee -i logfile.txt)
# Without this, only stdout would be captured - i.e. your
# log file would not contain any error messages.
# SEE (and upvote) the answer by Adam Spiers, which keeps STDERR
# as a separate stream - I did not want to steal from him by simply
# adding his answer to mine.
exec 2>&1
echo "foo"
echo "bar" >&2
Note that this is bash, not sh. If you invoke the script with sh myscript.sh, you will get an error along the lines of syntax error near unexpected token '>'.
If you are working with signal traps, you might want to use the tee -i option to avoid disruption of the output if a signal occurs. (Thanks to JamesThomasMoon1979 for the comment.)
Tools that change their output depending on whether they write to a pipe or a terminal (ls using colors and columnized output, for example) will detect the above construct as meaning that they output to a pipe.
There are options to enforce the colorizing / columnizing (e.g. ls -C --color=always). Note that this will result in the color codes being written to the logfile as well, making it less readable.
The accepted answer does not preserve STDERR as a separate file descriptor. That means
./script.sh >/dev/null
will not output bar to the terminal, only to the logfile, and
./script.sh 2>/dev/null
will output both foo and bar to the terminal. Clearly that's not
the behaviour a normal user is likely to expect. This can be
fixed by using two separate tee processes both appending to the same
log file:
#!/bin/bash
# See (and upvote) the comment by JamesThomasMoon1979
# explaining the use of the -i option to tee.
exec > >(tee -ia foo.log)
exec 2> >(tee -ia foo.log >&2)
echo "foo"
echo "bar" >&2
(Note that the above does not initially truncate the log file - if you want that behaviour you should add
>foo.log
to the top of the script.)
The POSIX.1-2008 specification of tee(1) requires that output is unbuffered, i.e. not even line-buffered, so in this case it is possible that STDOUT and STDERR could end up on the same line of foo.log; however that could also happen on the terminal, so the log file will be a faithful reflection of what could be seen on the terminal, if not an exact mirror of it. If you want the STDOUT lines cleanly separated from the STDERR lines, consider using two log files, possibly with date stamp prefixes on each line to allow chronological reassembly later on.
Solution for busybox, macOS bash, and non-bash shells
The accepted answer is certainly the best choice for bash. I'm working in a Busybox environment without access to bash, and it does not understand the exec > >(tee log.txt) syntax. It also does not do exec >$PIPE properly, trying to create an ordinary file with the same name as the named pipe, which fails and hangs.
Hopefully this would be useful to someone else who doesn't have bash.
Also, for anyone using a named pipe, it is safe to rm $PIPE, because that unlinks the pipe from the VFS, but the processes that use it still maintain a reference count on it until they are finished.
Note the use of $* is not necessarily safe.
#!/bin/sh
if [ "$SELF_LOGGING" != "1" ]
then
# The parent process will enter this branch and set up logging
# Create a named piped for logging the child's output
PIPE=tmp.fifo
mkfifo $PIPE
# Launch the child process with stdout redirected to the named pipe
SELF_LOGGING=1 sh $0 $* >$PIPE &
# Save PID of child process
PID=$!
# Launch tee in a separate process
tee logfile <$PIPE &
# Unlink $PIPE because the parent process no longer needs it
rm $PIPE
# Wait for child process, which is running the rest of this script
wait $PID
# Return the error code from the child process
exit $?
fi
# The rest of the script goes here
Inside your script file, put all of the commands within parentheses, like this:
(
echo start
ls -l
echo end
) | tee foo.log
Easy way to make a bash script log to syslog. The script output is available both through /var/log/syslog and through stderr. syslog will add useful metadata, including timestamps.
Add this line at the top:
exec &> >(logger -t myscript -s)
Alternatively, send the log to a separate file:
exec &> >(ts |tee -a /tmp/myscript.output >&2 )
This requires moreutils (for the ts command, which adds timestamps).
Using the accepted answer my script kept returning exceptionally early (right after 'exec > >(tee ...)') leaving the rest of my script running in the background. As I couldn't get that solution to work my way I found another solution/work around to the problem:
# Logging setup
logfile=mylogfile
mkfifo ${logfile}.pipe
tee < ${logfile}.pipe $logfile &
exec &> ${logfile}.pipe
rm ${logfile}.pipe
# Rest of my script
This makes output from script go from the process, through the pipe into the sub background process of 'tee' that logs everything to disc and to original stdout of the script.
Note that 'exec &>' redirects both stdout and stderr, we could redirect them separately if we like, or change to 'exec >' if we just want stdout.
Even thou the pipe is removed from the file system in the beginning of the script it will continue to function until the processes finishes. We just can't reference it using the file name after the rm-line.
Bash 4 has a coproc command which establishes a named pipe to a command and allows you to communicate through it.
Can't say I'm comfortable with any of the solutions based on exec. I prefer to use tee directly, so I make the script call itself with tee when requested:
# my script:
check_tee_output()
{
# copy (append) stdout and stderr to log file if TEE is unset or true
if [[ -z $TEE || "$TEE" == true ]]; then
echo '-------------------------------------------' >> log.txt
echo '***' $(date) $0 $# >> log.txt
TEE=false $0 $# 2>&1 | tee --append log.txt
exit $?
fi
}
check_tee_output $#
rest of my script
This allows you to do this:
your_script.sh args # tee
TEE=true your_script.sh args # tee
TEE=false your_script.sh args # don't tee
export TEE=false
your_script.sh args # tee
You can customize this, e.g. make tee=false the default instead, make TEE hold the log file instead, etc. I guess this solution is similar to jbarlow's, but simpler, maybe mine has limitations that I have not come across yet.
Neither of these is a perfect solution, but here are a couple things you could try:
exec >foo.log
tail -f foo.log &
# rest of your script
or
PIPE=tmp.fifo
mkfifo $PIPE
exec >$PIPE
tee foo.log <$PIPE &
# rest of your script
rm $PIPE
The second one would leave a pipe file sitting around if something goes wrong with your script, which may or may not be a problem (i.e. maybe you could rm it in the parent shell afterwards).
What is the cleanest (simplest, most efficient, shortest, quickest, easiest, most elegant) way to create a non-linear pipeline like this in Bash?
I have three commands: mksock, irclogin, and ircpingpong. I want to pipe stdin, irclogin, and ircpingpong into mksock, and pipe mksock into stdout and ircpingpong. This means that mksock and ircpingpong are in a loop. I drew a diagram:
irclogin only needs to be run once and be the first input into mksock. After that, ircpingpong, and stdin should be accepted at any time. I am currently using pipes and a temporary file like this:
#!/bin/bash
server=127.0.0.1
port=6667
infifo=/tmp/ircin
outfifo=/tmp/ircout
pongfifo=/tmp/ircpong
rm $infifo
rm $outfifo
rm $pongfifo
mkfifo $infifo
mkfifo $outfifo
touch $pongfifo
( irclogin | cat - $infifo & tail -f $pongfifo; ) | mksock $server $port | tee $outfifo | stdbuf -oL ircpingpong > $pongfifo &
cat < $outfifo &
cat > $infifo
pkill tail
This works, but I want to know if there is a better way to do this. It bothers me that I am using a file rather than a pipe for looping back from ircpingpong to mksock using tail. Using a pipe didn't work because, to my understanding, something is written to the pipe before tail -f starts reading from it, and so it misses it.
It also bothers me that I have to kill tail at the end of the script, because it doesn't stop on it's own and would leave the socket connected even after the script has ended.
I can suggest a version without temporary files, and with two fifo-s:
fifo1=/tmp/fifo1
fifo2=/tmp/fifo2
rm $fifo1
rm $fifo2
mkfifo $fifo1
mkfifo $fifo2
ircpingpong < $fifo2 > $fifo1 &
(mksock <$fifo1|tee $fifo2 )&
irclogin >$fifo1 &
cat >$fifo1
The idea is to run all programs separately, and only ensure that input and output of each program is redirected properly according to this diagram:
Of course, ircpingpong must read stdin and write to stdout, irclogin must write to stdout, and mksock must read from stdin and write to stdout.
Here's something that uses just one fifo.
fifo=/tmp/myfifo
rm $fifo
mkfifo $fifo
((ircpingpong < $fifo &) && irclogin && cat) | mksock | tee $fifo
Add stdbuf as needed.
I don't know whether you will get your "something doesn't die on its own" problem; when I ctrl-c'ed the script, everything seemed to die.
I have a streaming backup script which I'm running as follows:
./backup_script.sh | aws s3 cp - s3://bucket/path/to/backup
The aws command streams stdin to cloud storage in an atomic way. If the process is interrupted without an EOF, the upload is aborted.
I want the aws process to be killed if ./backup_script.sh exits with a non-zero exit code.
Any bash trick for doing this?
EDIT:
You can test your solution with this script:
#!/usr/bin/env python
import signal
import sys
import functools
def signal_handler(signame, signum, frame):
print "Got {}".format(signame)
sys.exit(0)
signal.signal(signal.SIGTERM, functools.partial(signal_handler, 'TERM'))
signal.signal(signal.SIGINT, functools.partial(signal_handler, 'INT'))
for i in sys.stdin:
pass
print "Got EOF"
Example:
$ grep --bla | ./sigoreof.py
grep: unrecognized option `--bla'
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
Got EOF
I want ./sigoreof.py to be terminated with a signal.
Adopting/correcting a solution originally given by #Dummy00001:
mkfifo aws.fifo
exec 3<>aws.fifo # open the FIFO read/write *in the shell itself*
aws s3 cp - s3://bucket/path/to/backup <aws.fifo 3>&- & aws_pid=$!
rm aws.fifo # everyone who needs a handle already has one; can remove the directory entry
if ./backup_script.sh >&3 3>&-; then
exec 3>&- # success: close the FIFO and let AWS exit successfully
wait "$aws_pid"
else
kill "$aws_pid" # send a SIGTERM...
wait "$aws_pid" # wait for the process to die...
exec 3>&- # only close the write end *after* the process is dead
fi
Important points:
The shell opens the FIFO r/w to avoid blocking (opening for write only would block for a reader; this could also be avoided by invoking the reader [that is, the s3 command] in the background prior to the exec opening the write side).
The write end of the FIFO is held by the script itself, so the read end never hits end-of-file until after the script intentionally closes it.
The aws command's handle on the write end of the FIFO is explicitly closed (3<&-), so it doesn't hold itself open (in which case the exec 3>&- done in the parent would not successfully allow it to finish reading and exit).
backup_script.sh should have a non-zero exit status if there is an error, so you script should look something like:
if ./backup_script.sh > output.txt; then
aws s3 cp output.txt s3://bucket/path/to/backup
fi
rm -f output.txt
A pipe isn't really appropriate here.
If you really need to conserve disk space locally, you'll have to "reverse" the upload; either remove the uploaded file in the event of an error in backup_script.sh, or upload to a temporary location, then move that to the final path once you've determined that the backup has succeeded.
(For simplicity, I'm ignoring the fact that by letting aws exit on its own in the event of an error, you may be uploading more of the partial backup than you need to. See Charles Duffy's answer for a more bandwidth-efficient approach.)
After starting the backup process with
mkfifo data
./backup_script.sh > data & writer_pid=$!
use one of the following to upload the data.
# Upload and remove if there was an error
aws s3 cp - s3://bucket/path/to/backup < data &
if ! wait $writer_pid; then
aws s3 rm s3://bucket/path/to/backup
fi
or
# Upload to a temporary file and move it into place
# once you know the backup succeeded.
aws s3 cp - s3://bucket/path/to/backup.tmp < data &
if wait $writer_pid; then
aws s3 mv s3://bucket/path/to/backup.tmp s3://bucket/path/to/backup
else
aws s3 rm s3://bucket/path/to/backup
fi
A short script which uses process substitution instead of named pipes would be:
#!/bin/bash
exec 4> >( ./second-process.sh )
./first-process.sh >&4 &
if ! wait $! ; then echo "error in first process" >&2; kill 0; wait; fi
It works much like with a fifo, basically using the fd as the information carrier for the IPC instead of a file name.
Two remarks: I wasn't sure whether it's necessary to close fd 4 ; I would assume that upon script exit the shell closes all open files.
And I couldn't figure out how to obtain the PID of the process in the process substitution (anybody? at least on my cygwin the usual $! didn't work.) Therefore I resorted to killing all processes in the group, which may not be desirable (but I'm not entirely sure about the semantics).
I think you need to spawn both processes from a third one and either use the named pipe approach from Lynch in the post mentioned by #tourism (further below in the answers); or keep piping directly but re-write backup_script.sh such that it stays alive in the error case, keeping stdout open. backup_script.sh would have to signal the error condition to the calling process (e.g. by sending a SIGUSR to the parent process ID), which in turn first kills the aws process (leading to an atomic abort) and only then backup_script.sh, unless it exited already because of the broken pipe.
I had a similar situation: a shell script contained a pipeline that used one of its own functions and that function wanted to be able to effect termination. A simple contrived example that finds and displays a file:
#!/bin/sh
a() { find . -maxdepth 1 -name "$1" -print -quit | grep . || exit 101; }
a "$1" | cat
echo done
Here, the function a needs to be able to effect termination which it tries to do by calling exit. However, when invoked through a pipeline (line 3), it only terminates its own (subshell) process. In the example, the done message still appears.
One way to work around this is to detect when in a subshell and send a signal to the parent:
#!/bin/sh
die() { [[ $$ == $(exec sh -c 'echo $PPID') ]] && exit $1 || kill $$; }
a() { find . -maxdepth 1 -name "$1" -print -quit | grep . || die 101; }
a "$1" | cat
echo done
When in a subshell the $$ is the pid of the parent and the construct $(exec sh -c 'echo $PPID') is a shell-agnostic way to obtain the pid of the subprocess. If using bash then this can be replaced by $BASHPID.
If the subprocess pid and $$ differ then it sends a SIGTERM signal to the parent (kill $$) instead of calling exit.
The given exit status (101) isn't propagated by kill so the script exits with a status of 143 (which is 128+15 where 15 is the id of SIGTERM).
I have a system that handles incoming emails to send them to a blackbox application at my job. The high level script is maintained by inittab to always run and runs a child script to do the actual work with this command:
$SCRIPT | nc -l -p $PORT
The script itself reads from a named pipe, does a bit of parsing and processing of the data before calling echo to shuffle the data back through netcat to the process connected on $PORT.
What I need is some method to handle incoming data from the far end of my pipe. When I make a request within the application to close the connection it sends back a string (I can define it to whatever I want) and waits for my script to close the pipe. I currently am struggling to understand how I can add in the functionality to read incoming data from the other end; verify it is the command to close the pipe, and then exit the script.
My script (in a nutshell) looks like this:
while true ; do
email_input="`cat "$pipe"`"
if [[ $email_input =~ .*escape_queue.* ]] ; then
break;
fi
echo "`parse`"
done
I'm open to the possibility of having to alter the program flow, I just can't wrap my head around how I would be able to read the data incoming asynchronously since the script blocks on cat $pipe until a new email is received to process.
If its not clear, I'm at a novice level with bash scripting and am always open to suggestions for improvement.
UPDATE
I've changed my script call to
$SCRIPT | nc -l -p $PORT > $nc_data
and within the script itself
netcat_response="`cat "$nc_data"`";
if [[ "$netcat_response" =~ "exit" ]] ; then
cat /dev/null > $nc_data
break;
fi
At this point the script terminates once a new message is piped into the fifo. This means that I will always lose 1 message as it gets read by the script and then the script terminates. The script still blocks on the cat until something is read. Worst case scenario this will have to do.
You can have nc quit after a certain time from the EOF of stdin.
$SCRIPT | nc -l -q 5 -p $PORT > $nc_data
-q being the option to quit after a certain amount of seconds.