I'd like to turn the following:
git status --short && (git status --short | xargs -Istr test -z str)
which gets me the desired result of mirroring the output to stdout and doing a zero length check on the result into something closer to:
git status --short | tee >(xargs -Istr test -z str)
which unfortunately returns the exit code of tee (always zero).
Is there any way to get at the exit code of the substituted process elegantly?
[EDIT]
I'm going with the following for now, it prevents running the same command twice but seems to beg for something better:
OUT=$(git status --short) && echo "${OUT}" && test -z "${OUT}"
Look here:
$ echo xxx | tee >(xargs test -n); echo $?
xxx
0
$ echo xxx | tee >(xargs test -z); echo $?
xxx
0
and look here:
$echo xxx | tee >(xargs test -z; echo "${PIPESTATUS[*]}")
xxx
123
$echo xxx | tee >(xargs test -n; echo "${PIPESTATUS[*]}")
xxx
0
Is it?
See also Pipe status after command substitution
I've been working on this for a while, and it seems that there is no way to do that with process substitution, except for resorting to inline signalling, and that can really be used only for input pipes, so I'm not going to expand on it.
However, bash-4.0 provides coprocesses which can be used to replace process substitution in this context and provide clean reaping.
The following snippet provided by you:
git status --short | tee >(xargs -Istr test -z str)
can be replaced by something alike:
coproc GIT_XARGS { xargs -Istr test -z str; }
{ git status --short | tee; } >&${GIT_XARGS[1]}
exec {GIT_XARGS[1]}>&-
wait ${GIT_XARGS_PID}
Now, for some explanation:
The coproc call creates a new coprocess, naming it GIT_XARGS (you can use any name you like), and running the command in braces. A pair of pipes is created for the coprocess, redirecting its stdin and stdout.
The coproc call sets two variables:
${GIT_XARGS[#]} containing pipes to process' stdin and stdout, appropriately ([0] to read from stdout, [1] to write to stdin),
${GIT_XARGS_PID} containing the coprocess' PID.
Afterwards, your command is run and its output is directed to the second pipe (i.e. coprocess' stdin). The cryptically looking >&${GIT_XARGS[1]} part is expanded to something like >&60 which is regular output-to-fd redirection.
Please note that I needed to put your command in braces. This is because a pipeline causes subprocesses to be spawned, and they don't inherit file descriptors from the parent process. In other words, the following:
git status --short | tee >&${GIT_XARGS[1]}
would fail with invalid file descriptor error, since the relevant fd exists in parent process and not the spawned tee process. Putting it in brace causes bash to apply the redirection to the whole pipeline.
The exec call is used to close the pipe to your coprocess. When you used process substitution, the process was spawned as part of output redirection and the pipe to it was closed immediately after the redirection no longer had effect. Since coprocess' pipe's lifetime extends beyond a single redirection, we need to close it explicitly.
Closing the output pipe should cause the process to get EOF condition on stdin and terminate gracefully. We use wait to wait for its termination and reap it. wait returns the coprocess' exit status.
As a last note, please note that in this case, you can't use kill to terminate the coprocess since that would alter its exit status.
#!/bin/bash
if read q < <(git status -s)
then
echo $q
exit
fi
Related
Let's say I have a line in my bash script with ssh bad#location "find -name 'fruit.txt' | grep "Apple" and I'm trying to retrieve the exit codes of ssh, find . -name 'fruit.txt', and "grep "Apple` to see which command went bad.
So far, I've tried something like echo $? ${PIPESTATUS[0]} ${PIPESTATUS[1]}, but it looks like $? returns the same thing as ${PIPESTATUS[0]} in this case. I only need to return the first non-zero exit code along with dmesg for debugging purposes.
I've also considered using set -o pipefail, which will return a failure exit code if any command errors, but I'd like to somehow know which command failed for debugging.
I'd like either get an exit code of 255 (from ssh) and its corresponding dmesg, or somehow get all of the exit codes.
ssh only returns one exit status (per channel) to the calling shell; if you want to get exit status for the individual pipeline components it ran remotely, you need to collect them remotely, put them in with the data, and then parse them back out. One way to do that, if you have a very new version of bash, is like so:
#!/usr/bin/env bash
# note <<'EOF' not just <<EOF; with the former, the local shell does not munge
# heredoc contents.
remote_script=$(cat <<'EOF'
tempfile=$(mktemp "${TMPDIR:-/tmp}/output.XXXXXX"); mktemp_rc=$?
find -name 'fruit.txt' | grep Apple >"$tempfile"
printf '%s\0' "$mktemp_rc" "${PIPESTATUS[#]}"
cat "$tempfile"
rm -f -- "$tempfile"
exit 0 # so a bad exit status will be from ssh itself
EOF
)
# note that collecting a process substitution PID needs bash 4.4!
exec {ssh_fd}< <(ssh bad#location "$remote_script" </dev/null); ssh_pid=$!
IFS= read -r -d '' mktemp_rc <&$ssh_fd # read $? of mktemp
IFS= read -r -d '' find_rc <&$ssh_fd # read $? of find
IFS= read -r -d '' grep_rc <&$ssh_fd # read $? of grep
cat <&$ssh_fd # spool output of grep to our own output
wait "$ssh_pid"; ssh_rc=$? # let ssh finish and read its $?
echo "mktemp exited with status $mktemp_rc" >&2
echo "find exited with status $find_rc" >&2
echo "grep exited with status $grep_rc" >&2
echo "ssh exited with status $ssh_rc" >&2
How does this work?
exec {fd_var_name}< <(...) uses the bash 4.1 automatic file descriptor allocation feature to generate a file descriptor number, and associate it with content read from the process substitution running ....
In bash 4.4 or newer, process substitutions also set $!, so their PIDs can be captured, to later wait for them and collect their exit status; this is what we're storing in ssh_pid.
IFS= read -r -d '' varname reads from stdin up to the next NUL (in read -d '', the first character of '' is treated as the end of input; as an empty string in a C-derived language, the first byte of the string is its NUL terminator).
This could theoretically be made easier by writing the output before the exit status values -- you wouldn't need a temporary file on the remote machine that way -- but the caveat there is that if there were a NUL anywhere in the find | grep output, then some of that output could be picked up by the reads. (Similarly, you could store output in a variable instead of a temporary file, but again, that would destroy any NULs in the stream's output).
The goal was to make a simple unintrusive wrapper that traces stdin and stdout to stderr:
#!/bin/bash
tee /dev/stderr | ./script.sh | tee /dev/stderr
exit ${PIPESTATUS[1]}
Test script script.sh:
#!/bin/bash
echo asd
sleep 1
exit 4
But when the script exits, it doesn't terminate the wrapper. Possible solution is to end the first tee from the second command of the pipe:
#!/bin/bash
# Second subshell will get the PID of the first one through the pipe.
# It will be able to kill the whole script by killing the first subshell.
# Create a temporary named pipe (it's safe, conflicts will throw an error).
pipe=$(mktemp -u)
if ! mkfifo $pipe; then
echo "ERROR: debug tracing pipe creation failed." >&2
exit 1
fi
# Attach it to file descriptor 3.
exec 3<>$pipe
# Unlink the named pipe.
rm $pipe
(echo $BASHPID >&3; tee /dev/stderr) | (./script.sh; r=$?; kill $(head -n1 <&3); exit $r) | tee /dev/stderr
exit ${PIPESTATUS[1]}
That's a lot of code. Is there another way?
I think that you're looking for the pipefail option. From the bash man page:
pipefail
If set, the return value of a pipeline is the value of the last (rightmost)
command to exit with a non-zero status, or zero if all commands in the
pipeline exit successfully. This option is disabled by default.
So if you start your wrapper script with
#!/bin/bash
set -e
set -o pipefail
Then the wrapper will exit when any error occurs (set -e) and will set the status of the pipeline in the way that you want.
The main issue at hand here is clearly the pipe. In bash, when executing a command of the form
command1 | command2
and command2 dies or terminates, the pipe which receives the output (/dev/stdout) from command1 becomes broken. The broken pipe, however, does not terminate command1. This will only happen when it tries to write to the broken pipe, upon which it will exit with sigpipe. A simple demonstration of this can be seen in this question.
If you want to avoid this problem, you should make use of process substitution in combination with input redirection. This way, you avoid pipes. The above pipeline is then written as:
command2 < <(command1)
In the case of the OP, this would become:
./script.sh < <(tee /dev/stderr) | tee /dev/stderr
which can also be written as:
./script.sh < <(tee /dev/stderr) > >(tee /dev/stderr)
In my script, If I want set a variable to the output of a command and avoid any errors from the command failing going to the screen, I can do something like:
var=$(command 2>/dev/null)
If I have commands piped together, i.e.
var=$(command1 | command2 | command3 2>/dev/null)
what's an elegant way to suppress any errors coming from any of the commands. I don't mind if var doesn't get set, I just don't want the user to see the errors from these "lower level commands" on the screen; I want to test var separately after.
Here's an example with two, but I've got a chain of command so I don't want to echo the variable results every time into the next command.
res=$(ls bogusfile | grep morebogus 2>/dev/null)
Put the whole pipeline in a group:
res=$( { ls bogusfile | grep morebogus; } 2>/dev/null)
You need to redirect stderr for each command in the pipeline:
res=$(ls bogusfile 2>/dev/null | grep morebogus 2>/dev/null)
Or you could wrap everything in a subshell whose output is redirected:
res=$( (ls bogusfile | grep morebogus) 2>/dev/null)
You should be able to use {} to group multiple commands:
var=$( { command1 | command2 | command3; } 2>/dev/null)
You can also just redirect it for the entire script, using exec 2>/dev/null, e.g.
#!/bin/bash
return 2>/dev/null # prevent sourcing
exec 3>&2 2>/dev/null
# file descriptor 2 is directed to /dev/null for any commands here
exec 2>&3
# fd 2 is directed back to where it was originally for any commands here
Note: This will prevent interactive output and displaying the prompt. So you can execute the script, but you shouldn't just run the commands in an interactive shell or source it without the initial return line. You also won't be able to use read normally without redirecting the file descriptor back
Is there any way to a piped commands to replicate its previous command exit status?
For example:
#/bin/bash
(...)
function customizedLog() {
# do something with the piped command output
exit <returned value from the last piped comand/script (script.sh)>
}
script.sh | customizedLog
echo ${?} # here I wanna show the script exit value
(...)
I know I could simply check the return using ${PIPESTATUS[0]}, but I really want to do this like the customizedLog function wasn't there.
Any thoughts?
In bash:
set -o pipefail
This will return the last non-zero exit status in a pipeline, or zero if all commands in the pipeline succeed.
set -o pipefail
script.sh | customizedLog
echo ${?}
Just make sure customizedLog succeeds (return 0), and you should pick up the exit status of script.sh. Test with false | customizedLog and true | customizedLog.
script.sh | customizedLog
The above will run in two separate processes (or 3, actually -- customizedLog will run in a bash fork as you can verify with something like ps -T --forest). As far as I know, with the UNIX process model, the only process that has access to a process's return information is its parent so there's no way customized log will be able to retrieve it.
So no, unless the previous command is run from a wrapper command that passes the exit status through the pipe (e.g., as the last line):
( command ; echo $? ) | piped_command_that_is_aware_of_such_an_arrangement
For following bash statement:
tail -Fn0 /tmp/report | while [ 1 ]; do echo "pre"; exit; echo "past"; done
I got "pre", but didn't quit to the bash prompt, then if I input something into /tmp/report, I could quit from this script and get into bash prompt.
I think that's reasonable. the 'exit' make the 'while' statement quit, but the 'tail' still alive. If something input into /tmp/report, the 'tail' will output to pipe, then 'tail' will detect the pipe is close, then 'tail' quits.
Am I right? If not, would anyone provide a correct interpretation?
Is it possible to add anything into 'while' statement to quit from the whole pipe statement immediately? I know I could save the pid of tail into a temporary file, then read this file in the 'while', then kill the tail. Is there a simpler way?
Let me enlarge my question. If use this tail|while in a script file, is it possible to fulfill following items simultaneously?
a. If Ctrl-C is inputed or signal the main shell process, the main shell and various subshells and background processes spawned by the main shell will quit
b. I could quit from tail|while only at a trigger case, and preserve other subprocesses keep running
c. It's better not use temporary file or pipe file.
You're correct. The while loop is executing in a subshell because its input is redirected, and exit just exits from that subshell.
If you're running bash 4.x, you may be able to achieve what you want with a coprocess.
coproc TAIL { tail -Fn0 /tmp/report.txt ;}
while [ 1 ]
do
echo "pre"
break
echo "past"
done <&${TAIL[0]}
kill $TAIL_PID
http://www.gnu.org/software/bash/manual/html_node/Coprocesses.html
With older versions, you can use a background process writing to a named pipe:
pipe=/tmp/tail.$$
mkfifo $pipe
tail -Fn0 /tmp/report.txt >$pipe &
TAIL_PID=$!
while [ 1 ]
do
echo "pre"
break
echo "past"
done <$pipe
kill $TAIL_PID
rm $pipe
You can (unreliably) get away with killing the process group:
tail -Fn0 /tmp/report | while :
do
echo "pre"
sh -c 'PGID=$( ps -o pgid= $$ | tr -d \ ); kill -TERM -$PGID'
echo "past"
done
This may send the signal to more processes than you want. If you run the above command in an interactive terminal you should be okay, but in a script it is entirely possible (indeed likely) the the process group will include the script running the command. To avoid sending the signal, it would be wise to enable monitoring and run the pipeline in the background to ensure that a new process group is formed for the pipeline:
#!/bin/sh
# In Posix shells that support the User Portability Utilities option
# this includes bash & ksh), executing "set -m" turns on job control.
# Background processes run in a separate process group. If the shell
# is interactive, a line containing their exit status is printed to
# stderr upon their completion.
set -m
tail -Fn0 /tmp/report | while :
do
echo "pre"
sh -c 'PGID=$( ps -o pgid= $$ | tr -d \ ); kill -TERM -$PGID'
echo "past"
done &
wait
Note that I've replaced the while [ 1 ] with while : because while [ 1 ] is poor style. (It behaves exactly the same as while [ 0 ]).