Say I start with the following statement, which echo-s a string into the ether:
$ echo "foo" 1>/dev/null
I then submit the following pipeline:
$ echo "foo" | cat -e - 1>/dev/null
I then leave the process out:
$ echo "foo" | 1>/dev/null
Why is this not returning an error message? The documentation on bash and piping doesn't seem to make direct mention of may be the cause. Is there an EOF sent before the first read from echo (or whatever the process is, which is running upstream of the pipe)?
A shell simple command is not required to have a command name. For a command without a command-name:
variable assignments apply to the current execution environment. The following will set two variables to argument values:
arg1=$1 arg3=$3
redirections occur in a subshell, but the subshell doesn't do anything other than initialize the redirect. The following will truncate or create the indicated file (if you have appropriate permissions):
>/file/to/empty
However, a command must have at least one word. A completely empty command is a syntax error (which is why it is occasionally necessary to use :).
Answer summarized from Posix XCU§2.9.1
Related
I have a shell script which writes all output to logfile
and terminal, this part works fine, but if I execute the script
a new shell prompt only appear if I press enter. Why is that and how do I fix it?
#!/bin/bash
exec > >(tee logfile)
echo "output"
First, when I'm testing this, there always is a new shell prompt, it's just that sometimes the string output comes after it, so the prompt isn't last. Did you happen to overlook it? If so, there seems to be a race where the shell prints the prompt before the tee in the background completes.
Unfortunately, that cannot fixed by waiting in the shell for tee, see this question on unix.stackexchange. Fragile workarounds aside, the easiest way to solve this that I see is to put your whole script inside a list:
{
your-code-here
} | tee logfile
If I run the following script (suppressing the newline from the echo), I see the prompt, but not "output". The string is still written to the file.
#!/bin/bash
exec > >(tee logfile)
echo -n "output"
What I suspect is this: you have three different file descriptors trying to write to the same file (that is, the terminal): standard output of the shell, standard error of the shell, and the standard output of tee. The shell writes synchronously: first the echo to standard output, then the prompt to standard error, so the terminal is able to sequence them correctly. However, the third file descriptor is written to asynchronously by tee, so there is a race condition. I don't quite understand how my modification affects the race, but it appears to upset some balance, allowing the prompt to be written at a different time and appear on the screen. (I expect output buffering to play a part in this).
You might also try running your script after running the script command, which will log everything written to the terminal; if you wade through all the control characters in the file, you may notice the prompt in the file just prior to the output written by tee. In support of my race condition theory, I'll note that after running the script a few times, it was no longer displaying "abnormal" behavior; my shell prompt was displayed as expected after the string "output", so there is definitely some non-deterministic element to this situation.
#chepner's answer provides great background information.
Here's a workaround - works on Ubuntu 12.04 (Linux 3.2.0) and on OS X 10.9.1:
#!/bin/bash
exec > >(tee logfile)
echo "output"
# WORKAROUND - place LAST in your script.
# Execute an executable (as opposed to a builtin) that outputs *something*
# to make the prompt reappear normally.
# In this case we use the printf *executable* to output an *empty string*.
# Use of `$ec` is to ensure that the script's actual exit code is passed through.
ec=$?; $(which printf) ''; exit $ec
Alternatives:
#user2719058's answer shows a simple alternative: wrapping the entire script body in a group command ({ ... }) and piping it to tee logfile.
An external solution, as #chepner has already hinted at, is to use the script utility to create a "transcript" of your script's output in addition to displaying it:
script -qc yourScript /dev/null > logfile # Linux syntax
This, however, will also capture stderr output; if you wanted to avoid that, use:
script -qc 'yourScript 2>/dev/null' /dev/null > logfile
Note, however, that this will suppress stderr output altogether.
As others have noted, it's not that there's no prompt printed -- it's that the last of the output written by tee can come after the prompt, making the prompt no longer visible.
If you have bash 4.4 or newer, you can wait for your tee process to exit, like so:
#!/usr/bin/env bash
case $BASH_VERSION in ''|[0-3].*|4.[0-3]) echo "ERROR: Bash 4.4+ needed" >&2; exit 1;; esac
exec {orig_stdout}>&1 {orig_stderr}>&2 # make a backup of original stdout
exec > >(tee -a "_install_log"); tee_pid=$! # track PID of tee after starting it
cleanup() { # define a function we'll call during shutdown
retval=$?
exec >&$orig_stdout # Copy your original stdout back to FD 1, overwriting the pipe to tee
exec 2>&$orig_stderr # If something overwrites stderr to also go through tee, fix that too
wait "$tee_pid" # Now, wait until tee exits
exit "$retval" # and complete exit with our original exit status
}
trap cleanup EXIT # configure the function above to be called during cleanup
echo "Writing something to stdout here"
I have this command sequence that I'm having trouble understanding:
[me#mine ~]$ (echo 'test'; cat) | bash
echo $?
1
echo 'this is the new shell'
this is the new shell
exit
[me#mine ~]$
As far as I can understand, here is what happens:
A pipe is created.
stdout of echo 'test' is sent to the pipe.
bash receives 'test' on stdin.
echo $? returns 1, which is what happens when you run test without args.
cat runs.
It is copying stdin to stdout.
stdout is sent to the pipe.
bash will execute whatever you type in, but stderr won't get printed to the screen (we used |, not |&).
I have three questions:
It looks like, even though we run two commands, we use the same pipe and bash process for both commands. Is that the case?
Where do the prompts go?
When something like cat uses stdin, does it take exclusive ownership of stdin as long as the shell runs, or can other things use it?
I suspect I'm missing some detail with ttys, but I'm not sure. Any help or details or man excerpt appreciated!
So...
Yes, there's a single pipe sending commands to a single instance of bash. Note:
$ echo 'date "+%T hello $$"; sleep 1; date "+%T world $$"' | bash
22:18:52 hello 72628
22:18:53 world 72628
There are no prompts. From the man page:
An interactive shell is one started without non-option arguments (unless -s is specified) and without the -c option whose standard input and error are both connected to terminals. PS1 is set and $- includes i if bash is interactive.
So a pipe is not an interactive shell, and therefore has no prompt.
Stdin and stdout can only connect to one thing at a time. cat will take stdin from the process that ran it (for example, your interactive shell) and send its stdout through the pipe to bash. If you need multiple things to be able to submit to the stdin of that cat, consider using a named pipe.
Does that cover it?
When I'm looking at bash script code, I sometimes see | and sometimes see ||, but I don't know which is preferable.
I'm trying to do something like ..
set -e;
ret=0 && { which ansible || ret=$?; }
if [[ ${ret} -ne 0 ]]; then
# install ansible here
fi
Please advise which OR operator is preferred in this scenario.
| isn't an OR operator at all. You could use ||, though:
which ansible || {
true # put your code to install ansible here
}
This is equivalent to an if:
if ! which ansible; then
true # put your code to install ansible here
fi
By the way -- consider making a habit of using type (a shell builtin) rather than which (an external command). type is both faster and has a better understanding of shell behavior: If you have an ansible command that's provided by, say, a shell function invoking the real command, which won't know that it's there, but type will correctly detect it as available.
There is a big difference between using a single pipe (pipe output from one command to be used as input for the next command) and a process control OR (double pipe).
cat /etc/issue | less
This runs the cat command on the /etc/issue file, and instead of immediately sending the output to stdout it is piped to be the input for the less command. Yes, this isn't a great example, since you could instead simply do less /etc/issue - but at least you can see how it works
touch /etc/testing || echo Did not work
For this one, the touch command is run, or attempted to run. If it has a non-zero exit status, then the double pipe OR kicks in, and tries to execute the echo command. If the touch command worked, then whatever the other choice is (our echo command in this case) is never attempted...
I have this bash script (actually a part of https://github.com/ddollar/heroku-buildpack-multi/blob/master/bin/compile with echo I've added myself):
echo "[DEBUG] chmod done"
framework=$($dir/bin/detect $1)
echo "[DEBUG] $framework done"
And I see in the log:
[DEBUG] chmod done
Staging failed: Buildpack compilation step failed
And I do not see the second echo in the logs at all.
I do not know much bash unfortunately. Could anybody explain to me in what case the first echo performs and second do not? I always thought that both echo should always work no matter whether the second line succeeds or not.
It's not visible in your question, but clicking on your link, it says in the third line
set -e
This means to stop processing the script immediately whenever an error occurs. Comment that line, and the script should run through and also print the second echo statement.
Note that I didn't inspect what the script actually does and I cannot tell you if commenting out set -e is actually good advice or not.
From man set:
−e: When this option is on, when any command fails (for any of the reasons listed
in Section 2.8.1, Consequences of Shell Errors or by returning an exit status
greater than zero), the shell immediately shall exit with the following excep‐
tions:
1. The failure of any individual command in a multi-command pipeline shall not
cause the shell to exit. Only the failure of the pipeline itself shall be
considered.
2. The −e setting shall be ignored when executing the compound list following
the while, until, if, or elif reserved word, a pipeline beginning with the
! reserved word, or any command of an AND-OR list other than the last.
3. If the exit status of a compound command other than a subshell command was
the result of a failure while −e was being ignored, then −e shall not apply
to this command.
This requirement applies to the shell environment and each subshell environment
separately. For example, in:
set -e; (false; echo one) | cat; echo two
the false command causes the subshell to exit without executing echo one; how‐
ever, echo two is executed because the exit status of the pipeline (false; echo
one) | cat is zero.
I am writing a script that among other things runs a shell command several times. This command doesn't handle exit codes very well and I need to know if the process ended successfully or not.
So what I was thinking is to analyze the stderr to find out the word error (using grep). I know this is not the best thing to do, I'm working on it....
Anyway, the only way I can imagine is to put the stderr of that program in a variable and then use grep to well, "grep" it and throw it to another variable. Then I can see if that variable is valorized, meaning that there was an error, and do my work.
The qustion is: how can I do this ?
I don't really want to run the program inside a variable, because it has got a lot of arguments (with special characters such as backslash, quotes, doublequotes...) and it's a memory and I/O intensive program.
Awaiting your reply, thanks.
Redirect the stderr of that command to a temporary file and check if the word "error" is present in that file.
mycommand 2> /tmp/temp.txt
grep error /tmp/temp.txt
Thanks #Jdamian, this was my answer too, in the end.
I asked my principal if I can write a temp file and it allowed, so this is the end result:
... script
command to be launched -argument -other "argument" -other "other" argument 2>&1 | tee $TEMPFILE
ERRORCODE=( `grep -i error "$TEMPFILE" `)
if [ -z $ERRORCODE ] ;
then
some actions ....
I didn't tested this yet because I got some other scripts involved that I need to write before.
What I'm trying to do is:
run the command, having its stderr redirected to stdout;
using tee, have the above result printed on screen and also to the temp file;
have grep to store the string error found on the temp file (if any) in a variable called ERRORCODE;
if that variable is populated (which mean if it has been created by grep), then the script stops, quitting with status 1, else it contiunes.
What do you think ?
If you don't need the standard output:
if mycommand 2>&1 >/dev/null | grep -q error; then
echo an error occurred
fi