I'm facing to a problem that I'm not able to explain. I've to recode a shell, and there is a weird behavior for me.
echo test >&2 2>&1
This kind of command is writing on stderr (so with the first redirection), why the second redirection isn't impacting the output destination ? Why the output isn't on stdout ?
I saw some stuff that redirection is happening before executing command, from left to right, so why the second redirection is not cancelling the first one ?
Thanks in advance.
edit: I'm running my script on bash.
x>&y means "redirect file descriptor x to whatever fd y is currently pointing to.
Redirections are processed strictly left to right.
So, >&2 2>&1 points fd 1 to something like /dev/stderr, and then points fd 2 to /dev/stderr also.
If you want to swap stderr and stdout, you need a 3rd file descriptor:
(echo "test to stdout"; echo "test to stderr" >&2) 3>&1 1>&2 2>&3 3>&-
# ................................................ A .. B .. C .. D
# A. fd 3 = /dev/stdout
# B. fd 1 = /dev/stderr
# C. fd 2 = /dev/stdout
# D. fd 3 is closed
Let's put that in a function for easier testing
fdtest() { (echo "test to stdout"; echo "test to stderr" >&2) 3>&1 1>&2 2>&3 3>&-; }
Run it
$ fdtest
test to stdout
test to stderr
Throw away standard error (we expect to see the "stderr" message on standard out).
$ fdtest 2>/dev/null
test to stderr
Throw away standard out (we expect to see the "stdout" message on standard err).
$ fdtest 1>/dev/null
test to stdout
cmd >&2 2>&1 is very different than cmd 2>&1 >&2, since the redirections occur in a different order. Suppose the command is invoked from a process which has fd 1 attached to a file named 'output' and fd 2 attached to a filed named 'error'. Then cmd >&2 2>&1 will redirect the stdout of cmd to the file named error and then redirect stderr of cmd to whatever fd 1 is connected to, namely the file named error. But cmd 2>&1 >&2 will first redirect fd 2 to 'output' and the redirect fd 1 to the same place. In other words, cmd >&2 2>&1 writes everything to stderr, and cmd 2>&1 >&2 writes everything on stdout.
Related
This question already has answers here:
How to redirect and append both standard output and standard error to a file with Bash
(8 answers)
Closed 1 year ago.
I want to redirect both standard output and standard error of a process to a single file. How do I do that in Bash?
Take a look here. It should be:
yourcommand &> filename
It redirects both standard output and standard error to file filename.
do_something 2>&1 | tee -a some_file
This is going to redirect standard error to standard output and standard output to some_file and print it to standard output.
You can redirect stderr to stdout and the stdout into a file:
some_command >file.log 2>&1
See Chapter 20. I/O Redirection
This format is preferred over the most popular &> format that only works in Bash. In Bourne shell it could be interpreted as running the command in background. Also the format is more readable - 2 (is standard error) redirected to 1 (standard output).
# Close standard output file descriptor
exec 1<&-
# Close standard error file descriptor
exec 2<&-
# Open standard output as $LOG_FILE file for read and write.
exec 1<>$LOG_FILE
# Redirect standard error to standard output
exec 2>&1
echo "This line will appear in $LOG_FILE, not 'on screen'"
Now, a simple echo will write to $LOG_FILE, and it is useful for daemonizing.
To the author of the original post,
It depends what you need to achieve. If you just need to redirect in/out of a command you call from your script, the answers are already given. Mine is about redirecting within current script which affects all commands/built-ins (includes forks) after the mentioned code snippet.
Another cool solution is about redirecting to both standard error and standard output and to log to a log file at once which involves splitting "a stream" into two. This functionality is provided by 'tee' command which can write/append to several file descriptors (files, sockets, pipes, etc.) at once: tee FILE1 FILE2 ... >(cmd1) >(cmd2) ...
exec 3>&1 4>&2 1> >(tee >(logger -i -t 'my_script_tag') >&3) 2> >(tee >(logger -i -t 'my_script_tag') >&4)
trap 'cleanup' INT QUIT TERM EXIT
get_pids_of_ppid() {
local ppid="$1"
RETVAL=''
local pids=`ps x -o pid,ppid | awk "\\$2 == \\"$ppid\\" { print \\$1 }"`
RETVAL="$pids"
}
# Needed to kill processes running in background
cleanup() {
local current_pid element
local pids=( "$$" )
running_pids=("${pids[#]}")
while :; do
current_pid="${running_pids[0]}"
[ -z "$current_pid" ] && break
running_pids=("${running_pids[#]:1}")
get_pids_of_ppid $current_pid
local new_pids="$RETVAL"
[ -z "$new_pids" ] && continue
for element in $new_pids; do
running_pids+=("$element")
pids=("$element" "${pids[#]}")
done
done
kill ${pids[#]} 2>/dev/null
}
So, from the beginning. Let's assume we have a terminal connected to /dev/stdout (file descriptor #1) and /dev/stderr (file descriptor #2). In practice, it could be a pipe, socket or whatever.
Create file descriptors (FDs) #3 and #4 and point to the same "location" as #1 and #2 respectively. Changing file descriptor #1 doesn't affect file descriptor #3 from now on. Now, file descriptors #3 and #4 point to standard output and standard error respectively. These will be used as real terminal standard output and standard error.
1> >(...) redirects standard output to command in parentheses
Parentheses (sub-shell) executes 'tee', reading from exec's standard output (pipe) and redirects to the 'logger' command via another pipe to the sub-shell in parentheses. At the same time it copies the same input to file descriptor #3 (the terminal)
the second part, very similar, is about doing the same trick for standard error and file descriptors #2 and #4.
The result of running a script having the above line and additionally this one:
echo "Will end up in standard output (terminal) and /var/log/messages"
...is as follows:
$ ./my_script
Will end up in standard output (terminal) and /var/log/messages
$ tail -n1 /var/log/messages
Sep 23 15:54:03 wks056 my_script_tag[11644]: Will end up in standard output (terminal) and /var/log/messages
If you want to see clearer picture, add these two lines to the script:
ls -l /proc/self/fd/
ps xf
bash your_script.sh 1>file.log 2>&1
1>file.log instructs the shell to send standard output to the file file.log, and 2>&1 tells it to redirect standard error (file descriptor 2) to standard output (file descriptor 1).
Note: The order matters as liw.fi pointed out, 2>&1 1>file.log doesn't work.
Curiously, this works:
yourcommand &> filename
But this gives a syntax error:
yourcommand &>> filename
syntax error near unexpected token `>'
You have to use:
yourcommand 1>> filename 2>&1
Short answer: Command >filename 2>&1 or Command &>filename
Explanation:
Consider the following code which prints the word "stdout" to stdout and the word "stderror" to stderror.
$ (echo "stdout"; echo "stderror" >&2)
stdout
stderror
Note that the '&' operator tells bash that 2 is a file descriptor (which points to the stderr) and not a file name. If we left out the '&', this command would print stdout to stdout, and create a file named "2" and write stderror there.
By experimenting with the code above, you can see for yourself exactly how redirection operators work. For instance, by changing which file which of the two descriptors 1,2, is redirected to /dev/null the following two lines of code delete everything from the stdout, and everything from stderror respectively (printing what remains).
$ (echo "stdout"; echo "stderror" >&2) 1>/dev/null
stderror
$ (echo "stdout"; echo "stderror" >&2) 2>/dev/null
stdout
Now, we can explain why the solution why the following code produces no output:
(echo "stdout"; echo "stderror" >&2) >/dev/null 2>&1
To truly understand this, I highly recommend you read this webpage on file descriptor tables. Assuming you have done that reading, we can proceed. Note that Bash processes left to right; thus Bash sees >/dev/null first (which is the same as 1>/dev/null), and sets the file descriptor 1 to point to /dev/null instead of the stdout. Having done this, Bash then moves rightwards and sees 2>&1. This sets the file descriptor 2 to point to the same file as file descriptor 1 (and not to file descriptor 1 itself!!!! (see this resource on pointers for more info)) . Since file descriptor 1 points to /dev/null, and file descriptor 2 points to the same file as file descriptor 1, file descriptor 2 now also points to /dev/null. Thus both file descriptors point to /dev/null, and this is why no output is rendered.
To test if you really understand the concept, try to guess the output when we switch the redirection order:
(echo "stdout"; echo "stderror" >&2) 2>&1 >/dev/null
stderror
The reasoning here is that evaluating from left to right, Bash sees 2>&1, and thus sets the file descriptor 2 to point to the same place as file descriptor 1, ie stdout. It then sets file descriptor 1 (remember that >/dev/null = 1>/dev/null) to point to >/dev/null, thus deleting everything which would usually be send to to the standard out. Thus all we are left with was that which was not send to stdout in the subshell (the code in the parentheses)- i.e. "stderror".
The interesting thing to note there is that even though 1 is just a pointer to the stdout, redirecting pointer 2 to 1 via 2>&1 does NOT form a chain of pointers 2 -> 1 -> stdout. If it did, as a result of redirecting 1 to /dev/null, the code 2>&1 >/dev/null would give the pointer chain 2 -> 1 -> /dev/null, and thus the code would generate nothing, in contrast to what we saw above.
Finally, I'd note that there is a simpler way to do this:
From section 3.6.4 here, we see that we can use the operator &> to redirect both stdout and stderr. Thus, to redirect both the stderr and stdout output of any command to \dev\null (which deletes the output), we simply type
$ command &> /dev/null
or in case of my example:
$ (echo "stdout"; echo "stderror" >&2) &>/dev/null
Key takeaways:
File descriptors behave like pointers (although file descriptors are not the same as file pointers)
Redirecting a file descriptor "a" to a file descriptor "b" which points to file "f", causes file descriptor "a" to point to the same place as file descriptor b - file "f". It DOES NOT form a chain of pointers a -> b -> f
Because of the above, order matters, 2>&1 >/dev/null is != >/dev/null 2>&1. One generates output and the other does not!
Finally have a look at these great resources:
Bash Documentation on Redirection, An Explanation of File Descriptor Tables, Introduction to Pointers
LOG_FACILITY="local7.notice"
LOG_TOPIC="my-prog-name"
LOG_TOPIC_OUT="$LOG_TOPIC-out[$$]"
LOG_TOPIC_ERR="$LOG_TOPIC-err[$$]"
exec 3>&1 > >(tee -a /dev/fd/3 | logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_OUT" )
exec 2> >(logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_ERR" )
It is related: Writing standard output and standard error to syslog.
It almost works, but not from xinetd ;(
For the situation when "piping" is necessary, you can use |&.
For example:
echo -ne "15\n100\n" | sort -c |& tee >sort_result.txt
or
TIMEFORMAT=%R;for i in `seq 1 20` ; do time kubectl get pods | grep node >>js.log ; done |& sort -h
These Bash-based solutions can pipe standard output and standard error separately (from standard error of "sort -c", or from standard error to "sort -h").
I wanted a solution to have the output from stdout plus stderr written into a log file and stderr still on console. So I needed to duplicate the stderr output via tee.
This is the solution I found:
command 3>&1 1>&2 2>&3 1>>logfile | tee -a logfile
First swap stderr and stdout
then append the stdout to the log file
pipe stderr to tee and append it also to the log file
Adding to what Fernando Fabreti did, I changed the functions slightly and removed the &- closing and it worked for me.
function saveStandardOutputs {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
exec 3>&1
exec 4>&2
trap restoreStandardOutputs EXIT
else
echo "[ERROR]: ${FUNCNAME[0]}: Cannot save standard outputs because they have been redirected before"
exit 1;
fi
}
# Parameters: $1 => logfile to write to
function redirectOutputsToLogfile {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
LOGFILE=$1
if [ -z "$LOGFILE" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: logfile empty [$LOGFILE]"
fi
if [ ! -f $LOGFILE ]; then
touch $LOGFILE
fi
if [ ! -f $LOGFILE ]; then
echo "[ERROR]: ${FUNCNAME[0]}: creating logfile [$LOGFILE]"
exit 1
fi
saveStandardOutputs
exec 1>>${LOGFILE}
exec 2>&1
OUTPUTS_REDIRECTED="true"
else
echo "[ERROR]: ${FUNCNAME[0]}: Cannot redirect standard outputs because they have been redirected before"
exit 1;
fi
}
function restoreStandardOutputs {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
exec 1>&3 #restore stdout
exec 2>&4 #restore stderr
OUTPUTS_REDIRECTED="false"
fi
}
LOGFILE_NAME="tmp/one.log"
OUTPUTS_REDIRECTED="false"
echo "this goes to standard output"
redirectOutputsToLogfile $LOGFILE_NAME
echo "this goes to logfile"
echo "${LOGFILE_NAME}"
restoreStandardOutputs
echo "After restore this goes to standard output"
The "easiest" way (Bash 4 only):
ls * 2>&- 1>&-
In situations when you consider using things like exec 2>&1, I find it easier to read, if possible, rewriting code using Bash functions like this:
function myfunc(){
[...]
}
myfunc &>mylog.log
The following functions can be used to automate the process of toggling outputs beetwen stdout/stderr and a logfile.
#!/bin/bash
#set -x
# global vars
OUTPUTS_REDIRECTED="false"
LOGFILE=/dev/stdout
# "private" function used by redirect_outputs_to_logfile()
function save_standard_outputs {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot save standard outputs because they have been redirected before"
exit 1;
fi
exec 3>&1
exec 4>&2
trap restore_standard_outputs EXIT
}
# Params: $1 => logfile to write to
function redirect_outputs_to_logfile {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot redirect standard outputs because they have been redirected before"
exit 1;
fi
LOGFILE=$1
if [ -z "$LOGFILE" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: logfile empty [$LOGFILE]"
fi
if [ ! -f $LOGFILE ]; then
touch $LOGFILE
fi
if [ ! -f $LOGFILE ]; then
echo "[ERROR]: ${FUNCNAME[0]}: creating logfile [$LOGFILE]"
exit 1
fi
save_standard_outputs
exec 1>>${LOGFILE%.log}.log
exec 2>&1
OUTPUTS_REDIRECTED="true"
}
# "private" function used by save_standard_outputs()
function restore_standard_outputs {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot restore standard outputs because they have NOT been redirected"
exit 1;
fi
exec 1>&- #closes FD 1 (logfile)
exec 2>&- #closes FD 2 (logfile)
exec 2>&4 #restore stderr
exec 1>&3 #restore stdout
OUTPUTS_REDIRECTED="false"
}
Example of usage inside script:
echo "this goes to stdout"
redirect_outputs_to_logfile /tmp/one.log
echo "this goes to logfile"
restore_standard_outputs
echo "this goes to stdout"
For tcsh, I have to use the following command:
command >& file
If using command &> file, it will give an "Invalid null command" error.
Perhaps it's just late and I'm having brain farts but shouldn't this
(>&2 echo dying) 2>&1 >/dev/null
produce no output in a normal shell?
Similarly if this is /tmp/x.pl
#!/usr/bin/perl
die "dying"
Then why does this
#> perl /tmp/x.pl 2>&1 >/dev/null
output
dying at /tmp/x.pl line 2.
?
Redirections are processed left-to-right. So you're doing 2>&1 before you do >/dev/null. This redirects FD 2 to the original connection of FD 1 (presumably the terminal), then redirects FD 1 to /dev/null. FD 2 is still connected to the terminal.
To redirect both stdout and stderr to /dev/null, you have to use
(>&2 echo dying) >/dev/null 2>&1
The order in which file descriptor redirections is done is very important.
Just switch the orders:
(>&2 echo dying) >/dev/null 2>&1
perl /tmp/x.pl >/dev/null 2>&1
While you are doing:
(>&2 echo dying) 2>&1 >/dev/null
the STDOUT of subshell (()) is redirected to where STDERR of the subshell is first. Then in the parent (main) shell, you have redirected STDERR to STDOUT, which is pointing to the terminal at that moment, so the STDERR from the subshell will get printed, then you are redirecting STDOUT to /dev/null which will send STDOUT to /dev/null from that time of evaluation, not before.
Similar note goes for the second case too.
So, always maintain order while manipulation file descriptors, and the order of evaluation if from left to right.
update: I think the below code may be headed in the wrong direction -- but the question remains, can I open a pipe to log all output (file&console), pause that log and log to a new log (new file&console), and then re-attach to the FD for the original logger just by moving FDs around and not re-opening the original log file
Trying to improve my knowledge of FDs in bash. I'm trying to log all output of the main "meta" test.sh -- but log to a different file when I get to "sections" -- e.g. functions, sourced scripts, etc. And then go back to appending to the "meta" log.
I know I could pretty easily accomplish this with subshells -- or by opening the 'meta' log again and append from there, but can anyone help accomplish this by switching FDs around?
#!/bin/bash
rm *.log
NAMED_PIPE="$(mktemp -u /tmp/pipe.XXXX)"
mknod $NAMED_PIPE p
tee <$NAMED_PIPE "./meta.log" &
section () {
echo SECTION: stdout
echo SECTION: stderr >&2
}
# link stdout->3 & stderr->4 and save stdout & stderr
exec 3>&1 4>&2 &> "$NAMED_PIPE"
echo METAstr: stdout
echo METAstr: stderr >&2
# restore stdout & stderr
exec 1>&3 2>&4
# sleep 1 # I think an additional delay prevents the possible race condition I'm seeing
# exec 1>&3- 2>&4- ... I think this would restore but close 3 & 4?
# do I need another named pipe here?
section 2>&1 # | tee section.log
# re-link to same pipe
exec 3>&1 4>&2 &> "$NAMED_PIPE"
echo METAend: stdout
echo METAend: stderr >&2
Without trying to log 'section' all the meta output gets printed after the return of the script:
-bash-4.2# ./test.sh
SECTION: stdout
SECTION: stderr
-bash-4.2# METAstr: stdout
METAstr: stderr
METAend: stdout
METAend: stderr
And trying to log 'section' I think fouls up my FDs so the following exec hangs me up:
-bash-4.2# ./test.sh
METAstr: stdout
METAstr: stderr
SECTION: stdout
SECTION: stderr
EDIT1:
Contents of meta.log after running the script without trying to tee section:
[root#master tmp]# cat meta.log
METAstr: stdout
METAstr: stderr
METAend: stdout
METAend: stderr
It logs the ending messages, the tee does not exit until the script does
EDIT2:
Revision of EDIT1. I think It's a race condition. I think the FDs are being closed -- but they're not closed by the time the final echo commands happen.
I was just going to write the same thing. It's a race condition. The second exec closes the writing end in your process, signalling an EOF to tee. tee will want to exit when it gets the EOF. If it does exit by the time you call the last exec, the last exec will block. If it hasn't exited yet at that point, it will not block, because a reading end of the FIFO will still be open.
Any delay will make it more likely tee will have exited by that time.
Spawning a process makes it very likely. I found with stracing (which slows the program down a little) it's about 50/50.
Given a shell script:
#!/bin/sh
echo "I'm stdout";
echo "I'm stderr" >&2;
Is there a way to call that script such that only stderr would print out, when the last part of the command is 2>/dev/null, ie
$ > sh myscript.sh SOME_OPTIONS_HERE 2>/dev/null
I'm stderr
Or, alternatively:
$ > sh myscript.sh SOME_OPTIONS_HERE >/dev/null
I'm stdout
It's a question at the end of a set of lecture slides, but after nearly a day working at this, I'm nearly certain it's some sort of typo. Pivoting doesn't work. 2>&- doesn't work. I'm out of ideas!
% (sh myscript.sh 3>&2 2>&1 1>&3) 2>/dev/null
I'm stderr
% (sh myscript.sh 3>&2 2>&1 1>&3) >/dev/null
I'm stdout
Explanation of 3>&2 2>&1 1>&3:
3>&2 means make a copy of file descriptor 2 (fd 2) (stderr), named fd 3 (file descriptor 3). It copies the file descriptor, it doesn't duplicate the stream as tee does.
2>&1 means that fd 2 of sh myscript.sh becomes a copy of it's fd 1 (stdout). Now, when myscript writes to it's stderr (it's fd 2), we receive it on stdout (our fd 1).
1>&3 means that fd 1 of sh myscript.sh becomes a copy of fd 3 (stderr). Now, when myscript writes to it's stdout (it's fd 1), we receive it on stderr (our fd 2).
For sake of completeness, based on a comment by #200_success above, it is probably better to move the file descriptor 3 using 1>&3- :
$ (sh myscript.sh 3>&2 2>&1 1>&3-) 2>/dev/null
I'm stderr
$ (sh myscript.sh 3>&2 2>&1 1>&3-) >/dev/null
I'm stdout
Instead of swapping file descriptors on a per-process basis, using exec you can swap stdin & stderr for all following commands launched by the current shell :
$ (exec 3>&2 2>&1 1>&3- ; sh myscript.sh ; sh myscript.sh ) 2>/dev/null
I'm stderr
I'm stderr
$ (exec 3>&2 2>&1 1>&3- ; sh myscript.sh ; sh myscript.sh ) >/dev/null
I'm stdout
I'm stdout
Moving a file descriptor (1>&3-) is not portable, not all POSIX shell implementations support it. It is a ksh93-ism and bash-ism. (more info here https://unix.stackexchange.com/questions/65000/practical-use-for-moving-file-descriptors)
It is also possible to close FD 3 instead after performing the redirections.
ls 3>&2 2>&1 1>&3 3>&-
prints the contents of the current working directory out stderr.
The syntax 3>&- or 3<&- closes file descriptor 3.
The bash hackers wiki can be very useful in this kind of things.
There's a way of doing it which is not mentioned among these answers, so
I'll put my two cents.
The semantics of >&N, for numeric N, means redirect to the target of
the file descriptor N. The word target is important since the descriptor can change target later, but once we copied that target we don't care. That's the reason why the order in which we declare of redirection is relevant.
So you can do it as follows:
./myscript.sh 2>&1 >/dev/null
That means:
redirect stderr to stdout's target, that is the stdout output stream. Now
stderr copied stdout's target
change stdout to /dev/null. This won't affect stderr, since it "copied"
the target before we changed it.
No need for a third file descriptor.
It is interesting how I can't simply do >&-, instead of >/dev/null. This actually closes stdout, so I'm getting an error (on stderr's target, that is the actual stdout, of course :D)
line 3: echo: write error: Bad file descriptor
You can see that order is relevant by trying to swap the redirections:
./myscript.sh >/dev/null 2>&1
This will not work, because:
We set the target of stdout to /dev/null
We set the target of stderr to stdout's target, that is /dev/null again.
This question already has answers here:
How to redirect and append both standard output and standard error to a file with Bash
(8 answers)
Closed 1 year ago.
I want to redirect both standard output and standard error of a process to a single file. How do I do that in Bash?
Take a look here. It should be:
yourcommand &> filename
It redirects both standard output and standard error to file filename.
do_something 2>&1 | tee -a some_file
This is going to redirect standard error to standard output and standard output to some_file and print it to standard output.
You can redirect stderr to stdout and the stdout into a file:
some_command >file.log 2>&1
See Chapter 20. I/O Redirection
This format is preferred over the most popular &> format that only works in Bash. In Bourne shell it could be interpreted as running the command in background. Also the format is more readable - 2 (is standard error) redirected to 1 (standard output).
# Close standard output file descriptor
exec 1<&-
# Close standard error file descriptor
exec 2<&-
# Open standard output as $LOG_FILE file for read and write.
exec 1<>$LOG_FILE
# Redirect standard error to standard output
exec 2>&1
echo "This line will appear in $LOG_FILE, not 'on screen'"
Now, a simple echo will write to $LOG_FILE, and it is useful for daemonizing.
To the author of the original post,
It depends what you need to achieve. If you just need to redirect in/out of a command you call from your script, the answers are already given. Mine is about redirecting within current script which affects all commands/built-ins (includes forks) after the mentioned code snippet.
Another cool solution is about redirecting to both standard error and standard output and to log to a log file at once which involves splitting "a stream" into two. This functionality is provided by 'tee' command which can write/append to several file descriptors (files, sockets, pipes, etc.) at once: tee FILE1 FILE2 ... >(cmd1) >(cmd2) ...
exec 3>&1 4>&2 1> >(tee >(logger -i -t 'my_script_tag') >&3) 2> >(tee >(logger -i -t 'my_script_tag') >&4)
trap 'cleanup' INT QUIT TERM EXIT
get_pids_of_ppid() {
local ppid="$1"
RETVAL=''
local pids=`ps x -o pid,ppid | awk "\\$2 == \\"$ppid\\" { print \\$1 }"`
RETVAL="$pids"
}
# Needed to kill processes running in background
cleanup() {
local current_pid element
local pids=( "$$" )
running_pids=("${pids[#]}")
while :; do
current_pid="${running_pids[0]}"
[ -z "$current_pid" ] && break
running_pids=("${running_pids[#]:1}")
get_pids_of_ppid $current_pid
local new_pids="$RETVAL"
[ -z "$new_pids" ] && continue
for element in $new_pids; do
running_pids+=("$element")
pids=("$element" "${pids[#]}")
done
done
kill ${pids[#]} 2>/dev/null
}
So, from the beginning. Let's assume we have a terminal connected to /dev/stdout (file descriptor #1) and /dev/stderr (file descriptor #2). In practice, it could be a pipe, socket or whatever.
Create file descriptors (FDs) #3 and #4 and point to the same "location" as #1 and #2 respectively. Changing file descriptor #1 doesn't affect file descriptor #3 from now on. Now, file descriptors #3 and #4 point to standard output and standard error respectively. These will be used as real terminal standard output and standard error.
1> >(...) redirects standard output to command in parentheses
Parentheses (sub-shell) executes 'tee', reading from exec's standard output (pipe) and redirects to the 'logger' command via another pipe to the sub-shell in parentheses. At the same time it copies the same input to file descriptor #3 (the terminal)
the second part, very similar, is about doing the same trick for standard error and file descriptors #2 and #4.
The result of running a script having the above line and additionally this one:
echo "Will end up in standard output (terminal) and /var/log/messages"
...is as follows:
$ ./my_script
Will end up in standard output (terminal) and /var/log/messages
$ tail -n1 /var/log/messages
Sep 23 15:54:03 wks056 my_script_tag[11644]: Will end up in standard output (terminal) and /var/log/messages
If you want to see clearer picture, add these two lines to the script:
ls -l /proc/self/fd/
ps xf
bash your_script.sh 1>file.log 2>&1
1>file.log instructs the shell to send standard output to the file file.log, and 2>&1 tells it to redirect standard error (file descriptor 2) to standard output (file descriptor 1).
Note: The order matters as liw.fi pointed out, 2>&1 1>file.log doesn't work.
Curiously, this works:
yourcommand &> filename
But this gives a syntax error:
yourcommand &>> filename
syntax error near unexpected token `>'
You have to use:
yourcommand 1>> filename 2>&1
Short answer: Command >filename 2>&1 or Command &>filename
Explanation:
Consider the following code which prints the word "stdout" to stdout and the word "stderror" to stderror.
$ (echo "stdout"; echo "stderror" >&2)
stdout
stderror
Note that the '&' operator tells bash that 2 is a file descriptor (which points to the stderr) and not a file name. If we left out the '&', this command would print stdout to stdout, and create a file named "2" and write stderror there.
By experimenting with the code above, you can see for yourself exactly how redirection operators work. For instance, by changing which file which of the two descriptors 1,2, is redirected to /dev/null the following two lines of code delete everything from the stdout, and everything from stderror respectively (printing what remains).
$ (echo "stdout"; echo "stderror" >&2) 1>/dev/null
stderror
$ (echo "stdout"; echo "stderror" >&2) 2>/dev/null
stdout
Now, we can explain why the solution why the following code produces no output:
(echo "stdout"; echo "stderror" >&2) >/dev/null 2>&1
To truly understand this, I highly recommend you read this webpage on file descriptor tables. Assuming you have done that reading, we can proceed. Note that Bash processes left to right; thus Bash sees >/dev/null first (which is the same as 1>/dev/null), and sets the file descriptor 1 to point to /dev/null instead of the stdout. Having done this, Bash then moves rightwards and sees 2>&1. This sets the file descriptor 2 to point to the same file as file descriptor 1 (and not to file descriptor 1 itself!!!! (see this resource on pointers for more info)) . Since file descriptor 1 points to /dev/null, and file descriptor 2 points to the same file as file descriptor 1, file descriptor 2 now also points to /dev/null. Thus both file descriptors point to /dev/null, and this is why no output is rendered.
To test if you really understand the concept, try to guess the output when we switch the redirection order:
(echo "stdout"; echo "stderror" >&2) 2>&1 >/dev/null
stderror
The reasoning here is that evaluating from left to right, Bash sees 2>&1, and thus sets the file descriptor 2 to point to the same place as file descriptor 1, ie stdout. It then sets file descriptor 1 (remember that >/dev/null = 1>/dev/null) to point to >/dev/null, thus deleting everything which would usually be send to to the standard out. Thus all we are left with was that which was not send to stdout in the subshell (the code in the parentheses)- i.e. "stderror".
The interesting thing to note there is that even though 1 is just a pointer to the stdout, redirecting pointer 2 to 1 via 2>&1 does NOT form a chain of pointers 2 -> 1 -> stdout. If it did, as a result of redirecting 1 to /dev/null, the code 2>&1 >/dev/null would give the pointer chain 2 -> 1 -> /dev/null, and thus the code would generate nothing, in contrast to what we saw above.
Finally, I'd note that there is a simpler way to do this:
From section 3.6.4 here, we see that we can use the operator &> to redirect both stdout and stderr. Thus, to redirect both the stderr and stdout output of any command to \dev\null (which deletes the output), we simply type
$ command &> /dev/null
or in case of my example:
$ (echo "stdout"; echo "stderror" >&2) &>/dev/null
Key takeaways:
File descriptors behave like pointers (although file descriptors are not the same as file pointers)
Redirecting a file descriptor "a" to a file descriptor "b" which points to file "f", causes file descriptor "a" to point to the same place as file descriptor b - file "f". It DOES NOT form a chain of pointers a -> b -> f
Because of the above, order matters, 2>&1 >/dev/null is != >/dev/null 2>&1. One generates output and the other does not!
Finally have a look at these great resources:
Bash Documentation on Redirection, An Explanation of File Descriptor Tables, Introduction to Pointers
LOG_FACILITY="local7.notice"
LOG_TOPIC="my-prog-name"
LOG_TOPIC_OUT="$LOG_TOPIC-out[$$]"
LOG_TOPIC_ERR="$LOG_TOPIC-err[$$]"
exec 3>&1 > >(tee -a /dev/fd/3 | logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_OUT" )
exec 2> >(logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_ERR" )
It is related: Writing standard output and standard error to syslog.
It almost works, but not from xinetd ;(
For the situation when "piping" is necessary, you can use |&.
For example:
echo -ne "15\n100\n" | sort -c |& tee >sort_result.txt
or
TIMEFORMAT=%R;for i in `seq 1 20` ; do time kubectl get pods | grep node >>js.log ; done |& sort -h
These Bash-based solutions can pipe standard output and standard error separately (from standard error of "sort -c", or from standard error to "sort -h").
I wanted a solution to have the output from stdout plus stderr written into a log file and stderr still on console. So I needed to duplicate the stderr output via tee.
This is the solution I found:
command 3>&1 1>&2 2>&3 1>>logfile | tee -a logfile
First swap stderr and stdout
then append the stdout to the log file
pipe stderr to tee and append it also to the log file
Adding to what Fernando Fabreti did, I changed the functions slightly and removed the &- closing and it worked for me.
function saveStandardOutputs {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
exec 3>&1
exec 4>&2
trap restoreStandardOutputs EXIT
else
echo "[ERROR]: ${FUNCNAME[0]}: Cannot save standard outputs because they have been redirected before"
exit 1;
fi
}
# Parameters: $1 => logfile to write to
function redirectOutputsToLogfile {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
LOGFILE=$1
if [ -z "$LOGFILE" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: logfile empty [$LOGFILE]"
fi
if [ ! -f $LOGFILE ]; then
touch $LOGFILE
fi
if [ ! -f $LOGFILE ]; then
echo "[ERROR]: ${FUNCNAME[0]}: creating logfile [$LOGFILE]"
exit 1
fi
saveStandardOutputs
exec 1>>${LOGFILE}
exec 2>&1
OUTPUTS_REDIRECTED="true"
else
echo "[ERROR]: ${FUNCNAME[0]}: Cannot redirect standard outputs because they have been redirected before"
exit 1;
fi
}
function restoreStandardOutputs {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
exec 1>&3 #restore stdout
exec 2>&4 #restore stderr
OUTPUTS_REDIRECTED="false"
fi
}
LOGFILE_NAME="tmp/one.log"
OUTPUTS_REDIRECTED="false"
echo "this goes to standard output"
redirectOutputsToLogfile $LOGFILE_NAME
echo "this goes to logfile"
echo "${LOGFILE_NAME}"
restoreStandardOutputs
echo "After restore this goes to standard output"
The "easiest" way (Bash 4 only):
ls * 2>&- 1>&-
In situations when you consider using things like exec 2>&1, I find it easier to read, if possible, rewriting code using Bash functions like this:
function myfunc(){
[...]
}
myfunc &>mylog.log
The following functions can be used to automate the process of toggling outputs beetwen stdout/stderr and a logfile.
#!/bin/bash
#set -x
# global vars
OUTPUTS_REDIRECTED="false"
LOGFILE=/dev/stdout
# "private" function used by redirect_outputs_to_logfile()
function save_standard_outputs {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot save standard outputs because they have been redirected before"
exit 1;
fi
exec 3>&1
exec 4>&2
trap restore_standard_outputs EXIT
}
# Params: $1 => logfile to write to
function redirect_outputs_to_logfile {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot redirect standard outputs because they have been redirected before"
exit 1;
fi
LOGFILE=$1
if [ -z "$LOGFILE" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: logfile empty [$LOGFILE]"
fi
if [ ! -f $LOGFILE ]; then
touch $LOGFILE
fi
if [ ! -f $LOGFILE ]; then
echo "[ERROR]: ${FUNCNAME[0]}: creating logfile [$LOGFILE]"
exit 1
fi
save_standard_outputs
exec 1>>${LOGFILE%.log}.log
exec 2>&1
OUTPUTS_REDIRECTED="true"
}
# "private" function used by save_standard_outputs()
function restore_standard_outputs {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot restore standard outputs because they have NOT been redirected"
exit 1;
fi
exec 1>&- #closes FD 1 (logfile)
exec 2>&- #closes FD 2 (logfile)
exec 2>&4 #restore stderr
exec 1>&3 #restore stdout
OUTPUTS_REDIRECTED="false"
}
Example of usage inside script:
echo "this goes to stdout"
redirect_outputs_to_logfile /tmp/one.log
echo "this goes to logfile"
restore_standard_outputs
echo "this goes to stdout"
For tcsh, I have to use the following command:
command >& file
If using command &> file, it will give an "Invalid null command" error.