update: I think the below code may be headed in the wrong direction -- but the question remains, can I open a pipe to log all output (file&console), pause that log and log to a new log (new file&console), and then re-attach to the FD for the original logger just by moving FDs around and not re-opening the original log file
Trying to improve my knowledge of FDs in bash. I'm trying to log all output of the main "meta" test.sh -- but log to a different file when I get to "sections" -- e.g. functions, sourced scripts, etc. And then go back to appending to the "meta" log.
I know I could pretty easily accomplish this with subshells -- or by opening the 'meta' log again and append from there, but can anyone help accomplish this by switching FDs around?
#!/bin/bash
rm *.log
NAMED_PIPE="$(mktemp -u /tmp/pipe.XXXX)"
mknod $NAMED_PIPE p
tee <$NAMED_PIPE "./meta.log" &
section () {
echo SECTION: stdout
echo SECTION: stderr >&2
}
# link stdout->3 & stderr->4 and save stdout & stderr
exec 3>&1 4>&2 &> "$NAMED_PIPE"
echo METAstr: stdout
echo METAstr: stderr >&2
# restore stdout & stderr
exec 1>&3 2>&4
# sleep 1 # I think an additional delay prevents the possible race condition I'm seeing
# exec 1>&3- 2>&4- ... I think this would restore but close 3 & 4?
# do I need another named pipe here?
section 2>&1 # | tee section.log
# re-link to same pipe
exec 3>&1 4>&2 &> "$NAMED_PIPE"
echo METAend: stdout
echo METAend: stderr >&2
Without trying to log 'section' all the meta output gets printed after the return of the script:
-bash-4.2# ./test.sh
SECTION: stdout
SECTION: stderr
-bash-4.2# METAstr: stdout
METAstr: stderr
METAend: stdout
METAend: stderr
And trying to log 'section' I think fouls up my FDs so the following exec hangs me up:
-bash-4.2# ./test.sh
METAstr: stdout
METAstr: stderr
SECTION: stdout
SECTION: stderr
EDIT1:
Contents of meta.log after running the script without trying to tee section:
[root#master tmp]# cat meta.log
METAstr: stdout
METAstr: stderr
METAend: stdout
METAend: stderr
It logs the ending messages, the tee does not exit until the script does
EDIT2:
Revision of EDIT1. I think It's a race condition. I think the FDs are being closed -- but they're not closed by the time the final echo commands happen.
I was just going to write the same thing. It's a race condition. The second exec closes the writing end in your process, signalling an EOF to tee. tee will want to exit when it gets the EOF. If it does exit by the time you call the last exec, the last exec will block. If it hasn't exited yet at that point, it will not block, because a reading end of the FIFO will still be open.
Any delay will make it more likely tee will have exited by that time.
Spawning a process makes it very likely. I found with stracing (which slows the program down a little) it's about 50/50.
Related
I am trying to save some logs from bash functions which execute tools (some of them run in subshells). In addition I would like to print all errors to the terminal.
My code leads to a sigpipe and exit code 141 upon hitting ctr-c plus a strange log file. The pipe fail seems to be caused by the redirection of stdout to stderr within the trap, which breaks the stdout stream of the tee command. Interestingly the code terminates as expected with exit code 130 without the redirection used in the trap or the cat command.
I am still unable to fix and explain the resulting log file. Why are there some echos twice and why are the trap echos written to the file as well?
Why isn't the sigpipe caused earlier by the redirection within the function?
trap '
echo trap_stdout
echo trap_stderr >&2
' INT
fun(){
echo fun_stdout
echo fun_stderr >&2
( sleep 10 | cat )
}
echo > log
fun >> log 2> >(tee -a log)
log file
fun_stdout
fun_stderr
fun_stderr
trap_stdout
EDIT: working example according to oguz ismail answer
exec 3>> log
exec 4> >(tee -ai log >&2)
fun 2>&4 >&3
exec 3>&-
exec 4>&-
Why are there some echos twice
fun's stdout is redirected to log before its stderr is redirected to the FIFO created for tee, thus tee inherits a stdout that is redirected to log. I can prove that like so:
$ : > file 2> >(date)
$ cat file
Sat Jul 25 18:46:31 +03 2020
Changing the order of redirections will fix that. E.g.:
fun 2> >(tee -a log) >> log
and why are the trap echos written to the file as well?
If the trap set for SIGINT is triggered while the shell is still executing fun, its perfectly normal that the redirections associated with fun takes effect.
To connect the trap action's stdout and stderr to those of the main shell, you can do:
exec 3>&1 4>&2
handler() {
: # handle SIGINT here
} 1>&3 2>&4
trap handler INT
Or something alike; the idea is making copies of the main shell's stdout and stderr.
Why isn't the sigpipe caused earlier by the redirection within the function?
Because tee is alive while echo fun_stderr >&2 is being executed. And sleep does not write anything to its stdout, so it can not trigger a SIGPIPE.
The reason why this script terminates due to a SIGPIPE is that tee receives the SIGINT generated by the keyboard as well and terminates before the trap action associated with SIGINT is executed. As a result, while executing echo trap_stderr >&2, since its stderr is connected to a pipe that has been closed moments ago, the shell receives the SIGPIPE.
To avoid this, as already suggested, you can make tee ignore SIGINT. You don't need to set an empty trap for that though, the -i option is enough.
fun 2> >(tee -a -i log) >> log
The source of the SIGPIPE is that the SIGINT (initiated by ctrl/c) is sent to ALL processes running: both the "main" bash process (executing the 'fun' function), and the sub shell executing the 'tee -a'. As a result, on Ctrl/C, both get killed. When the main process tries to send 'trap_stderr' to te "tee" process, it get SIGPIPE, because the "tee" has already died.
Given the role of the 'tee -a', it make sense to protect it from the SIGINT, and allow it to run until 'fun' complete (or killed). Consider the following change to the last line
fun >> log 2> >(trap '' INT ; tee -a log >&2)
which will produce the log file:
Console (stderr)
fun_stderr
^Ctrap_stderr
Log File: (no duplicates)
fun_stdout
fun_stderr
trap_stdout
trap_stderr
The above will also address the second question is about duplicate lines in the log file. This is the result of using tee to send each stderr line to log file AND stdout. Given that stdout just got redirect (by the '>>log') to the 'log' file, both copy of the output are sent to the log file, and none to the terminal
Given the redirection are performed sequentially, changing the 'tee' line to sent the output to the original stderr (instead of the already redirect stdout) will show the output on the terminal (or whatever stderr is)
I'm somewhat familiar with the common way of redirecting stdout to a file, and then redirecting stderr to stdout.
If I run a command such as ls > output.txt 2>&1, my guess is that under the hood, the shell is executing something like the following c code:
close(1)
open("output.txt") // assigned to fd 1
close(2)
dup2(1, 2)
Since fd 1 has already been replaced with output.txt, anything printed to stderr will be redirected to output.txt.
But, if I run ls 2>&1 > output.txt, I'm guessing that this is instead what happens:
close(2)
dup2(1, 2)
close(1)
open("output.txt")
But, since the shell prints out both stdout and stderr by default, is there any difference between ls 2>&1 output.txt and ls > output.txt? In both cases, stdout will be redirected to output.txt, while stderr will be printed to the console.
With ls >output.txt, the stderr from ls goes to the stderr inherited from the calling process. In contrast, with ls 2>&1 >output.txt, the stderr of ls is sent to the stdout of the calling process.
Let's try this with an example script that prints a line of output to each of stdout and stderr:
$ cat pr.sh
#!/bin/sh
echo "to stdout"
echo "to stderr" 1>&2
$ sh pr.sh >/dev/null
to stderr
$ sh pr.sh 2>/dev/null
to stdout
Now if we insert "2>&1" into the first command line, nothing appears different:
$ sh pr.sh 2>&1 >/dev/null
to stderr
But now let's run both of those inside a context where the inherited stdout is going someplace other than the console:
$ (sh pr.sh 2>&1 >/dev/null) >/dev/null
$ (sh pr.sh >/dev/null) >/dev/null
to stderr
The second command still prints because the inherited stderr is still going to the console. But the first prints nothing because the "2>&1" redirects the inner stderr to the outer stdout, which is going to /dev/null.
Although I've never used this construction, conceivably it could be useful in a situation where (in a script, most likely) you want to run a program, send its stdout to a file, but forward its stderr on to the caller as if it were "normal" output, perhaps because that program is being run along with some other programs and you want the first program's "error" output to be part of the same stream as the other programs' "normal" output. (Perhaps both programs are compilers, and you want to capture all the error messages, but they disagree about which stream errors are sent to.)
I'm facing to a problem that I'm not able to explain. I've to recode a shell, and there is a weird behavior for me.
echo test >&2 2>&1
This kind of command is writing on stderr (so with the first redirection), why the second redirection isn't impacting the output destination ? Why the output isn't on stdout ?
I saw some stuff that redirection is happening before executing command, from left to right, so why the second redirection is not cancelling the first one ?
Thanks in advance.
edit: I'm running my script on bash.
x>&y means "redirect file descriptor x to whatever fd y is currently pointing to.
Redirections are processed strictly left to right.
So, >&2 2>&1 points fd 1 to something like /dev/stderr, and then points fd 2 to /dev/stderr also.
If you want to swap stderr and stdout, you need a 3rd file descriptor:
(echo "test to stdout"; echo "test to stderr" >&2) 3>&1 1>&2 2>&3 3>&-
# ................................................ A .. B .. C .. D
# A. fd 3 = /dev/stdout
# B. fd 1 = /dev/stderr
# C. fd 2 = /dev/stdout
# D. fd 3 is closed
Let's put that in a function for easier testing
fdtest() { (echo "test to stdout"; echo "test to stderr" >&2) 3>&1 1>&2 2>&3 3>&-; }
Run it
$ fdtest
test to stdout
test to stderr
Throw away standard error (we expect to see the "stderr" message on standard out).
$ fdtest 2>/dev/null
test to stderr
Throw away standard out (we expect to see the "stdout" message on standard err).
$ fdtest 1>/dev/null
test to stdout
cmd >&2 2>&1 is very different than cmd 2>&1 >&2, since the redirections occur in a different order. Suppose the command is invoked from a process which has fd 1 attached to a file named 'output' and fd 2 attached to a filed named 'error'. Then cmd >&2 2>&1 will redirect the stdout of cmd to the file named error and then redirect stderr of cmd to whatever fd 1 is connected to, namely the file named error. But cmd 2>&1 >&2 will first redirect fd 2 to 'output' and the redirect fd 1 to the same place. In other words, cmd >&2 2>&1 writes everything to stderr, and cmd 2>&1 >&2 writes everything on stdout.
I want to write a shell script that runs a command, writing its stderr to my terminal as it arrives. However, I also want to save stderr to a variable, so I can inspect it later.
How can I achieve this? Should I use tee, or a subshell, or something else?
I've tried this:
# Create FD 3 that can be used so stdout still comes through
exec 3>&1
# Run the command, piping stdout to normal stdout, but saving stderr.
{ ERROR=$( $# 2>&1 1>&3) ; }
echo "copy of stderr: $ERROR"
However, this doesn't write stderr to the console, it only saves it.
I've also tried:
{ $#; } 2> >(tee stderr.txt >&2 )
echo "stderr was:"
cat stderr.txt
However, I don't want the temporary file.
I often want to do this, and find myself reaching for /dev/stderr, but there can be problems with this approach; for example, Nix build scripts give "permission denied" errors if they try to write to /dev/stdout or /dev/stderr.
After reinventing this wheel a few times, my current approach is to use process substitution as follows:
myCmd 2> >(tee >(cat 1>&2))
Reading this from the outside in:
This will run myCmd, leaving its stdout as-is. The 2> will redirect the stderr of myCmd to a different destination; the destination here is >(tee >(cat 1>&2)) which will cause it to be piped into the command tee >(cat 1>&2).
The tee command duplicates its input (in this case, the stderr of myCmd) to its stdout and to the given destination. The destination here is >(cat 1>&2), which will cause the data to be piped into the command cat 1>&2.
The cat command just passes its input straight to stdout. The 1>&2 redirects stdout to go to stderr.
Reading from the inside out:
The cat 1>&2 command redirects its stdin to stderr, so >(cat 1>&2) acts like /dev/stderr.
Hence tee >(cat 1>&2) duplicates its stdin to both stdout and stderr, acting like tee /dev/stderr.
We use 2> >(tee >(cat 1>&2)) to get 2 copies of stderr: one on stdout and one on stderr.
We can use the copy on stdout as normal, for example storing it in a variable. We can leave the copy on stderr to get printed to the terminal.
We can combine this with other redirections if we like, e.g.
# Create FD 3 that can be used so stdout still comes through
exec 3>&1
# Run the command, redirecting its stdout to the shell's stdout,
# duplicating its stderr and sending one copy to the shell's stderr
# and using the other to replace the command's stdout, which we then
# capture
{ ERROR=$( $# 2> >(tee >(cat 1>&2)) 1>&3) ; }
echo "copy of stderr: $ERROR"
Credit goes to #Etan Reisner for the fundamentals of the approach; however, it's better to use tee with /dev/stderr rather than /dev/tty in order to preserve normal behavior (if you send to /dev/tty, the outside world doesn't see it as stderr output, and can neither capture nor suppress it):
Here's the full idiom:
exec 3>&1 # Save original stdout in temp. fd #3.
# Redirect stderr to *captured* stdout, send stdout to *saved* stdout, also send
# captured stdout (and thus stderr) to original stderr.
errOutput=$("$#" 2>&1 1>&3 | tee /dev/stderr)
exec 3>&- # Close temp. fd.
echo "copy of stderr: $errOutput"
This is a task that I try to do pretty often.
I want to log both stderr and stdout to a log file. But I only want to print to console stderr.
I've tried with tee, but once I've merge stderr and stdout using "2>&1". I can not print stdout to the screen anymore since both my pipes are merged.
Here is a simple example of what I tried
./dosomething.sh | tee -a log 2>&1.
Now I have both stderr and stdout to the log and the screen.
Any Ideas?
Based on some reading on this web site, this question has been asked.
Write STDOUT & STDERR to a logfile, also write STDERR to screen
And also a question very similar here:
Save stdout, stderr and stdout+stderr synchronously
But neither of them are able to redirect both stdout+stderr to a log and stderr to the screen while stdoud and stderr are synchronously written to the log file.
I was able to get this working in bash:
(./tmp.sh 2> >(tee >(cat >&2) >&1)) > tmp.log
This does not work correctly in zsh (the prompt does not wait for the process to exit), and does not work at all in dash. A more portable solution may be to write a simple C program to do it.
I managed to get this working with this script in bash.
mkfifo stdout
mkfifo stderr
rm -f out
cat stderr | tee -a out &
cat stdout >> out &
(echo "stdout";
grep;
echo "an other stdout";
echo "again stdout";
stat) 2> stderr > stdout
rm -f stdout
rm -f stderr
The order of the output is preserved. With this script the process ends correctly.
Note: I used grep and stat without parameter to generate stdout.