I want to write a shell script that runs a command, writing its stderr to my terminal as it arrives. However, I also want to save stderr to a variable, so I can inspect it later.
How can I achieve this? Should I use tee, or a subshell, or something else?
I've tried this:
# Create FD 3 that can be used so stdout still comes through
exec 3>&1
# Run the command, piping stdout to normal stdout, but saving stderr.
{ ERROR=$( $# 2>&1 1>&3) ; }
echo "copy of stderr: $ERROR"
However, this doesn't write stderr to the console, it only saves it.
I've also tried:
{ $#; } 2> >(tee stderr.txt >&2 )
echo "stderr was:"
cat stderr.txt
However, I don't want the temporary file.
I often want to do this, and find myself reaching for /dev/stderr, but there can be problems with this approach; for example, Nix build scripts give "permission denied" errors if they try to write to /dev/stdout or /dev/stderr.
After reinventing this wheel a few times, my current approach is to use process substitution as follows:
myCmd 2> >(tee >(cat 1>&2))
Reading this from the outside in:
This will run myCmd, leaving its stdout as-is. The 2> will redirect the stderr of myCmd to a different destination; the destination here is >(tee >(cat 1>&2)) which will cause it to be piped into the command tee >(cat 1>&2).
The tee command duplicates its input (in this case, the stderr of myCmd) to its stdout and to the given destination. The destination here is >(cat 1>&2), which will cause the data to be piped into the command cat 1>&2.
The cat command just passes its input straight to stdout. The 1>&2 redirects stdout to go to stderr.
Reading from the inside out:
The cat 1>&2 command redirects its stdin to stderr, so >(cat 1>&2) acts like /dev/stderr.
Hence tee >(cat 1>&2) duplicates its stdin to both stdout and stderr, acting like tee /dev/stderr.
We use 2> >(tee >(cat 1>&2)) to get 2 copies of stderr: one on stdout and one on stderr.
We can use the copy on stdout as normal, for example storing it in a variable. We can leave the copy on stderr to get printed to the terminal.
We can combine this with other redirections if we like, e.g.
# Create FD 3 that can be used so stdout still comes through
exec 3>&1
# Run the command, redirecting its stdout to the shell's stdout,
# duplicating its stderr and sending one copy to the shell's stderr
# and using the other to replace the command's stdout, which we then
# capture
{ ERROR=$( $# 2> >(tee >(cat 1>&2)) 1>&3) ; }
echo "copy of stderr: $ERROR"
Credit goes to #Etan Reisner for the fundamentals of the approach; however, it's better to use tee with /dev/stderr rather than /dev/tty in order to preserve normal behavior (if you send to /dev/tty, the outside world doesn't see it as stderr output, and can neither capture nor suppress it):
Here's the full idiom:
exec 3>&1 # Save original stdout in temp. fd #3.
# Redirect stderr to *captured* stdout, send stdout to *saved* stdout, also send
# captured stdout (and thus stderr) to original stderr.
errOutput=$("$#" 2>&1 1>&3 | tee /dev/stderr)
exec 3>&- # Close temp. fd.
echo "copy of stderr: $errOutput"
Related
This is a follow-up to this question:
How do I get both STDOUT and STDERR to go to the terminal and a log file?
In short: I want to run a command and store both, STDOUT and STDERR in one log file (to keep time correlation) but I need STDERR output still to be printed on STDERR.
Motivation: there are tools which run a command and treat STDOUT and STDERR differently - e.g. printing STDERR in a different color or print STDERR when the command returns non-zero.
So I'd like to have a way to store all output in one log file but preserve the distinction between STDOUT and STDERR (as well as the return code).
log-output --file=command.log -c "make stuff-with-stderr"
From what I found in the above links answers there are at least two different approaches:
the_cmd 1> >(tee stdout.txt ) 2> >(tee stderr.txt >&2 )
will store STDOUT and STDERR in separate files, thus loosing time correlation. And unfortunately both STDOUT and STDERR will be printed on STDOUT only.
script -e -B build.log -c "the_cmd"
will store both STDOUT and STDERR in one file, keeping time correlation but still prints both STDOUT and STDERR on on STDOUT only.
So none of those approaches meets my requirements. Is there something else?
Edit:
Using my approach and your command from the comment below (pip3 install -U pytest) in a bash shell you get your desired outcome:
(((pip3 install -U pytest | tee -a log.txt) 3>&1 1>&2 2>&3 | tee -a log.txt) 3>&1 1>&2 2>&3)
The entries in log.txt are always in the correct order (same as executing pip3 install -U pytest in a terminal) and the output is printed to the terminal correctly (separate streams for stdout and stderr). I suspect a better solution exists, but this appears to be a 'workable' solution to the problem.
Original answer:
There's probably a less awkward solution than this, but you could tee stdout to log.txt, then swap stdout with stderr and tee the 'new' stdout (stderr) to log.txt, then swap stderr back with stdout, then print both streams to terminal (similar to the example here):
((({ echo "test_out"; echo "test_err" 1>&2; } | tee -a log.txt) 3>&1 1>&2 2>&3 | tee -a log.txt) 3>&1 1>&2 2>&3)
test_out
test_err
cat log.txt
test_out
test_err
# show stdout/stderr are still separate streams:
((({ echo "test_out"; echo "test_err" 1>&2; } | tee -a log.txt) 3>&1 1>&2 2>&3 | tee -a log.txt) 3>&1 1>&2 2>&3) 1>std.out 2>std.err
cat std.out
test_out
cat std.err
test_err
I know that it is possible to redirect both to a specific file:
./command 1> out.log 2> err.log
or
./command 1>test.log 2>&1
to write both to a file. But I don't know a way to write both to the same file (preserving the output order) while printing just one of both. tee isn't very helpful because it prints both file descriptors.
Writing standard output to one file and both to another is fairly simple with tee:
{ cmd | tee stdout.log; } &> both.log
Both descriptors of the compound command are redirected to both.log, but the standard output of cmd is first passed through tee to stdout.log before also being written to both.log.
Writing standard error to one and both to another is trickier.
{ foo 2>&1 1>&3 | tee stderr.log ; } 3>&1 | tee both.log > /dev/null
It's a little tricky to describe correctly. First, the command group's standard error is ignored; it's standard output is the pipe to tee both.log. But 3>&1 also copies its fd 3 to its standard output. So the question is, what gets written to that?
Inside the command group, foo's standard output is the pipe to tee stderr.log. The 2>&1 copies tee's standard error to that descriptor, and 1>&3 copies foo's standard output to the inherited fd 3.
It will be better to write them to different files, in this case it will be very easy:
cmd 2>/stderr.log| tee -a stdout.log
But if you want a single file, you will need some tricks here and additional process running for redirection.
You can use several redirections:
foo () { echo 1 ; echo 2 >&2 ; }
(( foo | tee >(cat) >&3) &>log ) 3>&1
The tee command sends the stdout to file descriptors 1 (through the process substitution) and 3. Both stdout and stderr are redirected to the log. In the end, 3, the copy of stdout, is sent back to the terminal.
Alternatively, you can do without the process substitution, redirect the output directly to log, but use -a and >> for append. You need to clear the log beforehand.
: > log; (( foo | tee -a log >&3) 2>> log ) 3>&1
stdout to the console and stdout and error to a file?
{ cmd | tee /dev/tty; } &> /tmp/both.log
To answer the question :
stderr to the console and stdout and error to a file?
{ cmd 3>&1 1>&2 2>&3 | tee /dev/tty; } &> /tmp/both.log
3>&1 1>&2 2>&3 means swapping stdout and stderr
I would like to save result of my script run with sh.exe on windows into three different files:
stdout to stdout.txt; stderr to stderr.txt and append stdout and stderr to all.txt. I tried to use
foo.sh &> all.txt 2> stderr.txt
or
foo.sh 2>&1 1>logfile | tee -a logfile
but it doen't even append stderr and stdout.
How can I do it?
Redirecting stdout+stderr such that both get written to a file while still outputting to stdout is simple enough:
cmd 2>&1 | tee output_file
But then now both stdout/stderr from cmd are coming on stdout. I'd like to write stdout+stderr to the same file (so ordering is preserved assuming cmd is single threaded) but then still be able to also separately redirect them, something like this:
some_magic_tee_variant combined_output cmd > >(command-expecting-stdout) 2> >(command-expecting-stderr)
So combined_output contains the both with order preserved, but the command-expecting-stdout only gets stdout and command-expecting-stderr only gets stderr. Basically, I want to log stdout+stderr while still allowing stdout and stderr to be separately redirected and piped. The problem with the tee approach is it globs them together. Is there a way to do this in bash/zsh?
From what I unterstand this is what you are looking for. First I made a litte script to write on stdout and stderr. It looks like this:
$ cat foo.sh
#!/bin/bash
echo foo 1>&2
echo bar
Then I ran it like this:
$ ./foo.sh 2> >(tee stderr | tee -a combined) 1> >(tee stdout | tee -a combined)
foo
bar
The results in my bash look like this:
$ cat stderr
foo
$ cat stdout
bar
$ cat combined
foo
bar
Note that the -a flag is required so the tees don't overwrite the other tee's content.
{ { cmd | tee out >&3; } 2>&1 | tee err >&2; } 3>&1
Or, to be pedantic:
{ { cmd 3>&- | tee out >&3 2> /dev/null; } 2>&1 | tee err >&2 3>&- 2> /dev/null; } 3>&1
Note that it's futile to try and preserve order. It is basically impossible. The only solution would be to modify "cmd" or use some LD_PRELOAD or gdb hack,
Order can indeed be preserved. Here's an example which captures the standard output and error, in the order in which they are generated, to a logfile, while displaying only the standard error on any terminal screen you like. Tweak to suit your needs.
1.Open two windows (shells)
2.Create some test files
touch /tmp/foo /tmp/foo1 /tmp/foo2
3.In window1:
mkfifo /tmp/fifo
</tmp/fifo cat - >/tmp/logfile
4.Then, in window2:
(ls -l /tmp/foo /tmp/nofile /tmp/foo1 /tmp/nofile /tmp/nofile; echo successful test; ls /tmp/nofile1111) 2>&1 1>/tmp/fifo | tee /tmp/fifo 1>/dev/pts/1
Where /dev/pts/1 can be whatever terminal display you want. The subshell runs some "ls" and "echo" commands in sequence, some succeed (providing stdout) and some fail (providing stderr) in order to generate a mingled stream of output and error messages, so that you can verify the correct ordering in the log file.
Here's how I do it:
exec 3>log ; example_command 2>&1 1>&3 | tee -a log ; exec 3>&-
Worked Example
bash$ exec 3>log ; { echo stdout ; echo stderr >&2 ; } 2>&1 1>&3 | \
tee -a log ; exec 3>&-
stderr
bash$ cat log
stdout
stderr
Here's how that works:
exec 3>log sets up file descriptor 3 to redirect into the file called log, until further notice.
example_command to make this a working example, I used { echo stdout ; echo stderr >&2 ; }. Or you could use ls /tmp doesnotexist to provide output instead.
Need to jump ahead to the pipe | at this point because bash does it first. The pipe sets up a pipe and redirects the file descriptor 1 into this pipe. So now, STDOUT is going into the pipe.
Now we can go back to where we were next in our left-to-right interpretation: 2>&1 this says errors from the program are to go to where STDOUT currently points, i.e. into the pipe we just set up.
1>&3 means STDOUT is redirected into file descriptor 3, which we earlier set up to output to the log file. So STDOUT from the command just goes into the log file, not to the terminal's STDOUT.
tee -a log takes it's input from the pipe (which you'll remember is now the errors from the command), and outputs it to STDOUT and also appends it to the log file.
exec 3>&- closes the file descriptor 3.
Victor Sergienko's comment is what worked for me, adding exec to the front of it makes this work for the entire script (instead of having to put it after individual commands)
exec 2> >(tee -a output_file >&2) 1> >(tee -a output_file)
This is a task that I try to do pretty often.
I want to log both stderr and stdout to a log file. But I only want to print to console stderr.
I've tried with tee, but once I've merge stderr and stdout using "2>&1". I can not print stdout to the screen anymore since both my pipes are merged.
Here is a simple example of what I tried
./dosomething.sh | tee -a log 2>&1.
Now I have both stderr and stdout to the log and the screen.
Any Ideas?
Based on some reading on this web site, this question has been asked.
Write STDOUT & STDERR to a logfile, also write STDERR to screen
And also a question very similar here:
Save stdout, stderr and stdout+stderr synchronously
But neither of them are able to redirect both stdout+stderr to a log and stderr to the screen while stdoud and stderr are synchronously written to the log file.
I was able to get this working in bash:
(./tmp.sh 2> >(tee >(cat >&2) >&1)) > tmp.log
This does not work correctly in zsh (the prompt does not wait for the process to exit), and does not work at all in dash. A more portable solution may be to write a simple C program to do it.
I managed to get this working with this script in bash.
mkfifo stdout
mkfifo stderr
rm -f out
cat stderr | tee -a out &
cat stdout >> out &
(echo "stdout";
grep;
echo "an other stdout";
echo "again stdout";
stat) 2> stderr > stdout
rm -f stdout
rm -f stderr
The order of the output is preserved. With this script the process ends correctly.
Note: I used grep and stat without parameter to generate stdout.