I would like to route a file descriptor to multiple places at the same time. For instance I would like every command in my script to print stdout to /dev/ps/9 and ./myscript.stdout at the same time.
I'm looking to achieve similar results as piping every command in a script (or a section of a script) into tee, perhaps with file descriptors. I also want the ability to restore default output behavior later in the script.
This code doesn't work, but it's an attempt at expressing my intent. To restore stdout as FD 1 later, I copy it into FD 4.
exec 3>(tee /dev/ps/9 ./myscript.stdout)
exec 4>&1
exec 1>&3
Restore normal output behavior, deleting FDs 3 and 4.
exec 1>&4
exec 4>&-
exec 3>&-
I would like every command in my script to print stdout to /dev/ps/9
and ./myscript.stdout at the same time.
exec 1> >(tee ./myscript.stdout >/dev/ps/9)
The above combines redirection and process substitution. With redirection alone, one can send stdout to a file. For example:
exec 1> filename
However, with bash, filenames can often be replaced with commands. This is called process substitution and it looks like >(some command) or <(some command) depending on whether one wants to write-to or read-from the process. In our case, we want to write to a tee command. Thus:
exec 1> >(some command)
Or, more specifically:
exec 1> >(tee ./myscript.stdout >/dev/ps/9)
Note that we have to maintain the space between the redirect (1>) and the process substitution (>(tee ./myscript.stdout >/dev/ps/9). Without the space, it would look like we were trying to append to a file whose name starts with a parens and this would generate a bash error.
For more information on this see the sections entitled "REDIRECTION" and "Process Substitution" in man bash.
#!/bin/bash
random=$$ # generate a random seed number to name the log files with
out=out.$random
err=err.$random
dev=`echo $(who -m) | cut -d' ' -f2` # for finding the right pseudo-terminal
: >$out # create the log files or empty their contents
: >$err # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
exec 1> >(tee ./$out >/dev/$dev) # I don't know how this works but it does
exec 2> >(tee ./$err >/dev/$dev) # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
echo # writing directly to the pts in /dev doesn't look right until sending a blank line
##########################################
echo 'hello'
for i in `seq 0 1 10`; do
echo $i
done
bad_command
THANKS #John1024
Here's a script for anybody else wishing to test this out.
Could somebody please explain in detail the exec lines to me.
For instance, why is there a blank space after the arrow in:
exec 1>
?
#!/bin/bash
logfile=$$.log
exec > $logfile 2>&1 | tee
echo "test"
The $$ does a random seed number which is optional.
Related
Is there a filename that is assignable to a variable (i.e. not a magic builtin shell token like &1) that will let me redirect to stdout?
What I finally want to do is run something like this in a cron script:
LOG=/tmp/some_file
...
some_command 2>&1 >> $LOG
echo "blah" >> $LOG
...
Conveniently, this lets me turn off log noise by redirecting to /dev/null later when I'm sure there is nothing that can fail (or, at least, nothing that I care about!) without rewriting the whole script. Yes, turning off logging isn't precisely best practice -- but once this script works, there is not much that can conceivably go wrong, and trashing the disk with megabytes of log info that nobody wants to read isn't desired.
In case something unexpectedly fails 5 years later, it is still possible to turn on logging again by flipping a switch.
On the other hand, while writing and debugging the script, which involves calling it manually from the shell, it would be extremely nice if it could just dump the output to the console. That way I wouldn't need to tail the logfile manually.
In other words, what I'm looking for is something like /proc/self/fd/0 in bash-talk that I can assign to LOG. As it happens, /proc/self/fd/0 works just fine on my Linux box, but I wonder if there isn't such a thing built into bash already (which would generally be preferrable).
Basic solution:
#!/bin/bash
LOG=/dev/null
# uncomment next line for debugging (logging)
# LOG=/tmp/some_file
{
some_command
echo "blah"
} | tee 1>$LOG 2>&1
More evolved:
#!/bin/bash
ENABLE_LOG=0 # 1 to log standard & error outputs
LOG=/tmp/some_file
{
some_command
echo "blah"
} | if (( $ENABLE_LOG ))
then
tee 1>$LOG 2>&1
fi
More elegant solution from DevSolar's idea:
#!/bin/bash
# uncomment next line for debugging (logging)
# exec 1> >(tee /tmp/some_file) 2>&1
some_command
echo "blah"
Thanks to the awesome hints by olibre and suvayu, I came up with this (for the record, version that I'm using now):
# log to file
# exec 1>> /tmp/logfile 2>&1
# be quiet
# exec 1> /dev/null 2>&1
# dump to console
exec 2>&1
Just need to uncomment one of the three, depending on what is desired, and don't worry about anything else, ever again. This either logs all subsequent output to a file or to console, or not at all.
No output duplicated, works universally the same for every command (without explicit redirects), no weird stuff, and as easy as it gets.
If I have understood your requirement clearly, the following should do what you want
exec 2>&1
exec >> $LOG
Stdout and stderr of all subsequent commands will be appended to the file $LOG.
Use /dev/stdout
Here's another SO answer that mentions this solution: Difference between stdout and /dev/stdout
In bash, calling foo would display any output from that command on the stdout.
Calling foo > output would redirect any output from that command to the file specified (in this case 'output').
Is there a way to redirect output to a file and have it display on stdout?
The command you want is named tee:
foo | tee output.file
For example, if you only care about stdout:
ls -a | tee output.file
If you want to include stderr, do:
program [arguments...] 2>&1 | tee outfile
2>&1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.
Furthermore, if you want to append to the log file, use tee -a as:
program [arguments...] 2>&1 | tee -a outfile
$ program [arguments...] 2>&1 | tee outfile
2>&1 dumps the stderr and stdout streams.
tee outfile takes the stream it gets and writes it to the screen and to the file "outfile".
This is probably what most people are looking for. The likely situation is some program or script is working hard for a long time and producing a lot of output. The user wants to check it periodically for progress, but also wants the output written to a file.
The problem (especially when mixing stdout and stderr streams) is that there is reliance on the streams being flushed by the program. If, for example, all the writes to stdout are not flushed, but all the writes to stderr are flushed, then they'll end up out of chronological order in the output file and on the screen.
It's also bad if the program only outputs 1 or 2 lines every few minutes to report progress. In such a case, if the output was not flushed by the program, the user wouldn't even see any output on the screen for hours, because none of it would get pushed through the pipe for hours.
Update: The program unbuffer, part of the expect package, will solve the buffering problem. This will cause stdout and stderr to write to the screen and file immediately and keep them in sync when being combined and redirected to tee. E.g.:
$ unbuffer program [arguments...] 2>&1 | tee outfile
Another way that works for me is,
<command> |& tee <outputFile>
as shown in gnu bash manual
Example:
ls |& tee files.txt
If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.
For more information, refer redirection
You can primarily use Zoredache solution, but If you don't want to overwrite the output file you should write tee with -a option as follow :
ls -lR / | tee -a output.file
Something to add ...
The package unbuffer has support issues with some packages under fedora and redhat unix releases.
Setting aside the troubles
Following worked for me
bash myscript.sh 2>&1 | tee output.log
Thank you ScDF & matthew your inputs saved me lot of time..
Using tail -f output should work.
In my case I had the Java process with output logs. The simplest solution to display output logs and redirect them into the file(named logfile here) was:
my_java_process_run_script.sh |& tee logfile
Result was Java process running with output logs displaying and
putting them into the file with name logfile
You can do that for your entire script by using something like that at the beginning of your script :
#!/usr/bin/env bash
test x$1 = x$'\x00' && shift || { set -o pipefail ; ( exec 2>&1 ; $0 $'\x00' "$#" ) | tee mylogfile ; exit $? ; }
# do whaetever you want
This redirect both stderr and stdout outputs to the file called mylogfile and let everything goes to stdout at the same time.
It is used some stupid tricks :
use exec without command to setup redirections,
use tee to duplicates outputs,
restart the script with the wanted redirections,
use a special first parameter (a simple NUL character specified by the $'string' special bash notation) to specify that the script is restarted (no equivalent parameter may be used by your original work),
try to preserve the original exit status when restarting the script using the pipefail option.
Ugly but useful for me in certain situations.
Bonus answer since this use-case brought me here:
In the case where you need to do this as some other user
echo "some output" | sudo -u some_user tee /some/path/some_file
Note that the echo will happen as you and the file write will happen as "some_user" what will NOT work is if you were to run the echo as "some_user" and redirect the output with >> "some_file" because the file redirect will happen as you.
Hint: tee also supports append with the -a flag, if you need to replace a line in a file as another user you could execute sed as the desired user.
< command > |& tee filename # this will create a file "filename" with command status as a content, If a file already exists it will remove existed content and writes the command status.
< command > | tee >> filename # this will append status to the file but it doesn't print the command status on standard_output (screen).
I want to print something by using "echo" on screen and append that echoed data to a file
echo "hi there, Have to print this on screen and append to a file"
tee is perfect for this, but this will also do the job
ls -lr / > output | cat output
I'm developing a BASH script which invokes another BASH script which prints a line to stdout. That output is captured by the first BASH script and used later. It works, but it has the downside that any other output which is printed by the second script will cause this part to behave unexpectedly, because there will be extra content.
main.sh
#!/bin/bash
# Invoke worker.sh and capture its standard output to stats
stats=$(worker.sh --generate-stats)
echo "stats=$stats"
worker.sh
#!/bin/bash
[[ $1 == "--generate-stats" ]] && echo "cpu=90 mem=50 disk=15"
In this over-simplified example, it's not a problem to use this construct, but as worker.sh grows in size and complexity, it's hard to remember that no other command can print to stdout without confounding the behavior, and if someone else works on worker.sh without realizing they can't print to stdout, it can easily get fouled. So what is considered good practice to generate output in one script and use it in the other?
I'm wondering if a fifo would be appropriate, or another file descriptor, or just a plain file. Or if exec should be used in this case, something like what is shown here https://www.tldp.org/LDP/abs/html/x17974.html:
#!/bin/bash
exec 6>&1 # Link file descriptor #6 with stdout.
# Saves stdout.
exec >&2 # stdout now goes to stderr
echo "Didn't know I shouldn't print to stdout"
exec 1>&6 6>&- # Restore stdout and close file descriptor #6.
[[ $1 == "--generate-stats" ]] && echo "cpu=90 mem=50 disk=15"
But I wouldn't want to use that if it's not considered good practice.
Many command-line utilities have quiet and verbose modes; it's generally considered good practice to have the most verbose output (debugging, tracing, etc.) be separated to standard error anyway, but it's common to have normal output be formatted for human legibility (e.g. include table headings and column separators) and quiet mode output be just the bare data for programmatic use. (For one example, see docker images vs docker images -q). So that would be my recommendation - have worker.sh take a flag indicating whether its output is being consumed programmatically, and write it such that its output is all sent via a function that checks that flag and filters appropriately.
Maybe a different approach would be for the second script to test to see if it's stdout is being used programatically:
gash.sh:
#!/bin/bash
data=$(./another.sh)
echo "Received $data"
another.sh:
#!/bin/bash
# for -t see man isatty(3). 1 is file descriptor 1 - stdout
if [ -t 1 ]; then
echo "stdout is a terminal"
else
echo "stdout is not a terminal"
fi
Gives (where $ is a generic keyboard prompt):
$ bash gash.sh
Received stdout is not a terminal
$ bash another.sh
stdout is a terminal
You could then set a flag to change script behaviour (ls(1) does a similar thing). However, you should be prepared for this:
$ bash another.sh|more
stdout is not a terminal
$ bash another.sh > out.txt
$ cat out.txt
stdout is not a terminal
My intent was to have all the output of my bash script displayed on the console and logged to a file.
Here is my script that works as expected.
#!/bin/bash
LOG_FILE="test_log.log"
touch $LOG_FILE
# output to console and to logfile
exec > >(tee $LOG_FILE) 2>&1
echo "Starting command ls"
ls -al
echo "End of script"
However I do not understand why it works that way.
I expected to have exec >>(tee $LOG_FILE) 2>&1 work but it fails although exec >>$LOG_FILE 2>&1 indeed works.
I could not find the reason for the construction exec > >(command ) in the bash manual nor in advanced bash scripting. Can you explain the logic behind it ?
The >(tee $LOG_FILE) is an example of Process substitution, you might wish to search for that. Advanced Shell Scriptng and Bash manual
Using the syntax, <(program) for capturing output and >(program) for feeding input, we can pass data just one record at a time. It is more powerful than command substitution (backticks, or $( )) because it substitutes for a filename, not text. Therefore anywhere a file is normally specified we can substitute a program's standard output or input (although process substitution on input is not all that common).
This is particularly useful where a program does not use standard streams for what you want.
Note that in your example you are missing a space, exec >>(tee $LOG_FILE) 2>&1 is wrong (you will get a syntax error). Rather,
exec > >(tee $LOG_FILE) 2>&1
is correct, that space is critical.
So, the exec > part changes file descriptor 1 (the default), also known as stdout or standard output, to refer to "whatever comes next", in this case it is the process substitution, although normally it would be a filename.
2>&1 redirects file descriptor 2 (stderr or standard error) to refer to the same place as file descriptor 1 (stdout or standard out). Important: if you omit the & you end-up with a file called 1 rather than successful redirection.
Once you have called the exec line above, then you have changed the current process's standard output, so output from the commands which follow go to that tee process instead of to regular stdout.
What we have is a set of commands including script files to be executed, a command string which can be executed.
We want to execute these commands and store the stderr output to a variable (lets say "err") and combined output of stderr and stdout to another variable ("combined").
e.g.
#!/bin/bash
cmds="<cmd1>; <cmd2>; <cmd3>;"
<cmd4>;
<cmd5>;
<cmd6>;
<cmd7>;
eval $cmds;
./myscript.sh
err=<some magic>
combined=<some magic>
So, variable $err should contain all the errors and $combined should contain combined output of the commands in exactly that order in which the commands were executed.
You can write your output and errors to separate files and at end read them and put in variable.
For collecting output use foo >>outputs.file and for error use foo 2>>errors.file so the command will be like this foo >>outputs.file 2>>errors.file
foo is your command and >> means append to file. if you use single > it will truncate the file before writing in it.
and for putting file contains in a variable use myvar=$(cat outputs.file)
. no space before and after equal sign
#!/bin/bash
cmds="<cmd1>; <cmd2>; <cmd3>;"
$(
exec 3>"combined.log" 2> >(tee "err.log" >&3) 1> >(tee "out.log" >&3)
<cmd4>;
<cmd5>;
<cmd6>;
<cmd7>;
eval $cmds;
./myscript.sh
)
out=$(cat "out.log")
err=$(cat "err.log")
combined=$(cat "combined.log")
echo -e "OUT :: \n\n$out \n\n"
echo -e "ERR :: \n\n$err \n\n"
echo -e "COMB :: \n\n$combined \n\n"
The line 3>"combined.log" set the stream 3 to a file combined.log.
The commands are executed in a subshell, in which we are "tee"ing stderr and stdout to separate files and finally redirecting all these outputs to fd 3.