This question already has answers here:
How to redirect and append both standard output and standard error to a file with Bash
(8 answers)
Closed 1 year ago.
I want to redirect both standard output and standard error of a process to a single file. How do I do that in Bash?
Take a look here. It should be:
yourcommand &> filename
It redirects both standard output and standard error to file filename.
do_something 2>&1 | tee -a some_file
This is going to redirect standard error to standard output and standard output to some_file and print it to standard output.
You can redirect stderr to stdout and the stdout into a file:
some_command >file.log 2>&1
See Chapter 20. I/O Redirection
This format is preferred over the most popular &> format that only works in Bash. In Bourne shell it could be interpreted as running the command in background. Also the format is more readable - 2 (is standard error) redirected to 1 (standard output).
# Close standard output file descriptor
exec 1<&-
# Close standard error file descriptor
exec 2<&-
# Open standard output as $LOG_FILE file for read and write.
exec 1<>$LOG_FILE
# Redirect standard error to standard output
exec 2>&1
echo "This line will appear in $LOG_FILE, not 'on screen'"
Now, a simple echo will write to $LOG_FILE, and it is useful for daemonizing.
To the author of the original post,
It depends what you need to achieve. If you just need to redirect in/out of a command you call from your script, the answers are already given. Mine is about redirecting within current script which affects all commands/built-ins (includes forks) after the mentioned code snippet.
Another cool solution is about redirecting to both standard error and standard output and to log to a log file at once which involves splitting "a stream" into two. This functionality is provided by 'tee' command which can write/append to several file descriptors (files, sockets, pipes, etc.) at once: tee FILE1 FILE2 ... >(cmd1) >(cmd2) ...
exec 3>&1 4>&2 1> >(tee >(logger -i -t 'my_script_tag') >&3) 2> >(tee >(logger -i -t 'my_script_tag') >&4)
trap 'cleanup' INT QUIT TERM EXIT
get_pids_of_ppid() {
local ppid="$1"
RETVAL=''
local pids=`ps x -o pid,ppid | awk "\\$2 == \\"$ppid\\" { print \\$1 }"`
RETVAL="$pids"
}
# Needed to kill processes running in background
cleanup() {
local current_pid element
local pids=( "$$" )
running_pids=("${pids[#]}")
while :; do
current_pid="${running_pids[0]}"
[ -z "$current_pid" ] && break
running_pids=("${running_pids[#]:1}")
get_pids_of_ppid $current_pid
local new_pids="$RETVAL"
[ -z "$new_pids" ] && continue
for element in $new_pids; do
running_pids+=("$element")
pids=("$element" "${pids[#]}")
done
done
kill ${pids[#]} 2>/dev/null
}
So, from the beginning. Let's assume we have a terminal connected to /dev/stdout (file descriptor #1) and /dev/stderr (file descriptor #2). In practice, it could be a pipe, socket or whatever.
Create file descriptors (FDs) #3 and #4 and point to the same "location" as #1 and #2 respectively. Changing file descriptor #1 doesn't affect file descriptor #3 from now on. Now, file descriptors #3 and #4 point to standard output and standard error respectively. These will be used as real terminal standard output and standard error.
1> >(...) redirects standard output to command in parentheses
Parentheses (sub-shell) executes 'tee', reading from exec's standard output (pipe) and redirects to the 'logger' command via another pipe to the sub-shell in parentheses. At the same time it copies the same input to file descriptor #3 (the terminal)
the second part, very similar, is about doing the same trick for standard error and file descriptors #2 and #4.
The result of running a script having the above line and additionally this one:
echo "Will end up in standard output (terminal) and /var/log/messages"
...is as follows:
$ ./my_script
Will end up in standard output (terminal) and /var/log/messages
$ tail -n1 /var/log/messages
Sep 23 15:54:03 wks056 my_script_tag[11644]: Will end up in standard output (terminal) and /var/log/messages
If you want to see clearer picture, add these two lines to the script:
ls -l /proc/self/fd/
ps xf
bash your_script.sh 1>file.log 2>&1
1>file.log instructs the shell to send standard output to the file file.log, and 2>&1 tells it to redirect standard error (file descriptor 2) to standard output (file descriptor 1).
Note: The order matters as liw.fi pointed out, 2>&1 1>file.log doesn't work.
Curiously, this works:
yourcommand &> filename
But this gives a syntax error:
yourcommand &>> filename
syntax error near unexpected token `>'
You have to use:
yourcommand 1>> filename 2>&1
Short answer: Command >filename 2>&1 or Command &>filename
Explanation:
Consider the following code which prints the word "stdout" to stdout and the word "stderror" to stderror.
$ (echo "stdout"; echo "stderror" >&2)
stdout
stderror
Note that the '&' operator tells bash that 2 is a file descriptor (which points to the stderr) and not a file name. If we left out the '&', this command would print stdout to stdout, and create a file named "2" and write stderror there.
By experimenting with the code above, you can see for yourself exactly how redirection operators work. For instance, by changing which file which of the two descriptors 1,2, is redirected to /dev/null the following two lines of code delete everything from the stdout, and everything from stderror respectively (printing what remains).
$ (echo "stdout"; echo "stderror" >&2) 1>/dev/null
stderror
$ (echo "stdout"; echo "stderror" >&2) 2>/dev/null
stdout
Now, we can explain why the solution why the following code produces no output:
(echo "stdout"; echo "stderror" >&2) >/dev/null 2>&1
To truly understand this, I highly recommend you read this webpage on file descriptor tables. Assuming you have done that reading, we can proceed. Note that Bash processes left to right; thus Bash sees >/dev/null first (which is the same as 1>/dev/null), and sets the file descriptor 1 to point to /dev/null instead of the stdout. Having done this, Bash then moves rightwards and sees 2>&1. This sets the file descriptor 2 to point to the same file as file descriptor 1 (and not to file descriptor 1 itself!!!! (see this resource on pointers for more info)) . Since file descriptor 1 points to /dev/null, and file descriptor 2 points to the same file as file descriptor 1, file descriptor 2 now also points to /dev/null. Thus both file descriptors point to /dev/null, and this is why no output is rendered.
To test if you really understand the concept, try to guess the output when we switch the redirection order:
(echo "stdout"; echo "stderror" >&2) 2>&1 >/dev/null
stderror
The reasoning here is that evaluating from left to right, Bash sees 2>&1, and thus sets the file descriptor 2 to point to the same place as file descriptor 1, ie stdout. It then sets file descriptor 1 (remember that >/dev/null = 1>/dev/null) to point to >/dev/null, thus deleting everything which would usually be send to to the standard out. Thus all we are left with was that which was not send to stdout in the subshell (the code in the parentheses)- i.e. "stderror".
The interesting thing to note there is that even though 1 is just a pointer to the stdout, redirecting pointer 2 to 1 via 2>&1 does NOT form a chain of pointers 2 -> 1 -> stdout. If it did, as a result of redirecting 1 to /dev/null, the code 2>&1 >/dev/null would give the pointer chain 2 -> 1 -> /dev/null, and thus the code would generate nothing, in contrast to what we saw above.
Finally, I'd note that there is a simpler way to do this:
From section 3.6.4 here, we see that we can use the operator &> to redirect both stdout and stderr. Thus, to redirect both the stderr and stdout output of any command to \dev\null (which deletes the output), we simply type
$ command &> /dev/null
or in case of my example:
$ (echo "stdout"; echo "stderror" >&2) &>/dev/null
Key takeaways:
File descriptors behave like pointers (although file descriptors are not the same as file pointers)
Redirecting a file descriptor "a" to a file descriptor "b" which points to file "f", causes file descriptor "a" to point to the same place as file descriptor b - file "f". It DOES NOT form a chain of pointers a -> b -> f
Because of the above, order matters, 2>&1 >/dev/null is != >/dev/null 2>&1. One generates output and the other does not!
Finally have a look at these great resources:
Bash Documentation on Redirection, An Explanation of File Descriptor Tables, Introduction to Pointers
LOG_FACILITY="local7.notice"
LOG_TOPIC="my-prog-name"
LOG_TOPIC_OUT="$LOG_TOPIC-out[$$]"
LOG_TOPIC_ERR="$LOG_TOPIC-err[$$]"
exec 3>&1 > >(tee -a /dev/fd/3 | logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_OUT" )
exec 2> >(logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_ERR" )
It is related: Writing standard output and standard error to syslog.
It almost works, but not from xinetd ;(
For the situation when "piping" is necessary, you can use |&.
For example:
echo -ne "15\n100\n" | sort -c |& tee >sort_result.txt
or
TIMEFORMAT=%R;for i in `seq 1 20` ; do time kubectl get pods | grep node >>js.log ; done |& sort -h
These Bash-based solutions can pipe standard output and standard error separately (from standard error of "sort -c", or from standard error to "sort -h").
I wanted a solution to have the output from stdout plus stderr written into a log file and stderr still on console. So I needed to duplicate the stderr output via tee.
This is the solution I found:
command 3>&1 1>&2 2>&3 1>>logfile | tee -a logfile
First swap stderr and stdout
then append the stdout to the log file
pipe stderr to tee and append it also to the log file
Adding to what Fernando Fabreti did, I changed the functions slightly and removed the &- closing and it worked for me.
function saveStandardOutputs {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
exec 3>&1
exec 4>&2
trap restoreStandardOutputs EXIT
else
echo "[ERROR]: ${FUNCNAME[0]}: Cannot save standard outputs because they have been redirected before"
exit 1;
fi
}
# Parameters: $1 => logfile to write to
function redirectOutputsToLogfile {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
LOGFILE=$1
if [ -z "$LOGFILE" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: logfile empty [$LOGFILE]"
fi
if [ ! -f $LOGFILE ]; then
touch $LOGFILE
fi
if [ ! -f $LOGFILE ]; then
echo "[ERROR]: ${FUNCNAME[0]}: creating logfile [$LOGFILE]"
exit 1
fi
saveStandardOutputs
exec 1>>${LOGFILE}
exec 2>&1
OUTPUTS_REDIRECTED="true"
else
echo "[ERROR]: ${FUNCNAME[0]}: Cannot redirect standard outputs because they have been redirected before"
exit 1;
fi
}
function restoreStandardOutputs {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
exec 1>&3 #restore stdout
exec 2>&4 #restore stderr
OUTPUTS_REDIRECTED="false"
fi
}
LOGFILE_NAME="tmp/one.log"
OUTPUTS_REDIRECTED="false"
echo "this goes to standard output"
redirectOutputsToLogfile $LOGFILE_NAME
echo "this goes to logfile"
echo "${LOGFILE_NAME}"
restoreStandardOutputs
echo "After restore this goes to standard output"
The "easiest" way (Bash 4 only):
ls * 2>&- 1>&-
In situations when you consider using things like exec 2>&1, I find it easier to read, if possible, rewriting code using Bash functions like this:
function myfunc(){
[...]
}
myfunc &>mylog.log
The following functions can be used to automate the process of toggling outputs beetwen stdout/stderr and a logfile.
#!/bin/bash
#set -x
# global vars
OUTPUTS_REDIRECTED="false"
LOGFILE=/dev/stdout
# "private" function used by redirect_outputs_to_logfile()
function save_standard_outputs {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot save standard outputs because they have been redirected before"
exit 1;
fi
exec 3>&1
exec 4>&2
trap restore_standard_outputs EXIT
}
# Params: $1 => logfile to write to
function redirect_outputs_to_logfile {
if [ "$OUTPUTS_REDIRECTED" == "true" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot redirect standard outputs because they have been redirected before"
exit 1;
fi
LOGFILE=$1
if [ -z "$LOGFILE" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: logfile empty [$LOGFILE]"
fi
if [ ! -f $LOGFILE ]; then
touch $LOGFILE
fi
if [ ! -f $LOGFILE ]; then
echo "[ERROR]: ${FUNCNAME[0]}: creating logfile [$LOGFILE]"
exit 1
fi
save_standard_outputs
exec 1>>${LOGFILE%.log}.log
exec 2>&1
OUTPUTS_REDIRECTED="true"
}
# "private" function used by save_standard_outputs()
function restore_standard_outputs {
if [ "$OUTPUTS_REDIRECTED" == "false" ]; then
echo "[ERROR]: ${FUNCNAME[0]}: Cannot restore standard outputs because they have NOT been redirected"
exit 1;
fi
exec 1>&- #closes FD 1 (logfile)
exec 2>&- #closes FD 2 (logfile)
exec 2>&4 #restore stderr
exec 1>&3 #restore stdout
OUTPUTS_REDIRECTED="false"
}
Example of usage inside script:
echo "this goes to stdout"
redirect_outputs_to_logfile /tmp/one.log
echo "this goes to logfile"
restore_standard_outputs
echo "this goes to stdout"
For tcsh, I have to use the following command:
command >& file
If using command &> file, it will give an "Invalid null command" error.
I'm trying to create a bash script that prints results optionally to the console (STDOUT and STDERR), a log file or both.
I've got the getops working to set the options, but I can't seem to figure out the command. So far, I have something like this:
# default logs to both
export LOG_COMMAND=" 2>&1 | tee ${INSTALL_LOG}"
if [ "$ONLY_CONSOLE" == "true" ] ; then
export LOG_COMMAND=""
elif [ "$ONLY_LOG" == "true" ] ; then
export LOG_COMMAND=" | tee ${INSTALL_LOG}"
fi
echo "Starting script" ${LOG_COMMAND}
# ... then do more stuff ...
This script prints this to the console only and nothing to the log file:
Starting script 2>&1 | tee scriptfile.log
But I would like the script to print to the console and to the file scriptfile.log.
Has anyone done this before? It appears that the echo command is processing the ${LOG_COMMAND} as a variable.
Replacing the echo command with this works will print both:
echo "Starting script" 2>&1 | tee -a ${INSTALL_LOG}
However, this means that there is no option to print to console, log file, or both. It is hard coded to print to both.
There is definitely more than one way to do this, but the easiest may be to keep things simple.
After getopts is done and you've figured out what goes where, you use that to call main in the correct manner, and put all the rest of your logic in main (or called from main). For example:
main() {
# do stuff
# call other functions
# do more stuff
}
if [ $ONLY_CONSOLE -eq 0 ]
then
main
elif [ $ONLY_LOG -eq 0 ]
then
main > $INSTALL_LOG 2>&1
else # both
main 2>&1 | tee $INSTALL_LOG
fi
The alternative is an arcane combination of execs and other flow, but for a small number of options, this just makes the most sense to me.
#!/bin/bash
ip route add 10.105.8.100 via 192.168.1.100
date
cat /home/xxx/Documents/list.txt | while read output
do
ping="ping -c 3 -w 3 -q 'output'"
if $ping | grep -E "min/avg/max/mdev" > /dev/null; then
echo 'connection is ok'
else
echo "router $output is down"
then
cat /home/xxx/Documents/roots.txt | while read outputs
do
cd /home/xxx/Documents/routers
php rebootRouter.php "outputs" admin admin
done
fi
done
The other documents are:
lists.txt
10.105.8.100
roots.txt
192.168.1.100
when i run the script, the result is a reboot of the router am trying to ping. It doesn't ping.
Is there a problem with the bash script.??
If your files only contain a single line, there's no need for the while-loop, just use read:
read -r router_addr < /home/xxx/Documents/list.txt
# the grep is unnecessary, the return-code of the ping will be non-zero if the host is down
if ping -c 3 -w 3 -q "$router_addr" &> /dev/null; then
echo "connection to $router_addr is ok"
else
echo "router $router_addr is down"
read -r outputs < /home/xxx/Documents/roots.txt
cd /home/xxx/Documents/routers
php rebootRouter.php "$outputs" admin admin
fi
If your files contain multiple lines, you should redirect the file from the right-side of the while-loop:
while read -r output; do
...
done < /foo/bar/baz
Also make sure your files contain a newline at the end, or use the following pattern in your while-loops:
while read -r output || [[ -n $output ]]; do
...
done < /foo/bar/baz
where || [[ -n $output ]] is true even if the file doesn't end in a newline.
Note that the way you're checking for your routers status is somewhat brittle as even a single missed ping will force it to reboot (for example the checking computer returns from a sleep-state just as the script is running, the ping fails as the network is still down but the admin script succeeds as the network just comes up at that time).
To check if mdnsd is in probing mode we are using below command to browse for service and redirect its output a file, and hostname of the device is found in the command we decide that mdnsd is in probing mode.
command used for publishing service
dns-sd -R "Test status" "_mytest._tcp." "local." "22"
To browse the service following command is used (Running in background)
dns-sd -lo -Z _mytest._tcp > /tmp/myfile &
To display the content of the file used cat.
cat /tmp/myfile
myfile is empty, if > replaced with tee , I see output on console myfile remains empty.
I am unable to understand what is going on.
Is there any pointer, help
EDIT
Just for completeness adding output, which i missed adding before.
# dns-sd -lo -Z _mytest._tcp local
Using LocalOnly
Using interface -1
Browsing for _mytest._tcp
DATE: ---Tue 25 Apr 2017---
11:09:24.775 ...STARTING...
; To direct clients to browse a different domain, substitute that domain in place of '#'
lb._dns-sd._udp PTR #
; In the list of services below, the SRV records will typically reference dot-local Multicast DNS names.
; When transferring this zone file data to your unicast DNS server, you'll need to replace those dot-local
; names with the correct fully-qualified (unicast) domain name of the target host offering the service.
_mytest._tcp PTR Test\032status._mytest._tcp
Test\032status._mytest._tcp SRV 0 0 22 DevBoard.local. ; Replace with unicast FQDN of target host
Test\032status._mytest._tcp TXT ""
You appear to have a program with behavior that differs based on whether its output is to a TTY. One workaround is to use a tool such as unbuffer or script to simulate a TTY.
Moreover, inasmuch as the use of a file at all is done as a workaround, I suggest using a FIFO to actually capture the line you want without needing to write to a file and poll that file's contents.
#!/bin/sh
newline='
'
# Let's define some helpers...
cleanup() {
[ -e /proc/self/fd/3 ] && exec 3<&- ## close FD 3 if it's open
rm -f "fifo.$$" ## delete the FIFO from disk
if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then ## if our pid is still running...
kill "$pid" ## ...then shut it down.
fi
}
die() { cleanup; echo "$*" >&2; exit 1; }
# Create a FIFO, and start `dns-sd` in the background *thinking* it's writing to a TTY
# but actually writing to that FIFO
mkfifo "fifo.$$"
script -q -f -c 'dns-sd -lo -Z _mytest._tcp local' /proc/self/fd/1 |
tr -d '\r' >"fifo.$$" & pid=$!
exec 3<"fifo.$$"
while read -t 1 -r line <&3; do
case $line in
"Script started on"*|";"*|"") continue;;
"Using "*|DATE:*|[[:digit:]]*) continue;;
*) result="${result}${line}${newline}"; break
esac
done
if [ -z "$result" ]; then
die "Timeout before receiving a non-boilerplate line"
fi
printf '%s\n' "Retrieved a result:" "$result"
cleanup
I'm writing a shell script, and I'm trying to check if the output of a command contains a certain string. I'm thinking I probably have to use grep, but I'm not sure how. Does anyone know?
Testing $? is an anti-pattern.
if ./somecommand | grep -q 'string'; then
echo "matched"
fi
Test the return value of grep:
./somecommand | grep 'string' &> /dev/null
if [ $? == 0 ]; then
echo "matched"
fi
which is done idiomatically like so:
if ./somecommand | grep -q 'string'; then
echo "matched"
fi
and also:
./somecommand | grep -q 'string' && echo 'matched'
Another option is to check for regular expression match on the command output.
For example:
[[ "$(./somecommand)" =~ "sub string" ]] && echo "Output includes 'sub string'"
A clean if/else conditional shell script:
if (ls | grep '$1')
then
echo "exists"
else
echo "doesn't exist"
fi
SHORT ANSWER
All the above (very excellent) answers all assume that grep can "see" the output of the command, which isn't always true:
SUCCESS can be sent to STDOUT while FAILURE to STDERR.
So depending on which direction you test, your grep can fail. That's to say that if you are testing for the case of FAILURE you must redirect the output of the command to STDOUT using 2>&1 in such a case as this.
LONGER ANSWER w/ PROOFS
I had what I thought was a very simple test in a bash script using grep and it kept failing. Much head scratching followed. Use of set -x in my script revealed that the variable was empty! So I created the following test to understand how things were breaking.
NOTE: iscsiadm is a Linux tool from the "open-iscsi" package used to connect/disconnect a host to SAN storage. The command iscsiadm -m session is used to show if any LUN connections are established):
#!/bin/bash
set -x
TEST1=$(iscsiadm -m session)
TEST2=$(iscsiadm -m session 2>&1)
echo
echo 'Print TEST1'
echo $TEST1
echo
echo 'Print TEST2'
echo $TEST2
echo
If a LUN WAS connected, BOTH variables were successfully populated with values:
Print TEST1
tcp: [25] 192.168.X.XX:3260,1 iqn.2000-01.com.synology:ipdisk.Target-LUN1 (non-flash) tcp: [26] 192.168.X.XX:3260,1 iqn.2000-01.com.synology:storagehost.Target-LUN1 (non-flash)
Print TEST2
tcp: [25] 192.168.X.XX:3260,1 iqn.2000-01.com.synology:ipdisk.Target-LUN1 (non-flash) tcp: [26] 192.168.X.XX:3260,1 iqn.2000-01.com.synology:storagehost.Target-LUN1 (non-flash)
However, if a LUN WASN'T connected, iscsiadm sent the output to STDERR, and only the "TEST2" variable was populated where we had redirected to STDOUT using 2>&1; "TEST1" variable which had no redirection to STDOUT was empty:
iscsiadm: No active sessions.
Print TEST1
Print TEST2
iscsiadm: No active sessions.
CONCLUSION
If you have a funky, half-broken- works in one direction but not the other- situation such as this, try the above test replacing iscsiadm with your own command and you should get the proper visibility to rewrite your test to work correctly.