Let's say you have a series of scripts that you don't own and, therefore, can't modify, that may spawn background processes without redirecting stdout and stderr. I've noticed that in bash, tee'ing the output, as shown in the following example, does not return when the script is done if the background process is still running (and has open file descriptors for stdout or stderr).
./runme.sh 2>&1| tee runme.out
Where runme.sh is defined as:
#!/bin/bash
# Start a fake daemon
perl -e 'while(1) { sleep(1) }' &
printf "Enter your name: "
read name
echo "Goodbye $name"
How can I run scripts like this in bash while capturing all output and get back to the prompt when the script is done?
alternative syntax could be to use process substitution
./runme.sh > >(tee runme.out) 2>&1
this way tee is no more a child process of current shell and shell will wait only for runme.sh termination whereas in a pipeline it's waiting for all process termination.
Note that tee and subprocesses are still running after runme.sh terminates.
does not return when the script is done if the background process is still running (and has open file descriptors for stdout or stderr)
So don't do that. Daemon tools will generally redirect stdout/err for this reason, and you can do it manually too:
perl -e 'while(1) { sleep(1) }' < /dev/null > mydaemon.log 2>&1 &
Now that it's not keeping the pipe open, you can tee robustly without hacks.
Related
I know how to redirect stdout to a file:
exec > foo.log
echo test
this will put the 'test' into the foo.log file.
Now I want to redirect the output into the log file AND keep it on stdout
i.e. it can be done trivially from outside the script:
script | tee foo.log
but I want to do declare it within the script itself
I tried
exec | tee foo.log
but it didn't work.
#!/usr/bin/env bash
# Redirect stdout ( > ) into a named pipe ( >() ) running "tee"
exec > >(tee -i logfile.txt)
# Without this, only stdout would be captured - i.e. your
# log file would not contain any error messages.
# SEE (and upvote) the answer by Adam Spiers, which keeps STDERR
# as a separate stream - I did not want to steal from him by simply
# adding his answer to mine.
exec 2>&1
echo "foo"
echo "bar" >&2
Note that this is bash, not sh. If you invoke the script with sh myscript.sh, you will get an error along the lines of syntax error near unexpected token '>'.
If you are working with signal traps, you might want to use the tee -i option to avoid disruption of the output if a signal occurs. (Thanks to JamesThomasMoon1979 for the comment.)
Tools that change their output depending on whether they write to a pipe or a terminal (ls using colors and columnized output, for example) will detect the above construct as meaning that they output to a pipe.
There are options to enforce the colorizing / columnizing (e.g. ls -C --color=always). Note that this will result in the color codes being written to the logfile as well, making it less readable.
The accepted answer does not preserve STDERR as a separate file descriptor. That means
./script.sh >/dev/null
will not output bar to the terminal, only to the logfile, and
./script.sh 2>/dev/null
will output both foo and bar to the terminal. Clearly that's not
the behaviour a normal user is likely to expect. This can be
fixed by using two separate tee processes both appending to the same
log file:
#!/bin/bash
# See (and upvote) the comment by JamesThomasMoon1979
# explaining the use of the -i option to tee.
exec > >(tee -ia foo.log)
exec 2> >(tee -ia foo.log >&2)
echo "foo"
echo "bar" >&2
(Note that the above does not initially truncate the log file - if you want that behaviour you should add
>foo.log
to the top of the script.)
The POSIX.1-2008 specification of tee(1) requires that output is unbuffered, i.e. not even line-buffered, so in this case it is possible that STDOUT and STDERR could end up on the same line of foo.log; however that could also happen on the terminal, so the log file will be a faithful reflection of what could be seen on the terminal, if not an exact mirror of it. If you want the STDOUT lines cleanly separated from the STDERR lines, consider using two log files, possibly with date stamp prefixes on each line to allow chronological reassembly later on.
Solution for busybox, macOS bash, and non-bash shells
The accepted answer is certainly the best choice for bash. I'm working in a Busybox environment without access to bash, and it does not understand the exec > >(tee log.txt) syntax. It also does not do exec >$PIPE properly, trying to create an ordinary file with the same name as the named pipe, which fails and hangs.
Hopefully this would be useful to someone else who doesn't have bash.
Also, for anyone using a named pipe, it is safe to rm $PIPE, because that unlinks the pipe from the VFS, but the processes that use it still maintain a reference count on it until they are finished.
Note the use of $* is not necessarily safe.
#!/bin/sh
if [ "$SELF_LOGGING" != "1" ]
then
# The parent process will enter this branch and set up logging
# Create a named piped for logging the child's output
PIPE=tmp.fifo
mkfifo $PIPE
# Launch the child process with stdout redirected to the named pipe
SELF_LOGGING=1 sh $0 $* >$PIPE &
# Save PID of child process
PID=$!
# Launch tee in a separate process
tee logfile <$PIPE &
# Unlink $PIPE because the parent process no longer needs it
rm $PIPE
# Wait for child process, which is running the rest of this script
wait $PID
# Return the error code from the child process
exit $?
fi
# The rest of the script goes here
Inside your script file, put all of the commands within parentheses, like this:
(
echo start
ls -l
echo end
) | tee foo.log
Easy way to make a bash script log to syslog. The script output is available both through /var/log/syslog and through stderr. syslog will add useful metadata, including timestamps.
Add this line at the top:
exec &> >(logger -t myscript -s)
Alternatively, send the log to a separate file:
exec &> >(ts |tee -a /tmp/myscript.output >&2 )
This requires moreutils (for the ts command, which adds timestamps).
Using the accepted answer my script kept returning exceptionally early (right after 'exec > >(tee ...)') leaving the rest of my script running in the background. As I couldn't get that solution to work my way I found another solution/work around to the problem:
# Logging setup
logfile=mylogfile
mkfifo ${logfile}.pipe
tee < ${logfile}.pipe $logfile &
exec &> ${logfile}.pipe
rm ${logfile}.pipe
# Rest of my script
This makes output from script go from the process, through the pipe into the sub background process of 'tee' that logs everything to disc and to original stdout of the script.
Note that 'exec &>' redirects both stdout and stderr, we could redirect them separately if we like, or change to 'exec >' if we just want stdout.
Even thou the pipe is removed from the file system in the beginning of the script it will continue to function until the processes finishes. We just can't reference it using the file name after the rm-line.
Bash 4 has a coproc command which establishes a named pipe to a command and allows you to communicate through it.
Can't say I'm comfortable with any of the solutions based on exec. I prefer to use tee directly, so I make the script call itself with tee when requested:
# my script:
check_tee_output()
{
# copy (append) stdout and stderr to log file if TEE is unset or true
if [[ -z $TEE || "$TEE" == true ]]; then
echo '-------------------------------------------' >> log.txt
echo '***' $(date) $0 $# >> log.txt
TEE=false $0 $# 2>&1 | tee --append log.txt
exit $?
fi
}
check_tee_output $#
rest of my script
This allows you to do this:
your_script.sh args # tee
TEE=true your_script.sh args # tee
TEE=false your_script.sh args # don't tee
export TEE=false
your_script.sh args # tee
You can customize this, e.g. make tee=false the default instead, make TEE hold the log file instead, etc. I guess this solution is similar to jbarlow's, but simpler, maybe mine has limitations that I have not come across yet.
Neither of these is a perfect solution, but here are a couple things you could try:
exec >foo.log
tail -f foo.log &
# rest of your script
or
PIPE=tmp.fifo
mkfifo $PIPE
exec >$PIPE
tee foo.log <$PIPE &
# rest of your script
rm $PIPE
The second one would leave a pipe file sitting around if something goes wrong with your script, which may or may not be a problem (i.e. maybe you could rm it in the parent shell afterwards).
I'm writing a bash script and using the following trick to redirect standard output into a named pipe which is consumed by tee:
exec > >(tee -a $LOGFILE) 2>&1
However, when the script exits, it does not return the shell until I press enter. Is there a simple way to fix this while still using this approach?
Edit: This is the environment I'm running this in:
Centos 7
Bash version 4.2.45
Contents of simple script called redirect.sh:
#!/bin/bash
exec > >(tee -a /tmp/haha) 2>&1
echo "hi there"
exit 0
Sample session:
[root#linux-ha-1 ~]# ./redirect.sh
[root#linux-ha-1 ~]# hi there
[root#linux-ha-1 ~]#
The prompt is being printed; unfortunately, it is printed before tee's output is printed (which is why it appears before hi there in the sample output).
Since the tee process is running asynchronously, there is no guarantee that it will send its output to the console before the script terminates. What you really want to do is to close the tee process and then wait for it to terminate before exiting from the script. This cannot be done with process substitution, unfortunately, but it can be accomplished either with coprocesses (in bash 4) or using named pipes, as is explained in the answer to bash: How do I ensure termination of process substitution used with exec?
For a simpler (but unreliable) solution, close the pipes feeding the tee process (which will force it to close) and then wait a few milliseconds:
#!/bin/bash
exec 3>&1 > >(tee -a /tmp/haha) 2>&1
echo "hi there"
exec 1>&3 2>&3
sleep 0.1
I have a bash script which I want it to run as a daemon as it checks for a condition does some work and sleeps for some time and the cycle repeats.
I demonize the script as : nohup myScript.sh &
How can I setup the logging for the same script as it has same echo commands which prints some statements and how to make sure I have the pid of the script if I want to kill it later.
Can anyone suggest how I can achieve the same
If you look at the I/O redirection chapter in the Advanced Bash-Scripting Guide, then you see that you can redirect all output to a file by doing something like, in the top of your script:
exec 2>&1
exec 1>>out.log
The first line redirects stderr to stdout, and the second line appends stdout to out.log. If you do this, then you can use echo to write specific messages to the log-file, and the output of all commands, which you haven't silenced will be written to the log-file as well.
Bash has a lot of special variables, you can get an overview of these in the Special Shell Variables reference card. One of these is $$, which hold the scripts PID. To save this, you can do the following:
echo $$ > ${0}.pid
This will create a file name <file name of script>.pid, containing the PID of the script.
Put the following line in the beginning of your script. It saves the pid for later :
echo $$ > myScript.pid
Start your script capturing output for logging (from the nohup man page example) :
nohup myScript.sh > myScript.log &
Kill the script later by :
kill -9 `cat myScript.pid`
Simple myScript.sh example :
#!/bin/bash
echo $$ > myScript.pid
while :; do echo "Hello"; sleep 1; done
Suppose I have test.sh as below. The intent is to run some background task(s) by this script, that continuously updates some file. If the background task is terminated for some reason, it should be started again.
#!/bin/sh
if [ -f pidfile ] && kill -0 $(cat pidfile); then
cat somewhere
exit
fi
while true; do
echo "something" >> somewhere
sleep 1
done &
echo $! > pidfile
and want to call it like ./test.sh | otherprogram, e. g. ./test.sh | cat.
The pipe is not being closed as the background process still exists and might produce some output. How can I tell the pipe to close at the end of test.sh? Is there a better way than checking for existence of pidfile before calling the pipe command?
As a variant I tried using #!/bin/bash and disown at the end of test.sh, but it is still waiting for the pipe to be closed.
What I actually try to achieve: I have a "status" script which collects the output of various scripts (uptime, free, date, get-xy-from-dbus, etc.), similar to this test.sh here. The output of the script is passed to my window manager, which displays it. It's also used in my GNU screen bottom line.
Since some of the scripts that are used might take some time to create output, I want to detach them from output collection. So I put them in a while true; do script; sleep 1; done loop, which is started if it is not running yet.
The problem here is now that I don't know how to tell the calling script to "really" detach the daemon process.
See if this serves your purpose:
(I am assuming that you are not interested in any stderr of commands in while loop. You would adjust the code, if you are. :-) )
#!/bin/bash
if [ -f pidfile ] && kill -0 $(cat pidfile); then
cat somewhere
exit
fi
while true; do
echo "something" >> somewhere
sleep 1
done >/dev/null 2>&1 &
echo $! > pidfile
If you want to explicitly close a file descriptor, like for example 1 which is standard output, you can do it with:
exec 1<&-
This is valid for POSIX shells, see: here
When you put the while loop in an explicit subshell and run the subshell in the background it will give the desired behaviour.
(while true; do
echo "something" >> somewhere
sleep 1
done)&
I'm looking for a way to capture all standard output and standard error to a file, while also outputting it to console. So:
(set it up here)
set -x # I want to capture every line that's executed too
cat 'foo'
echo 'bar'
Now the output from foo and bar, as well as the debugging output from set -x, will be logged to some log file and shown on the console.
I can't control how the file is invoked, so it needs to be set up at the start of the file.
You can use exec and process substitution to send stdout and stderr inside of the script to tee. The process substitution is a bashism, so it is not portable and will not work if bash is called as /bin/sh or with --posix.
exec > >(tee foo.log) 2>&1
set -x # I want to capture every line that's executed too
cat 'foo'
echo 'bar'
sleep 2
The sleep is added to the end because the output to the console will be buffered by the tee. The sleep will help prevent the prompt from returning before the output has finished.
Maybe create a proxy-script that calls the real script, redirecting stderr to stdout and piping it to tee?
Something like this:
#!/bin/bash
/path/to/real/script "$#" 2>&1 | tee file
If you like only STDERR on your console you can:
#!/bin/bash
set -e
outfile=logfile
exec > >(cat >> $outfile)
exec 2> >(tee -a $outfile >&2)
# write your code here