I've used this answer, which only copies stdout to file:
$ cat /etc/dehydrated/syncNexusCertificatesHook.sh
#!/bin/bash -ex
exec &> >(ts '[%Y-%m-%d %H:%M:%S]' | tee -a /var/log/dehydrated.log >&2 )
...
When running the script, there was a perl: warning: Setting locale failed. printed to the terminal, but not to the log file.
I want to have all output from the script printed with a timestamp printed to the console as well as saved to the log file. How can I achieve this?
To redirect both stdout and stderr of a command to a sequence of commands, and finally to both the console and a file, you can do it with this syntax
command |& ts '[%Y-%m-%d %H:%M:%S]' | tee file
which is the synonym of this
command 2>&1 | ts '[%Y-%m-%d %H:%M:%S]' | tee file
or redirecting both streams to a process substitution
command &> >(ts '[%Y-%m-%d %H:%M:%S]' | tee file)
or redirecting the output and then make the stderr a copy of stdout
command > >(ts '[%Y-%m-%d %H:%M:%S]' | tee file) 2>&1
Related
I've a Bash command with runs a python script and produce outputs on stdout and stderror.
Normal and error logs are written to separate files with this command:
python3 file.py >> normal.log 2>> error.log
Additionally, stdout and stderr shall be forwarded to an external program (e.g. logToTelegram.sh):
python3 file.py 2>&1 | logToTelegram.sh
Is there a way to implement both during one execution, write log files (normal and error) and pipe stdout and stderror together to the program logToTelegram.sh?
This should do the trick:
( python3 file.py 2> >(tee -a error.log) > >(tee -a normal.log) ) | logToTelegram.sh
Note that 2> >(tee ...) has to be placed before > >(tee ...) for this to work correctly. For simplicity, both tee commands output to stdout, eliminating the need for 2>&1 before piping to logToTelegram.sh.
Or, to stay closer to the original code and to be more precise:
( python3 file.py > >(tee -a normal.log) 2> >(tee -a error.log >&2) ) 2>&1 | logToTelegram.sh
Here, the first tee outputs to stdout, while the second tee outputs to stderr, thus 2>&1 is required to send all output to logToTelegram.sh.
I need to redirect stdout to console along with stdout and stderr redirected to a file. This needs to be done inside a shell script.
I found the below code to redirect both to console and log file, now I need to remove stderr to console.
exec > >(tee -i "output.log") 2>&1
Could you please help me here?
Derived from this answer:
How do I get both STDOUT and STDERR to go to the terminal and a log file?
I tested the following code:
echo '' > out.log
exec 1> >(tee -a -i out.log) 2> >(tee -a -i out.log > /dev/null)
>&2 echo yay
echo nay
The STDERR content goes to file and not to console, and STDOUT goes to both.
I think you should launch it like this:
your-command 2>&1 | tee -i "output.log"
It will pipe both stdin and stderr to tee -i output.log, which will echo text before writing.
Or, to ignore stderr (pipe to /dev/null):
your-command 2>/dev/null | tee -i "output.log"
Or your command modified with /dev/null redirection:
exec > >(tee -i "output.log") 2>/dev/null
It'd be something like this, but you don't have to tee that second one.
exec 1> >( tee stdout.txt ) 2> stderr.txt
output 1 is tee'd so you see it on screen, and in stdout.txt
errors 2 just goes straight to stderr.txt
The script should redirect all the output (stdout and stderr) to a log file, and only display stderr to the screen (notifying user if an error happens). The command tee may help but don't know how to write it.
Thanks.
P.S., thanks lihao and konsolebox for the answer, but is there a way to keep the output in order. For example:
$ cat test.sh
echo "to stdout..1"
echo "to stderr..1" >&2
echo "to stdout..2"
echo "to stderr..2" >&2
$ sh test.sh 2>&1 >test.log | tee -a test.log
to stderr..1
to stderr..2
$ cat test.log
to stdout..1
to stdout..2
to stderr..1
to stderr..2
Command: { sh test.sh 2> >(tee /dev/fd/4); } 4>&1 >test.log has the same output.
how about the following:
cmd args 2>&1 >logfile | tee -a logfile
You should map normal stdout to another file descriptor (4), make the file the default output, then use tee to redirect output to the new file descriptor through /dev/fd. Of course you'd need process substitution to pass stderr output to tee:
{ cmd args 2> >(exec tee /dev/fd/4); } 4>&1 >file
If you want to make a general redirection for the script, place this at the beginning of it:
exec 4>&1 >file 2> >(exec tee /dev/fd/4)
You can restore normal output with:
exec >&4 4>&-
I am running a program called stm. I want to save only those stderr messages that contain the text "ERROR" in a text file. I also want the messages on the console.
How do I do that in bash?
Use the following pipeline if only messages containing ERROR should be displayed on the console (stderr):
stm |& grep ERROR | tee -a /path/to/logfile
Use the following command if all messages should be displayed on the console (stderr):
stm |& tee /dev/stderr | grep ERROR >> /path/to/logfile
Edit: Versions without connecting standard output and standard error:
stm 2> >( grep --line-buffered ERROR | tee -a /path/to/logfile >&2 )
stm 2> >( tee /dev/stderr | grep --line-buffered ERROR >> /path/to/logfile )
This looks like a duplicate of How to pipe stderr, and not stdout?
Redirect stderr to "&1", which means "the same place where stdout is going".
Then redirect stdout to /dev/null. Then use a normal pipe.
$ date -g
date: invalid option -- 'g'
Try `date --help' for more information.
$
$ (echo invent ; date -g)
invent (stdout)
date: invalid option -- 'g' (stderr)
Try `date --help' for more information. (stderr)
$
$ (echo invent ; date -g) 2>&1 >/dev/null | grep inv
date: invalid option -- 'g'
$
To copy the output from the above command to a file, you can use a > redirection or "tee". The tee command will print one copy of the output to the console and second copy to the file.
$ stm 2>&1 >/dev/null | grep ERROR > errors.txt
or
$ stm 2>&1 >/dev/null | grep ERROR | tee errors.txt
Are you saying that you want both stderr and stdout to appear in the console, but only stderr (not stdout) that contains "ERROR" to be logged to a file? It is that last condition that makes it difficult to find an elegant solution. If that is what you are looking for, here is my very ugly solution:
touch stm.out stm.err
stm 1>stm.out 2>stm.err & tail -f stm.out & tail -f stm.err & \
wait `pgrep stm`; pkill tail; grep ERROR stm.err > error.log; rm stm.err stm.out
I warned you about it being ugly. You could hide it in a function, use mktemp to create the temporary filenames, etc. If you don't want to wait for stm to exit before logging the ERROR text to a file, you could add tail -f stm.err | grep ERROR > error.log & after the other tail commands, and remove the grep command from the last line.
Redirecting stdout+stderr such that both get written to a file while still outputting to stdout is simple enough:
cmd 2>&1 | tee output_file
But then now both stdout/stderr from cmd are coming on stdout. I'd like to write stdout+stderr to the same file (so ordering is preserved assuming cmd is single threaded) but then still be able to also separately redirect them, something like this:
some_magic_tee_variant combined_output cmd > >(command-expecting-stdout) 2> >(command-expecting-stderr)
So combined_output contains the both with order preserved, but the command-expecting-stdout only gets stdout and command-expecting-stderr only gets stderr. Basically, I want to log stdout+stderr while still allowing stdout and stderr to be separately redirected and piped. The problem with the tee approach is it globs them together. Is there a way to do this in bash/zsh?
From what I unterstand this is what you are looking for. First I made a litte script to write on stdout and stderr. It looks like this:
$ cat foo.sh
#!/bin/bash
echo foo 1>&2
echo bar
Then I ran it like this:
$ ./foo.sh 2> >(tee stderr | tee -a combined) 1> >(tee stdout | tee -a combined)
foo
bar
The results in my bash look like this:
$ cat stderr
foo
$ cat stdout
bar
$ cat combined
foo
bar
Note that the -a flag is required so the tees don't overwrite the other tee's content.
{ { cmd | tee out >&3; } 2>&1 | tee err >&2; } 3>&1
Or, to be pedantic:
{ { cmd 3>&- | tee out >&3 2> /dev/null; } 2>&1 | tee err >&2 3>&- 2> /dev/null; } 3>&1
Note that it's futile to try and preserve order. It is basically impossible. The only solution would be to modify "cmd" or use some LD_PRELOAD or gdb hack,
Order can indeed be preserved. Here's an example which captures the standard output and error, in the order in which they are generated, to a logfile, while displaying only the standard error on any terminal screen you like. Tweak to suit your needs.
1.Open two windows (shells)
2.Create some test files
touch /tmp/foo /tmp/foo1 /tmp/foo2
3.In window1:
mkfifo /tmp/fifo
</tmp/fifo cat - >/tmp/logfile
4.Then, in window2:
(ls -l /tmp/foo /tmp/nofile /tmp/foo1 /tmp/nofile /tmp/nofile; echo successful test; ls /tmp/nofile1111) 2>&1 1>/tmp/fifo | tee /tmp/fifo 1>/dev/pts/1
Where /dev/pts/1 can be whatever terminal display you want. The subshell runs some "ls" and "echo" commands in sequence, some succeed (providing stdout) and some fail (providing stderr) in order to generate a mingled stream of output and error messages, so that you can verify the correct ordering in the log file.
Here's how I do it:
exec 3>log ; example_command 2>&1 1>&3 | tee -a log ; exec 3>&-
Worked Example
bash$ exec 3>log ; { echo stdout ; echo stderr >&2 ; } 2>&1 1>&3 | \
tee -a log ; exec 3>&-
stderr
bash$ cat log
stdout
stderr
Here's how that works:
exec 3>log sets up file descriptor 3 to redirect into the file called log, until further notice.
example_command to make this a working example, I used { echo stdout ; echo stderr >&2 ; }. Or you could use ls /tmp doesnotexist to provide output instead.
Need to jump ahead to the pipe | at this point because bash does it first. The pipe sets up a pipe and redirects the file descriptor 1 into this pipe. So now, STDOUT is going into the pipe.
Now we can go back to where we were next in our left-to-right interpretation: 2>&1 this says errors from the program are to go to where STDOUT currently points, i.e. into the pipe we just set up.
1>&3 means STDOUT is redirected into file descriptor 3, which we earlier set up to output to the log file. So STDOUT from the command just goes into the log file, not to the terminal's STDOUT.
tee -a log takes it's input from the pipe (which you'll remember is now the errors from the command), and outputs it to STDOUT and also appends it to the log file.
exec 3>&- closes the file descriptor 3.
Victor Sergienko's comment is what worked for me, adding exec to the front of it makes this work for the entire script (instead of having to put it after individual commands)
exec 2> >(tee -a output_file >&2) 1> >(tee -a output_file)