Super simple, but I have NO IDEA how to fix this. I have a super simple pv line (or so I thought) that writes to a file, but the output appends "^M" to the end of each line which the tail command doesn't interpret properly.
To explain, I'm writing an image to a compute module (CM3 or CM4) with pv and outputting the stderr to a temp file I can read from a dialog box lower in the code.
For some reason, the tailboxbg I'm using doesn't realize it's changing and just keeps the first line posted (since it's appended with "^M" which is not a newline) but I never specify anywhere to add that to the end of the line.
The code:
pv -teabf -s "$copySize" "$copyImage" > "devCompMod" 2> _temp &
The _temp file:
429MiB 0:00:01 [ 429MiB/s] ETA 0:00:33^M 819MiB 0:00:02 [ 408MiB/s] ETA 0:00:34^M 880MiB 0:00:03 [ 292MiB/s] ETA 0:00:47^M 939MiB 0:00:04 [ 234MiB/s] ETA 0:00:59^M 999MiB 0:00:05 [ 199MiB/s] ETA 0:01:09^M 1.34GiB 0:00:06 [ 229MiB/s] ETA 0:00:59^M
pv: write failed: No space left on device
Any help would be greatly appreciated, I'm losing my mind here.
I tried different redirections, but I have no idea how to pipe stderr through something else without completely screwing up my fileread, since it's running in a subshell ("&").
This has nothing to do with dos or mac. It is just that it really intends to add a \r to have a status line updating, instead of repeating them many times. I do that often in my own code.
One solution could be to add another command to replace \r by \n.
pv -teabf -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | tr $'\r' $'\n'
Btw, that's how you pipe stderr. Just redirect it to stdout. But be sure, of course to redirect stdout on something else. Which you already did.
The thing that appears unnatural to most people with this is the order.
Instinct could say that I should cmd > stdout.txt 2>&1 | stderrfilter.
That is, first get take the original content from stdout to a file, and now that this content has be "displaced", use the now empty stdout to hold stderr with 2>&1, and then pipe it.
But that is not what redirections mean. It is not content that you displace. It is filedescriptors. a>&b, is the bash version of dup2(b,a)
So see it more some sort of variables holding the destination of outputs. Whose initial values are dest1 = STDOUT, dest2 = STDERR. Everything that is printed to stdout goes to dest1. Everything that is printed to stderr goes to dest2. Unless your redirect something. > outfile.txt, changes dest1 = outfile.txt. 2>&1 changes dest2 to whatever is in dest1, like dest2 = dest1.
So, doing cmd > stdout.txt 2>&1 | filter, is like dest1=stdout.txt ; dest2=dest1. So that order doesn't work. It means that every outputs goes (mixed) to stdout.txt.
Instead cmd 2>&1 > stdout.txt | filter, is like dest2=dest1; dest1=stdout.txt so first changes dest2 to initial value of dest1, that is STDOUT then dest1 to stdout.txt. So std output of cmd goest to stdout.txt, and its err output goes to STDOUT, that is, to the pipe.
Of course, that is not really how it works. But that is schematically how to understand in which order to use the pipes and redirections. Again, think of them as variables, holding destinations. That you have to assign in a logical order. And not as content, that you have to displace.
And anyway, the correct order to redirect standard output to a file and standard error to a pipe is pv ... 2>&1 > outfile | filter
Another way
If you don't want to struggle with redirection, another way could be to tell pv to add a \n itself, by creating the output string with format option.
pv -F $'%t %e %a %b\n' -f -s "$copySize" "$copyImage" > "devCompMod"
But it is not ideal. First of all, it doesn't remove the \r (on a terminal, the only consequence is that the cursor is at the beginning of the line, not the end. I suppose most tool parsing lines wouldn't care neither. But you never know). It just add a \n.
And secondly, I suppose that in reality you would like to redirect the result anyway, since I take your objective is to feed another tool with the progress indication, that is stderr of pv. Your _temp was probably a fix because you couldn't redirect stderr in a pipe... but now that you know how to, I surmise you wouldn't use that _temp anymore.
But, well, if it works for you, then it is probably preferable over adding another command in your pipe chain (besides, I don't know if tr doesn't add a latency)
So, tl;dr
pv -F $'%t %e %a %b\n' -f -s "$copySize" "$copyImage" > "devCompMod" 2> _temp
or
pv -teabf -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | tr $'\r' $'\n' > _temp
if you want the progress into _temp. Preferably the first; the 2nd if the \r that are still (in addition to added \n) in the 1st output are a problem.
And if you want to redirect that output to a dialog command, instead of using a temp file.
pv -F $'%t %e %a %b\n' -f -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | dialog
or
pv -teabf -s "$copySize" "$copyImage" 2>&1 > "devCompMod" | tr $'\r' $'\n' | dialog
Related
In bash, calling foo would display any output from that command on the stdout.
Calling foo > output would redirect any output from that command to the file specified (in this case 'output').
Is there a way to redirect output to a file and have it display on stdout?
The command you want is named tee:
foo | tee output.file
For example, if you only care about stdout:
ls -a | tee output.file
If you want to include stderr, do:
program [arguments...] 2>&1 | tee outfile
2>&1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.
Furthermore, if you want to append to the log file, use tee -a as:
program [arguments...] 2>&1 | tee -a outfile
$ program [arguments...] 2>&1 | tee outfile
2>&1 dumps the stderr and stdout streams.
tee outfile takes the stream it gets and writes it to the screen and to the file "outfile".
This is probably what most people are looking for. The likely situation is some program or script is working hard for a long time and producing a lot of output. The user wants to check it periodically for progress, but also wants the output written to a file.
The problem (especially when mixing stdout and stderr streams) is that there is reliance on the streams being flushed by the program. If, for example, all the writes to stdout are not flushed, but all the writes to stderr are flushed, then they'll end up out of chronological order in the output file and on the screen.
It's also bad if the program only outputs 1 or 2 lines every few minutes to report progress. In such a case, if the output was not flushed by the program, the user wouldn't even see any output on the screen for hours, because none of it would get pushed through the pipe for hours.
Update: The program unbuffer, part of the expect package, will solve the buffering problem. This will cause stdout and stderr to write to the screen and file immediately and keep them in sync when being combined and redirected to tee. E.g.:
$ unbuffer program [arguments...] 2>&1 | tee outfile
Another way that works for me is,
<command> |& tee <outputFile>
as shown in gnu bash manual
Example:
ls |& tee files.txt
If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.
For more information, refer redirection
You can primarily use Zoredache solution, but If you don't want to overwrite the output file you should write tee with -a option as follow :
ls -lR / | tee -a output.file
Something to add ...
The package unbuffer has support issues with some packages under fedora and redhat unix releases.
Setting aside the troubles
Following worked for me
bash myscript.sh 2>&1 | tee output.log
Thank you ScDF & matthew your inputs saved me lot of time..
Using tail -f output should work.
In my case I had the Java process with output logs. The simplest solution to display output logs and redirect them into the file(named logfile here) was:
my_java_process_run_script.sh |& tee logfile
Result was Java process running with output logs displaying and
putting them into the file with name logfile
You can do that for your entire script by using something like that at the beginning of your script :
#!/usr/bin/env bash
test x$1 = x$'\x00' && shift || { set -o pipefail ; ( exec 2>&1 ; $0 $'\x00' "$#" ) | tee mylogfile ; exit $? ; }
# do whaetever you want
This redirect both stderr and stdout outputs to the file called mylogfile and let everything goes to stdout at the same time.
It is used some stupid tricks :
use exec without command to setup redirections,
use tee to duplicates outputs,
restart the script with the wanted redirections,
use a special first parameter (a simple NUL character specified by the $'string' special bash notation) to specify that the script is restarted (no equivalent parameter may be used by your original work),
try to preserve the original exit status when restarting the script using the pipefail option.
Ugly but useful for me in certain situations.
Bonus answer since this use-case brought me here:
In the case where you need to do this as some other user
echo "some output" | sudo -u some_user tee /some/path/some_file
Note that the echo will happen as you and the file write will happen as "some_user" what will NOT work is if you were to run the echo as "some_user" and redirect the output with >> "some_file" because the file redirect will happen as you.
Hint: tee also supports append with the -a flag, if you need to replace a line in a file as another user you could execute sed as the desired user.
< command > |& tee filename # this will create a file "filename" with command status as a content, If a file already exists it will remove existed content and writes the command status.
< command > | tee >> filename # this will append status to the file but it doesn't print the command status on standard_output (screen).
I want to print something by using "echo" on screen and append that echoed data to a file
echo "hi there, Have to print this on screen and append to a file"
tee is perfect for this, but this will also do the job
ls -lr / > output | cat output
I typically use tee to receive a piped output data, echo it to standard output, and forward it to the actual intended recipient of the piped data. But sometimes this fails, and I cannot exactly understand why.
I'll try to demonstrate with a series of examples:
$ echo testing with this string | tee
testing with this string
So, just echoing some data to tee without arguments, is replicated/printed on the terminal/stdout. Note that this should be tee printing the output, as the output from echo is now "piped"/redirected, and therefore not present in stdout anymore (the same that happens here:
$ echo aa | echo bb
bb
... i.e. echo aa output got redirected to the next command, - which, being echo b, does not care about the input, and outputs just its own output.)
$ echo testing with this string | tee | python3 -c 'a=1'
$
Now here, piping data into tee without arguments, - and then piping, from tee, to a program that does not provide any output to terminal/stdout - prints nothing. I would have expected tee here to duplicate to stdout, and then forward to the next command in the pipeline, but apparently that does not happen.
$ echo testing with this string | tee /dev/stdout
testing with this string
testing with this string
Right, so if we pipe to tee with command line argument /dev/stdout, we get the printout twice - and as concluded earlier, it must be tee that produces both printed lines. That means, that when used without an argument, |tee basically does not open any file for duplicating, and simply forwards what it receives on its input, to its output; but as it is the last in the pipeline, its output is stdout in that case, so we get a single printout.
Here we get double printout, because
tee duplicated its input stream to /dev/stdout due to the argument (which ends up as the first printout); and then
forwarded the same input to its output, which here being stdout (as tee is again last in the pipeline), results with the second printout.
This also would explain why the previous ...| tee | python3 -c 'a=1' did not print anything: tee without arguments did not open any file for duplication, and merely forwarded to next command in the toolchain - and as the next one does not print any output either, no output is generated whatsoever.
Well, if the above understanding is correct, then this:
$ echo testing with this string | tee /dev/stdout | python3 -c 'a=1'
$
... should print at least one line (from tee copying to /dev/stdout; the "forwarded" part will end up being "gulped" by the final command as it prints nothing), but it does not.
So, why does this happen - where am I going wrong in my understanding of what tee does?
And how can I use tee, to print to stdout, also when its output is forwarded to a command that doesn't print anything to stdout on its own?
You aren't misunderstanding tee, you're misunderstanding what stdout is. In a pipe, like echo testing | tee | python3 -c 'a=1', the tee command's stdout is not the terminal, it's the pipe going to the python command (and the echo command's stdout is the pipe going to tee).
So tee /dev/stdout sends two copies of its input (on stdin) to the exact same place: its stdout, whether that's the terminal, or a pipe, or whatever.
If you want to send a copy of the input to tee someplace other than down the pipe, you need to send it somewhere other than stdout. Where that is depends on where you actually want to send it (i.e. why you want to copy it). If you specifically want to send it to the terminal, you could do this:
echo testing | tee /dev/tty | python3 -c 'a=1'
...while if you want to send it to the outer context's stdout (which might or might not be a terminal), you can duplicate the outer context's stdin to a different file descriptor (#3 is handy for this), and then have tee write a copy to that:
{ echo testing | tee /dev/fd/3 | python3 -c 'a=1'; } 3>&1
Yet another option is to redirect it to stderr (aka FD #2, which is also the terminal by default, but redirectable separately from stdout) with tee /dev/fd/2.
Note that the various /dev entries I'm using here are supported by most unixish OSes, but they aren't universal. Check to see what your specific OS provides.
I think I got it, but am not sure if it is correct: I saw this: 19.8. Forgetting That Pipelines Make Subshells - bash Cookbook [Book].
So, if pipelines make subshells, then
echo testing with this string | tee /dev/stdout | python3 -c 'a=1'
... is conceptually equal to:
echo testing with this string | (tee /dev/stdout | (python3 -c 'a=1'))
Note that the second pipe | redirects stdout of the subshell tee runs in, and as /dev/stdout is just an interface to stdout, it is redirected too, so we get nothing printed.
So, while stdout (and /dev/stdout) is local to the (sub)shell, /dev/tty is local to the terminal - and therefore the following:
$ echo testing with this string | tee /dev/tty | python3 -c 'a=1'
testing with this string
... in fact prints a line, as expected.
I have a script that stores the output of commands, functions, and other scripts in a log file.
I want to avoid capturing user input.
The line that is in charge of storing the output of the commands to a logfile is this one:
$command 2>&1 | tee /dev/tty | ruby -pe 'print Time.now.strftime("[%s] ") if !$stdin.tty?' >> "$tempfile"
If the command is a function or a script that asks for user input and prints out those data, that input is stored in temporary file. I would like to avoid that since I don't want to capture sensible data.
I can't modify the commands, functions that I'm wrapping.
Your command only saves program output, not user input. The problem you're seeing is that the command has chosen to output whatever the user inputs, merging it into its own output that is then obviously logged.
There is no good way around this. Please fix your command.
Anyways. Here's a bad, fragile, hacky way around it:
tempfile=test.txt
command='read -ep Enter_some_input: '
$command 2>&1 |
tee /dev/tty |
python3 -c $'import os\nwhile s:=os.read(0, 1024):\n if len(s) > 3: os.write(1, s)' |
ruby -pe 'print Time.now.strftime("[%s] ") if !$stdin.tty?' >> "$tempfile"
The Python command drops all reads of 3 bytes or less. This aims to remove character by character echo as would happen in the most basic cases of a user typing into readline and similar, while hopefully not removing too much intentional output.
I want to execute a command and want to redirect stderr and stdout as below:
stderr and stdout -> should be written ONLY to logs.log file while keeping the order
stderr -> should be printed to SCREEN and also written to errors.log
So far I can redirect them both to screen and the file log.txt like this:
command 2>&1 | tee logs.log
But the above is not what I need.
To make it more clear again what the results needs to be.
After command is executed I need to see on screen only the result of stderr, I need to have one file named errors.log with stderr, and I need to have another file named logs.log with the result of both stdout and stderr in the original order in which they were created.
Preserving perfect order while performing separate redirections is not even theoretically possible without some ugly hackery. Ordering is only preserved in writes (in O_APPEND mode) directly to the same file; as soon as you put something like tee in one process but not the other, ordering guarantees go out the window and can't be retrieved without keeping information about which syscalls were invoked in what order.
So, what would that hackery look like? It might look something like this:
# eat our initialization time *before* we start the background process
sudo sysdig-probe-loader
# now, start monitoring syscalls made by children of this shell that write to fd 1 or 2
# ...funnel content into our logs.log file
sudo sysdig -s 32768 -b -p '%evt.buffer' \
"proc.apid=$$ and evt.type=write and (fd.num=1 or fd.num=2)" \
> >(base64 -i -d >logs.log) \
& sysdig_pid=$!
# Run your-program, with stderr going both to console and to errors.log
./your-program >/dev/null 2> >(tee errors.log)
That said, this remains ugly hackery: It only catches writes direct to FDs 1 and 2, and doesn't track any further redirections that may take place. (This could be improved by performing the writes to FIFOs, and using sysdig to track writes to those FIFOs; that way fdup() and similar operations would work as-expected; but the above suffices to prove the concept).
Making Separate Handling Explicit
Here we demonstrate how to use this to colorize only stderr, and leave stdout alone -- by telling sysdig to generate a stream of JSON as output, and then iterating over that:
exec {colorizer_fd}> >(
jq --unbuffered --arg startColor "$(tput setaf 1)" --arg endColor "$(tput sgr0)" -r '
if .["fd.filename"] == "stdout" then
("STDOUT: " + .["evt.buffer"])
else
("STDERR: " + $startColor + .["evt.buffer"] + $endColor)
end
'
)
sudo sysdig -s 32768 -j -p '%fd.filename %evt.buffer' \
"proc.apid=$$ and evt.type=write and proc.name != jq and (fd.num=1 or fd.num=2)" \
>&$colorizer_fd \
& sysdig_pid=$!
# Run your-program, with stdout and stderr going to two separately-named destinations
./your-program >stdout 2>stderr
Because we're keying off the output filenames (stdout and stderr), these need to be constant for the above code to work -- any temporary directory desired can be used.
Obviously, you shouldn't actually do any of this. Update your program to support whatever logging infrastructure is available in its native language (Log4j in Java, the Python logging module, etc) to allow its logging to be configured explicitly.
This will get you most of the way there:
your_command 2> >(tee -a logs.log errors.log) 1>>logs.log
but I don't think you'll be able to exactly preserve the order of the output in the logs.log file.
I have a program that outputs to stdout and stderr but doesn't make use of them in the correct way. Some errors go to stdout, some go do stderr, non error stuff goes to stderr and it prints way to much info on stdout. To fix this I want to make a pipeline to do:
Save all output of $cmd (from both stderr and stdout) to a file $logfile (don't print it to screen).
Filter out all warning and error messages on stderr and stdout (from warning|error to empty line) and colorize only "error" words (redirect output to stderr).
Save output of step 2 to a file $logfile:r.stderr.
Exit with the correct exit code from the command.
So far I have this:
$!/bin/zsh
# using zsh 4.2.0
setopt no_multios
# Don't error out if sed or grep don't find a match:
alias -g grep_err_warn="(sed -n '/error\|warning/I,/^$/p' || true)"
alias -g color_err="(grep --color -i -C 1000 error 1>&2 || true)"
alias -g filter='tee $logfile | grep_err_warn | tee $logfile:r.stderr | color_err'
# use {} around command to avoid possible race conditions:
{ eval $cmd } 2>&1 | filter
exit $pipestatus[1]
I've tried many things but can't get it to work. I've read "From Bash to Z Shell", many posts, etc. My problems currently are:
Only stdin goes into the filter
Note: the $cmd is a shell script that calls a binary with a /usr/bin/time -p prefix. This seems to cause issues with pipelines and is why I'm wrapping the command in {…} all the output goes into the pipe.
I don't have zsh available.
I did notice that your {..}'d statement is not correct.
You always need a semicolon before the closing `}'.
When I added that in bash, I could prove to my satisfaction that stderr was being redirected to stdout.
Try
{ eval $cmd ; } 2>&1 | filter
# ----------^
Also, you wrote
Save all output of $cmd (form stderr
and stdout) to a file $logfile
I don't see any mention of $logfile in your code.
You should be able to get all output into logfile (while losing the specficity of stderr stream), with
yourCommand 2>&1 | tee ${logFile} | ....
I hope this helps.
P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, and/or give it a + (or -) as a useful answer.
Don't use aliases in scripts, use functions (global aliases are especially looking for trouble). Not that you actually need functions here. You also don't need || true (unless you're running under set -e, in which case you should turn it off here). Other than that, your script looks ok; what is it choking on?
{ eval $cmd } |
tee $logfile |
sed -n '/error\|warning/I,/^$/p' |
tee $logfile:r.stderr |
grep --color -i -C 1000 error 1>&2
exit $pipestatus[1]
I'm also not sure what you meant by the sed expression; I don't quite understand your requirement 2.
The original post was mostly correct, except for an optimization by Gilles (to turn off set -e so the || true's are not needed.
#!/bin/zsh
# using zsh 4.2.0
setopt no_multios
#setopt no_errexit # set -e # don't turn this on
{ eval $cmd } 2>&1 |
tee $logfile |
sed -n '/error\|warning/I,/^$/p' |
tee $logfile:r.stderr |
grep --color -i -C 1000 error 1>&2
exit $pipestatus[1]
The part that confused me was the mixing of stdout and stderr led to them being interleaved and the sed -n '/error\|warning/I,/^$/p' (which prints out from and error || warning to the next empty line) was printing out a lot more than expected which made it seem like the command wasn't working.