I have a perl script which writes messages to STDOUT and STDERR (via print/croak statements), but I also redirect the STDOUT and STDERR to a logfile:
File::Tee::tee STDOUT, ">>", "$logFile" ;
File::Tee::tee STDERR, ">>", "$logFile" ;
Now the output logfile has messages from STDOUT and STDERR displayed out of order. Also the actual output on terminal is also out of order. I have tried flushing the buffers (as recommended here: https://perl.plover.com/FAQs/Buffering.html) but it doesn't help:
select(STDERR) ;
$| = 1 ;
select(STDOUT) ;
$| = 1 ;
Does anyone know what I must do to see the output in order (I also tried additionally flushing the filehandle corresponding to $logfile, but its still the same)?
EDIT:
Thanks to all of you who have replied. A lot of the discussion over this ended up in comments, so I am going to list the few things which I tried based on feedback from all of you.
I already was flushing the STDOUT and STDERR before I used File::Tee. As #jimtut suspected, File::Tee was indeed the culprit - removing it restored the ordering on console. But I did want to redirect the STDOUT and STDERR.
#mob suggested using IO::Tee instead, but I haven't fully understood how to make that work they way I want in my code.
#briandfoy pointed out that there isn't a reliable way to ensure 2 seperate filehandles are seen in the correct order in realtime and also suggested using a logging routine which is the only place which can write to STDOUT/STDERR. #zimd further pointed out that File::Tee uses fork which is the heart of the issue since 2 processes cannot guarantee any order on output.
Since the File::Tee is to blame, I attempted to remove this from the code. I updated my logger function to print to STDOUT/STDERR as well as to additionally print to $log filehandle. Further for capturing the warns in the log, I did the following:
sub warning_handler {
my $msg = $_[0] ;
print STDERR $msg ;
print $log $msg if defined $log ;
}
$SIG{__WARN__} = \&warning_handler ;
This worked great for all of the code under my control. Everything was printing in order now both on console and logfile. However I realized I can't use this solution since I was also calling someone else's perl packages for some functionality and clearly I couldn't intercept the print/croak etc which wrote to STDOUT/STDERR within the 'off the shelf' package. So right now, I don't have a good solution. However I suspect if I can find someway to intercept STDOUT/STDERR within perl, I might be able to get what I need.
EDIT2:
I added my own answer which is probably the closest I got to solving the problem by modifying mob's solution of using IO::Tee instead of File::Tee, but even this misses some messages (though it fixes ordering).
EDIT3:
Finally found the 'solution'
use IO::Tee ;
use Capture::Tiny qw(capture);
...
...
select(STDERR) ;
$| = 1 ;
select(STDOUT) ;
$| = 1 ;
open (my $log, ">", $logfilename) ;
*REALSTDOUT = *STDOUT ;
*REALSTDERR = *STDERR ;
*STDOUT = IO::Tee->new(\*REALSTDOUT, $log);
*STDERR = IO::Tee->new(\*REALSTDERR, $log);
# Regular Perl code here which sends output to STDOUT/STDERR
...
...
# system calls / calls to .so needs to be catpured
&log_streams(sub { &some_func_which_calls_shared_object() ; }) ;
sub log_streams {
my ($cr, #args) = #_; # code reference, with its arguments
my ($out, $err, $exit) = capture { $cr->(#args) };
if ($out) {
print STDOUT $out;
}
if ($err) {
print STDERR $err;
}
}
The use of IO::Tee ensures all perl generated outputs to console also go to logfile, and this happens immediately thereby updating the log and console in realtime. Since IO::Tee is changing the meaning of STDOUT/STDERR filehandles to now refer to the teed handles, it can only intercept stdio from perl statements, it misses sys calls since they bypass perl's STDOUT/STDERR handles. So we capture the syscall output and then use the log_streams routine to forward it to the now aliased STDOUT/STDERR streams. This creates a delay in the system call generated output showing up in log/terminal but there is no delay for perl generated output - i.e. best of both worlds. Do note that the ordering of stderr and stdout generated by an invocation of subroutine some_func_which_calls_shared_object is not preserved since in the log_streams routine, we first print to STDOUT and then to STDERR - as long as the system call is atomic and doesn't do much in terms of interleaving stdout/stderr messages we should be ok.
Appreciate solutions from briandfoy, mob and zimd whose answers I combined to arrive at this solution! Never thought it would require to go through this detail for what seems a very simple problem.
With two separate file handles, there's no contract or guarantee that you'll see them in real time. Various settings and buffers affect that, which is why you see the auto flush stuff ($|). It's the same idea for files or the terminal.
Realize this is an architectural issue rather than a syntactic one. You have two things competing for the same resource. That usually ends in tears. I hesitate to suggest a solution when I don't know what the problem is, but consider having whatever is trying to write to STDOUT or STDERR write to some sort of message broker that collects all the messages and is the only thing that writes to the final (shared) destination. For example, things that want to add entries to the syslog don't write to the syslog; they send messages to the thing that writes to the syslog.
A more Perly example: in Log4perl, you don't write to the final destinations. You simply log a message and the logger is the single thing figures out how to handle it. When I want this sort of behavior with the module, I don't use output facilities directly:
debug( "Some debug message" );
sub debug {
my $message = shift;
output( "DEBUG: $message" );
}
sub output { # single thing that can output message
...
}
Then do whatever you need to do in output.
But, you can't always control that in other things that are also trying to output things. Perl let's you get around this by redefining what warn and friends do by putting a coderef in $SIG{__WARN__}. You can capture warning messages and do whatever you like with them (such as sending them to standard output). Beyond that is black magic that reopens STDERR onto something you can control. It's not that bad and it's isolated in one place.
At some point where another person doesn't want merged output, and the intrusive solutions make it impossible to separate them. I'd much prefer flexibility than hard-coded constraint. If I want just the errors, I want a way to get just the errors. There are many other sorts of workarounds, such as wrappers that collect both output streams (so, not at all intrusive) and various command-redirections.
You will have two filehandles writing to $logfile. Unless File::Tee takes care to seek to the end of the filehandle before every write (which it doesn't appear to), you will get a race condition where one filehandle will overwrite the other.
A workaround would be to use the reopen option to the File::Tee::tee function -- that will close the file after each write and reopen it (at the proper end of the file) before the next write. That could hurt your performance though, depending on how often you write to those filehandles.
You might also have better luck with IO::Tee, which is a more straightforward implementation (using tied filehandles) than what File::Tee uses (a background process for each File::Tee::tee call), so you may get fewer surprises. Here is how an IO::Tee solution might look:
use IO::Tee;
$| = 1;
open my $stdout, ">&=1"; # to write to original stdout
open my $stderr, ">&=2"; # to write to original stderr
open my $fh_log, ">>", $logfile;
*STDOUT = IO::Tee->new($stdout, $fh_log);
*STDERR = IO::Tee->new($stderr, $fh_log);
...
There are no background process, extra threads, or anything else to cause a race condition. Both STDOUT and STDERR will both write to the same log filehandle from the same process.
After taking hint from #mob's answer to use IO::Tie instead of File::Tee (since the latter uses fork causing out of order STDERR vs STDOUT), I modified mob's original solution a bit and it worked (almost - read on):
use IO::Tee
...
...
open (my $log, ">", $logfilename) ;
*MYSTDOUT = *STDOUT ;
*MYSTDERR = *STDERR ;
*STDOUT = IO::Tee->new(\*MYSTDOUT, $log);
*STDERR = IO::Tee->new(\*MYSTDERR, $log);
This resulted in the correct ordering on both console and in the logfile (mob's original solution using open to dup the STDOUT/STDERR didn't work - it resulted in correct order in logfile, but out of order on console. Using a typeglob alias instead of dup works for some strange reason).
However, as good as this solution sounds, it missed printing some messages from a package which I call in the logfile (though they get printed on console). My original code which had File::Tee did result in these same messages from package being shown in the logfile, so there is some voodoo going on somewhere. The specific package in question is a .so file so I have no visibility into how exactly it prints its messages.
EDIT:
I guess that the .so file is as good as an external system command printing to stdout/stderr. Since its not going through perl IO, the handles pointed to by STDOUT/STDERR typeglobs within perl will not reflect the output of external programs called from perl.
I guess the best solution would be to use a combination of this solution for messages coming from within the perl code, and using Capture::Tiny::capture as pointed out by #zdim for capturing and redirecting messages from system calls/calls going through the swig interface.
Note The first part is done via tie-d handles; solution in the second part uses Capture::Tiny
A bare-bones proof-of-concept for an approach using tie-d handles.
The package that ties a handle, by printing from it to a file and to (a copy of) STDOUT stream
package DupePrints;
use warnings;
use strict;
use feature 'say';
my $log = 't_tee_log.out';
open my $fh_out, '>', $log or die $!; # for logging
# An independent copy of STDOUT (via dup2), for prints to terminal
open my $stdout, '>&', STDOUT or die $!;
sub TIEHANDLE { bless {} }
sub PRINT {
my $self = shift;
print $fh_out #_;
print $stdout #_;
}
1;
A program that uses it
use warnings;
use strict;
use feature 'say';
use DupePrints;
$| = 1;
tie *STDERR, 'DupePrints';
tie *STDOUT, 'DupePrints';
say "hi";
warn "\t==> ohno";
my $y;
my $x = $y + 7;
say "done";
This prints to both the terminal and to t_tee_log.out the text
hi
==> ohno at main_DupePrints.pl line 14.
Use of uninitialized value $y in addition (+) at main_DupePrints.pl line 17.
done
See perltie and Tie::Handle, and this post with related examples, and perhaps this post
The logging to a file of STDOUT and STDERR streams (along with a copied printout) works across other modules that may be used in the main program, as well.
To also have "clean" prints, that don't get logged, copy the STDOUT handle in the main program, like it's done in the module, and print to that. If you need to use this in a more selective and sophisticated manner please modify as needed -- as it stands it is meant to be only a basic demo.
With the clarification in the question's edit, here is a different approach: wrap a call to Capture::Tiny, which captures all output from any code, and then manage the captured prints as needed
use warnings;
use strict;
use feature qw(say state);
use Capture::Tiny qw(capture);
sub log_streams {
my ($cr, #args) = #_; # code reference, with its arguments
# Initialize "state" variable, so it runs once and stays open over calls
state $fh_log = do {
open my $fh, '>', 'tee_log.txt' or die $!;
$fh;
};
my ($out, $err, $exit) = capture { $cr->(#args) };
if ($out) {
print $fh_log $out;
print $out;
}
if ($err) {
print $fh_log $err;
print $err;
}
}
log_streams( sub { say "hi" } );
log_streams( sub { warn "==> ohno" } );
log_streams( sub { my $y; my $x = $y + 7; } );
log_streams( sub { system('perl', '-wE', q(say "external perl one-liner")) } );
log_streams( sub { say "done" } );
The downside of all this is that everything needs to run via that sub. But then again, that's actually a good thing, even if sometimes inconvenient.
The state feature is used to "initialize" the filehandle because a variable declared as state is never re-initialized; so the file is opened only once, on the first call, and stays opened.
This is also a demo in need of completion.
I'm defining a variable as a composition of other variables and some text, and I'm trying to get this variable to not expand its containing variables on the assigning. But I want it to expand when called later. That way I could reuse the same template to print different results as the inner variables keep changing. I'm truing to avoid eval as much as possible as I will be receiving some of the inner variables from third parties, and I do not know what to expect.
My use case, as below, is to have some "calling stack" so I can log all messages with the same format and keep a record of the script, function, and line of the logged message in some format like this: script.sh:this_function:42.
My attempted solution
called.sh:
#!/bin/bash
SCRIPT_NAME="`basename "${BASH_SOURCE[0]}"`"
CURR_STACK="${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"
echo "${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"
echo "${CURR_STACK}"
echo
function _func_1 {
echo "${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"
echo "${CURR_STACK}"
}
_func_1
So, I intend to get the same results while printing the "${CURR_STACK}" as when printing the previous line.
If there is some built-in or other clever way to log this 'call stack', by all means, let me know! I'll gladly wave my code good-bye, but I'd still like to know how to prevent the variables from expanding right away on the assigning of CURR_STACK, but still keep them able to expand further ahead.
Am I missing some shopt?
What I've tried:
Case 1 (expanding on line 4):
CURR_STACK="${SCRIPT_NAME}:${FUNNAME[0]}:${LINENO[0]}"
CURR_STACK="`echo "${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"`"
CURR_STACK="`echo "\${SCRIPT_NAME}:\${FUNCNAME[0]}:\${LINENO[0]}"`"
called.sh::7 <------------------| These are control lines
called.sh::4 <---------------. .------------| With the results I expect to get.
X
called.sh:_func_1:12 <---´ `-------| Both indicate that the values expanded
called.sh::4 <-------------------------| on line 4 - when CURR_STACK was set.
Case 2 (not expanding at all):
CURR_STACK="\${SCRIPT_NAME}:\${FUNNAME[0]}:\${LINENO[0]}"
CURR_STACK=\${SCRIPT_NAME}:\${FUNCNAME[0]}:\${LINENO[0]}
CURR_STACK="`echo '${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}'`"
called.sh::7
${SCRIPT_NAME}:${FUNNAME[0]}:${LINENO[0]} <-------.----| No expansion at all!...
/
called.sh::12 /
${SCRIPT_NAME}:${FUNNAME[0]}:${LINENO[0]} <----´
Shell variables are store plain inert text(*), not executable code; there isn't really any concept of delayed evaluation here. To make something that does something when used, create a function instead of a variable:
print_curr_stack() {
echo "$(basename "${BASH_SOURCE[1]}"):${FUNCNAME[1]}:${BASH_LINENO[0]}"
}
# ...
echo "We are now at $(print_curr_stack)"
# Or just run it directly:
print_curr_stack
Note: using BASH_SOURCE[1] and FUNCNAME[1] gets info about context the function was run from, rather than where it is in the function itself. But for some reason I'm not clear on, BASH_LINENO[1] gets the wrong info, and BASH_LINENO[0] is what you want.
You could also write it to allow the caller to specify additional text to print:
print_curr_stack() {
echo "$#" "$(basename "${BASH_SOURCE[1]}"):${FUNCNAME[1]}:${BASH_LINENO[0]}"
}
# ...
print_curr_stack "We are now at"
(* There's an exception to what I said about variables just contain inert text: some variables -- like $LINENO, $RANDOM, etc -- are handled specially by the shell itself. But you can't create new ones like this except by modifying the shell itself.)
Are you familiar with eval?
$ a=this; b=is; c=a; d=test;
$ e='echo "$a $b $c $d"';
$ eval $e;
this is a test
$ b='is NOT'; # modify one of the variables
$ eval $e;
this is NOT a test
$ f=$(eval $e); # capture the value of the "eval" statement
$ echo $f;
this is NOT a test
I want to test that echo 1 outputs 1, but
expect { `echo 1` }.to output("1").to_stdout
does not work. It says it outputs nothing to stdout, while
expect { print 1 }.to output("1").to_stdout
works just fine. Why doesn't the first one work?
expect { `echo 1` }.to output("1").to_stdout
doesn't work for two reasons:
echo runs in a subprocess. RSpec's output matcher doesn't handle output from subprocesses by default. But you can use to_stdout_from_any_process instead of to_stdout to handle subprocesses, although it's a bit slower.
output only works for output sent to the same standard output stream as the Ruby process. Backticks open a new standard output stream, send the command's standard output to it and return the contents of the stream when the command completes. I don't think you care whether you run your subprocess with backticks or some other way, so just use system (which sends the command's standard output to the Ruby process's standard output stream) instead.
Addressing those two points gives us this expectation, which passes:
expect { system("echo 1") }.to output("1\n").to_stdout_from_any_process
(I had to change the expected value for it to pass, since echo adds a newline.)
As MilesStanfield pointed out, in the case you gave it's equivalent and easier to just test the output of backticks rather than use output:
expect { `echo 1` }.to eq "1\n"
That might or might not work in the more complicated case that you presumably have in mind.
In a shell script I wrote to test how functions are returning values I came across an odd unexpected behavior. The code below assumes that when entering the function fnttmpfile the first echo statement would print to the console and then the second echo statement would actually return the string to the calling main. Well that's what I assumed, but I was wrong!
#!/bin/sh
fntmpfile() {
TMPFILE=/tmp/$1.$$
echo "This is my temp file dude!"
echo "$TMPFILE"
}
mainname=main
retval=$(fntmpfile "$mainname")
echo "main retval=$retval"
What actually happens is the reverse. The first echo goes to the calling function and the second echo goes to STDOUT. why is this and is there a better way....
main retval=This is my temp file dude!
/tmp/main.19121
The whole reason for this test is because I am writing a shell script to do some database backups and decided to use small functions to do specific things, ya know make it clean instead of spaghetti code. One of the functions I was using was this:
log_to_console() {
# arg1 = calling function name
# arg2 = message to log
printf "$1 - $2\n"
}
The whole problem with this is that the function that was returning a string value is getting the log_to_console output instead depending on the order of things. I guess this is one of those gotcha things about shell scripting that I wasn't aware of.
No, what's happening is that you are running your function, and it outputs two lines to stdout:
This is my temp file dude!
/tmp/main.4059
When you run it $(), bash will intercept the output and store it in the value. The string that is stored in the variable contains the first linebreak (the last one is removed). So what is really in your "retval" variable is the following C-style string:
"This is my temp file dude!\n/tmp/main.4059"
This is not really returning a string (can't do that in a shell script), it's just capturing whatever output your function returns. Which is why it doesn't work. Call your function normally if you want to log to console.
I've written a bash logging library to be implemented with some complex scripts that my company is currently using. I've been deadset on providing the script filename (${BASH_SOURCE}) and the line number (${LINENO}) of the calling script when making the log calls. However, I didn't want to have to rely on the user or implementing script to pass in these two variables as parameters. If this were C/C++, I would just create a macro that prepends "__FILE__" and "__LINE__" to the parameter list.
I was finally able to get this part working. Here are some much-simplified abstracts as a proof of concept:
Here's the logging library:
# log.sh
LOG="eval _log \${BASH_SOURCE} \${LINENO}"
_log () {
_BASH_SOURCE=`basename "${1}"` && shift
_LINENO=${1} && shift
echo "(${_BASH_SOURCE}:${_LINENO}) $#"
}
And an implementing test script:
# MyTest.sh
. ./log.sh
${LOG} "This is a log message"
# (test.sh:5) This is a log message
This works pretty well (and, I was thrilled to get it working at first). However, this has one glaring problem: the interaction between quotes and eval. If I make the call:
${LOG} "I'm thrilled that I got this working"
# ./test.sh: eval: line 5: unexected EOF while looking for matching `''
# ./test.sh: eval: line 6: syntax error: unexpected end of file
Now, I believe that I understand why this is happening. The quoted parameters are kept intact as they're passed to eval, but at that point, the content is placed as-is into the resulting command string. I know that I can fix this by doing some escaping; however, I REALLY do not want to force the implementing scripts to have to do this. Before I implemented this "eval macro" capability, I had users make calls directly to "_log" and allowed them to optionally pass in "${LINENO}." With this implementation, The failing call above (with only a quoted sentence) worked just fine.
At the most basic level, all I really want is for a script to be able to call [log function/macro] "String to log with special characers" and have the resulting log message contain the filename and line number of the calling script, followed by the log message. If it's possible, I would assume that I'm very close, but if there's something I'm overlooking that would require a different approach, I'm open to that as well. I can't force the users to escape all of their messages, as that will likely cause them to not use this library. This is such a problem that if I can't find a solution to this, I'll likely revert to the old capability (which required ${LINENO} to be passed as a function parameter - this was much less intrusive).
TLDR: Is there any way to get eval to respect special characters within a quoted parameter, without having to escape them?
I recommend avoiding eval if possible. For your logging use case, you could take a look at the shell builtin caller. If you need more information, you can use the variables BASH_SOURCE, BASH_LINENO and FUNCNAME. Note that all of these variables are arrays and contain the full call stack. See the following example:
#! /bin/bash
function log() {
echo "[$( caller )] $*" >&2
echo "BASH_SOURCE: ${BASH_SOURCE[*]}"
echo "BASH_LINENO: ${BASH_LINENO[*]}"
echo "FUNCNAME: ${FUNCNAME[*]}"
}
function foobar() {
log "failed:" "$#"
}
foobar "$#"
(Note: this solves the immediate problem of quoting, but #nosid's answer about
accessing the call stack is much better)
Change your definition of _log slightly, to read from standard input instead of
taking the log message from positional parameters:
_log () {
# Set up _BASH_SOURCE and _LINENO the same way
cat <(echo -n "$(_BASH_SOURCE:$_LINENO) ") -
}
Then pass your log message via standard input using a here doc or a here string:
${LOG} <<<"This is a log message"
${LOG} <<<"I'm thrilled this works, too!"
${LOG} <<HERE
Even this long
message works as
intended!
HERE
For your specific case of logging, you may want to look at this function which prints the caller's context's file function and line number.
Reusable quoting function
This function will do correct quoting for you:
function token_quote {
local quoted=()
for token; do
quoted+=( "$(printf '%q' "$token")" )
done
printf '%s\n' "${quoted[*]}"
}
Example usage:
Example usage:
$ token_quote token 'single token' token
token single\ token token
Above, note the single token's space is quoted as \.
$ set $(token_quote token 'single token' token)
$ eval printf '%s\\n' "$#"
token
single token
token
$
This shows that the spaces in tokens are preserved.