I have a huge bash script and I want to log specific blocks of code to a specific & small log files (instead of just one huge log file).
I have the following two methods:
# in this case, 'log' is a bash function
# Using code block & piping
{
# ... bash code ...
} | log "file name"
# Using Process Substitution
log "file name" < <(
# ... bash code ...
)
Both methods may interfere with the proper execution of the bash script, e.g. when assigning values to a variable (like the problem presented here).
How do you suggest to log the output of commands to log files?
Edit:
This is what I tried to do (besides many other variations), but doesn't work as expected:
function log()
{
if [ -z "$counter" ]; then
counter=1
echo "" >> "./General_Log_File" # Create the summary log file
else
(( ++counter ))
fi
echo "" > "./${counter}_log_file" # Create specific log file
# Display text-to-be-logged on screen & add it to the summary log file
# & write text-to-be-logged to it's corresponding log file
exec 1> >(tee "./${counter}_log_file" | tee -a "./General_Log_File") 2>&1
}
log # Logs the following code block
{
# ... Many bash commands ...
}
log # Logs the following code block
{
# ... Many bash commands ...
}
The results of executions varies: sometimes the log files are created and sometimes they don't (which raise an error).
You could try something like this:
function log()
{
local logfile=$1
local errfile=$2
exec > $logfile
exec 2> $errfile # if $errfile is not an empty string
}
log $fileA $errfileA
echo stuff
log $fileB $errfileB
echo more stuff
This would redirect all stdout/stderr from current process to a file without any subprocesses.
Edit: The below might be a good solution then, but not tested:
pipe=$(mktemp)
mknod $pipe p
exec 1>$pipe
function log()
{
if ! [[ -z "$teepid2" ]]; then
kill $teepid2
else
tee <$pipe general_log_file &
teepid1=$!
count=1
fi
tee <$pipe ${count}_logfile &
teepid2=$!
(( ++count ))
}
log
echo stuff
log
echo stuff2
if ! [[ -z "$teepid1" ]]; then kill $teepid1; fi
Thanks to Sahas, I managed to achieve the following solution:
function log()
{
[ -z "$counter" ] && counter=1 || (( ++counter ))
if [ -n "$teepid" ]; then
exec 1>&- 2>&- # close file descriptors to signal EOF to the `tee`
# command in the bg process
wait $teepid # wait for bg process to exit
fi
# Display text-to-be-logged on screen and
# write it to the summary log & to it's corresponding log file
( tee "${counter}.log" < "$pipe" | tee -a "Summary.log" 1>&4 ) &
teepid=$!
exec 1>"$pipe" 2>&1 # redirect stdout & stderr to the pipe
}
# Create temporary FIFO/pipe
pipe_dir=$(mktemp -d)
pipe="${pipe_dir}/cmds_output"
mkfifo "$pipe"
exec 4<&1 # save value of FD1 to FD4
log # Logs the following code block
{
# ... Many bash commands ...
}
log # Logs the following code block
{
# ... Many bash commands ...
}
if [ -n "$teepid" ]; then
exec 1>&- 2>&- # close file descriptors to signal EOF to the `tee`
# command in the bg process
wait $teepid # wait for bg process to exit
fi
It works - I tested it.
References:
Force bash script to use tee without piping from the command line # superuser.com - helped a lot
I/O Redirection # tldp.org
$! - PID Variable # tldp.org
TEST Operators: Binary Comparison # tldp.org
For simple redirection of bash code block, without using a dedicated function, do:
(
echo "log this block of code"
# commands ...
# ...
# ...
) &> output.log
Related
I am assigning the output of a command to variable A:
A=$(some_command)
How can I "capture" stderr into a variable B ?
I have tried some variations with 2>&1 and read but that does not work:
A=$(some_command) 2>&1 | read B
echo $B
Here's a code snippet that might help you
# capture stderr into a variable and print it
echo "capture stderr into a variable and print it"
var=$(lt -l /tmp 2>&1)
echo $var
capture stderr into a variable and print it
zsh: command not found: lt
# capture stdout into a variable and print it
echo "capture stdout into a variable and print it"
var=$(ls -l /tmp)
echo $var
# capture both stderr and stdout into a variable and print it
echo "capture both stderr and stdout into a variable and print it"
var=$(ls -l /tmp 2>&1)
echo $var
# more classic way of executing a command which I always follow is as follows. This way I am always in control of what is going on and can act accordingly
if somecommand ; then
echo "command succeeded"
else
echo "command failed"
fi
If you have to capture the output and stderr in different variables, then the following might help as well
## create a file using file descriptor for stdout
exec 3> stdout.txt
# create a file using file descriptor for stderr
exec 4> stderr.txt
A=$($1 /tmp 2>&4 >&3);
## close file descriptor
exec 3>&-
exec 4>&-
## open file descriptor for reading
exec 3< stdout.txt
exec 4< stderr.txt
## read from file using file descriptor
read line <&3
read line2 <&4
## close file descriptor
exec 3<&-
exec 4<&-
## print line read from file
echo "stdout: $line"
echo "stderr: $line2"
## delete file
rm stdout.txt
rm stderr.txt
You can try running it with the following
╰─ bash test.sh pwd
stdout: /tmp/somedir
stderr:
╰─ bash test.sh pwdd
stdout:
stderr: test.sh: line 8: pwdd: command not found
As noted in a comment your use case may be better served in other scripting languages. An example: in Perl you can achieve what you want quite simple:
#!/usr/bin/env perl
use v5.26; # or earlier versions
use Capture::Tiny 'capture'; # library is not in core
my $cmd = 'date';
my #arg = ('-R', '-u');
my ($stdout, $stderr, $exit) = capture {
system( $cmd, #arg );
};
say "STDOUT: $stdout";
say "STDERR: $stderr";
say "EXIT: $exit";
I'm sure similar solutions are available in python, ruby, and all the rest.
I gave it another try using process substitution and came up with this:
# command with no error
date +%b > >(read A; if [ "$A" = 'Sep' ]; then echo 'September'; fi ) 2> >(read B; if [ ! -z "$B" ]; then echo "$B"; fi >&2)
September
# command with error
date b > >(read A; if [ "$A" = 'Sep' ]; then echo 'September'; fi ) 2> >(read B; if [ ! -z "$B" ]; then echo "$B"; fi >&2)
date: invalid date “b“
# command with both at the same time should work too
I had no success "exporting" the variables from the subprocesses back to the original script. It might be possible though. I just couldn't figure it out.
But this gives you at least access to stdout and stderr as a variable. This means you can do whatever processing you want on them as variables. It depends on your use case if this is of any help to you. Good luck :-)
I have two bash script.
One script write in a fifo. The second one read from the fifo, but AFTER the first one end to write.
But something does not work. I do not understand where the problem is. Here the code.
The first script is (the writer):
#!/bin/bash
fifo_name="myfifo";
# Se non esiste, crea la fifo;
[ -p $fifo_name ] || mkfifo $fifo_name;
exec 3<> $fifo_name;
echo "foo" > $fifo_name;
echo "bar" > $fifo_name;
The second script is (the reader):
#!/bin/bash
fifo_name="myfifo";
while true
do
if read line <$fifo_name; then
# if [[ "$line" == 'ar' ]]; then
# break
#fi
echo $line
fi
done
Can anyone help me please?
Thank you
Replace the second script with:
#!/bin/bash
fifo_name="myfifo"
while true
do
if read line; then
echo $line
fi
done <"$fifo_name"
This opens the fifo only once and reads every line from it.
The problem with your setup is that you have fifo creation in the wrong script if you wish to control fifo access to time when the reader is actually running. In order to correct the problem you will need to do something like this:
reader: fifo_read.sh
#!/bin/bash
fifo_name="/tmp/myfifo" # fifo name
trap "rm -f $fifo_name" EXIT # set trap to rm fifo_name at exit
[ -p "$fifo_name" ] || mkfifo "$fifo_name" # if fifo not found, create
exec 3< $fifo_name # redirect fifo_name to fd 3
# (not required, but makes read clearer)
while :; do
if read -r -u 3 line; then # read line from fifo_name
if [ "$line" = 'quit' ]; then # if line is quit, quit
printf "%s: 'quit' command received\n" "$fifo_name"
break
fi
printf "%s: %s\n" "$fifo_name" "$line" # print line read
fi
done
exec 3<&- # reset fd 3 redirection
exit 0
writer: fifo_write.sh
#!/bin/bash
fifo_name="/tmp/myfifo"
# Se non esiste, exit :);
[ -p "$fifo_name" ] || {
printf "\n Error fifo '%s' not found.\n\n" "$fifo_name"
exit 1
}
[ -n "$1" ] &&
printf "%s\n" "$1" > "$fifo_name" ||
printf "pid: '%s' writing to fifo\n" "$$" > "$fifo_name"
exit 0
operation: (start reader in 1st terminal)
$ ./fifo_read.sh # you can background with & at end
(launch writer in second terminal)
$ ./fifo_write.sh "message from writer" # second terminal
$ ./fifo_write.sh
$ ./fifo_write.sh quit
output in 1st terminal:
$ ./fifo_read.sh
/tmp/myfifo: message from writer
/tmp/myfifo: pid: '28698' writing to fifo
/tmp/myfifo: 'quit' command received
The following script should do the job:
#!/bin/bash
FIFO="/tmp/fifo"
if [ ! -e "$FIFO" ]; then
mkfifo "$FIFO"
fi
for script in "$#"; do
echo $script > $FIFO &
done
while read script; do
/bin/bash -c $script
done < $FIFO
Given two script a.sh and b.sh where both scripts pass "a" and "b" to stdout, respectively, one will get the following result (given that the script above is called test.sh):
./test.sh /tmp/a.sh /tmp/b.sh
a
b
Best,
Julian
My aim is to monitor child process without creating any pipe while still being able to discriminate stderr from stdout, and being able to retrieve exit code.
I would like to avoid using named pipes or /dev/shm since they wouldn't be cleaned up in case of SIGKILL (yes, I have nice and tactful users)
After picking ideas on stackoverflow, I came to this. Not sure it is the smartest way, feel free to correct/enhance !
% monitor "echo stdout:" "echo stderr:" /bin/bash -c "echo FOO;BAR"
stdout: FOO
stderr: /bin/bash: BAR: command not found
% echo Exit-code: $?
Exit-code: 127
There's room for improvement as it keeps spawning 2 child processes, 1 per redirection I suspect.
#
# PURPOSE
# redirect stdout and stderr to 2 distinct functions and retrieve exit code
#
# USAGE
# monitor HANDLER_OUT HANDLER_ERR COMMAND ...
# HANDLER_OUT and HANDLER_ERR can be commands or functions
# COMMAND is the full command which outputs are being monitors
#
# RESULT
# returns COMMAND exit-code
#
# EXAMPLES
# monitor "echo OUT=" "echo ERR=" /bin/bash -c "FOOBAR; echo Done."
# monitor myfunction1 myfunction2 /bin/bash -c "FOOBAR; echo Done.;/bin/false"
#
monitor () {
monitor_exit_code "$#" 1001>&1 1002>&2
}
monitor_exit_code () {
local code=0
while read data
do
code=$data
done < <( monitor_stdout "$#" 1000>&1 )
return $code
}
monitor_stdout () {
local parser_out=$1;shift
while read data
do
($parser_out "$data") >&1001
done < <( monitor_stderr "$#" 1001>&1 )
}
monitor_stderr () {
local parser_err=$1;shift
while read data
do
($parser_err "$data") >&1002
done < <( "$#" 2>&1 1>&1001 ; echo >&1000 $? )
}
I have written a wrapper in bash, which calls other shell scripts. However, I need to print only the output from the wrapper, avoiding the output from called scripts, which I am basically logging into a log file.
Elaborating…..
Basically I am using a function as
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 | tee -a ${LOGFILE}
Where start logging is define as:- (I could only understand this function partially)
start_logging()
{
## usage: start_logging
## start a new log or append to existing log file
declare -i rc=0
if [ ! "${LOGFILE}" ];then
## display error and bail
fi
local TIME_STAMP=$(date +%Y%m%d:%H:%M:%S)
## open ${LOGFILE} or append to existing ${LOGFILE} with timestamp and actual command line
if [ ${DRY_RUN} ]; then
echo "DRY_RUN set..."
echo "${TIME_STAMP} Starting $(basename ${0}) run: '${0} ${ORIG_ARGS}'" { I}
echo "DRY_RUN set..."
echo "Please ignore \"No such file or directory\" from tee..."
else
echo "${TIME_STAMP} Starting $(basename ${0}) run: '${0} ${ORIG_ARGS}'"
echo "${TIME_STAMP} Starting $(basename ${0}) run: '${0} ${ORIG_ARGS}'"
fi
return ${rc}
}
LOGFILE is defined in the wrapper as
{
TMPDIR ="$/tmp"
LOGFILE="${TMPDIR}/${$}/${BASENAME%.*}.log
}
Now when its calling funtion1, funtion2 which basically calls other bash scripts its logging all the output in the file .i.e. { TMPDIR}/${$}/${BASENAME%.*}.log } as well as on the bash terminal.
I wanted that it should only echo what I have written in the wrapper on to the bash terminal and rest should be recorded in the log.
PleaseNote:- the called scripts from wrapper have echo function within but I don’t wanted that there output should be displayed on the terminal
Is it possible to achieve....
You need to redirect both stdout + stderr of your called scripts into your logfile.
./your_other_script.sh 2&>1 >> /var/log/mylogfile.txt
You get output to the terminal, because of tee. So you can:
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 | tee -a ${LOGFILE} >/dev/null
^^^^^^^^^^ - redirect the output from a tee to /dev/null
or simply remove the tee and all logs redirect only into the file
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 >${LOGFILE}
or for bigger parts of script, enclose the part into ( ) pair, will executed in a subshell and redirect the output to /dev/null, so:
(
start_logging ${LOGFILE}
{
Funtion1
Funtion2
} 2>&1 | tee -a ${LOGFILE}
) >/dev/null
I have a script that prints in a loop. I want the loop to print differently the first time from all other times (i.e., it should print differently if anything has been printed at all). I am thinking a simple way would be to check whether anything has been printed yet (i.e., stdout has been written to). Is there any way to determine that?
I know I could also write to a variable and test whether it's empty, but I'd like to avoid a variable if I can.
I think that will do what you need. If you echo something between # THE SCRIPT ITSELF and # END, THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT will be printed STDOUT HAS NOT BEEN TOUCHED else...
#!/bin/bash
readonly TMP=$(mktemp /tmp/test_XXXXXX)
exec 3<> "$TMP" # open tmp file as fd 3
exec 4>&1 # save current value of stdout as fd 4
exec >&3 # redirect stdout to fd 3 (tmp file)
# THE SCRIPT ITSELF
echo Hello World
# END
exec >&4 # restore save stdout
exec 3>&- # close tmp file
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo "THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT"
echo
cat "$TMP"
else
echo "STDOUT HAS NOT BEEN TOUCHED"
fi
rm "$TMP"
So, output of the script as is:
THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT
Hello World
and if you remove the echo Hello World line:
STDOUT HAS NOT BEEN TOUCHED
And if you really want to test that while running the script itself, you can do that, too :-)
#!/bin/bash
#FIRST ELSE
function echo_fl() {
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo $2
else
echo $1
fi
}
TMP=$(mktemp /tmp/test_XXXXXX)
exec 3 "$TMP" # open tmp file as fd 3
exec 4>&1 # save current value of stdout as fd 4
exec >&3 # redirect stdout to fd 3 (tmp file)
# THE SCRIPT ITSELF
for f in fst snd trd; do
echo_fl "$(echo $f | tr a-z A-Z)" "$f"
done
# END
exec >&4 # restore save stdout
exec 3>&- # close tmp file
TMP_SIZE=$(stat -f %z "$TMP")
if [ $TMP_SIZE -gt 0 ]; then
echo "THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT"
echo
cat "$TMP"
else
echo "STDOUT HAS NOT BEEN TOUCHED"
fi
rm "$TMP"
output is:
THE FOLLOWING DATA HAS BEEN WRITTEN TO STDOUT
FST
snd
trd
as you can see: Only the first line (FST) has caps on. That's what the echo_fl function does for you: If it's the first line of output, if echoes the first argument, if it's not it echoes the second argument :-)
It's hard to tell what you are trying to do here, but if your script is printing to stdout, you could simply pipe it to perl:
yourcommand | perl -pe 'if ($. == 1) { print "First line is: $_" }'
It all depends on what kind of changes you are attempting to do.
You cannot use the -f option with %z. The line TMP_SIZE=$(stat -f %z "$TMP") produces a long string that fails the test in if [ $TMP_SIZE -gt 0 ].