The program:
function import {
set -x
# read NAME < <(/usr/bin/pwd)
NAME=$(/usr/bin/pwd)
echo 123 >&2
set +x
}
echo aaaaaaaaaaa
import
echo bbbbbbbbbbb
OUT=$( import 2>&1 )
echo "$OUT"
echo ccccccccccc
I hoped to have the output between 'aaa' and 'bbb' to be the same as in between 'bbb' and 'ccc'. But it is not the case with ksh:
aaaaaaaaaaa
+ /usr/bin/pwd
+ NAME=/home/neuron
+ echo 123
+ 1>& 2
123
bbbbbbbbbbb
+ /usr/bin/pwd
ccccccccccc
If I change $( ... ) into < <(...), stderr works as usual and I have the same output. I tried that on solaris and linux and it behaves the same, so I guess it's not ksh bug. Please note that it's not just 'set -x' being disabled, also the 'echo 123 1>&2' output disapears. In bash the code works as I would suppose.
My questions are 'why' and 'how to capture the function's stdout and stderr?'
Thank you
Vlad
It's confirmed to be a bug. And we can reproduce it very simply:
Something appears broken with nested command substitution:
for command in true /bin/true; do
a=$( ( b=$( $command ); echo 123 >& 3; ) 3>& 1 ) &&
echo a=$a command=$command
done
a=123 command=true
a= command=/bin/true
We run the same assignment twice, once with a builtin, and once with
an external command. We would expect the same results, but this fails
when we include the external command.
Glenn Fowler:
I believe this was fixed between 2012-08-23 and 2012-10-12
Tested with the latest beta:
$ ${ksh} test.sh
Version AIJM 93v- 2014-01-14
a=123 command=true
a=123 command=/bin/true
$
$ ksh test.sh
Version JM 93u 2011-02-08
a=123 command=true
a= command=/bin/true
$
This looks like a bug in ksh (My version is u), and it is not specific to pwd or stderr.
Below I can reproduce the effect with true and fd 3. (This allows us to keep use the shell trace)
The effect appears to be triggered by assigning output from an external process
into a variable inside a function. If the assignment is then followed by output
to some other file descriptor, that output gets lost.
The idea here is that the $( ... ) construct somehow conflicts with the subsequent redirect,
but only when the code in ... is run externally. The true and pwd builtins don't trigger a subshell in ksh93.
I will ask David Korn for confirmation.
function f1 {
var=$( /bin/true )
echo 123 >& 3
}
function f2 {
var=$( true )
echo 123 >& 3
}
function f3 {
typeset var
var=$( /bin/true )
echo 123 >& 3
}
functions="f1 f2 f3"
typeset -tf ${functions}
exec 3>& 1
echo ${.sh.version}
for f in ${functions}; do
echo TEST $f
functions $f
echo "123 expected: "
$f
OUT=$( $f 3>& 1 )
echo "OUT='123' expected"
echo "OUT='$OUT'" Captured output
echo
done
output:
Version JM 93u 2011-02-08
TEST f1
function f1 {
var=$( /bin/true )
echo 123 >& 3
}
123 expected:
123
OUT='123' expected
OUT='' Captured output
TEST f2
function f2 {
var=$( true )
echo 123 >& 3
}
123 expected:
123
OUT='123' expected
OUT='123' Captured output
TEST f3
function f3 {
typeset var
var=$( /bin/true )
echo 123 >& 3
}
123 expected:
123
OUT='123' expected
OUT='' Captured output
Related
Can somebody explain the following behaviour with a here-string, please?
$ echo "$SHLVL $BASH_SUBSHELL $BASHPID $$"
1 0 18689 18689
$ cat <<< "$SHLVL $BASH_SUBSHELL $BASHPID $$"
1 0 19078 18689
$ cat <<< "$SHLVL $BASH_SUBSHELL $BASHPID $$"
1 0 19079 18689
The BASHPID is different from the shell PID in the here-string, and change each time, but I don't understand why.
The same happens in here-documents:
$ cat << EOT
> $SHLVL $BASH_SUBSHELL $BASHPID $$
> EOT
1 0 19096 18689
Surprisingly, BASHPID doesn't change in a command group:
$ { cat ;} <<< "$SHLVL $BASH_SUBSHELL $BASHPID $$"
1 0 18689 18689
On the other hand, it changes in a subshell
$ (echo $BASHPID ; cat) <<< "$SHLVL $BASH_SUBSHELL $BASHPID $$"
20465
1 1 20465 18689
while the here-string is supposed to expand in the current shell.
Note: my bash version is 4.3+
(Just guessing ...)
The behavior is similar to this:
# echo $$
35130
# echo $( echo $$ $BASHPID )
35130 88025
# echo $( echo $$ $BASHPID )
35130 88026
#
# # or
#
# echo $$ $BASHPID | cat
35130 88028
# echo $$ $BASHPID | cat
35130 88030
Apparently $BASHPID is not expanded at the same time as $$. According to man bash:
BASHPID
Expands to the process ID of the current bash process. This differs from $$ under certain circumstances, such as subshells that do not require bash to be re-initialized.
This implies $BASHPID is not expanded the time Bash parses the command line otherwise it'll be the same as $$. And in Bash source code there's a func initialize_dynamic_variables() (in file variables.c):
1905 static void
1906 initialize_dynamic_variables ()
1907 {
1908 SHELL_VAR *v;
1909
1910 v = init_seconds_var ();
1911
1912 INIT_DYNAMIC_VAR ("BASH_ARGV0", (char *)NULL, get_bash_argv0, assign_bash_argv0);
....
....
1924 INIT_DYNAMIC_VAR ("BASHPID", (char *)NULL, get_bashpid, null_assign);
1925 VSETATTR (v, att_integer);
....
As it shows, vars like BASHPID are called dynamic variables. I guess these vars are handled specially and are expanded in the last minute when it knows that no more sub-shell will be forked (the fork may be followed by exec(), e.g., to run external commands).
I generally have -e set in my Bash scripts, but occasionally I would like to run a command and get the return value.
Without doing the set +e; some-command; res=$?; set -e dance, how can I do that?
From the bash manual:
The shell does not exit if the command that fails is [...] part of any command executed in a && or || list [...].
So, just do:
#!/bin/bash
set -eu
foo() {
# exit code will be 0, 1, or 2
return $(( RANDOM % 3 ))
}
ret=0
foo || ret=$?
echo "foo() exited with: $ret"
Example runs:
$ ./foo.sh
foo() exited with: 1
$ ./foo.sh
foo() exited with: 0
$ ./foo.sh
foo() exited with: 2
This is the canonical way of doing it.
as an alternative
ans=0
some-command || ans=$?
Maybe try running the commands in question in a subshell, like this?
res=$(some-command > /dev/null; echo $?)
Given behavior of shell described at this question it's possible to use following construct:
#!/bin/sh
set -e
{ custom_command; rc=$?; } || :
echo $rc
Another option is to use simple if. It is a bit longer, but fully supported by bash, i.e. that the command can return non-zero value, but the script doesn't exit even with set -e. See it in this simple script:
#! /bin/bash -eu
f () {
return 2
}
if f;then
echo Command succeeded
else
echo Command failed, returned: $?
fi
echo Script still continues.
When we run it, we can see that script still continues after non-zero return code:
$ ./test.sh
Command failed, returned: 2
Script still continues.
Use a wrapper function to execute your commands:
function __e {
set +e
"$#"
__r=$?
set -e
}
__e yourcommand arg1 arg2
And use $__r instead of $?:
if [[ __r -eq 0 ]]; then
echo "success"
else
echo "failed"
fi
Another method to call commands in a pipe, only that you have to quote the pipe. This does a safe eval.
function __p {
set +e
local __A=() __I
for (( __I = 1; __I <= $#; ++__I )); do
if [[ "${!__I}" == '|' ]]; then
__A+=('|')
else
__A+=("\"\$$__I\"")
fi
done
eval "${__A[#]}"
__r=$?
set -e
}
Example:
__p echo abc '|' grep abc
And I actually prefer this syntax:
__p echo abc :: grep abc
Which I could do with
...
if [[ ${!__I} == '::' ]]; then
...
I have some scripts that work with parameters, they work just fine but i would like them to be able to read from stdin, from a pipe for example, an example, suppose this is called read:
#!/bin/bash
function read()
{
echo $*
}
read $*
Now this works with read "foo" "bar", but I would like to use it as:
echo "foo" | read
How do I accomplish this?
It's a little tricky to write a function which can read standard input, but works properly when no standard input is given. If you simply try to read from standard input, it will block until it receives any, much like if you simply type cat at the prompt.
In bash 4, you can work around this by using the -t option to read with an argument of 0. It succeeds if there is any input available, but does not consume any of it; otherwise, it fails.
Here's a simple function that works like cat if it has anything from standard input, and echo otherwise.
catecho () {
if read -t 0; then
cat
else
echo "$*"
fi
}
$ catecho command line arguments
command line arguments
$ echo "foo bar" | catecho
foo bar
This makes standard input take precedence over command-line arguments, i.e., echo foo | catecho bar would output foo. To make arguments take precedence over standard input (echo foo | catecho bar outputs bar), you can use the simpler function
catecho () {
if [ $# -eq 0 ]; then
cat
else
echo "$*"
fi
}
(which also has the advantage of working with any POSIX-compatible shell, not just certain versions of bash).
You can use <<< to get this behaviour. read <<< echo "text" should make it.
Test with readly (I prefer not using reserved words):
function readly()
{
echo $*
echo "this was a test"
}
$ readly <<< echo "hello"
hello
this was a test
With pipes, based on this answer to "Bash script, read values from stdin pipe":
$ echo "hello bye" | { read a; echo $a; echo "this was a test"; }
hello bye
this was a test
To combine a number of other answers into what worked for me (this contrived example turns lowercase input to uppercase):
uppercase() {
local COMMAND='tr [:lower:] [:upper:]'
if [ -t 0 ]; then
if [ $# -gt 0 ]; then
echo "$*" | ${COMMAND}
fi
else
cat - | ${COMMAND}
fi
}
Some examples (the first has no input, and therefore no output):
:; uppercase
:; uppercase test
TEST
:; echo test | uppercase
TEST
:; uppercase <<< test
TEST
:; uppercase < <(echo test)
TEST
Step by step:
test if file descriptor 0 (/dev/stdin) was opened by a terminal
if [ -t 0 ]; then
tests for CLI invocation arguments
if [ $# -gt 0 ]; then
echo all CLI arguments to command
echo "$*" | ${COMMAND}
else if stdin is piped (i.e. not terminal input), output stdin to command (cat - and cat are shorthand for cat /dev/stdin)
else
cat - | ${COMMAND}
Here is example implementation of sprintf function in bash which uses printf and standard input:
sprintf() { local stdin; read -d '' -u 0 stdin; printf "$#" "$stdin"; }
Example usage:
$ echo bar | sprintf "foo %s"
foo bar
This would give you an idea how function can read from standard input.
Late to the party here. Building off of #andy's answer, here's how I define my to_uppercase function.
if stdin is not empty, use stdin
if stdin is empty, use args
if args are empty, do nothing
to_uppercase() {
local input="$([[ -p /dev/stdin ]] && cat - || echo "$#")"
[[ -n "$input" ]] && echo "$input" | tr '[:lower:]' '[:upper:]'
}
Usages:
$ to_uppercase
$ to_uppercase abc
ABC
$ echo abc | to_uppercase
ABC
$ to_uppercase <<< echo abc
ABC
Bash version info:
$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
I've discovered that this can be done in one line using test and awk...
test -p /dev/stdin && awk '{print}' /dev/stdin
The test -p tests for input on a pipe, which accepts input via stdin. Only if input is present do we want to run the awk since otherwise it will hang indefinitely waiting for input which will never come.
I've put this into a function to make it easy to use...
inputStdin () {
test -p /dev/stdin && awk '{print}' /dev/stdin && return 0
### accepts input if any but does not hang waiting for input
#
return 1
}
Usage...
_stdin="$(inputStdin)"
Another function uses awk without the test to wait for commandline input...
inputCli () {
local _input=""
local _prompt="$1"
#
[[ "$_prompt" ]] && { printf "%s" "$_prompt" > /dev/tty; }
### no prompt at all if none supplied
#
_input="$(awk 'BEGIN {getline INPUT < "/dev/tty"; print INPUT}')"
### accept input (used in place of 'read')
### put in a BEGIN section so will only accept 1 line and exit on ENTER
### WAITS INDEFINITELY FOR INPUT
#
[[ "$_input" ]] && { printf "%s" "$_input"; return 0; }
#
return 1
}
Usage...
_userinput="$(inputCli "Prompt string: ")"
Note that the > /dev/tty on the first printf seems to be necessary to get the prompt to print when the function is called in a Command Substituion $(...).
This use of awk allows the elimination of the quirky read command for collecting input from keyboard or stdin.
Yet another version that:
works by passing text through a pipe or from arguments
easy to copy and paste by changing command in last line
works in bash, zsh
# Prints a text in a decorated ballon
function balloon()
{
(test -p /dev/stdin && cat - || echo $#) figlet -t | cowsay -n -f eyes | toilet -t --gay -f term
}
Usage:
# Using with a pipe
$ fortune -s | balloon
# Passing text as parameter
balloon "$(fortune -s )"
To test if a variable is read-only, there are the following ugly hacks:
# True if readonly
readonly -p | egrep "declare -[:lower:]+ ${var}="
# False if readonly
temp="$var"; eval $var=x 2>/dev/null && eval $var=\$temp
Is there a more elegant solution?
Using a subshell seems to work. Both with local and exported variables.
$ foo=123
$ bar=456
$ readonly foo
$ echo $foo $bar
123 456
$ (unset foo 2> /dev/null) || echo "Read only"
Read only
$ (unset bar 2> /dev/null) || echo "Read only"
$
$ echo $foo $bar
123 456 # Still intact :-)
The important thing is that even is that the subshell salvages your RW ($bar in this case) from being unset in your current shell.
Tested with bash and ksh.
You can also add an empty string to the variable, which still leaves its value alone, but is faster than using a subshell, e.g.:
foo+= 2>/dev/null || echo "Read only"
Captured as a function, it'd be:
is-writable() { eval "$1+=" >2/dev/null; }
In bash/ksh can we add timestamp to STDERR redirection?
E.g. myscript.sh 2> error.log
I want to get a timestamp written on the log too.
If you're talking about an up-to-date timestamp on each line, that's something you'd probably want to do in your actual script (but see below for a nifty solution if you have no power to change it). If you just want a marker date on its own line before your script starts writing, I'd use:
( date 1>&2 ; myscript.sh ) 2>error.log
What you need is a trick to pipe stderr through another program that can add timestamps to each line. You could do this with a C program but there's a far more devious way using just bash.
First, create a script which will add the timestamp to each line (called predate.sh):
#!/bin/bash
while read line ; do
echo "$(date): ${line}"
done
For example:
( echo a ; sleep 5 ; echo b ; sleep 2 ; echo c ) | ./predate.sh
produces:
Fri Oct 2 12:31:39 WAST 2009: a
Fri Oct 2 12:31:44 WAST 2009: b
Fri Oct 2 12:31:46 WAST 2009: c
Then you need another trick that can swap stdout and stderr, this little monstrosity here:
( myscript.sh 3>&1 1>&2- 2>&3- )
Then it's simple to combine the two tricks by timestamping stdout and redirecting it to your file:
( myscript.sh 3>&1 1>&2- 2>&3- ) | ./predate.sh >error.log
The following transcript shows this in action:
pax> cat predate.sh
#!/bin/bash
while read line ; do
echo "$(date): ${line}"
done
pax> cat tstdate.sh
#!/bin/bash
echo a to stderr then wait five seconds 1>&2
sleep 5
echo b to stderr then wait two seconds 1>&2
sleep 2
echo c to stderr 1>&2
echo d to stdout
pax> ( ( ./tstdate.sh ) 3>&1 1>&2- 2>&3- ) | ./predate.sh >error.log
d to stdout
pax> cat error.log
Fri Oct 2 12:49:40 WAST 2009: a to stderr then wait five seconds
Fri Oct 2 12:49:45 WAST 2009: b to stderr then wait two seconds
Fri Oct 2 12:49:47 WAST 2009: c to stderr
As already mentioned, predate.sh will prefix each line with a timestamp and the tstdate.sh is simply a test program to write to stdout and stderr with specific time gaps.
When you run the command, you actually get "d to stdout" written to stderr (but that's your TTY device or whatever else stdout may have been when you started). The timestamped stderr lines are written to your desired file.
The devscripts package in Debian/Ubuntu contains a script called annotate-output which does that (for both stdout and stderr).
$ annotate-output make
21:41:21 I: Started make
21:41:21 O: gcc -Wall program.c
21:43:18 E: program.c: Couldn't compile, and took me ages to find out
21:43:19 E: collect2: ld returned 1 exit status
21:43:19 E: make: *** [all] Error 1
21:43:19 I: Finished with exitcode 2
Here's a version that uses a while read loop like pax's, but doesn't require extra file descriptors or a separate script (although you could use one). It uses process substitution:
myscript.sh 2> >( while read line; do echo "$(date): ${line}"; done > error.log )
Using pax's predate.sh:
myscript.sh 2> >( predate.sh > error.log )
The program ts from the moreutils package pipes standard input to standard output, and prefixes each line with a timestamp.
To prefix stdout lines: command | ts
To prefix both stdout and stderr: command 2>&1 | ts
I like those portable shell scripts but a little disturbed that they fork/exec date(1) for every line. Here is a quick perl one-liner to do the same more efficiently:
perl -p -MPOSIX -e 'BEGIN {$!=1} $_ = strftime("%T ", localtime) . $_'
To use it, just feed this command input through its stdin:
(echo hi; sleep 1; echo lo) | perl -p -MPOSIX -e 'BEGIN {$|=1} $_ = strftime("%T ", localtime) . $_'
Rather than writing a script to pipe to, I prefer to write the logger as a function inside the script, and then send the entirety of the process into it with brackets, like so:
# Vars
logfile=/path/to/scriptoutput.log
# Defined functions
teelogger(){
log=$1
while read line ; do
print "$(date +"%x %T") :: $line" | tee -a $log
done
}
# Start process
{
echo 'well'
sleep 3
echo 'hi'
sleep 3
echo 'there'
sleep 3
echo 'sailor'
} | teelogger $logfile
If you want to redirect back to stdout I found you have to do this:
myscript.sh >> >( while read line; do echo "$(date): ${line}"; done )
Not sure why I need the > in front of the (, so <(, but thats what works.
I was too lazy for all current solutions... So I figured out new one (works for stdout could be adjusted for stderr as well):
echo "output" | xargs -L1 -I{} bash -c "echo \$(date +'%x %T') '{}'" | tee error.log
would save to file and print something like that:
11/3/16 16:07:52 output
Details:
-L1 means "for each new line"
-I{} means "replace {} by input"
bash -c is used to update $(date) each time for new call
%x %T formats timestamp to minimal form.
It should work like a charm if stdout and stderr doesn't have quotes (" or `). If it has (or could have) it's better to use:
echo "output" | awk '{cmd="(date +'%H:%M:%S')"; cmd | getline d; print d,$0; close(cmd)} | tee error.log'
(Took from my answer in another topic: https://stackoverflow.com/a/41138870/868947)
How about timestamping the remaining output, redirecting all to stdout?
This answer combines some techniques from above, as well as from unix stackexchange here and here. bash >= 4.2 is assumed, but other advanced shells may work. For < 4.2, replace printf with a (slower) call to date.
: ${TIMESTAMP_FORMAT:="%F %T"} # override via environment
_loglines() {
while IFS= read -r _line ; do
printf "%(${TIMESTAMP_FORMAT})T#%s\n" '-1' "$_line";
done;
}
exec 7<&2 6<&1
exec &> >( _loglines )
# Logit
To restore stdout/stderr:
exec 1>&6 2>&7
You can then use tee to send the timestamps to stdout and a logfile.
_tmpfile=$(mktemp)
exec &> >( _loglines | tee $_tmpfile )
Not a bad idea to have cleanup code if the process exited without error:
trap "_cleanup \$?" 0 SIGHUP SIGINT SIGABRT SIGBUS SIGQUIT SIGTRAP SIGUSR1 SIGUSR2 SIGTERM
_cleanup() {
exec >&6 2>&7
[[ "$1" != 0 ]] && cat "$_logtemp"
rm -f "$_logtemp"
exit "${1:-0}"
}
#!/bin/bash
DEBUG=1
LOG=$HOME/script_${0##*/}_$(date +%Y.%m.%d-%H.%M.%S-%N).log
ERROR=$HOME/error.log
exec 2> $ERROR
exec 1> >(tee -ai $LOG)
if [ $DEBUG = 0 ] ; then
exec 4> >(xargs -i echo -e "[ debug ] {}")
else
exec 4> /dev/null
fi
# test
echo " debug sth " >&4
echo " log sth normal "
type -f this_is_error
echo " errot sth ..." >&2
echo " finish ..." >&2>&4
# close descriptor 4
exec 4>&-
This thing: nohup myscript.sh 2> >( while read line; do echo "$(date): ${line}"; done > mystd.err ) < /dev/null &
Works as such but when I log out and log back in to the server, it does not work. that is mystd.err stop getting populated with stderr stream even though my process (myscript.sh here) still runs.
Does someone know how to get back the lost stderr in the mystd.err file back?
Thought I would add my 2 cents worth..
#!/bin/sh
timestamp(){
name=$(printf "$1%*s" `expr 15 - ${#1}`)
awk "{ print strftime(\"%b %d %H:%M:%S\"), \"- $name -\", $$, \"- INFO -\", \$0; fflush() }";
}
echo "hi" | timestamp "process name" >> /tmp/proccess.log
printf "$1%*s" `expr 15 - ${#1}`
Spaces the name out so it looks nice, where 15 is the max space issued, increase if desired
outputs >> Date - Process name - Process ID - INFO - Message
Jun 27 13:57:20 - process name - 18866 - INFO - hi
Redirections are taken in order. Try this:
Given a script -
$: cat tst
echo a
sleep 2
echo 1 >&2
echo b
sleep 2
echo 2 >&2
echo c
sleep 2
echo 3 >&2
echo d
I get the following
$: ./tst 2>&1 1>stdout | sed 's/^/echo $(date +%Y%m%dT%H%M%S) /; e'
20180925T084008 1
20180925T084010 2
20180925T084012 3
And as much as I dislike awk, it does avoid the redundant subcalls to date.
$: ./tst 2>&1 1>stdout | awk "{ print strftime(\"%Y%m%dT%H%M%S \") \$0; fflush() }"; >stderr
$: cat stderr
20180925T084414 1
20180925T084416 2
20180925T084418 3
$: cat stdout
a
b
c
d