Is it possible to add spaces to the left of every output to stdout (and stderr if possible) when I run commands in a bash shell script?
I'd like to do something like:
#!/bin/bash
echo Installing: Something
echo " => installing prerequisite1"
## INSERT MAGICAL LEFT SPACES COMMAND HERE ##
apt-get install -q -y prerequisite
## ANOTHER MAGICAL CANCELLING LEFT SPACES COMMAND HERE ##
echo " => installing prerequisite2"
# ... the padding again ...
wget http://abc.com/lostzilla.tar.gz
tar vzxf lostzilla.tar.gz
cd lostzilla-1.01
./configure
make && make install
# ... end of padding ...
echo Done.
Any idea?
EDIT: Added quotes to the echo command, otherwise they won't be padded.
Yes, you can quote them for simple things:
echo ' => installing prerequisite1'
and pipe the output through sed for complex things:
tar vzxf lostzilla.tar.gz 2>&1 | sed 's/^/ /'
The 2>&1 puts stdout and stderr onto the stdout stream and the sed replaces every start-of-line marker with three spaces.
How well this will work on something like wget which does all sorts of cursor manipulations I'm not sure.
Example shown here:
pax> ls -1 p*
phase1.py
phase1.sh
phase2.py
phase2.sh
primes.c
primes.exe
primes.sh
primes.stat
pax> ls -1 p* | sed 's/^/ /'
phase1.py
phase1.sh
phase2.py
phase2.sh
primes.c
primes.exe
primes.sh
primes.stat
One trick I've used in the past is to ensure that the scripts themselves take care of the indentation:
#!/bin/bash
if [[ "${DONT_EVER_SET_THIS_VAR}" = "" ]] ; then
export DONT_EVER_SET_THIS_VAR=except_for_here
$0 | sed 's/^/ /'
exit
fi
ls -1 p*
This will re-run the script with indentation through sed if it's not already doing so. That way, you don't have to worry about changing all your output statements. A bit of a hack, I know, but I tend to just do what's necessary for quick-and-dirty shell scripts.
If you want to turn spacing on and off, use the following awk script:
#!/usr/bin/gawk -f
/^#SPACEON/ { spaces=1; }
/^#SPACEOFF/ { spaces=0; }
!/^#SPACE/ {
if(spaces) {
print " " $0;
} else {
print $0;
}
}
Note that there are slight problems with your bash scipt. Notably, the use of => in your echo statements will output the character = to the file "installing".
#!/bin/bash
echo Installing: Something
echo '=> installing prerequisite1'
echo '#SPACEON'
echo You would see apt-get install -q -y prerequisite
echo '#SPACEOFF'
echo '=> installing prerequisite2'
echo '#SPACEON'
echo You would see wget http://abc.com/lostzilla.tar.gz
echo You would see tar vzxf lostzilla.tar.gz
echo You would see cd lostzilla-1.01
echo You would see ./configure
echo You would see make \&\& make install
echo '#SPACEOFF'
echo Done.
Combining the two gives me:
$ ./do-stuff | ./magic-spacing
Installing: Something
=> installing prerequisite1
You would see apt-get install -q -y prerequisite
=> installing prerequisite2
You would see wget http://abc.com/lostzilla.tar.gz
You would see tar vzxf lostzilla.tar.gz
You would see cd lostzilla-1.01
You would see ./configure
You would see make && make install
Done.
Where do-stuff is your bash script and magic-spacing is my awk script above.
Depending on how the command writes to stdout, you can just indent with a simple awk script:
$ echo -e 'hello\nworld' | awk '{print " ",$0}'
hello
world
Quite un-magical you can use printf to do the following:
# space padding for single string
printf "%-4s%s\n" "" "=> installing prerequisite1"
# space padding for single command output
# use of subshell leaves original IFS intact
( IFS=$'\n'; printf " %s\n" $(command ls -ld * 2>&1) )
# note: output to stderr is unbuffered
( IFS=$'\n'; printf " %s\n" $(command ls -ld * 1>&2) )
It's also possible to group commands by enclosing them in curly braces and space-padd their output like so:
{
cmd1 1>&2
cmd2 1>&2
cmd3 1>&2
} 2>&1 | sed 's/.*/ &/'
It's possible to redirect stdout to stderr script/shell-wide using exec ...
(
exec 1>&2
command ls -ld *
) 2>&1 | sed 's/^/ /'
Use python pyp (The Pyed Piper):
ls -ld | pyp "' '+p"
Related
I would like to run commands such as "history" or "!23" off of a pipe.
How might I achieve this?
Why does the following command not work?
echo "history" | xargs eval $1
To answer (2) first:
history and eval are both bash builtins. So xargs cannot run either of them.
xargs does not use $1 arguments. man xargs for the correct syntax.
For (1), it doesn't really make much sense to do what you are attempting because shell history is not likely to be synchronised between invocations, but you could try something like:
{ echo 'history'; echo '!23'; } | bash -i
or:
{ echo 'history'; echo '!23'; } | while read -r cmd; do eval "$cmd"; done
Note that pipelines run inside subshells. Environment changes are not retained:
x=1; echo "x=2" | while read -r cmd; do eval "$cmd"; done; echo "$x"
You can try like this
First redirect the history commands to a file (cut out the line numbers)
history | cut -c 8- > cmd.txt
Now Create this script hcmd.sh(Referred to this Read a file line by line assigning the value to a variable)
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
$line
done < "cmd.txt"
Run it like this
./hcmd.sh
I have an executable that accepts queries from stdin and responds to them, reading until EOF. Additionally I have an input file and a special command, let's call those EXEC, FILE and CMD respectively.
What I need to do is:
Pass FILE to EXEC as input.
Disregard all the output corresponding to commands read from FILE (/dev/null/).
Pass CMD as the last command.
Fetch output for the last command and save it in a variable.
EXEC's output can be multiline for each query.
I know how to pass FILE + CMD into the EXEC:
echo ${CMD} | cat ${FILE} - | ${EXEC}
but I have no idea how to fetch only output resulting from CMD.
Is there a magical one-liner that does this?
After looking around I've found the following partial solution:
mkfifo mypipe
(tail -f mypipe) | ${EXEC} &
cat ${FILE} | while read line; do
echo ${line} > mypipe
done
echo ${CMD} > mypipe
This allows me to redirect my input, but now the output gets printed to screen. I want to ignore all the output produced by EXEC in the while loop and get only what it prints for the last line.
I tried what first came into my mind, which is:
(tail -f mypipe) | ${EXEC} > somefile &
But it didn't work, the file was empty.
This is race-prone -- I'd suggest putting in a delay after the kill, or using an explicit sigil to determine when it's been received. That said:
#!/usr/bin/env bash
# route FD 4 to your output routine
exec 4> >(
output=; trap 'output=1' USR1
while IFS= read -r line; do
[[ $output ]] && printf '%s\n' "$line"
done
); out_pid=$!
# Capture the PID for the process substitution above; note that this requires a very
# new version of bash (4.4?)
[[ $out_pid ]] || { echo "ERROR: Your bash version is too old" >&2; exit 1; }
# Run your program in another process substitution, and close the parent's handle on FD 4
exec 3> >("$EXEC" >&4) 4>&-
# cat your file to FD 3...
cat "$file" >&3
# UGLY HACK: Wait to let your program finish flushing output from those commands
sleep 0.1
# notify the subshell writing output to disk that the ignored input is done...
kill -USR1 "$out_pid"
# UGLY HACK: Wait to let the subprocess actually receive the signal and set output=1
sleep 0.1
# ...and then write the command for which you actually want content logged.
echo "command" >&3
In validating this answer, I'm doing the following:
EXEC=stub_function
stub_function() {
local count line
count=0
while IFS= read -r line; do
(( ++count ))
printf '%s: %s\n' "$count" "$line"
done
}
cat >file <<EOF
do-not-log-my-output-1
do-not-log-my-output-2
do-not-log-my-output-3
EOF
file=file
export -f stub_function
export file EXEC
Output is only:
4: command
You could pipe it into a sed:
var=$(YOUR COMMAND | sed '$!d')
This will put only the last line into the variable
I think, that your proram EXEC does something special (open connection or remember state). When that is not the case, you can use
${EXEC} < ${FILE} > /dev/null
myvar=$(echo ${CMD} | ${EXEC})
Or with normal commands:
# Do not use (printf "==%s==\n" 1 2 3 ; printf "oo%soo\n" 4 5 6) | cat
printf "==%s==\n" 1 2 3 | cat > /dev/null
myvar=$(printf "oo%soo\n" 4 5 6 | cat)
When you need to give all input to one process, perhaps you can think of a marker that you can filter on:
(printf "==%s==\n" 1 2 3 ; printf "%s\n" "marker"; printf "oo%soo\n" 4 5 6) | cat | sed '1,/marker/ d'
You should examine your EXEC what could be used. When it is running SQL, you might use something like
(cat ${FILE}; echo 'select "DamonMarker" from dual;' ; echo ${CMD} ) |
${EXEC} | sed '1,/DamonMarker/ d'
and write this in a var with
myvar=$( (cat ${FILE}; echo 'select "DamonMarker" from dual;' ; echo ${CMD} ) |
${EXEC} | sed '1,/DamonMarker/ d' )
There are a few layers here, so bear with me.
My docker-container ssh -c"echo 'YAY!'; exit 25;" command executes echo 'YAY!'; exit 25; in my docker container. It returns:
YAY
error: message=YAY!
, code=25
I need to know if the command within the container was successful, so I append the following to the command:
docker-container ssh -c"echo 'YAY!'; exit 25;" >&1 2>/tmp/stderr; cat /tmp/stderr | grep 'code=' | cut -d'=' -f2 | { read exitStatus; echo $exitStatus; }
This sends the stderr to /tmp/stderr and, with the echo $exitStatus returns:
YAY!
25
So, this is exactly what I want. I want the $exitStatus saved to a variable. My problem is, I am placing this into a bash script (GIT pre-commit) and when this exact code is executed, the exit status is null.
Here is my bash script:
# .git/hooks/pre-commit
if [ -z ${DOCKER_MOUNT+x} ];
then
docker-container ssh -c"echo 'YAY!'; exit 25;" >&1 2>/tmp/stderr; cat /tmp/stderr | grep 'code=' | cut -d'=' -f2 | { read exitStatus; echo $exitStatus; }
exit $exitStatus;
else
echo "Container detected!"
fi;
That's because you're setting the variable in a pipeline. Each command in the pipeline is run in a subshell, and when the subshell exits the variable are no longer available.
bash allows you to run the pipeline's last command in the current shell, but you also have to turn off job control
An example
# default bash
$ echo foo | { read x; echo x=$x; } ; echo x=$x
x=foo
x=
# with "lastpipe" configuration
$ set +m; shopt -s lastpipe
$ echo foo | { read x; echo x=$x; } ; echo x=$x
x=foo
x=foo
Add set +m; shopt -s lastpipe to your script and you should be good.
And as Charles comments, there are more efficient ways to do it. Like this:
source <(docker-container ssh -c "echo 'YAY!'; exit 25;" 2>&1 1>/dev/null | awk -F= '/code=/ {print "exitStatus=" $2}')
echo $exitStatus
I had an issue with a ksh93 code. As it was very complex I started reducing it to an example code that will reproduce the same issue. I ended up with this:
set -o pipefail;
{
echo "progress" 1>&3 | false
} 3>&1 | cat | \
while read pv_output; do
echo "meanwhile ... we got "
echo $pv_output | cat
done
echo $?
When I run this code with ksh93 it outputs "0", when I run it with bash it outputs "1".
# echo "ksh93";ksh93 ./x1.sh ;echo "bash"; bash ./x1.sh
ksh93
meanwhile ... we got
progress
0
bash
meanwhile ... we got
progress
1
However if I start fiddling with the code, and remove the first cat, both shells return "1"
set -o pipefail;
{
echo "progress" 1>&3 | false
} 3>&1 | \
while read pv_output; do
echo "meanwhile ... we got "
echo $pv_output | cat
done
echo $?
Or... if I leave in the first cat, but I remove the second one from inside the while, they work the same way again.
set -o pipefail;
{
echo "progress" 1>&3 | false
} 3>&1 | cat | \
while read pv_output; do
echo "meanwhile ... we got "
echo $pv_output
done
echo $?
Now, for this example I used cat for simplicity's sake. In real life the first cat is actually an awk processing the output of a complicated command. The second cat is actually a sed. I mention these, so that it is clear that the cat command in itself is not the culprit.
Normally in my bash scripts I'm used to do some_command >> log.log. This works fine, however how can I append more data like time and command name?
My goal is to have a log like this
2012-01-01 00:00:01 [some_command] => some command output...
2012-01-01 00:01:01 [other_command] => other command output...
The processes should running and writing to the file concurrently.
The final solution, pointed by William Pursell in my case would be:
some_command 2>&1 | perl -ne '$|=1; print localtime . ": [somme_command] $_"' >> /log.log &
I also added 2>&1 to redirect the STDOUTand STDERR to the file and an & on the end to keep the program on background.
Thank you!
Given your comments, it seems that you want multiple processes to be writing to the file concurrently, and have a timestamp on each individual line. Something like this might suffice:
some_cmd | perl -ne '$|=1; print localtime . ": [some_cmd] $_"' >> logfile
If you want to massage the format of the date, use POSIX::strftime
some_cmd | perl -MPOSIX -ne 'BEGIN{ $|=1 }
print strftime( "%Y-%m-%d %H:%M:%S", localtime ) . " [some_cmd] $_"' >> logfile
An alternative solution using sed would be:
some_command 2>&1 | sed "s/^/`date '+%Y-%m-%d %H:%M:%S'`: [some_command] /" >> log.log &
It works by replacing the beginning of line "character" (^). Might come in handy if you don't want to depend on Perl.
On Ubuntu:
sudo apt-get install moreutils
echo "cat" | ts
Mar 26 09:43:00 cat
something like this:
(echo -n $(date); echo -n " ls => "; ls) >> /tmp/log
however, your command output is multiple lines and it will not have the format above you are showing. you may want to replace the newline in output with some other character with a command like tr or sed in that case.
One approach is to use logger(1).
Another might be something like this:
stamp () {
( echo -n "`date +%T` "
"$#" ) >> logfile
}
stamp echo how now brown cow
A better alternative using GNU sed would be:
some_command 2>&1 | sed 'h; s/.*/date "+%Y-%m-%d %H:%M:%S"/e; G; s/\n/ [some_command]: /'
Breaking down how this sed program works:
# store the current line (a.k.a. "pattern space") in the "hold space"
h;
# replace the current line with the date command and execute it
# note the e command is available in GNU sed, but not in some other version
s/.*/date "+%Y-%m-%d %H:%M:%S"/e;
# Append a newline and the contents of the "hold space" to the "pattern space"
G;
# Replace that newline inserted by the G command with whatever text you want
s/\n/ [some_command]: /