In a shell script, how do I echo all shell commands called and expand any variable names?
For example, given the following line:
ls $DIRNAME
I would like the script to run the command and display the following
ls /full/path/to/some/dir
The purpose is to save a log of all shell commands called and their arguments. Is there perhaps a better way of generating such a log?
set -x or set -o xtrace expands variables and prints a little + sign before the line.
set -v or set -o verbose does not expand the variables before printing.
Use set +x and set +v to turn off the above settings.
On the first line of the script, one can put #!/bin/sh -x (or -v) to have the same effect as set -x (or -v) later in the script.
The above also works with /bin/sh.
See the bash-hackers' wiki on set attributes, and on debugging.
$ cat shl
#!/bin/bash
DIR=/tmp/so
ls $DIR
$ bash -x shl
+ DIR=/tmp/so
+ ls /tmp/so
$
set -x will give you what you want.
Here is an example shell script to demonstrate:
#!/bin/bash
set -x #echo on
ls $PWD
This expands all variables and prints the full commands before output of the command.
Output:
+ ls /home/user/
file1.txt file2.txt
I use a function to echo and run the command:
#!/bin/bash
# Function to display commands
exe() { echo "\$ $#" ; "$#" ; }
exe echo hello world
Which outputs
$ echo hello world
hello world
For more complicated commands pipes, etc., you can use eval:
#!/bin/bash
# Function to display commands
exe() { echo "\$ ${#/eval/}" ; "$#" ; }
exe eval "echo 'Hello, World!' | cut -d ' ' -f1"
Which outputs
$ echo 'Hello, World!' | cut -d ' ' -f1
Hello
You can also toggle this for select lines in your script by wrapping them in set -x and set +x, for example,
#!/bin/bash
...
if [[ ! -e $OUT_FILE ]];
then
echo "grabbing $URL"
set -x
curl --fail --noproxy $SERV -s -S $URL -o $OUT_FILE
set +x
fi
shuckc's answer for echoing select lines has a few downsides: you end up with the following set +x command being echoed as well, and you lose the ability to test the exit code with $? since it gets overwritten by the set +x.
Another option is to run the command in a subshell:
echo "getting URL..."
( set -x ; curl -s --fail $URL -o $OUTFILE )
if [ $? -eq 0 ] ; then
echo "curl failed"
exit 1
fi
which will give you output like:
getting URL...
+ curl -s --fail http://example.com/missing -o /tmp/example
curl failed
This does incur the overhead of creating a new subshell for the command, though.
According to TLDP's Bash Guide for Beginners: Chapter 2. Writing and debugging scripts:
2.3.1. Debugging on the entire script
$ bash -x script1.sh
...
There is now a full-fledged debugger for Bash, available at SourceForge. These debugging features are available in most modern versions of Bash, starting from 3.x.
2.3.2. Debugging on part(s) of the script
set -x # Activate debugging from here
w
set +x # Stop debugging from here
...
Table 2-1. Overview of set debugging options
Short | Long notation | Result
-------+---------------+--------------------------------------------------------------
set -f | set -o noglob | Disable file name generation using metacharacters (globbing).
set -v | set -o verbose| Prints shell input lines as they are read.
set -x | set -o xtrace | Print command traces before executing command.
...
Alternatively, these modes can be specified in the script itself, by
adding the desired options to the first line shell declaration.
Options can be combined, as is usually the case with UNIX commands:
#!/bin/bash -xv
Another option is to put "-x" at the top of your script instead of on the command line:
$ cat ./server
#!/bin/bash -x
ssh user#server
$ ./server
+ ssh user#server
user#server's password: ^C
$
You can execute a Bash script in debug mode with the -x option.
This will echo all the commands.
bash -x example_script.sh
# Console output
+ cd /home/user
+ mv text.txt mytext.txt
You can also save the -x option in the script. Just specify the -x option in the shebang.
######## example_script.sh ###################
#!/bin/bash -x
cd /home/user
mv text.txt mytext.txt
##############################################
./example_script.sh
# Console output
+ cd /home/user
+ mv text.txt mytext.txt
Type "bash -x" on the command line before the name of the Bash script. For instance, to execute foo.sh, type:
bash -x foo.sh
Combining all the answers I found this to be the best, simplest
#!/bin/bash
# https://stackoverflow.com/a/64644990/8608146
exe(){
set -x
"$#"
{ set +x; } 2>/dev/null
}
# example
exe go generate ./...
{ set +x; } 2>/dev/null from https://stackoverflow.com/a/19226038/8608146
If the exit status of the command is needed, as mentioned here
Use
{ STATUS=$?; set +x; } 2>/dev/null
And use the $STATUS later like exit $STATUS at the end
A slightly more useful one
#!/bin/bash
# https://stackoverflow.com/a/64644990/8608146
_exe(){
[ $1 == on ] && { set -x; return; } 2>/dev/null
[ $1 == off ] && { set +x; return; } 2>/dev/null
echo + "$#"
"$#"
}
exe(){
{ _exe "$#"; } 2>/dev/null
}
# examples
exe on # turn on same as set -x
echo This command prints with +
echo This too prints with +
exe off # same as set +x
echo This does not
# can also be used for individual commands
exe echo what up!
For zsh, echo
setopt VERBOSE
And for debugging,
setopt XTRACE
To allow for compound commands to be echoed, I use eval plus Soth's exe function to echo and run the command. This is useful for piped commands that would otherwise only show none or just the initial part of the piped command.
Without eval:
exe() { echo "\$ $#" ; "$#" ; }
exe ls -F | grep *.txt
Outputs:
$
file.txt
With eval:
exe() { echo "\$ $#" ; "$#" ; }
exe eval 'ls -F | grep *.txt'
Which outputs
$ exe eval 'ls -F | grep *.txt'
file.txt
For csh and tcsh, you can set verbose or set echo (or you can even set both, but it may result in some duplication most of the time).
The verbose option prints pretty much the exact shell expression that you type.
The echo option is more indicative of what will be executed through spawning.
http://www.tcsh.org/tcsh.html/Special_shell_variables.html#verbose
http://www.tcsh.org/tcsh.html/Special_shell_variables.html#echo
Special shell variables
verbose
If set, causes the words of each command to be printed, after history substitution (if any). Set by the -v command line option.
echo
If set, each command with its arguments is echoed just before it is executed. For non-builtin commands all expansions occur before echoing. Builtin commands are echoed before command and filename substitution, because these substitutions are then done selectively. Set by the -x command line option.
$ cat exampleScript.sh
#!/bin/bash
name="karthik";
echo $name;
bash -x exampleScript.sh
Output is as follows:
Related
In a script.sh,
source a.sh
source b.sh
CMD1
CMD2
CMD3
how can I replace the source *.sh with their content (without executing the commands)?
I would like to see what the bash interpreter executes after sourcing the files and expanding all variables.
I know I can use set -n -v or run bash -n -v script.sh 2>output.sh, but that would not replace the source commands (and even less if a.sh or b.sh contain variables).
I thought of using a subshell, but that still doesn't expand the source lines. I tried a combination of set +n +v and set -n -v before and after the source lines, but that still does not work.
I'm going to send that output to a remote machine using ssh.
I could use <<output.sh to pipe the content into the ssh command, but I can't log as root onto the remote machine, but I am however a sudoer.
Therefore, I thought I could create the script and send it as a base64-encoded string (using that clever trick )
base64 script | ssh remotehost 'base64 -d | sudo bash'
Is there a solution?
Or do you have a better idea?
You can do something like this:
inline.sh:
#!/usr/bin/env bash
while read line; do
if [[ "$line" =~ (\.|source)\s+.+ ]]; then
file="$(echo $line | cut -d' ' -f2)"
echo "$(cat $file)"
else
echo "$line"
fi
done < "$1"
Note this assumes the sourced files exist, and doesn't handle errors. You should also handle possible hashbangs. If the sourced files contain themselves source, you need to apply the script recursively, e.g. something like (not tested):
while egrep -q '^(source|\.)' main.sh; do
bash inline.sh main.sh > main.sh
done
Let's test it
main.sh:
source a.sh
. b.sh
echo cc
echo "$var_a $var_b"
a.sh:
echo aa
var_a="stack"
b.sh:
echo bb
var_b="overflow"
Result:
bash inline.sh main.sh
echo aa
var_a="stack"
echo bb
var_b="overflow"
echo cc
echo "$var_a $var_b"
bash inline.sh main.sh | bash
aa
bb
cc
stack overflow
BTW, if you just want to see what bash executes, you can run
bash -x [script]
or remotely
ssh user#host -t "bash -x [script]"
I'm writing a shell script that loops over some values and run a long command line for each value. I'd like to print out these commands along the way, just like make does when running a makefile. I know I could just "echo" all commands before running them, but it feels inelegant. So I'm looking at set -x and similar mechanisms instead :
#!/bin/sh
for value in a long list of values
do
set -v
touch $value # imagine a complicated invocation here
set +v
done
My problem is: at each iteration, not only is the interresting line printed out, but also the set +x line as well. Is it somehow possible to prevent that ? If not, what workaround do you recommend ?
PS: the MWE above uses sh, but I also have bash and zsh installed in case that helps.
Sandbox it in a subshell:
(set -x; do_thing_you_want_traced)
Of course, changes to variables or the environment made in that subshell will be lost.
If you REALLY care about this, you could also use a DEBUG trap (using set -T to cause it to be inherited by functions) to implement your own set -x equivalent.
For instance, if using bash:
trap_fn() {
[[ $DEBUG && $BASH_COMMAND != "unset DEBUG" ]] && \
printf "[%s:%s] %s\n" "$BASH_SOURCE" "$LINENO" "$BASH_COMMAND"
return 0 # do not block execution in extdebug mode
}
trap trap_fn DEBUG
DEBUG=1
# ...do something you want traced...
unset DEBUG
That said, emitting BASH_COMMAND (as a DEBUG trap can do) is not fully equivalent of set -x; for instance, it does not show post-expansion values.
You want to try using a single-line xtrace:
function xtrace() {
# Print the line as if xtrace was turned on, using perl to filter out
# the extra colon character and the following "set +x" line.
(
set -x
# Colon is a no-op in bash, so nothing will execute.
: "$#"
set +x
) 2>&1 | perl -ne 's/^[+] :/+/ and print' 1>&2
# Execute the original line unmolested
"$#"
}
The original command executes in the same shell under an identity transformation. Just prior to running, you get a non-recursive xtrace of the arguments. This allows you to xtrace the commands you care about without spamming stederr with duplicate copies of every "echo" command.
# Example
for value in $long_list; do
computed_value=$(echo "$value" | sed 's/.../...')
xtrace some_command -x -y -z $value $computed_value ...
done
Next command disables 'xtrace' option:
$ set +o xtrace
I thought of
set -x >/dev/null 2>1; echo 1; echo 2; set +x >/dev/null 2>&1
but got
+ echo 1
1
+ echo 2
2
+ 1> /dev/null 2>& 1
I'm surprised by these results. .... But
set -x ; echo 1; echo 2; set +x
+ echo 1
1
+ echo 2
2
looks to meet your requirement.
I saw similar results when I put each statement on its only line (excepting the set +x)
IHTH.
How many times have you seen someone trying to "Log the command I run and the output of the command"? It happens often, and for seeing the command you're running set -v is nice, (set -x is nice, too, but it can be harder to read), but what happens when you want to capture the command being run... but not all commands being run?
Running interactively I don't see a way to capture the set -v output at all.
set -v
echo a 1>/dev/null # 'echo a 1>/dev/null' is printed to the screen
echo a 2>/dev/null # 'echo a 2>/dev/null\na' is printed to the screen
I can put this in a script and things get better:
echo 'set -v'$'\n''echo a' > setvtest.sh
bash setvtest.sh 1>/dev/null # 'echo a' is printed to the screen
bash setvtest.sh 2>/dev/null # 'a' is printed to the screen
Aha, so from a script it goes to stderr. What about inline?
set +v
{ set -v ; echo a ; } 1>/dev/null # no output
set +v
( set -v ; echo a ; ) 1>/dev/null # no output
Hmm, no luck there.
Interestingly, and as a side note, this produces no output:
echo 'set -v ; echo a' > setvtest.sh
bash setvtest.sh 1>/dev/null
I'm not sure why, but perhaps that's also why the subshell version returns nothing.
What about shell functions?
setvtest2 () {
set -v
echo a
}
setvtest2 # 'a'
set +v
setvtest2 1>/dev/null # nothing
set +v
setvtest2 2>/dev/null # nothing
Now the question: Is there a nice way to capture the output of set -v?
Here's my not-nice hack, so I'm looking for something less insane:
#!/usr/bin/env bash
script=/tmp/$$.script
output=/tmp/$$.out
echo 'set -v'$'\n'"$1" >"$script"
bash "$script" 1>"$output"
cat "$output"
rm -f "$script" "$output"
Now I can execute simple scripts
bash gen.sh 'echo a' 1>/dev/null # prints 'echo a'
bash gen.sh 'echo a' 2>/dev/null # prints 'a'
But surely there are better ways.
You can run bash with option -v instead of turning on and off via set:
bash -v -c "echo a" 1>/dev/null # prints 'echo a'
bash -v -c "echo a" 2>/dev/null # prints 'a'
The dark side of this solution is that each such line will require to create new bash process, but you will not have to remember to switch off the v option back, since it's switched on only in a child process.
how about
#!/bin/bash
set -o xtrace
Stuff.....
I'm looking for a way to create a switch for this bash script so that I have the option of either printing (echo) it to stdout or executing the command for debugging purposes. As you can see below, I am just doing this manually by commenting out one statement over the other to achieve this.
Code:
#!/usr/local/bin/bash
if [ $# != 2 ]; then
echo "Usage: testcurl.sh <localfile> <projectname>" >&2
echo "sample:testcurl.sh /share1/data/20110818.dat projectZ" >&2
exit 1
fi
echo /usr/bin/curl -c $PROXY --certkey $CERT --header "Test:'${AUTH}'" -T $localfile $fsProxyURL
#/usr/bin/curl -c $PROXY --certkey $CERT --header "Test:'${AUTH}'" -T $localfile $fsProxyURL
I'm simply looking for an elegant/better way to create like a switch from the command line. Print or execute.
One possible trick, though it will only work for simple commands (e.g., no pipes or redirection (a)) is to use a prefix variable like:
pax> cat qq.sh
${PAXPREFIX} ls /tmp
${PAXPREFIX} printf "%05d\n" 72
${PAXPREFIX} echo 3
What this will do is to insert you specific variable (PAXPREFIX in this case) before the commands. If the variable is empty, it will not affect the command, as follows:
pax> ./qq.sh
my_porn.gz copy_of_the_internet.gz
00072
3
However, if it's set to echo, it will prefix each line with that echo string.
pax> PAXPREFIX=echo ./qq.sh
ls /tmp
printf %05d\n 72
echo 3
(a) The reason why it will only work for simple commands can be seen if you have something like:
${PAXPREFIX} ls -1 | tr '[a-z]' '[A-Z]'
When PAXPREFIX is empty, it will simply give you the list of your filenames in uppercase. When it's set to echo, it will result in:
echo ls -1 | tr '[a-z]' '[A-Z]'
giving:
LS -1
(not quite what you'd expect).
In fact, you can see a problem with even the simple case above, where %05d\n is no longer surrounded by quotes.
If you want a more robust solution, I'd opt for:
if [[ ${PAXDEBUG:-0} -eq 1 ]] ; then
echo /usr/bin/curl -c $PROXY --certkey $CERT --header ...
else
/usr/bin/curl -c $PROXY --certkey $CERT --header ...
fi
and use PAXDEBUG=1 myscript.sh to run it in debug mode. This is similar to what you have now but with the advantage that you don't need to edit the file to switch between normal and debug modes.
For debugging output from the shell itself, you can run it with bash -x or put set -x in your script to turn it on at a specific point (and, of course, turn it off with set +x).
#!/usr/local/bin/bash
if [[ "$1" == "--dryrun" ]]; then
echoquoted() {
printf "%q " "$#"
echo
}
maybeecho=echoquoted
shift
else
maybeecho=""
fi
if [ $# != 2 ]; then
echo "Usage: testcurl.sh <localfile> <projectname>" >&2
echo "sample:testcurl.sh /share1/data/20110818.dat projectZ" >&2
exit 1
fi
$maybeecho /usr/bin/curl "$1" -o "$2"
Try something like this:
show=echo
$show /usr/bin/curl ...
Then set/unset $show accordingly.
This does not directly answer your specific question, but I guess you're trying to see what command gets executed for debugging. If you replace #!/usr/local/bin/bash with #!/usr/local/bin/bash -x bash will run and echo the commands in your script.
I do not know of a way for "print vs execute" but I know of a way for "print and execute", and it is using "bash -x". See this link for example.
This must either be really simple or really complex, but I couldn't find anything about it... I am trying to open a new bash instance, then run a few commands inside it, and give the control back to the user inside that same instance.
I tried:
$ bash -lic "some_command"
but this executes some_command inside the new instance, then closes it. I want it to stay open.
One more detail which might affect answers: if I can get this to work I will use it in my .bashrc as alias(es), so bonus points for an alias implementation!
bash --rcfile <(echo '. ~/.bashrc; some_command')
dispenses the creation of temporary files. Question on other sites:
https://serverfault.com/questions/368054/run-an-interactive-bash-subshell-with-initial-commands-without-returning-to-the
https://unix.stackexchange.com/questions/123103/how-to-keep-bash-running-after-command-execution
This is a late answer, but I had the exact same problem and Google sent me to this page, so for completeness here is how I got around the problem.
As far as I can tell, bash does not have an option to do what the original poster wanted to do. The -c option will always return after the commands have been executed.
Broken solution: The simplest and obvious attempt around this is:
bash -c 'XXXX ; bash'
This partly works (albeit with an extra sub-shell layer). However, the problem is that while a sub-shell will inherit the exported environment variables, aliases and functions are not inherited. So this might work for some things but isn't a general solution.
Better: The way around this is to dynamically create a startup file and call bash with this new initialization file, making sure that your new init file calls your regular ~/.bashrc if necessary.
# Create a temporary file
TMPFILE=$(mktemp)
# Add stuff to the temporary file
echo "source ~/.bashrc" > $TMPFILE
echo "<other commands>" >> $TMPFILE
echo "rm -f $TMPFILE" >> $TMPFILE
# Start the new bash shell
bash --rcfile $TMPFILE
The nice thing is that the temporary init file will delete itself as soon as it is used, reducing the risk that it is not cleaned up correctly.
Note: I'm not sure if /etc/bashrc is usually called as part of a normal non-login shell. If so you might want to source /etc/bashrc as well as your ~/.bashrc.
You can pass --rcfile to Bash to cause it to read a file of your choice. This file will be read instead of your .bashrc. (If that's a problem, source ~/.bashrc from the other script.)
Edit: So a function to start a new shell with the stuff from ~/.more.sh would look something like:
more() { bash --rcfile ~/.more.sh ; }
... and in .more.sh you would have the commands you want to execute when the shell starts. (I suppose it would be elegant to avoid a separate startup file -- you cannot use standard input because then the shell will not be interactive, but you could create a startup file from a here document in a temporary location, then read it.)
bash -c '<some command> ; exec /bin/bash'
will avoid additional shell sublayer
You can get the functionality you want by sourcing the script instead of running it. eg:
$cat script
cmd1
cmd2
$ . script
$ at this point cmd1 and cmd2 have been run inside this shell
Append to ~/.bashrc a section like this:
if [ "$subshell" = 'true' ]
then
# commands to execute only on a subshell
date
fi
alias sub='subshell=true bash'
Then you can start the subshell with sub.
The accepted answer is really helpful! Just to add that process substitution (i.e., <(COMMAND)) is not supported in some shells (e.g., dash).
In my case, I was trying to create a custom action (basically a one-line shell script) in Thunar file manager to start a shell and activate the selected Python virtual environment. My first attempt was:
urxvt -e bash --rcfile <(echo ". $HOME/.bashrc; . %f/bin/activate;")
where %f is the path to the virtual environment handled by Thunar.
I got an error (by running Thunar from command line):
/bin/sh: 1: Syntax error: "(" unexpected
Then I realized that my sh (essentially dash) does not support process substitution.
My solution was to invoke bash at the top level to interpret the process substitution, at the expense of an extra level of shell:
bash -c 'urxvt -e bash --rcfile <(echo "source $HOME/.bashrc; source %f/bin/activate;")'
Alternatively, I tried to use here-document for dash but with no success. Something like:
echo -e " <<EOF\n. $HOME/.bashrc; . %f/bin/activate;\nEOF\n" | xargs -0 urxvt -e bash --rcfile
P.S.: I do not have enough reputation to post comments, moderators please feel free to move it to comments or remove it if not helpful with this question.
With accordance with the answer by daveraja, here is a bash script which will solve the purpose.
Consider a situation if you are using C-shell and you want to execute a command
without leaving the C-shell context/window as follows,
Command to be executed: Search exact word 'Testing' in current directory recursively only in *.h, *.c files
grep -nrs --color -w --include="*.{h,c}" Testing ./
Solution 1: Enter into bash from C-shell and execute the command
bash
grep -nrs --color -w --include="*.{h,c}" Testing ./
exit
Solution 2: Write the intended command into a text file and execute it using bash
echo 'grep -nrs --color -w --include="*.{h,c}" Testing ./' > tmp_file.txt
bash tmp_file.txt
Solution 3: Run command on the same line using bash
bash -c 'grep -nrs --color -w --include="*.{h,c}" Testing ./'
Solution 4: Create a sciprt (one-time) and use it for all future commands
alias ebash './execute_command_on_bash.sh'
ebash grep -nrs --color -w --include="*.{h,c}" Testing ./
The script is as follows,
#!/bin/bash
# =========================================================================
# References:
# https://stackoverflow.com/a/13343457/5409274
# https://stackoverflow.com/a/26733366/5409274
# https://stackoverflow.com/a/2853811/5409274
# https://stackoverflow.com/a/2853811/5409274
# https://www.linuxquestions.org/questions/other-%2Anix-55/how-can-i-run-a-command-on-another-shell-without-changing-the-current-shell-794580/
# https://www.tldp.org/LDP/abs/html/internalvariables.html
# https://stackoverflow.com/a/4277753/5409274
# =========================================================================
# Enable following line to see the script commands
# getting printing along with their execution. This will help for debugging.
#set -o verbose
E_BADARGS=85
if [ ! -n "$1" ]
then
echo "Usage: `basename $0` grep -nrs --color -w --include=\"*.{h,c}\" Testing ."
echo "Usage: `basename $0` find . -name \"*.txt\""
exit $E_BADARGS
fi
# Create a temporary file
TMPFILE=$(mktemp)
# Add stuff to the temporary file
#echo "echo Hello World...." >> $TMPFILE
#initialize the variable that will contain the whole argument string
argList=""
#iterate on each argument
for arg in "$#"
do
#if an argument contains a white space, enclose it in double quotes and append to the list
#otherwise simply append the argument to the list
if echo $arg | grep -q " "; then
argList="$argList \"$arg\""
else
argList="$argList $arg"
fi
done
#remove a possible trailing space at the beginning of the list
argList=$(echo $argList | sed 's/^ *//')
# Echoing the command to be executed to tmp file
echo "$argList" >> $TMPFILE
# Note: This should be your last command
# Important last command which deletes the tmp file
last_command="rm -f $TMPFILE"
echo "$last_command" >> $TMPFILE
#echo "---------------------------------------------"
#echo "TMPFILE is $TMPFILE as follows"
#cat $TMPFILE
#echo "---------------------------------------------"
check_for_last_line=$(tail -n 1 $TMPFILE | grep -o "$last_command")
#echo $check_for_last_line
#if tail -n 1 $TMPFILE | grep -o "$last_command"
if [ "$check_for_last_line" == "$last_command" ]
then
#echo "Okay..."
bash $TMPFILE
exit 0
else
echo "Something is wrong"
echo "Last command in your tmp file should be removing itself"
echo "Aborting the process"
exit 1
fi
$ bash --init-file <(echo 'some_command')
$ bash --rcfile <(echo 'some_command')
In case you can't or don't want to use process substitution:
$ cat script
some_command
$ bash --init-file script
Another way:
$ bash -c 'some_command; exec bash'
$ sh -c 'some_command; exec sh'
sh-only way (dash, busybox):
$ ENV=script sh
Here is yet another (working) variant:
This opens a new gnome terminal, then in the new terminal it runs bash. The user's rc file is read first, then a command ls -la is sent for execution to the new shell before it turns interactive.
The last echo adds an extra newline that is needed to finish execution.
gnome-terminal -- bash -c 'bash --rcfile <( cat ~/.bashrc; echo ls -la ; echo)'
I also find it useful sometimes to decorate the terminal, e.g. with colorfor better orientation.
gnome-terminal --profile green -- bash -c 'bash --rcfile <( cat ~/.bashrc; echo ls -la ; echo)'