How to correctly pass to the script and substitute a variable that is already defined there?
My script test.sh:
#!/bin/bash
TARGETARCH=amd64
echo $1
When I enter:
bash test.sh https://example/$TARGETARCH
I want to see
https://example/amd64
but I actually see
https://example/
What am I doing wrong?
The first problem with the original approach is that the $TARGETARCH is removed by your calling shell before your script is ever invoked. To prevent that, you need to use quotes:
./yourscript 'https://example.com/$TARGETARCH'
The second problem is that parameter expansions only happen in code, not in data. This is, from a security perspective, a Very Good Thing -- if data were silently treated as code it would be impossible to write secure scripts handling untrusted data -- but it does mean you need to do some more work. The easy thing, in this case, is to export your variable and use GNU envsubst, as long as your operating system provides it:
#!/bin/bash
export TARGETARCH=amd64
substitutedValue=$(envsubst <<<"$1")
echo "Original value was: $1"
echo "Substituted value is: $substitutedValue"
See the above running in an online sandbox at https://replit.com/#CharlesDuffy2/EcstaticAfraidComputeranimation#replit.nix
Note the use of yourscript instead of test.sh here -- using .sh file extensions, especially for bash scripts as opposed to sh scripts, is an antipattern; the essay at https://www.talisman.org/~erlkonig/documents/commandname-extensions-considered-harmful/ has been linked by the #bash IRC channel on this topic for over a decade.
For similar reasons, changing bash yourscript to ./yourscript lets the #!/usr/bin/env bash line select an interpreter, so you aren't repeating the "bash" name in multiple places, leading to the risk of those places getting out of sync with each other.
Is there any way to make the script:
diff <(echo 'hello') <(echo 'hello-2')
work, as currently it fails with error in Dart when run using Process.run or Process.start:
syntax error near unexpected token `('
I tried to drill down on it and found out that since process substitution is not available when running the script through Process.run or Process.start, hence it's failing. So, is there any way to make it work?
I found out that we need to use set +o posix to make process substitution available if it's not, but I don't know how to do it in Dart.
According to the documentation and common expectations, the shell will be /bin/sh; and so, you can't use Bash features like process substitutions.
You can always force it by running Bash explicitly:
Process.run("bash", ["-c", "diff <(echo 'hello') <(echo 'hello-2')"])
You could add set +o posix; to the beginning of the argument to bash -c (the argument can be an arbitrarily complex string with multiple commands) but of course, it's not necessary here, and actually a red herring in this context.
Here's an example of a slightly more complex command which uses the Bash-only "C-style" for loop syntax.
Process.run("bash", ["-c", "for ((i=0; i<=255; i++)); do ping -c 3 10.9.8.$i || break; done; echo 'finished'"])
... though generally speaking, it's probably better to keep your subprocesses as simple as possible, and write any necessary plumbing in the host language.
Perhaps see also Difference between sh and bash
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Is there a way to debug a Bash script?
E.g., something that prints a sort of an execution log, like "calling line 1", "calling line 2", etc.
sh -x script [arg1 ...]
bash -x script [arg1 ...]
These give you a trace of what is being executed. (See also 'Clarification' near the bottom of the answer.)
Sometimes, you need to control the debugging within the script. In that case, as Cheeto reminded me, you can use:
set -x
This turns debugging on. You can then turn it off again with:
set +x
(You can find out the current tracing state by analyzing $-, the current flags, for x.)
Also, shells generally provide options '-n' for 'no execution' and '-v' for 'verbose' mode; you can use these in combination to see whether the shell thinks it could execute your script — occasionally useful if you have an unbalanced quote somewhere.
There is contention that the '-x' option in Bash is different from other shells (see the comments). The Bash Manual says:
-x
Print a trace of simple commands, for commands, case commands, select commands, and arithmetic for commands and their arguments or associated word lists after they are expanded and before they are executed. The value of the PS4 variable is expanded and the resultant value is printed before the command and its expanded arguments.
That much does not seem to indicate different behaviour at all. I don't see any other relevant references to '-x' in the manual. It does not describe differences in the startup sequence.
Clarification: On systems such as a typical Linux box, where '/bin/sh' is a symlink to '/bin/bash' (or wherever the Bash executable is found), the two command lines achieve the equivalent effect of running the script with execution trace on. On other systems (for example, Solaris, and some more modern variants of Linux), /bin/sh is not Bash, and the two command lines would give (slightly) different results. Most notably, '/bin/sh' would be confused by constructs in Bash that it does not recognize at all. (On Solaris, /bin/sh is a Bourne shell; on modern Linux, it is sometimes Dash — a smaller, more strictly POSIX-only shell.) When invoked by name like this, the 'shebang' line ('#!/bin/bash' vs '#!/bin/sh') at the start of the file has no effect on how the contents are interpreted.
The Bash manual has a section on Bash POSIX mode which, contrary to a long-standing but erroneous version of this answer (see also the comments below), does describe in extensive detail the difference between 'Bash invoked as sh' and 'Bash invoked as bash'.
When debugging a (Bash) shell script, it will be sensible and sane — necessary even — to use the shell named in the shebang line with the -x option. Otherwise, you may (will?) get different behaviour when debugging from when running the script.
I've used the following methods to debug my script.
set -e makes the script stop immediately if any external program returns a non-zero exit status. This is useful if your script attempts to handle all error cases and where a failure to do so should be trapped.
set -x was mentioned above and is certainly the most useful of all the debugging methods.
set -n might also be useful if you want to check your script for syntax errors.
strace is also useful to see what's going on. Especially useful if you haven't written the script yourself.
Jonathan Leffler's answer is valid and useful.
But, I find that the "standard" script debugging methods are inefficient, unintuitive, and hard to use. For those used to sophisticated GUI debuggers that put everything at your fingertips and make the job a breeze for easy problems (and possible for hard problems), these solutions aren't very satisfactory.
I do use a combination of DDD and bashdb. The former executes the latter, and the latter executes your script. This provides a multi-window UI with the ability to step through code in context and view variables, stack, etc., without the constant mental effort to maintain context in your head or keep re-listing the source.
There is guidance on setting that up in DDD and BASHDB.
I found the shellcheck utility and maybe some folks find it interesting.
A little example:
$ cat test.sh
ARRAY=("hello there" world)
for x in $ARRAY; do
echo $x
done
$ shellcheck test.sh
In test.sh line 3:
for x in $ARRAY; do
^-- SC2128: Expanding an array without an index only gives the first element.
Fix the bug. First try...
$ cat test.sh
ARRAY=("hello there" world)
for x in ${ARRAY[#]}; do
echo $x
done
$ shellcheck test.sh
In test.sh line 3:
for x in ${ARRAY[#]}; do
^-- SC2068: Double quote array expansions, otherwise they're like $* and break on spaces.
Let's try again...
$ cat test.sh
ARRAY=("hello there" world)
for x in "${ARRAY[#]}"; do
echo $x
done
$ shellcheck test.sh
Found now!
It's just a small example.
You can also write "set -x" within the script.
Install Visual Studio Code, and then add the Bash debug extension and you are ready to debug in visual mode. See it here in action.
Use Eclipse with the plugins Shelled and BashEclipse.
DVKit - Eclipse-based IDE for design verification tasks
BashEclipse
For Shelled: Download the ZIP file and import it into Eclipse via menu Help → Install new software: local archive. For BashEclipse: Copy the JAR files into the dropins directory of Eclipse
Follow the steps provided in BashEclipse files
I wrote a tutorial with many screenshots at Bash: enabling Eclipse for Bash Programming | Plugin Shelled (shell editor)
I built a Bash debugger: Bash Debuging Bash. Just give it a try.
set +x = #ECHO OFF, set -x = #ECHO ON.
You can add the -xv option to the standard shebang as follows:
#!/bin/bash -xv
-x : Display commands and their arguments as they are executed.
-v : Display shell input lines as they are read.
ltrace is another Linux utility similar to strace. However, ltrace lists all the library calls being called in an executable or a running process. Its name itself comes from library-call tracing.
For example:
ltrace ./executable <parameters>
ltrace -p <PID>
Source
I think you can try this Bash debugger: http://bashdb.sourceforge.net/.
Some trick to debug Bash scripts:
Using set -[nvx]
In addition to
set -x
and
set +x
for stopping dump.
I would like to speak about set -v which dump as smaller as less developed output.
bash <<<$'set -x\nfor i in {0..9};do\n\techo $i\n\tdone\nset +x' 2>&1 >/dev/null|wc -l
21
for arg in x v n nx nv nvx;do echo "- opts: $arg"
bash 2> >(wc -l|sed s/^/stderr:/) > >(wc -l|sed s/^/stdout:/) <<eof
set -$arg
for i in {0..9};do
echo $i
done
set +$arg
echo Done.
eof
sleep .02
done
- opts: x
stdout:11
stderr:21
- opts: v
stdout:11
stderr:4
- opts: n
stdout:0
stderr:0
- opts: nx
stdout:0
stderr:0
- opts: nv
stdout:0
stderr:5
- opts: nvx
stdout:0
stderr:5
Dump variables or tracing on the fly
For testing some variables, I use sometime this:
bash <(sed '18ideclare >&2 -p var1 var2' myscript.sh) args
for adding:
declare >&2 -p var1 var2
at line 18 and running the resulting script (with args), without having to edit them.
Of course, this could be used for adding set [+-][nvx]:
bash <(sed '18s/$/\ndeclare -p v1 v2 >\&2/;22s/^/set -x\n/;26s/^/set +x\n/' myscript) args
It will add declare -p v1 v2 >&2 after line 18, set -x before line 22 and set +x before line 26.
A little sample:
bash <(sed '2,3s/$/\ndeclare -p LINENO i v2 >\&2/;5s/^/set -x\n/;7s/^/set +x\n/' <(
seq -f 'echo $#, $((i=%g))' 1 8)) arg1 arg2
arg1 arg2, 1
arg1 arg2, 2
declare -i LINENO="3"
declare -- i="2"
/dev/fd/63: line 3: declare: v2: not found
arg1 arg2, 3
declare -i LINENO="5"
declare -- i="3"
/dev/fd/63: line 5: declare: v2: not found
arg1 arg2, 4
+ echo arg1 arg2, 5
arg1 arg2, 5
+ echo arg1 arg2, 6
arg1 arg2, 6
+ set +x
arg1 arg2, 7
arg1 arg2, 8
Note: Care about $LINENO. It will be affected by on-the-fly modifications!
(To see resulting script without executing, simply drop bash <( and ) arg1 arg2)
Step by step, execution time
Have a look at my answer about how to profile Bash scripts.
There's a good amount of detail on logging for shell scripts via the global variables of the shell. We can emulate a similar kind of logging in a shell script: Log tracing mechanism for shell scripts
The post has details on introducing log levels, like INFO, DEBUG, and ERROR. Tracing details like script entry, script exit, function entry, function exit.
Sample log:
It seems that the set -e option in Bash gets reset inside of functions, when those functions are invoked via a $(...) expansion.
This surprises me, and I'm not sure if it is a bug or not.
I have not been able to find a description of this behavior in the (usually quite thorough) Bash manpage.
Note: here are some other similar SO posts:
Bash functions ignore set -e
Using set -e / set +e in bash with functions
But neither of them deals with $(...), which is not really discussed in the manpage either.
I also cannot find reference to this issue in the excellent Bash FAQ 105.
Here is a small program to demonstrate the issue:
echo "Initial: $-"
set -eu
echo "After set: $-"
function foo() {
echo "Inside foo: $-"
}
foo
function bar() {
false # I'd expect this to immediately fail
echo "Inside bar: $-"
}
# When a $(...) construct is involved, 'bar' runs to completion!
x=$(bar)
echo "We should never get here ... but we do."
echo "$x"
For me, on Bash version 5.0.11(0)-release, I get the following output:
Initial: hB
After set: ehuB
Inside foo: ehuB
We should never get here ... but we do.
Inside bar: huB
So, as you can see, the -u option does get "passed through" to the function in all cases. And the -e option gets passed through when the function is called normally. But only in the special case of $(bar) does the -e option get reset.
Does anyone know if this is documented behavior, or otherwise explainable?
It makes no sense to me (:
The behaviour of set -e in conjunction with Command Substitution is documented in
Command Execution Environment:
Subshells spawned to execute command substitutions inherit the value of the -e option from the parent shell. When not in POSIX mode, Bash clears the -e option in such subshells.
That seems to say that the behaviour you see is expected — unless you're running in POSIX mode, the -e option is unset in command substitution subshells in Bash (even though the -e setting is initially inherited, it is changed soon after the subshell commences execution). It's a funny way of writing it, though.
The relevant man page quotes for you, first from the Command Substiution section.
Command substitution allows the output of a command to replace the command name. There are two forms:
$(command)
or
`command`
Bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command, with any trailing newlines deleted.
And from Command Execution Environment:
Subshells spawned to execute command substitutions inherit the value of the -e option from the parent shell. When not in posix mode, bash clears the -e option in such subshells.
So bar get executed in a subshell, and since your are not in posix mode, the -e option gets cleared.
Add set -o posix to the start of you script and it will behave as expected, although expect other differences when using this mode.
I am trying to do the following command in bash and dash:
x="env PATH=\"$PATH:/dir with space\""
cmd="ls"
"$x" $cmd
This fails with
-bash: env PATH="/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/usr/local/go/bin:/dir
with space": No such file or directory
Note the following works:
env PATH="$PATH:/dir with space" $cmd
The reason I am assigning to variable x env, is because it is part of a larger command wrapper to $cmd, which is also a complicated variable.
It's more complex than the initial example. I have logic in setting these variables,, instead of repeating them each time. Eventually the invocation is as shown here:
path_value="$PATH"
invocation="env PATH=\"$path_value\" $other_val1 $other_val2"
base="python python_script.py --opt1=a,b,c script_args"
add_on="$base more_arg1 more_arg2"
"$invocation" $base
You can use shell array to store and reuse:
x=(env PATH="$PATH:/dir with space")
cmd="ls"
"${x[#]}" "$cmd"
anubhava's helpful array-based bash answer is the best choice if you can assume only bash (which appeared to be the case initially).
Given that you must also support dash, which almost exclusively supports POSIX sh features only, arrays are not an option.
Assuming that you fully control or trust the values that you use to build your command line in a string, you can use eval, which should generally be avoided:
path_value="$PATH"
invocation="env PATH=\"$path_value\" $other_val1 $other_val2"
base="python python_script.py --opt1=a,b,c script_args"
add_on="$base more_arg1 more_arg2"
eval "$invocation $add_on"