eval not working in shellscript - shell

I am trying to get the value of a variable to be selected by name at runtime, using eval, but I don't get its value if - (hyphen) is in the name.
ENV=dev
REGION=us-east-1
DBUSERNAME=DB_USER_${ENV}_$REGION
DBPASSWORD=DB_PASS_${ENV}_$REGION
eval "USERNAME=\${${DBUSERNAME}}"
eval "PASSWORD=\${${DBPASSWORD}}"
echo USERNAME=$USERNAME
echo PASSWORD=$PASSWORD
RESULT
echo USERNAME=east-1
echo PASSWORD=east-1
EXPECTED RESULT
echo USERNAME=DB_USER_dev_us-east-1
echo PASSWORD=DB_USER_dev_us-east-1
It's working fine if there is no hyphen present in the name.

Investigation
We can see what's happening by running this in shell with -x option to trace execution:
$ sh -x ./36332134.sh
+ ENV=dev
+ REGION=us-east-1
+ DBUSERNAME=DB_USER_dev_us-east-1
+ DBPASSWORD=DB_PASS_dev_us-east-1
+ eval USERNAME=${DB_USER_dev_us-east-1}
+ USERNAME=east-1
+ eval PASSWORD=${DB_PASS_dev_us-east-1}
+ PASSWORD=east-1
+ echo USERNAME=east-1
USERNAME=east-1
+ echo PASSWORD=east-1
PASSWORD=east-1
Notice that eval USERNAME=${DB_USER_dev_us-east-1} gives us USERNAME=east-1. That's parameter expansion in effect, as described in the Bash manual:
When not performing substring expansion, using the forms documented below (e.g., :-), bash tests for a parameter that is unset
or null. Omitting the colon results in a test only for a parameter
that is unset.
${parameter:-word}
Use Default Values. If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter
is substituted.
Since $DB_USER_dev_us is unset, then the expansion of ${DB_USER_dev_us-east-1} is east-1.
Workarounds
Shell doesn't allow - in variable names (including environment variables). I guess DB_USER_dev_us-east-1 was set by some non-shell program? In which case, you'll need a similar non-shell program to retrieve it, I think. I tested quoting the -, but to no avail.
If you can use Bash as your shell, you might want to use an associative array instead of composing variable names.
If you are able to change the environment variables, you might consider changing the - to (say) _, then using (Bash) ${REGION//-/_} or (otherwise) tr to transform the name:
REGION="${REGION//-/_}" # Bash
REGION="$(echo "$REGION"|tr - _)" # POSIX

You are evaluating/expanding a few too many times.
Toby's answer is exactly correct (and shows the proper debugging technique for this sort of issue) but the solution to the problem is to unwrap one level of expansion.
You wrote
eval "USERNAME=\${${DBUSERNAME}}"
which becomes
eval "USERNAME=\${DB_USER_dev_us-east-1}"
which then gets run through eval as
USERNAME=${DB_USER_dev_us-east-1}
which becomes
USERNAME=east-1
but you wanted to stop after the first expansion. That is
eval "USERNAME=\${DB_USER_dev_us-east-1}"
has already performed the expansion you wanted and gotten you the result you needed. So you don't want the \${...} bit or eval. Just
USERNAME=DB_USER_dev_us-east-1
which you get from
USERNAME=${DBUSERNAME}
Unless I've missed something or your example isn't accurate.

As they have explained the problem, here is what you can do
$> more a.sh
ENV=dev
REGION=us-east-1
DBUSERNAME=DB_USER_${ENV}_$REGION
DBPASSWORD=DB_PASS_${ENV}_$REGION
eval "USERNAME=${DBUSERNAME}"
eval "PASSWORD=${DBPASSWORD}"
echo USERNAME=$USERNAME
echo PASSWORD=$PASSWORD
Results
$> ./a.sh
USERNAME=DB_USER_dev_us-east-1
PASSWORD=DB_PASS_dev_us-east-1

Related

Equal/minus sign without a colon in a parameter expasion in bash

I found a snippet like this in a Bash script recently:
$ echo ${A=3}
Now, I know that ${A:=3} would set the variable A if A is "falsy", or ${A:-3} would return 3 if A is "falsy." I have never seen these similar expressions without the colon though, and I cannot find the explanation for these colon-less expressions in the Bash's documentation.
What is going on here?
Actually, the documentation does explain what is going on here, even if burying the lede a bit:
When not performing substring expansion, using the form described below (e.g., ‘:-’), Bash tests for a parameter that is unset or null. Omitting the colon results in a test only for a parameter that is unset. Put another way, if the colon is included, the operator tests for both parameters’ existence and that its value is not null; if the colon is omitted, the operator tests only for existence.
In practice, this means that they behave the same way if the variables are unset:
$ echo ${A=no-colon}
no-colon
$ echo ${B:=with-colon}
with-colon
$ echo $A
no-colon
$ echo $B
with-colon
However, if the variables are set to the empty string, then the behavior is different. The expression with a colon will set the variable and return the value, and the one without will leave the variable as is (i.e., set to the empty string) and return its empty value:
$ A='' ; B=''
$ echo ${A=no-colon}
$ echo ${B:=with-colon}
with-colon
$ echo $A
$ echo $B
with-colon
As stated in the documentation, the same behavior applies to the other "operators" (-, ?, +).
Posting it in the spirit of Can I answer my own question? and because it took a surprisingly long time for me to learn it, even after finding it in code. Maybe making it a bit more explicit, with some examples, can help somebody else out there :)

Bash assignment value to variable as command substitution and print value output

I would like to achieve this in Bash: echo $(a=1)and print the value of variable a
I test eval, $$a,{}, $() but none of them work as most of the times either I got literally a=1 or in one case (I don't remember which) it tried to execute the value.
I known that I can do: a=1;echo $a but because I'm little fun one command per line (even if sometimes is getting little complicated) I was wondering if is possible to do this either with echo or with printf
If you know that $a is previously unset, you can do this using the following syntax:
echo ${a:=1}
This, and other types of parameter expansion, are defined in the POSIX shell command language specification.
If you want to assign a numeric value, another option, which doesn't depend on the value previously being unset, would be to use an arithmetic expansion:
echo $(( a = 1 ))
This assigns the value and echoes the number that has been assigned.
It's worth mentioning that what you're trying to do cannot be done in a subshell by design because a child process cannot modify the environment of its parent.

variable of Variable

I have to print a variable value which is art variable.
eg. variables are
A = X
Z_X = Test
Set in shell using
setenv A X
setenv Z_X Test
I want to print $Z_X value using $A
I am trying but without any success.
echo ${Z_${A}}
echo ${Z_[$A]}
could anyone tell me where I am wrong.
regards
A = X
Z_X = Test
This seems wrong; in csh, you need to use the set keyword to assign variables; in addition, the convention is also to use lower case for "normal" variables, and UPPER CASE for environment variables:
set a = x
set z_x = Test
You can then use eval to get what you want:
% eval echo \$z_$a
Test
% set x = `eval echo \$z_$a`
% echo $s
Test
This may be dangerous if you don't trust the source of $a, since it may also do a rm -rf / or something similarly dangerous (but if you trust the source of $a, it's perfectly fine).
You can get a list of all variables with set:
% set | grep ^z_$a
z_x Test
% set | grep ^z_$a | awk '{print $1}'
z_x
Which is the only safe way I can figure out to do what you want.
Generally this is a bad idea, and you should rather rethink your approach.
You can use indirect expansion:
name=Z_$A
echo ${!name}
From the manual:
If the first character of parameter is an exclamation point (!), a level of variable indirection is introduced. Bash uses the value of the variable formed from the rest of parameter as the name of the variable; this variable is then expanded and that value is used in the rest of the substitution, rather than the value of parameter itself. This is known as indirect expansion. The exceptions to this are the expansions of ${!prefix*} and
${!name[#]} described below. The exclamation point must immediately follow the left brace
in order to introduce indirection.
Or indirect references:
name=Z_$A
eval echo \$$name
You can find more about indirect references in the guide.

How to use eval to force variable update

I was reading the bash advanced scripting guide (if memory serves me right), and it said something to the extent that eval can be used to force variable updates.
So I tried this:
randomPath="/path/$var/here/" # var is not defined at this point
echo $randomPath
/path//here/
var="is" # initially defining var
eval $randomPath
zsh: no such file or directory: /path//here/
I don't understand the error message, and I'm wondering if I'm using eval properly.
The output I was expecting is:
eval $randomPath
echo $randomPath
/path/is/here
The problem is that $var is already being substituted in randomPath="/path/$var/here/", and because it is blank, randomPath is set to /path//here. You want to use single quotes to prevent the early substitution:
randomPath='/path/$var/here/'
The second problem is that eval x runs x as a command. What you want to do is return the newly evaluated variable as a string:
eval echo $randomPath
You can store it in a variable in the usual way:
randomPath=`eval echo $randomPath`

What is the purpose of the : (colon) GNU Bash builtin?

What is the purpose of a command that does nothing, being little more than a comment leader, but is actually a shell builtin in and of itself?
It's slower than inserting a comment into your scripts by about 40% per call, which probably varies greatly depending on the size of the comment. The only possible reasons I can see for it are these:
# poor man's delay function
for ((x=0;x<100000;++x)) ; do : ; done
# inserting comments into string of commands
command ; command ; : we need a comment in here for some reason ; command
# an alias for `true'
while : ; do command ; done
I guess what I'm really looking for is what historical application it might have had.
Historically, Bourne shells didn't have true and false as built-in commands. true was instead simply aliased to :, and false to something like let 0.
: is slightly better than true for portability to ancient Bourne-derived shells. As a simple example, consider having neither the ! pipeline operator nor the || list operator (as was the case for some ancient Bourne shells). This leaves the else clause of the if statement as the only means for branching based on exit status:
if command; then :; else ...; fi
Since if requires a non-empty then clause and comments don't count as non-empty, : serves as a no-op.
Nowadays (that is: in a modern context) you can usually use either : or true. Both are specified by POSIX, and some find true easier to read. However there is one interesting difference: : is a so-called POSIX special built-in, whereas true is a regular built-in.
Special built-ins are required to be built into the shell; Regular built-ins are only "typically" built in, but it isn't strictly guaranteed. There usually shouldn't be a regular program named : with the function of true in PATH of most systems.
Probably the most crucial difference is that with special built-ins, any variable set by the built-in - even in the environment during simple command evaluation - persists after the command completes, as demonstrated here using ksh93:
$ unset x; ( x=hi :; echo "$x" )
hi
$ ( x=hi true; echo "$x" )
$
Note that Zsh ignores this requirement, as does GNU Bash except when operating in POSIX compatibility mode, but all other major "POSIX sh derived" shells observe this including dash, ksh93, and mksh.
Another difference is that regular built-ins must be compatible with exec - demonstrated here using Bash:
$ ( exec : )
-bash: exec: :: not found
$ ( exec true )
$
POSIX also explicitly notes that : may be faster than true, though this is of course an implementation-specific detail.
I use it to easily enable/disable variable commands:
#!/bin/bash
if [[ "$VERBOSE" == "" || "$VERBOSE" == "0" ]]; then
vecho=":" # no "verbose echo"
else
vecho=echo # enable "verbose echo"
fi
$vecho "Verbose echo is ON"
Thus
$ ./vecho
$ VERBOSE=1 ./vecho
Verbose echo is ON
This makes for a clean script. This cannot be done with '#'.
Also,
: >afile
is one of the simplest ways to guarantee that 'afile' exists but is 0 length.
A useful application for : is if you're only interested in using parameter expansions for their side-effects rather than actually passing their result to a command.
In that case, you use the parameter expansion as an argument to either : or false depending upon whether you want an exit status of 0 or 1. An example might be
: "${var:=$1}"
Since : is a builtin, it should be pretty fast.
: can also be for block comment (similar to /* */ in C language). For example, if you want to skip a block of code in your script, you can do this:
: << 'SKIP'
your code block here
SKIP
Two more uses not mentioned in other answers:
Logging
Take this example script:
set -x
: Logging message here
example_command
The first line, set -x, makes the shell print out the command before running it. It's quite a useful construct. The downside is that the usual echo Log message type of statement now prints the message twice. The colon method gets round that. Note that you'll still have to escape special characters just like you would for echo.
Cron job titles
I've seen it being used in cron jobs, like this:
45 10 * * * : Backup for database ; /opt/backup.sh
This is a cron job that runs the script /opt/backup.sh every day at 10:45. The advantage of this technique is that it makes for better looking email subjects when the /opt/backup.sh prints some output.
It's similar to pass in Python.
One use would be to stub out a function until it gets written:
future_function () { :; }
If you'd like to truncate a file to zero bytes, useful for clearing logs, try this:
:> file.log
You could use it in conjunction with backticks (``) to execute a command without displaying its output, like this:
: `some_command`
Of course you could just do some_command > /dev/null, but the :-version is somewhat shorter.
That being said I wouldn't recommend actually doing that as it would just confuse people. It just came to mind as a possible use-case.
It's also useful for polyglot programs:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
~function(){ ... }
This is now both an executable shell-script and a JavaScript program: meaning ./filename.js, sh filename.js, and node filename.js all work.
(Definitely a little bit of a strange usage, but effective nonetheless.)
Some explication, as requested:
Shell-scripts are evaluated line-by-line; and the exec command, when run, terminates the shell and replaces it's process with the resultant command. This means that to the shell, the program looks like this:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
As long as no parameter expansion or aliasing is occurring in the word, any word in a shell-script can be wrapped in quotes without changing its' meaning; this means that ':' is equivalent to : (we've only wrapped it in quotes here to achieve the JavaScript semantics described below)
... and as described above, the first command on the first line is a no-op (it translates to : //, or if you prefer to quote the words, ':' '//'. Notice that the // carries no special meaning here, as it does in JavaScript; it's just a meaningless word that's being thrown away.)
Finally, the second command on the first line (after the semicolon), is the real meat of the program: it's the exec call which replaces the shell-script being invoked, with a Node.js process invoked to evaluate the rest of the script.
Meanwhile, the first line, in JavaScript, parses as a string-literal (':'), and then a comment, which is deleted; thus, to JavaScript, the program looks like this:
':'
~function(){ ... }
Since the string-literal is on a line by itself, it is a no-op statement, and is thus stripped from the program; that means that the entire line is removed, leaving only your program-code (in this example, the function(){ ... } body.)
Self-documenting functions
You can also use : to embed documentation in a function.
Assume you have a library script mylib.sh, providing a variety of functions. You could either source the library (. mylib.sh) and call the functions directly after that (lib_function1 arg1 arg2), or avoid cluttering your namespace and invoke the library with a function argument (mylib.sh lib_function1 arg1 arg2).
Wouldn't it be nice if you could also type mylib.sh --help and get a list of available functions and their usage, without having to manually maintain the function list in the help text?
#!/bin/bash
# all "public" functions must start with this prefix
LIB_PREFIX='lib_'
# "public" library functions
lib_function1() {
: This function does something complicated with two arguments.
:
: Parameters:
: ' arg1 - first argument ($1)'
: ' arg2 - second argument'
:
: Result:
: " it's complicated"
# actual function code starts here
}
lib_function2() {
: Function documentation
# function code here
}
# help function
--help() {
echo MyLib v0.0.1
echo
echo Usage: mylib.sh [function_name [args]]
echo
echo Available functions:
declare -f | sed -n -e '/^'$LIB_PREFIX'/,/^}$/{/\(^'$LIB_PREFIX'\)\|\(^[ \t]*:\)/{
s/^\('$LIB_PREFIX'.*\) ()/\n=== \1 ===/;s/^[ \t]*: \?['\''"]\?/ /;s/['\''"]\?;\?$//;p}}'
}
# main code
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
# the script was executed instead of sourced
# invoke requested function or display help
if [ "$(type -t - "$1" 2>/dev/null)" = function ]; then
"$#"
else
--help
fi
fi
A few comments about the code:
All "public" functions have the same prefix. Only these are meant to be invoked by the user, and to be listed in the help text.
The self-documenting feature relies on the previous point, and uses declare -f to enumerate all available functions, then filters them through sed to only display functions with the appropriate prefix.
It is a good idea to enclose the documentation in single quotes, to prevent undesired expansion and whitespace removal. You'll also need to be careful when using apostrophes/quotes in the text.
You could write code to internalize the library prefix, i.e. the user only has to type mylib.sh function1 and it gets translated internally to lib_function1. This is an exercise left to the reader.
The help function is named "--help". This is a convenient (i.e. lazy) approach that uses the library invoke mechanism to display the help itself, without having to code an extra check for $1. At the same time, it will clutter your namespace if you source the library. If you don't like that, you can either change the name to something like lib_help or actually check the args for --help in the main code and invoke the help function manually.
I saw this usage in a script and thought it was a good substitute for invoking basename within a script.
oldIFS=$IFS
IFS=/
for basetool in $0 ; do : ; done
IFS=$oldIFS
...
this is a replacement for the code: basetool=$(basename $0)
Another way, not yet mentioned here is the initialisation of parameters in infinite while-loops. Below is not the cleanest example, but it serves it's purpose.
#!/usr/bin/env bash
[ "$1" ] && foo=0 && bar="baz"
while : "${foo=2}" "${bar:=qux}"; do
echo "$foo"
(( foo == 3 )) && echo "$bar" && break
(( foo=foo+1 ))
done

Resources