In my bash script I have two arrays. Depending on some logic either one or another shall be used, so I'm getting the name of a required array in a variable varName. I can surely get the values of this array with the code below but is there a way to get it's keys? Tried several options but no luck.
declare -A foo=([a]=b [c]=d)
declare -A bar=([e]=f [g]=h)
varName=foo
varArray=$varName[#]
echo ${!varArray}
Thanks.
Not without resorting to eval, unfortunately. To be safe, make sure varName is just a single valid identifier.
[[ varName =~ ^[a-zA-Z_][a-zA-Z_0-9]+$ ]] && eval "echo \${!$varName[#]}"
eval is necessary to provide a second round of parsing and evaluation. In the first round, the shell performs the usual parameter expansion, resulting in the string echo ${!foo[#]} being passed as the single argument to eval. (In particular, the first dollar sign was escaped and so is passed literally; $varName is expanded to foo; and the quotes are removed as part of quote removal. eval then parses that string and evaluates it.
$ eval "echo \${!$varName[#]}"
# echo ${!foo [#]}
# The above is the argument that `eval` sees, after the shell
# does the normal evaluation before calling `eval`. Parameter
# expansion replaces $varName with foo and quote removal gets
# rid of the backslash before `$` and the double quotes.
a c
If you are using bash 4.3 or later, you can use a nameref.
declare -n varName=foo
for key in "${!varName[#]}"; do
echo "$key"
done
There is an IFS-safe method to get the keys (or values) of an array indirectly:
declare -a things=("an apple" "a banana")
declare -a tmp
arrayName=things
eval "tmp=(\"\${$arrayName[#]}\")"
# "${tmp[#]}" is now ("an apple" "a banana")
Related
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
Example:
testing="test"
var="\"${testing} the variable\""
testing="Update"
echo "$var"
Output:
test the variable
Required output:
Update the variable
Variables are expanded when the string is used, they're not templates that remember the variable substitution.
If you need to do this, you need to put the variable literally into the string, and use eval.
testing="test"
var='"${testing} the variable"'
testing="Update"
eval "echo $var"
But eval is dangerous -- it will execute any shell commands in the string. A better solution would probably be to use some kind of placeholder string and replace it using the shell expansion operator.
var='#testing# the variable'
testing="Update"
echo "${var//#testing#/$testing}"
So I got the following alias:
alias current_dir="pwd | sed -e 's/ /\\ /'"
In another place, I first safe my returned string in order to use parameter expansion to lowercase the string, like so:
CURRENT_DIR=$(current_dir)
echo "${CURRENT_DIR,,}"
But I wonder if it is possible to directly use parameter expansion on a alias/function call? I tried the following possibilities but they all didn't work for me:
echo "${current_dir,,}" # just an empty echo
echo "${$(current_dir),,}" # bad substitution
echo "${"$(current_dir)",,}" # bad substitution
No, it's not possible. You have to save the output in an intermediate variable. It's unavoidable.
You could use
declare -l CURRENT_DIR=$(current_dir)
Although Shellcheck has some sage words about declare and command substitution on the same line
However, to get a properly shell-quoted/escaped version of the string, use one of
$ mkdir '/tmp/a dir "with quotes and spaces"'
$ cd !$
$ printf -v CURRENT_DIR "%q" "$PWD"
$ echo "$CURRENT_DIR"
/tmp/a\ dir\ \"with\ quotes\ and\ spaces\"
$ CURRENT_DIR=${PWD#Q}
$ echo "$CURRENT_DIR"
'/tmp/a dir "with quotes and spaces"'
Get out of the habit of using ALLCAPS variable names, leave those as
reserved by the shell. One day you'll write PATH=something and then
wonder why
your script is broken.
${var#operator} showed up in bash 4.4:
${parameter#operator}
Parameter transformation. The expansion is either a transformation of the
value of parameter or information about parameter itself, depending on the
value of operator. Each operator is a single letter:
Q The expansion is a string that is the value of parameter quoted in
a format that can be reused as input.
E The expansion is a string that is the value of parameter with back-
slash escape sequences expanded as with the $'...' quoting mechan-
sim.
P The expansion is a string that is the result of expanding the value
of parameter as if it were a prompt string (see PROMPTING below).
A The expansion is a string in the form of an assignment statement or
declare command that, if evaluated, will recreate parameter with
its attributes and value.
a The expansion is a string consisting of flag values representing
parameter's attributes.
If parameter is # or *, the operation is applied to each positional param-
eter in turn, and the expansion is the resultant list. If parameter is an
array variable subscripted with # or *, the case modification operation is
applied to each member of the array in turn, and the expansion is the
resultant list.
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
I have a variable that has a hash value assigned to it. For example :
hash=$1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/
now when I do echo $hash in bash it returns:
/jV.7qpaKlCTsBXOYu2op/
How do I have hash value not escape any characters? or have echo $hash return the entire string $1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/?
Any help is highly appreciated.
The definition needs single quotes:
$ hash='$1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/'
$ echo "$hash"
$1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/
Without the single quotes, the shell performs variable substitution and the result depends on the value returned by $1, $qGqTE and $syM when the definition statement is executed.
I also added double-quotes to the echo statement. This stops the shell from performing word splitting and pathname expansion. While it may be unlikely that a hash value would be affected by these, it is safer to use the double quotes. As an example of the potential problem:
$ hash='/bin/le*'
$ echo $hash
/bin/less /bin/lessecho /bin/lessfile /bin/lesskey /bin/lesspipe
$ echo "$hash"
/bin/le*
As you can see, in this case, echo $hash performs pathname expansion and returns a list of files. echo "$hash", however, works as desired. To avoid surprises such as this, it is best to put references to shell variables in double quotes.