I have a variable that has a hash value assigned to it. For example :
hash=$1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/
now when I do echo $hash in bash it returns:
/jV.7qpaKlCTsBXOYu2op/
How do I have hash value not escape any characters? or have echo $hash return the entire string $1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/?
Any help is highly appreciated.
The definition needs single quotes:
$ hash='$1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/'
$ echo "$hash"
$1$qGqTE/jV$syM.7qpaKlCTsBXOYu2op/
Without the single quotes, the shell performs variable substitution and the result depends on the value returned by $1, $qGqTE and $syM when the definition statement is executed.
I also added double-quotes to the echo statement. This stops the shell from performing word splitting and pathname expansion. While it may be unlikely that a hash value would be affected by these, it is safer to use the double quotes. As an example of the potential problem:
$ hash='/bin/le*'
$ echo $hash
/bin/less /bin/lessecho /bin/lessfile /bin/lesskey /bin/lesspipe
$ echo "$hash"
/bin/le*
As you can see, in this case, echo $hash performs pathname expansion and returns a list of files. echo "$hash", however, works as desired. To avoid surprises such as this, it is best to put references to shell variables in double quotes.
Related
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
So I got the following alias:
alias current_dir="pwd | sed -e 's/ /\\ /'"
In another place, I first safe my returned string in order to use parameter expansion to lowercase the string, like so:
CURRENT_DIR=$(current_dir)
echo "${CURRENT_DIR,,}"
But I wonder if it is possible to directly use parameter expansion on a alias/function call? I tried the following possibilities but they all didn't work for me:
echo "${current_dir,,}" # just an empty echo
echo "${$(current_dir),,}" # bad substitution
echo "${"$(current_dir)",,}" # bad substitution
No, it's not possible. You have to save the output in an intermediate variable. It's unavoidable.
You could use
declare -l CURRENT_DIR=$(current_dir)
Although Shellcheck has some sage words about declare and command substitution on the same line
However, to get a properly shell-quoted/escaped version of the string, use one of
$ mkdir '/tmp/a dir "with quotes and spaces"'
$ cd !$
$ printf -v CURRENT_DIR "%q" "$PWD"
$ echo "$CURRENT_DIR"
/tmp/a\ dir\ \"with\ quotes\ and\ spaces\"
$ CURRENT_DIR=${PWD#Q}
$ echo "$CURRENT_DIR"
'/tmp/a dir "with quotes and spaces"'
Get out of the habit of using ALLCAPS variable names, leave those as
reserved by the shell. One day you'll write PATH=something and then
wonder why
your script is broken.
${var#operator} showed up in bash 4.4:
${parameter#operator}
Parameter transformation. The expansion is either a transformation of the
value of parameter or information about parameter itself, depending on the
value of operator. Each operator is a single letter:
Q The expansion is a string that is the value of parameter quoted in
a format that can be reused as input.
E The expansion is a string that is the value of parameter with back-
slash escape sequences expanded as with the $'...' quoting mechan-
sim.
P The expansion is a string that is the result of expanding the value
of parameter as if it were a prompt string (see PROMPTING below).
A The expansion is a string in the form of an assignment statement or
declare command that, if evaluated, will recreate parameter with
its attributes and value.
a The expansion is a string consisting of flag values representing
parameter's attributes.
If parameter is # or *, the operation is applied to each positional param-
eter in turn, and the expansion is the resultant list. If parameter is an
array variable subscripted with # or *, the case modification operation is
applied to each member of the array in turn, and the expansion is the
resultant list.
Seems that the recommended way of doing indirect variable setting in bash is to use eval:
var=x; val=foo
eval $var=$val
echo $x # --> foo
The problem is the usual one with eval:
var=x; val=1$'\n'pwd
eval $var=$val # bad output here
(and since it is recommended in many places, I wonder just how many scripts are vulnerable because of this...)
In any case, the obvious solution of using (escaped) quotes doesn't really work:
var=x; val=1\"$'\n'pwd\"
eval $var=\"$val\" # fail with the above
The thing is that bash has indirect variable reference baked in (with ${!foo}), but I don't see any such way to do indirect assignment -- is there any sane way to do this?
For the record, I did find a solution, but this is not something that I'd consider "sane"...:
eval "$var='"${val//\'/\'\"\'\"\'}"'"
A slightly better way, avoiding the possible security implications of using eval, is
declare "$var=$val"
Note that declare is a synonym for typeset in bash. The typeset command is more widely supported (ksh and zsh also use it):
typeset "$var=$val"
In modern versions of bash, one should use a nameref.
declare -n var=x
x=$val
It's safer than eval, but still not perfect.
Bash has an extension to printf that saves its result into a variable:
printf -v "${VARNAME}" '%s' "${VALUE}"
This prevents all possible escaping issues.
If you use an invalid identifier for $VARNAME, the command will fail and return status code 2:
$ printf -v ';;;' '%s' foobar; echo $?
bash: printf: `;;;': not a valid identifier
2
eval "$var=\$val"
The argument to eval should always be a single string enclosed in either single or double quotes. All code that deviates from this pattern has some unintended behavior in edge cases, such as file names with special characters.
When the argument to eval is expanded by the shell, the $var is replaced with the variable name, and the \$ is replaced with a simple dollar. The string that is evaluated therefore becomes:
varname=$value
This is exactly what you want.
Generally, all expressions of the form $varname should be enclosed in double quotes, to prevent accidental expansion of filename patterns like *.c.
There are only two places where the quotes may be omitted since they are defined to not expand pathnames and split fields: variable assignments and case. POSIX 2018 says:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
This list of expansions is missing the parameter expansion and the field splitting. Sure, that's hard to see from reading this sentence alone, but that's the official definition.
Since this is a variable assignment, the quotes are not needed here. They don't hurt, though, so you could also write the original code as:
eval "$var=\"the value is \$val\""
Note that the second dollar is escaped using a backslash, to prevent it from being expanded in the first run. What happens is:
eval "$var=\"the value is \$val\""
The argument to the command eval is sent through parameter expansion and unescaping, resulting in:
varname="the value is $val"
This string is then evaluated as a variable assignment, which assigns the following value to the variable varname:
the value is value
The main point is that the recommended way to do this is:
eval "$var=\$val"
with the RHS done indirectly too. Since eval is used in the same
environment, it will have $val bound, so deferring it works, and since
now it's just a variable. Since the $val variable has a known name,
there are no issues with quoting, and it could have even been written as:
eval $var=\$val
But since it's better to always add quotes, the former is better, or
even this:
eval "$var=\"\$val\""
A better alternative in bash that was mentioned for the whole thing that
avoids eval completely (and is not as subtle as declare etc):
printf -v "$var" "%s" "$val"
Though this is not a direct answer what I originally asked...
Newer versions of bash support something called "parameter transformation", documented in a section of the same name in bash(1).
"${value#Q}" expands to a shell-quoted version of "${value}" that you can re-use as input.
Which means the following is a safe solution:
eval="${varname}=${value#Q}"
Just for completeness I also want to suggest the possible use of the bash built in read. I've also made corrections regarding -d'' based on socowi's comments.
But much care needs to be exercised when using read to ensure the input is sanitized (-d'' reads until null termination and printf "...\0" terminates the value with a null), and that read itself is executed in the main shell where the variable is needed and not a sub-shell (hence the < <( ... ) syntax).
var=x; val=foo0shouldnotterminateearly
read -d'' -r "$var" < <(printf "$val\0")
echo $x # --> foo0shouldnotterminateearly
echo ${!var} # --> foo0shouldnotterminateearly
I tested this with \n \t \r spaces and 0, etc it worked as expected on my version of bash.
The -r will avoid escaping \, so if you had the characters "\" and "n" in your value and not an actual newline, x will contain the two characters "\" and "n" also.
This method may not be aesthetically as pleasing as the eval or printf solution, and would be more useful if the value is coming in from a file or other input file descriptor
read -d'' -r "$var" < <( cat $file )
And here are some alternative suggestions for the < <() syntax
read -d'' -r "$var" <<< "$val"$'\0'
read -d'' -r "$var" < <(printf "$val") #Apparently I didn't even need the \0, the printf process ending was enough to trigger the read to finish.
read -d'' -r "$var" <<< $(printf "$val")
read -d'' -r "$var" <<< "$val"
read -d'' -r "$var" < <(printf "$val")
Yet another way to accomplish this, without eval, is to use "read":
INDIRECT=foo
read -d '' -r "${INDIRECT}" <<<"$(( 2 * 2 ))"
echo "${foo}" # outputs "4"
In my bash script I have two arrays. Depending on some logic either one or another shall be used, so I'm getting the name of a required array in a variable varName. I can surely get the values of this array with the code below but is there a way to get it's keys? Tried several options but no luck.
declare -A foo=([a]=b [c]=d)
declare -A bar=([e]=f [g]=h)
varName=foo
varArray=$varName[#]
echo ${!varArray}
Thanks.
Not without resorting to eval, unfortunately. To be safe, make sure varName is just a single valid identifier.
[[ varName =~ ^[a-zA-Z_][a-zA-Z_0-9]+$ ]] && eval "echo \${!$varName[#]}"
eval is necessary to provide a second round of parsing and evaluation. In the first round, the shell performs the usual parameter expansion, resulting in the string echo ${!foo[#]} being passed as the single argument to eval. (In particular, the first dollar sign was escaped and so is passed literally; $varName is expanded to foo; and the quotes are removed as part of quote removal. eval then parses that string and evaluates it.
$ eval "echo \${!$varName[#]}"
# echo ${!foo [#]}
# The above is the argument that `eval` sees, after the shell
# does the normal evaluation before calling `eval`. Parameter
# expansion replaces $varName with foo and quote removal gets
# rid of the backslash before `$` and the double quotes.
a c
If you are using bash 4.3 or later, you can use a nameref.
declare -n varName=foo
for key in "${!varName[#]}"; do
echo "$key"
done
There is an IFS-safe method to get the keys (or values) of an array indirectly:
declare -a things=("an apple" "a banana")
declare -a tmp
arrayName=things
eval "tmp=(\"\${$arrayName[#]}\")"
# "${tmp[#]}" is now ("an apple" "a banana")
In shell scripting, what is the difference between these two when assigning one variable to another:
a=$b
and
a="$b"
and when should I use one over the other?
I think there is no big difference here. Yes, it is advisable to enclose a variable in double quotes when that variable is being referenced. However, $x does not seem to be referenced here in your question.
y=$x does not by itself affect how whitespaces will be handled. It is only when $y is actually used that quoting matters. For example:
$ x=" a b "
$ y=$x
$ echo $y
a b
$ echo "$y"
a b
From section 2.9.1 of the POSIX shell syntax specification:
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
String-splitting and globbing (the steps which double quotes suppress) are not in this list.
Thus, the quotes are superfluous in all simple assignments (not speaking here to those implemented with arguments to declare, export or similar commands) except those where (1) the behavior of single-quoted, not double-quoted, strings are desired; or (2) whitespace or other content in the value would be otherwise parsed as syntactic rather than literal.
(Note that the decision on how to parse a command -- thus, whether it is an assignment, a simple command, a compound command, or something else -- takes place before parameter expansions; thus, var=$1 is determined to be an assignment before the value of $1 is ever considered! Were this untrue, such that data could silently become syntax, it would be far more difficult -- if not impossible -- to write secure code handling untrusted data in bash).
There are no (good) reasons to double quote the RHS of a variable assignment when used as a statement on its own.
The RHS of an assignment statement is not subject to word splitting (or brace expansion), etc. so cannot need quotes to assign correctly. All other expansions (as far as I'm aware) do occur on the RHS but also occur in double quotes so the quoting serves no purpose.
That being said there are reasons not to quote the RHS. Namely how to address error "bash: !d': event not found" in Bash command substitution (specifically see my answer and rici's answer).
Here are some other examples: ( having two files in the current directory t.sh and file)
a='$(ls)' # no command substitution
b="$(ls)" # command substitution, no word splitting
c='*' # no filename expansion
d="*" # no filename expansion
e=* # no filename expansion
f=$a # no expansions or splittings
g="$a" # no expansions or splittings
h=$d # no expansions or splittings
echo ---'$a'---
echo $a # no command substitution
echo ---'$b'---
echo $b # word splitting
echo ---'"$b"'---
echo "$b" # no word splitting
echo ---'$c'---
echo $c # filename expansion, word splitting
echo ---'"$c"'---
echo "$c" # no filename expansion, no word splitting
echo ---'$d'---
echo $d # filename expansion, word splitting
echo ---'"$d"'---
echo "$d" # no filename expansion, no word splitting
echo ---'"$e"'---
echo "$e" # no filename expansion, no word splitting
echo ---'$e'---
echo $e # filename expansion, word splitting
echo ---'"$f"'---
echo "$f" # no filename expansion, no word splitting
echo ---'"$g"'---
echo "$g" # no filename expansion, no word splitting
echo ---'$h'---
echo $h # filename expansion, word splitting
echo ---'"$h"'---
echo "$h" # no filename expansion, no word splitting
Output:
---$a---
$(ls)
---$b---
file t.sh
---"$b"---
file
t.sh
---$c---
file t.sh
---"$c"---
*
---$d---
file t.sh
---"$d"---
*
---"$e"---
*
---$e---
file t.sh
---"$f"---
$(ls)
---"$g"---
$(ls)
---$h---
file t.sh
---"$h"---
*
One interesting thing to notice is that command substitution occurs in variable assignments if they are in double quotes, and if the RHS is given explicitly as "$(ls)" and not implicitly as "$a"..
Advanced Bash-Scripting Guide: Chapter 5: Quoting
When referencing a variable, it is
generally advisable to enclose its
name in double quotes. This prevents
reinterpretation of all special
characters within the quoted string.
Use double quotes to prevent word
splitting. An argument enclosed in
double quotes presents itself as a
single word, even if it contains
whitespace separators.