how to echo out associative part of associative array bash [duplicate] - bash

Using:
set -o nounset
Having an indexed array like:
myArray=( "red" "black" "blue" )
What is the shortest way to check if element 1 is set?
I sometimes use the following:
test "${#myArray[#]}" -gt "1" && echo "1 exists" || echo "1 doesn't exist"
I would like to know if there's a preferred one.
How to deal with non-consecutive indexes?
myArray=()
myArray[12]="red"
myArray[51]="black"
myArray[129]="blue"
How to quick check that 51 is already set for example?
How to deal with associative arrays?
declare -A myArray
myArray["key1"]="red"
myArray["key2"]="black"
myArray["key3"]="blue"
How to quick check that key2 is already used for example?

To check if the element is set (applies to both indexed and associative array)
[ "${array[key]+abc}" ] && echo "exists"
Basically what ${array[key]+abc} does is
if array[key] is set, return abc
if array[key] is not set, return nothing
References:
See Parameter Expansion in Bash manual and the little note
if the colon is omitted, the operator tests only for existence [of parameter]
This answer is actually adapted from the answers for this SO question: How to tell if a string is not defined in a bash shell script?
A wrapper function:
exists(){
if [ "$2" != in ]; then
echo "Incorrect usage."
echo "Correct usage: exists {key} in {array}"
return
fi
eval '[ ${'$3'[$1]+muahaha} ]'
}
For example
if ! exists key in array; then echo "No such array element"; fi

From man bash, conditional expressions:
-v varname
True if the shell variable varname is set (has been assigned a value).
example:
declare -A foo
foo[bar]="this is bar"
foo[baz]=""
if [[ -v "foo[bar]" ]] ; then
echo "foo[bar] is set"
fi
if [[ -v "foo[baz]" ]] ; then
echo "foo[baz] is set"
fi
if [[ -v "foo[quux]" ]] ; then
echo "foo[quux] is set"
fi
This will show that both foo[bar] and foo[baz] are set (even though the latter is set to an empty value) and foo[quux] is not.

New answer
From version 4.2 of bash (and newer), there is a new -v option to built-in test command.
From version 4.3, this test could address element of arrays.
array=([12]="red" [51]="black" [129]="blue")
for i in 10 12 30 {50..52} {128..131};do
if [ -v 'array[i]' ];then
echo "Variable 'array[$i]' is defined"
else
echo "Variable 'array[$i]' not exist"
fi
done
Variable 'array[10]' not exist
Variable 'array[12]' is defined
Variable 'array[30]' not exist
Variable 'array[50]' not exist
Variable 'array[51]' is defined
Variable 'array[52]' not exist
Variable 'array[128]' not exist
Variable 'array[129]' is defined
Variable 'array[130]' not exist
Variable 'array[131]' not exist
Note: regarding ssc's comment, I've single quoted 'array[i]' in -v test, in order to satisfy shellcheck's error SC2208. This seem not really required here, because there is no glob character in array[i], anyway...
This work with associative arrays in same way:
declare -A aArray=([foo]="bar" [bar]="baz" [baz]=$'Hello world\041')
for i in alpha bar baz dummy foo test;do
if [ -v 'aArray[$i]' ];then
echo "Variable 'aArray[$i]' is defined"
else
echo "Variable 'aArray[$i]' not exist"
fi
done
Variable 'aArray[alpha]' not exist
Variable 'aArray[bar]' is defined
Variable 'aArray[baz]' is defined
Variable 'aArray[dummy]' not exist
Variable 'aArray[foo]' is defined
Variable 'aArray[test]' not exist
With a little difference:In regular arrays, variable between brackets ([i]) is integer, so dollar symbol ($) is not required, but for associative array, as key is a word, $ is required ([$i])!
Old answer for bash prior to V4.2
Unfortunately, bash give no way to make difference betwen empty and undefined variable.
But there is some ways:
$ array=()
$ array[12]="red"
$ array[51]="black"
$ array[129]="blue"
$ echo ${array[#]}
red black blue
$ echo ${!array[#]}
12 51 129
$ echo "${#array[#]}"
3
$ printf "%s\n" ${!array[#]}|grep -q ^51$ && echo 51 exist
51 exist
$ printf "%s\n" ${!array[#]}|grep -q ^52$ && echo 52 exist
(give no answer)
And for associative array, you could use the same:
$ unset array
$ declare -A array
$ array["key1"]="red"
$ array["key2"]="black"
$ array["key3"]="blue"
$ echo ${array[#]}
blue black red
$ echo ${!array[#]}
key3 key2 key1
$ echo ${#array[#]}
3
$ set | grep ^array=
array=([key3]="blue" [key2]="black" [key1]="red" )
$ printf "%s\n" ${!array[#]}|grep -q ^key2$ && echo key2 exist || echo key2 not exist
key2 exist
$ printf "%s\n" ${!array[#]}|grep -q ^key5$ && echo key5 exist || echo key5 not exist
key5 not exist
You could do the job without the need of externals tools (no printf|grep as pure bash), and why not, build checkIfExist() as a new bash function:
$ checkIfExist() {
eval 'local keys=${!'$1'[#]}';
eval "case '$2' in
${keys// /|}) return 0 ;;
* ) return 1 ;;
esac";
}
$ checkIfExist array key2 && echo exist || echo don\'t
exist
$ checkIfExist array key5 && echo exist || echo don\'t
don't
or even create a new getIfExist bash function that return the desired value and exit with false result-code if desired value not exist:
$ getIfExist() {
eval 'local keys=${!'$1'[#]}';
eval "case '$2' in
${keys// /|}) echo \${$1[$2]};return 0 ;;
* ) return 1 ;;
esac";
}
$ getIfExist array key1
red
$ echo $?
0
$ # now with an empty defined value
$ array["key4"]=""
$ getIfExist array key4
$ echo $?
0
$ getIfExist array key5
$ echo $?
1

What about a -n test and the :- operator?
For example, this script:
#!/usr/bin/env bash
set -e
set -u
declare -A sample
sample["ABC"]=2
sample["DEF"]=3
if [[ -n "${sample['ABC']:-}" ]]; then
echo "ABC is set"
fi
if [[ -n "${sample['DEF']:-}" ]]; then
echo "DEF is set"
fi
if [[ -n "${sample['GHI']:-}" ]]; then
echo "GHI is set"
fi
Prints:
ABC is set
DEF is set

tested in bash 4.3.39(1)-release
declare -A fmap
fmap['foo']="boo"
key='foo'
# should echo foo is set to 'boo'
if [[ -z "${fmap[${key}]}" ]]; then echo "$key is unset in fmap"; else echo "${key} is set to '${fmap[${key}]}'"; fi
key='blah'
# should echo blah is unset in fmap
if [[ -z "${fmap[${key}]}" ]]; then echo "$key is unset in fmap"; else echo "${key} is set to '${fmap[${key}]}'"; fi

Reiterating this from Thamme:
[[ ${array[key]+Y} ]] && echo Y || echo N
This tests if the variable/array element exists, including if it is set to a null value. This works with a wider range of bash versions than -v and doesn't appear sensitive to things like set -u. If you see a "bad array subscript" using this method please post an example.

This is the easiest way I found for scripts.
<search> is the string you want to find, ASSOC_ARRAY the name of the variable holding your associative array.
Dependign on what you want to achieve:
key exists:
if grep -qe "<search>" <(echo "${!ASSOC_ARRAY[#]}"); then echo key is present; fi
key exists not:
if ! grep -qe "<search>" <(echo "${!ASSOC_ARRAY[#]}"); then echo key not present; fi
value exists:
if grep -qe "<search>" <(echo "${ASSOC_ARRAY[#]}"); then echo value is present; fi
value exists not:
if ! grep -qe "<search>" <(echo "${ASSOC_ARRAY[#]}"); then echo value not present; fi

I wrote a function to check if a key exists in an array in Bash:
# Check if array key exists
# Usage: array_key_exists $array_name $key
# Returns: 0 = key exists, 1 = key does NOT exist
function array_key_exists() {
local _array_name="$1"
local _key="$2"
local _cmd='echo ${!'$_array_name'[#]}'
local _array_keys=($(eval $_cmd))
local _key_exists=$(echo " ${_array_keys[#]} " | grep " $_key " &>/dev/null; echo $?)
[[ "$_key_exists" = "0" ]] && return 0 || return 1
}
Example
declare -A my_array
my_array['foo']="bar"
if [[ "$(array_key_exists 'my_array' 'foo'; echo $?)" = "0" ]]; then
echo "OK"
else
echo "ERROR"
fi
Tested with GNU bash, version 4.1.5(1)-release (i486-pc-linux-gnu)

For all time people, once and for all.
There's a "clean code" long way, and there is a shorter, more concise, bash centered way.
$1 = The index or key you are looking for.
$2 = The array / map passed in by reference.
function hasKey ()
{
local -r needle="${1:?}"
local -nr haystack=${2:?}
for key in "${!haystack[#]}"; do
if [[ $key == $needle ]] ;
return 0
fi
done
return 1
}
A linear search can be replaced by a binary search, which would perform better with larger data sets. Simply count and sort the keys first, then do a classic binary halving of of the haystack as you get closer and closer to the answer.
Now, for the purist out there that is like "No, I want the more performant version because I may have to deal with large arrays in bash," lets look at a more bash centered solution, but one that maintains clean code and the flexibility to deal with arrays or maps.
function hasKey ()
{
local -r needle="${1:?}"
local -nr haystack=${2:?}
[ -n ${haystack["$needle"]+found} ]
}
The line [ -n ${haystack["$needle"]+found} ]uses the ${parameter+word} form of bash variable expansion, not the ${parameter:+word} form, which attempts to test the value of a key, too, which is not the matter at hand.
Usage
local -A person=(firstname Anthony lastname Rutledge)
if hasMapKey "firstname" person; then
# Do something
fi
When not performing substring expansion, using the form described
below (e.g., ‘:-’), Bash tests for a parameter that is unset or null.
Omitting the colon results in a test only for a parameter that is
unset. Put another way, if the colon is included, the operator tests
for both parameter’s existence and that its value is not null; if the
colon is omitted, the operator tests only for existence.
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted.
${parameter:=word}
If parameter is unset or null, the expansion of word is assigned to parameter. The value of parameter is then substituted. Positional
parameters and special parameters may not be assigned to in this way.
${parameter:?word}
If parameter is null or unset, the expansion of word (or a message to that effect if word is not present) is written to the standard
error and the shell, if it is not interactive, exits. Otherwise, the
value of parameter is substituted. ${parameter:+word}
If parameter is null or unset, nothing is substituted, otherwise the expansion of word is substituted.
https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameter-Expansion
If $needle does not exist expand to nothing, otherwise expand to the non-zero length string, "found". This will make the -n test succeed if the $needle in fact does exist (as I say "found"), and fail otherwise.

Both in the case of arrays and hash maps I find the easiest and more straightforward solution is to use the matching operator =~.
For arrays:
myArray=("red" "black" "blue")
if [[ " ${myArray[#]} " =~ " blue " ]]; then
echo "blue exists in myArray"
else
echo "blue does not exist in myArray"
fi
NOTE: The spaces around the array guarantee the first and last element can match. The spaces around the value guarantee an exact match.
For hash maps, it's actually the same solution since printing a hash map as a string gives you a list of its values.
declare -A myMap
myMap=(
["key1"]="red"
["key2"]="black"
["key3"]="blue"
)
if [[ " ${myMap[#]} " =~ " blue " ]]; then
echo "blue exists in myMap"
else
echo "blue does not exist in myMap"
fi
But what if you would like to check whether a key exists in a hash map? In the case you can use the ! operator which gives you the list of keys in a hash map.
if [[ " ${!myMap[#]} " =~ " key3 " ]]; then
echo "key3 exists in myMap"
else
echo "key3 does not exist in myMap"
fi

I get bad array subscript error when the key I'm checking is not set. So, I wrote a function that loops over the keys:
#!/usr/bin/env bash
declare -A helpList
function get_help(){
target="$1"
for key in "${!helpList[#]}";do
if [[ "$key" == "$target" ]];then
echo "${helpList["$target"]}"
return;
fi
done
}
targetValue="$(get_help command_name)"
if [[ -z "$targetvalue" ]];then
echo "command_name is not set"
fi
It echos the value when it is found & echos nothing when not found. All the other solutions I tried gave me that error.

Related

Bash - check if argument is string

I am attempting to check if an argument is an array with the following code:
if [[ $(declare -p $1) ]] != *-a*;
Here $1 is a string with the value "123". I get the following error message from bash:
`arrays.bash: line 23: declare: 123: not found
This code works if I pass an array as an argument but not a string. I want to verify that the argument is either an array or an associative array. I have no concern with the contents at this point, I only want the type. Any ideas on how to do this?
After all, why worry about the types, if you are relying on it perhaps your approach is wrong or you may need a strong-typed language
% v=1
% declare -p v
declare -- v="1"
% echo $v
1
% echo ${v[#]}
1
% v[1]=2
% declare -p v
declare -a v=([0]="1" [1]="2")
% echo ${v[#]}
1 2
The Error In The Question
You called yourfunction "$a" instead of yourfunction a, when a=123. Don't do that: You need to pass the name of the variable, not its value.
General Solution: Bash 5.x+
Bash 5 has a new feature called parameter transformation, whereby ${parameter#operator} can perform a variety of actions; one of these is checking the type of the parameter.
myfunc() {
[[ -v "$1" ]] || { echo "No variable named $1 exists" >&2; return 1; }
case ${!1#a} in
*a*) echo "Array";;
*A*) echo "Associative array";;
*i*) echo "Integer";;
"") echo "Default string";;
*) echo "Other/unknown flag set: ${!1#a}";;
esac
}
Older Solution
myfunc() {
local typedesc
typedesc=$(declare -p "$1" 2>/dev/null) || {
echo "No variable named $1 is set" >&2
return 1
}
case $typedesc in
"declare -a"*) echo "Array";;
"declare -A"*) echo "Associative array";;
"declare -i"*) echo "Integer";;
"declare --"*) echo "Regular (default) string variable";;
*) echo "Other/unrecognized type";;
esac
}

How to check if dictionary contains a key in bash?

I would like to check if a dictionary contains a key, but i dont know how.
I tried this:
if [ -z "${codeDict["$STR_ARRAY[2]"]+xxx}" ]
then
echo "codeDict not contains ${STR_ARRAY[2]}"
codeDict["${STR_ARRAY[2]}"]="${STR_ARRAY[3]}"
fi
There's nothing wrong with your approach (using -z), as this example shows:
$ declare -A a
$ a=( [a]=1 [b]=2 [d]=4 )
$ [[ -z ${a[a]} ]] && echo unset
$ [[ -z ${a[c]} ]] && echo unset
unset
However, there are a couple of issues with the code in your question. You're missing the curly braces around your inner array and personally I'd suggest that you use extended tests ([[ instead of [) to avoid having to mess around with quotes:
$ str=( a b c )
$ [[ -z ${a[${str[0]}]} ]] && echo unset
$ [[ -z ${a[${str[2]}]} ]] && echo unset
unset
If you are using bash 4.3, you can use the -v test:
if [[ -v codeDict["${STR_ARRAY[2]}"] ]]; then
# codeDict has ${STR_ARRAY[2]} as a key
else
# codeDict does not have ${STR_ARRAY[2]} as a key
fi
Otherwise, you need to take care to distinguish between keys that map to an empty string and keys that are not in the array at all.
key=${STR_ARRARY[2]}
tmp=codeDict["$key"] # Save a lot of typing
# ${!tmp} expands to the actual value (which may be the empty string),
# or the empty string if the key does not exist
# ${!tmp-foo} expands to the actual value (which may be the empty string),
# or "foo" if the key does not exist
# ${!tmp:-foo} expands to the actual value if it is a non-empty string,
# or "foo" if the key does not exist *or* the key maps
# to the empty string.
if [[ ${!tmp} = ${!tmp-foo} || ${!tmp} = ${!tmp:-foo} ]]; then
# $key is a key
else
# $key is not a key
fi
In any version of bash that supports associative arrays, you can use a simple one liner if all you want to do is provide a default value for a key that does not exist.
: ${codeDict["${STR_ARRAY[2]}"]="${STR_ARRAY[3]}"}

How to get arguments with flags in Bash

I know that I can easily get positioned parameters like this in bash:
$0 or $1
I want to be able to use flag options like this to specify for what each parameter is used:
mysql -u user -h host
What is the best way to get -u param value and -h param value by flag instead of by position?
This example uses Bash's built-in getopts command and is from the Google Shell Style Guide:
a_flag=''
b_flag=''
files=''
verbose='false'
print_usage() {
printf "Usage: ..."
}
while getopts 'abf:v' flag; do
case "${flag}" in
a) a_flag='true' ;;
b) b_flag='true' ;;
f) files="${OPTARG}" ;;
v) verbose='true' ;;
*) print_usage
exit 1 ;;
esac
done
Note: If a character is followed by a colon (e.g. f:), that option is expected to have an argument.
Example usage: ./script -v -a -b -f filename
Using getopts has several advantages over the accepted answer:
the while condition is a lot more readable and shows what the accepted options are
cleaner code; no counting the number of parameters and shifting
you can join options (e.g. -a -b -c → -abc)
However, a big disadvantage is that it doesn't support long options, only single-character options.
This is the idiom I usually use:
while test $# -gt 0; do
case "$1" in
-h|--help)
echo "$package - attempt to capture frames"
echo " "
echo "$package [options] application [arguments]"
echo " "
echo "options:"
echo "-h, --help show brief help"
echo "-a, --action=ACTION specify an action to use"
echo "-o, --output-dir=DIR specify a directory to store output in"
exit 0
;;
-a)
shift
if test $# -gt 0; then
export PROCESS=$1
else
echo "no process specified"
exit 1
fi
shift
;;
--action*)
export PROCESS=`echo $1 | sed -e 's/^[^=]*=//g'`
shift
;;
-o)
shift
if test $# -gt 0; then
export OUTPUT=$1
else
echo "no output dir specified"
exit 1
fi
shift
;;
--output-dir*)
export OUTPUT=`echo $1 | sed -e 's/^[^=]*=//g'`
shift
;;
*)
break
;;
esac
done
Key points are:
$# is the number of arguments
while loop looks at all the arguments supplied, matching on their values inside a case statement
shift takes the first one away. You can shift multiple times inside of a case statement to take multiple values.
getopt is your friend.. a simple example:
function f () {
TEMP=`getopt --long -o "u:h:" "$#"`
eval set -- "$TEMP"
while true ; do
case "$1" in
-u )
user=$2
shift 2
;;
-h )
host=$2
shift 2
;;
*)
break
;;
esac
done;
echo "user = $user, host = $host"
}
f -u myself -h some_host
There should be various examples in your /usr/bin directory.
I propose a simple TLDR:; example for the un-initiated.
Create a bash script called greeter.sh
#!/bin/bash
while getopts "n:" arg; do
case $arg in
n) Name=$OPTARG;;
esac
done
echo "Hello $Name!"
You can then pass an optional parameter -n when executing the script.
Execute the script as such:
$ bash greeter.sh -n 'Bob'
Output
$ Hello Bob!
Notes
If you'd like to use multiple parameters:
extend while getops "n:" arg: do with more paramaters such as
while getops "n:o:p:" arg: do
extend the case switch with extra variable assignments. Such as o) Option=$OPTARG and p) Parameter=$OPTARG
To make the script executable:
chmod u+x greeter.sh
I think this would serve as a simpler example of what you want to achieve. There is no need to use external tools. Bash built in tools can do the job for you.
function DOSOMETHING {
while test $# -gt 0; do
case "$1" in
-first)
shift
first_argument=$1
shift
;;
-last)
shift
last_argument=$1
shift
;;
*)
echo "$1 is not a recognized flag!"
return 1;
;;
esac
done
echo "First argument : $first_argument";
echo "Last argument : $last_argument";
}
This will allow you to use flags so no matter which order you are passing the parameters you will get the proper behavior.
Example :
DOSOMETHING -last "Adios" -first "Hola"
Output :
First argument : Hola
Last argument : Adios
You can add this function to your profile or put it inside of a script.
Thanks!
Edit :
Save this as a a file and then execute it as yourfile.sh -last "Adios" -first "Hola"
#!/bin/bash
while test $# -gt 0; do
case "$1" in
-first)
shift
first_argument=$1
shift
;;
-last)
shift
last_argument=$1
shift
;;
*)
echo "$1 is not a recognized flag!"
return 1;
;;
esac
done
echo "First argument : $first_argument";
echo "Last argument : $last_argument";
Another alternative would be to use something like the below example which would allow you to use long --image or short -i tags and also allow compiled -i="example.jpg" or separate -i example.jpg methods of passing in arguments.
# declaring a couple of associative arrays
declare -A arguments=();
declare -A variables=();
# declaring an index integer
declare -i index=1;
# any variables you want to use here
# on the left left side is argument label or key (entered at the command line along with it's value)
# on the right side is the variable name the value of these arguments should be mapped to.
# (the examples above show how these are being passed into this script)
variables["-gu"]="git_user";
variables["--git-user"]="git_user";
variables["-gb"]="git_branch";
variables["--git-branch"]="git_branch";
variables["-dbr"]="db_fqdn";
variables["--db-redirect"]="db_fqdn";
variables["-e"]="environment";
variables["--environment"]="environment";
# $# here represents all arguments passed in
for i in "$#"
do
arguments[$index]=$i;
prev_index="$(expr $index - 1)";
# this if block does something akin to "where $i contains ="
# "%=*" here strips out everything from the = to the end of the argument leaving only the label
if [[ $i == *"="* ]]
then argument_label=${i%=*}
else argument_label=${arguments[$prev_index]}
fi
# this if block only evaluates to true if the argument label exists in the variables array
if [[ -n ${variables[$argument_label]} ]]
then
# dynamically creating variables names using declare
# "#$argument_label=" here strips out the label leaving only the value
if [[ $i == *"="* ]]
then declare ${variables[$argument_label]}=${i#$argument_label=}
else declare ${variables[$argument_label]}=${arguments[$index]}
fi
fi
index=index+1;
done;
# then you could simply use the variables like so:
echo "$git_user";
I like Robert McMahan's answer the best here as it seems the easiest to make into sharable include files for any of your scripts to use. But it seems to have a flaw with the line if [[ -n ${variables[$argument_label]} ]] throwing the message, "variables: bad array subscript". I don't have the rep to comment, and I doubt this is the proper 'fix,' but wrapping that if in if [[ -n $argument_label ]] ; then cleans it up.
Here's the code I ended up with, if you know a better way please add a comment to Robert's answer.
Include File "flags-declares.sh"
# declaring a couple of associative arrays
declare -A arguments=();
declare -A variables=();
# declaring an index integer
declare -i index=1;
Include File "flags-arguments.sh"
# $# here represents all arguments passed in
for i in "$#"
do
arguments[$index]=$i;
prev_index="$(expr $index - 1)";
# this if block does something akin to "where $i contains ="
# "%=*" here strips out everything from the = to the end of the argument leaving only the label
if [[ $i == *"="* ]]
then argument_label=${i%=*}
else argument_label=${arguments[$prev_index]}
fi
if [[ -n $argument_label ]] ; then
# this if block only evaluates to true if the argument label exists in the variables array
if [[ -n ${variables[$argument_label]} ]] ; then
# dynamically creating variables names using declare
# "#$argument_label=" here strips out the label leaving only the value
if [[ $i == *"="* ]]
then declare ${variables[$argument_label]}=${i#$argument_label=}
else declare ${variables[$argument_label]}=${arguments[$index]}
fi
fi
fi
index=index+1;
done;
Your "script.sh"
. bin/includes/flags-declares.sh
# any variables you want to use here
# on the left left side is argument label or key (entered at the command line along with it's value)
# on the right side is the variable name the value of these arguments should be mapped to.
# (the examples above show how these are being passed into this script)
variables["-gu"]="git_user";
variables["--git-user"]="git_user";
variables["-gb"]="git_branch";
variables["--git-branch"]="git_branch";
variables["-dbr"]="db_fqdn";
variables["--db-redirect"]="db_fqdn";
variables["-e"]="environment";
variables["--environment"]="environment";
. bin/includes/flags-arguments.sh
# then you could simply use the variables like so:
echo "$git_user";
echo "$git_branch";
echo "$db_fqdn";
echo "$environment";
#!/bin/bash
if getopts "n:" arg; then
echo "Welcome $OPTARG"
fi
Save it as sample.sh
and try running
sh sample.sh -n John
in your terminal.
If you're familiar with Python argparse, and don't mind calling python to parse bash arguments, there is a piece of code I found really helpful and super easy to use called argparse-bash
https://github.com/nhoffman/argparse-bash
Example take from their example.sh script:
#!/bin/bash
source $(dirname $0)/argparse.bash || exit 1
argparse "$#" <<EOF || exit 1
parser.add_argument('infile')
parser.add_argument('outfile')
parser.add_argument('-a', '--the-answer', default=42, type=int,
help='Pick a number [default %(default)s]')
parser.add_argument('-d', '--do-the-thing', action='store_true',
default=False, help='store a boolean [default %(default)s]')
parser.add_argument('-m', '--multiple', nargs='+',
help='multiple values allowed')
EOF
echo required infile: "$INFILE"
echo required outfile: "$OUTFILE"
echo the answer: "$THE_ANSWER"
echo -n do the thing?
if [[ $DO_THE_THING ]]; then
echo " yes, do it"
else
echo " no, do not do it"
fi
echo -n "arg with multiple values: "
for a in "${MULTIPLE[#]}"; do
echo -n "[$a] "
done
echo
I had trouble using getopts with multiple flags, so I wrote this code. It uses a modal variable to detect flags, and to use those flags to assign arguments to variables.
Note that, if a flag shouldn't have an argument, something other than setting CURRENTFLAG can be done.
for MYFIELD in "$#"; do
CHECKFIRST=`echo $MYFIELD | cut -c1`
if [ "$CHECKFIRST" == "-" ]; then
mode="flag"
else
mode="arg"
fi
if [ "$mode" == "flag" ]; then
case $MYFIELD in
-a)
CURRENTFLAG="VARIABLE_A"
;;
-b)
CURRENTFLAG="VARIABLE_B"
;;
-c)
CURRENTFLAG="VARIABLE_C"
;;
esac
elif [ "$mode" == "arg" ]; then
case $CURRENTFLAG in
VARIABLE_A)
VARIABLE_A="$MYFIELD"
;;
VARIABLE_B)
VARIABLE_B="$MYFIELD"
;;
VARIABLE_C)
VARIABLE_C="$MYFIELD"
;;
esac
fi
done
So here it is my solution. I wanted to be able to handle boolean flags without hyphen, with one hyphen, and with two hyphen as well as parameter/value assignment with one and two hyphens.
# Handle multiple types of arguments and prints some variables
#
# Boolean flags
# 1) No hyphen
# create Assigns `true` to the variable `CREATE`.
# Default is `CREATE_DEFAULT`.
# delete Assigns true to the variable `DELETE`.
# Default is `DELETE_DEFAULT`.
# 2) One hyphen
# a Assigns `true` to a. Default is `false`.
# b Assigns `true` to b. Default is `false`.
# 3) Two hyphens
# cats Assigns `true` to `cats`. By default is not set.
# dogs Assigns `true` to `cats`. By default is not set.
#
# Parameter - Value
# 1) One hyphen
# c Assign any value you want
# d Assign any value you want
#
# 2) Two hyphens
# ... Anything really, whatever two-hyphen argument is given that is not
# defined as flag, will be defined with the next argument after it.
#
# Example:
# ./parser_example.sh delete -a -c VA_1 --cats --dir /path/to/dir
parser() {
# Define arguments with one hyphen that are boolean flags
HYPHEN_FLAGS="a b"
# Define arguments with two hyphens that are boolean flags
DHYPHEN_FLAGS="cats dogs"
# Iterate over all the arguments
while [ $# -gt 0 ]; do
# Handle the arguments with no hyphen
if [[ $1 != "-"* ]]; then
echo "Argument with no hyphen!"
echo $1
# Assign true to argument $1
declare $1=true
# Shift arguments by one to the left
shift
# Handle the arguments with one hyphen
elif [[ $1 == "-"[A-Za-z0-9]* ]]; then
# Handle the flags
if [[ $HYPHEN_FLAGS == *"${1/-/}"* ]]; then
echo "Argument with one hyphen flag!"
echo $1
# Remove the hyphen from $1
local param="${1/-/}"
# Assign true to $param
declare $param=true
# Shift by one
shift
# Handle the parameter-value cases
else
echo "Argument with one hyphen value!"
echo $1 $2
# Remove the hyphen from $1
local param="${1/-/}"
# Assign argument $2 to $param
declare $param="$2"
# Shift by two
shift 2
fi
# Handle the arguments with two hyphens
elif [[ $1 == "--"[A-Za-z0-9]* ]]; then
# NOTE: For double hyphen I am using `declare -g $param`.
# This is the case because I am assuming that's going to be
# the final name of the variable
echo "Argument with two hypens!"
# Handle the flags
if [[ $DHYPHEN_FLAGS == *"${1/--/}"* ]]; then
echo $1 true
# Remove the hyphens from $1
local param="${1/--/}"
# Assign argument $2 to $param
declare -g $param=true
# Shift by two
shift
# Handle the parameter-value cases
else
echo $1 $2
# Remove the hyphens from $1
local param="${1/--/}"
# Assign argument $2 to $param
declare -g $param="$2"
# Shift by two
shift 2
fi
fi
done
# Default value for arguments with no hypheb
CREATE=${create:-'CREATE_DEFAULT'}
DELETE=${delete:-'DELETE_DEFAULT'}
# Default value for arguments with one hypen flag
VAR1=${a:-false}
VAR2=${b:-false}
# Default value for arguments with value
# NOTE1: This is just for illustration in one line. We can well create
# another function to handle this. Here I am handling the cases where
# we have a full named argument and a contraction of it.
# For example `--arg1` can be also set with `-c`.
# NOTE2: What we are doing here is to check if $arg is defined. If not,
# check if $c was defined. If not, assign the default value "VD_"
VAR3=$(if [[ $arg1 ]]; then echo $arg1; else echo ${c:-"VD_1"}; fi)
VAR4=$(if [[ $arg2 ]]; then echo $arg2; else echo ${d:-"VD_2"}; fi)
}
# Pass all the arguments given to the script to the parser function
parser "$#"
echo $CREATE $DELETE $VAR1 $VAR2 $VAR3 $VAR4 $cats $dir
Some references
The main procedure was found here.
More about passing all the arguments to a function here.
More info regarding default values here.
More info about declare do $ bash -c "help declare".
More info about shift do $ bash -c "help shift".

How to parse $QUERY_STRING from a bash CGI script?

I have a bash script that is being used in a CGI. The CGI sets the $QUERY_STRING environment variable by reading everything after the ? in the URL. For example, http://example.com?a=123&b=456&c=ok sets QUERY_STRING=a=123&b=456&c=ok.
Somewhere I found the following ugliness:
b=$(echo "$QUERY_STRING" | sed -n 's/^.*b=\([^&]*\).*$/\1/p' | sed "s/%20/ /g")
which will set $b to whatever was found in $QUERY_STRING for b. However, my script has grown to have over ten input parameters. Is there an easier way to automatically convert the parameters in $QUERY_STRING into environment variables usable by bash?
Maybe I'll just use a for loop of some sort, but it'd be even better if the script was smart enough to automatically detect each parameter and maybe build an array that looks something like this:
${parm[a]}=123
${parm[b]}=456
${parm[c]}=ok
How could I write code to do that?
Try this:
saveIFS=$IFS
IFS='=&'
parm=($QUERY_STRING)
IFS=$saveIFS
Now you have this:
parm[0]=a
parm[1]=123
parm[2]=b
parm[3]=456
parm[4]=c
parm[5]=ok
In Bash 4, which has associative arrays, you can do this (using the array created above):
declare -A array
for ((i=0; i<${#parm[#]}; i+=2))
do
array[${parm[i]}]=${parm[i+1]}
done
which will give you this:
array[a]=123
array[b]=456
array[c]=ok
Edit:
To use indirection in Bash 2 and later (using the parm array created above):
for ((i=0; i<${#parm[#]}; i+=2))
do
declare var_${parm[i]}=${parm[i+1]}
done
Then you will have:
var_a=123
var_b=456
var_c=ok
You can access these directly:
echo $var_a
or indirectly:
for p in a b c
do
name="var$p"
echo ${!name}
done
If possible, it's better to avoid indirection since it can make code messy and be a source of bugs.
you can break $QUERY down using IFS. For example, setting it to &
$ QUERY="a=123&b=456&c=ok"
$ echo $QUERY
a=123&b=456&c=ok
$ IFS="&"
$ set -- $QUERY
$ echo $1
a=123
$ echo $2
b=456
$ echo $3
c=ok
$ array=($#)
$ for i in "${array[#]}"; do IFS="=" ; set -- $i; echo $1 $2; done
a 123
b 456
c ok
And you can save to a hash/dictionary in Bash 4+
$ declare -A hash
$ for i in "${array[#]}"; do IFS="=" ; set -- $i; hash[$1]=$2; done
$ echo ${hash["b"]}
456
Please don't use the evil eval junk.
Here's how you can reliably parse the string and get an associative array:
declare -A param
while IFS='=' read -r -d '&' key value && [[ -n "$key" ]]; do
param["$key"]=$value
done <<<"${QUERY_STRING}&"
If you don't like the key check, you could do this instead:
declare -A param
while IFS='=' read -r -d '&' key value; do
param["$key"]=$value
done <<<"${QUERY_STRING:+"${QUERY_STRING}&"}"
Listing all the keys and values from the array:
for key in "${!param[#]}"; do
echo "$key: ${param[$key]}"
done
I packaged the sed command up into another script:
$cat getvar.sh
s='s/^.*'${1}'=\([^&]*\).*$/\1/p'
echo $QUERY_STRING | sed -n $s | sed "s/%20/ /g"
and I call it from my main cgi as:
id=`./getvar.sh id`
ds=`./getvar.sh ds`
dt=`./getvar.sh dt`
...etc, etc - you get idea.
works for me even with a very basic busybox appliance (my PVR in this case).
To converts the contents of QUERY_STRING into bash variables use the following command:
eval $(echo ${QUERY_STRING//&/;})
The inner step, echo ${QUERY_STRING//&/;}, substitutes all ampersands with semicolons producing a=123;b=456;c=ok which the eval then evaluates into the current shell.
The result can then be used as bash variables.
echo $a
echo $b
echo $c
The assumptions are:
values will never contain '&'
values will never contain ';'
QUERY_STRING will never contain malicious code
While the accepted answer is probably the most beautiful one, there might be cases where security is super-important, and it needs to be also well-visible from your script.
In such a case, first I wouldn't use bash for the task, but if it should be done on some reason, it might be better to avoid these new array - dictionary features, because you can't be sure, how exactly are they escaped.
In this case, the good old primitive solutions might work:
QS="${QUERY_STRING}"
while [ "${QS}" != "" ]
do
nameval="${QS%%&*}"
QS="${QS#$nameval}"
QS="${QS#&}"
name="${nameval%%=*}"
val="${nameval#$name}"
val="${nameval#=}"
# and here we have $name and $val as names and values
# ...
done
This iterates on the name-value pairs of the QUERY_STRING, and there is no way to circumvent it with any tricky escape sequence - the " is a very strong thing in bash, except a single variable name substitution, which is fully controlled by us, nothing can be tricked.
Furthermore, you can inject your own processing code into "# ...". This enables you to allow only your own, well-defined (and, ideally, short) list of the allowed variable names. Needless to say, LD_PRELOAD shouldn't be one of them. ;-)
Furthermore, no variable will be exported, and exclusively QS, nameval, name and val is used.
Following the correct answer, I've done myself some changes to support array variables like in this other question. I added also a decode function of which I can not find the author to give some credit.
Code appears somewhat messy, but it works. Changes and other recommendations would be greatly appreciated.
function cgi_decodevar() {
[ $# -ne 1 ] && return
local v t h
# replace all + with whitespace and append %%
t="${1//+/ }%%"
while [ ${#t} -gt 0 -a "${t}" != "%" ]; do
v="${v}${t%%\%*}" # digest up to the first %
t="${t#*%}" # remove digested part
# decode if there is anything to decode and if not at end of string
if [ ${#t} -gt 0 -a "${t}" != "%" ]; then
h=${t:0:2} # save first two chars
t="${t:2}" # remove these
v="${v}"`echo -e \\\\x${h}` # convert hex to special char
fi
done
# return decoded string
echo "${v}"
return
}
saveIFS=$IFS
IFS='=&'
VARS=($QUERY_STRING)
IFS=$saveIFS
for ((i=0; i<${#VARS[#]}; i+=2))
do
curr="$(cgi_decodevar ${VARS[i]})"
next="$(cgi_decodevar ${VARS[i+2]})"
prev="$(cgi_decodevar ${VARS[i-2]})"
value="$(cgi_decodevar ${VARS[i+1]})"
array=${curr%"[]"}
if [ "$curr" == "$next" ] && [ "$curr" != "$prev" ] ;then
j=0
declare var_${array}[$j]="$value"
elif [ $i -gt 1 ] && [ "$curr" == "$prev" ]; then
j=$((j + 1))
declare var_${array}[$j]="$value"
else
declare var_$curr="$value"
fi
done
I would simply replace the & to ;. It will become to something like:
a=123;b=456;c=ok
So now you need just evaluate and read your vars:
eval `echo "${QUERY_STRING}"|tr '&' ';'`
echo $a
echo $b
echo $c
A nice way to handle CGI query strings is to use Haserl which acts as a wrapper around your Bash cgi script, and offers convenient and secure query string parsing.
To bring this up to date, if you have a recent Bash version then you can achieve this with regular expressions:
q="$QUERY_STRING"
re1='^(\w+=\w+)&?'
re2='^(\w+)=(\w+)$'
declare -A params
while [[ $q =~ $re1 ]]; do
q=${q##*${BASH_REMATCH[0]}}
[[ ${BASH_REMATCH[1]} =~ $re2 ]] && params+=([${BASH_REMATCH[1]}]=${BASH_REMATCH[2]})
done
If you don't want to use associative arrays then just change the penultimate line to do what you want. For each iteration of the loop the parameter is in ${BASH_REMATCH[1]} and its value is in ${BASH_REMATCH[2]}.
Here is the same thing as a function in a short test script that iterates over the array outputs the query string's parameters and their values
#!/bin/bash
QUERY_STRING='foo=hello&bar=there&baz=freddy'
get_query_string() {
local q="$QUERY_STRING"
local re1='^(\w+=\w+)&?'
local re2='^(\w+)=(\w+)$'
while [[ $q =~ $re1 ]]; do
q=${q##*${BASH_REMATCH[0]}}
[[ ${BASH_REMATCH[1]} =~ $re2 ]] && eval "$1+=([${BASH_REMATCH[1]}]=${BASH_REMATCH[2]})"
done
}
declare -A params
get_query_string params
for k in "${!params[#]}"
do
v="${params[$k]}"
echo "$k : $v"
done
Note the parameters end up in the array in reverse order (it's associative so that shouldn't matter).
why not this
$ echo "${QUERY_STRING}"
name=carlo&last=lanza&city=pfungen-CH
$ saveIFS=$IFS
$ IFS='&'
$ eval $QUERY_STRING
$ IFS=$saveIFS
now you have this
name = carlo
last = lanza
city = pfungen-CH
$ echo "name is ${name}"
name is carlo
$ echo "last is ${last}"
last is lanza
$ echo "city is ${city}"
city is pfungen-CH
#giacecco
To include a hiphen in the regex you could change the two lines as such in answer from #starfry.
Change these two lines:
local re1='^(\w+=\w+)&?'
local re2='^(\w+)=(\w+)$'
To these two lines:
local re1='^(\w+=(\w+|-|)+)&?'
local re2='^(\w+)=((\w+|-|)+)$'
For all those who couldn't get it working with the posted answers (like me),
this guy figured it out.
Can't upvote his post unfortunately...
Let me repost the code here real quick:
#!/bin/sh
if [ "$REQUEST_METHOD" = "POST" ]; then
if [ "$CONTENT_LENGTH" -gt 0 ]; then
read -n $CONTENT_LENGTH POST_DATA <&0
fi
fi
#echo "$POST_DATA" > data.bin
IFS='=&'
set -- $POST_DATA
#2- Value1
#4- Value2
#6- Value3
#8- Value4
echo $2 $4 $6 $8
echo "Content-type: text/html"
echo ""
echo "<html><head><title>Saved</title></head><body>"
echo "Data received: $POST_DATA"
echo "</body></html>"
Hope this is of help for anybody.
Cheers
Actually I liked bolt's answer, so I made a version which works with Busybox as well (ash in Busybox does not support here string).
This code will accept key1 and key2 parameters, all others will be ignored.
while IFS= read -r -d '&' KEYVAL && [[ -n "$KEYVAL" ]]; do
case ${KEYVAL%=*} in
key1) KEY1=${KEYVAL#*=} ;;
key2) KEY2=${KEYVAL#*=} ;;
esac
done <<END
$(echo "${QUERY_STRING}&")
END
One can use the bash-cgi.sh, which processes :
the query string into the $QUERY_STRING_GET key and value array;
the post request data (x-www-form-urlencoded) into the $QUERY_STRING_POST key and value array;
the cookies data into the $HTTP_COOKIES key and value array.
Demands bash version 4.0 or higher (to define the key and value arrays above).
All processing is made by bash only (i.e. in an one process) without any external dependencies and additional processes invoking.
It has:
the check for max length of data, which can be transferred to it's input,
as well as processed as query string and cookies;
the redirect() procedure to produce redirect to itself with the extension changed to .html (it is useful for an one page's sites);
the http_header_tail() procedure to output the last two strings of the HTTP(S) respond's header;
the $REMOTE_ADDR value sanitizer from possible injections;
the parser and evaluator of the escaped UTF-8 symbols embedded into the values passed to the $QUERY_STRING_GET, $QUERY_STRING_POST and $HTTP_COOKIES;
the sanitizer of the $QUERY_STRING_GET, $QUERY_STRING_POST and $HTTP_COOKIES values against possible SQL injections (the escaping like the mysql_real_escape_string php function does, plus the escaping of # and $).
It is available here:
https://github.com/VladimirBelousov/fancy_scripts
This works in dash using for in loop
IFS='&'
for f in $query_string; do
value=${f##*=}
key=${f%%=*}
# if you need environment variable -> eval "qs_$key=$value"
done

Test for a Bash variable being unset, using a function

A simple Bash variable test goes:
${varName:? "${varName} is not defined"}
I'd like to reuse this, by putting it in a function. How can I do it?
The following fails
#
# Test a variable exists
tvar(){
val=${1:? "${1} must be defined, preferably in $basedir"}
if [ -z ${val} ]
then
echo Zero length value
else
echo ${1} exists, value ${1}
fi
}
I.e., I need to exit if the test fails.
Thanks to lhunath's answer, I was led to a part of the Bash man page that I've overlooked hundreds of times:
When not performing substring expansion, bash tests for a parameter that is unset or null; omitting the colon results in a test only for a parameter that is unset.
This prompted me to create the following truth table:
Unset
Set, but null
Set and not null
Meaning
${var-_}
T
F
T
Not null or not set
${var:-_}
T
T
T
Always true, use for subst.
$var
F
F
T
'var' is set and not null
${!var[#]}
F
T
T
'var' is set
This table introduces the specification in the last row. The Bash man page says "If name is not an array, expands to 0 if name is set and null otherwise." For purposes of this truth table, it behaves the same even if it's an array.
You're looking for indirection.
assertNotEmpty() {
: "${!1:? "$1 is empty, aborting."}"
}
That causes the script to abort with an error message if you do something like this:
$ foo=""
$ assertNotEmpty foo
bash: !1: foo is empty, aborting.
If you just want to test whether foo is empty, instead of aborting the script, use this instead of a function:
[[ $foo ]]
For example:
until read -p "What is your name? " name && [[ $name ]]; do
echo "You didn't enter your name. Please, try again." >&2
done
Also, note that there is a very important difference between an empty and an unset parameter. You should take care not to confuse these terms! An empty parameter is one that is set, but just set to an empty string. An unset parameter is one that doesn't exist at all.
The previous examples all test for empty parameters. If you want to test for unset parameters and consider all set parameters OK, whether they're empty or not, use this:
[[ ! $foo && ${foo-_} ]]
Use it in a function like this:
assertIsSet() {
[[ ! ${!1} && ${!1-_} ]] && {
echo "$1 is not set, aborting." >&2
exit 1
}
}
Which only aborts the script when the parameter name you pass denotes a parameter that isn't set:
$ ( foo="blah"; assertIsSet foo; echo "Still running." )
Still running.
$ ( foo=""; assertIsSet foo; echo "Still running." )
Still running.
$ ( unset foo; assertIsSet foo; echo "Still running." )
foo is not set, aborting.
You want to use [ -z ${parameter+word} ]
Some part of man bash:
Parameter Expansion
...
In each of the cases below, word is subject to tilde expansion, parameter expansion, command substitution, and
arithmetic expansion. When not performing substring expansion, bash tests for a parameter that is unset or null;
omitting the colon results in a test only for a parameter that is unset.
...
${parameter:+word}
Use Alternate Value. If parameter is null or unset, nothing is substituted, otherwise the expansion of
word is substituted.
...
in other words:
${parameter+word}
Use Alternate Value. If parameter is unset, nothing is substituted, otherwise the expansion of
word is substituted.
some examples:
$ set | grep FOOBAR
$ if [ -z "${FOOBAR+something}" ]; then echo "it is unset"; fi
it is unset
$ declare FOOBAR
$ if [ -z "${FOOBAR+something}" ]; then echo "it is unset"; fi
$ FOOBAR=
$ if [ -z "${FOOBAR+something}" ]; then echo "it is unset"; fi
$ FOOBAR=1
$ if [ -z "${FOOBAR+something}" ]; then echo "it is unset"; fi
$ unset FOOBAR
$ if [ -z "${FOOBAR+something}" ]; then echo "it is unset"; fi
it is unset
$
This function tests for variables that are currently set. The variable may even be an array. Note that in Bash: 0 == TRUE, 1 == FALSE.
function var.defined {
eval '[[ ${!'$1'[#]} ]]'
}
# Typical usage of var.defined {}
declare you="Your Name Here" ref='you';
read -p "What's your name: " you;
if var.defined you; then # Simple demo using literal text
echo "BASH recognizes $you";
echo "BASH also knows a reference to $ref as ${!ref}, by indirection.";
fi
unset you # Have just been killed by a master :D
if ! var.defined $ref; then # Standard demo using an expanded literal value
echo "BASH doesn't know $ref any longer";
fi
read -s -N 1 -p "Press any key to continue...";
echo "";
So to be clear here, the function tests literal text. Every time a command is called in Bash, variables are generally 'swapped-out' or 'substituted' with the underlying value unless:
$varRef ($) is escaped: $varRef
$varRef is single quoted '$varRef'
I.e., I need to exit if the test fails.
The code:
${varName:? "${varName} is not defined"}
will return a nonzero exit code when there is not a variable named "varName". The exit code of the last command is saved in $?.
About your code:
val=${1:? "${1} must be defined, preferably in $basedir"}
Maybe it is not doing what you need. In the case that $1 is not defined, the "${1}" will be substituted with nothing. Probably you want use the single quotes that literally writes ${1} without substitution.
val=${1:? '${1} must be defined, preferably in $basedir'
I am unsure if this is exactly what you want, but a handy trick I use when writing a new and complex script is to use "set -o":
set -o # Will make the script bomb out when it finds an unset variable
For example,
$ grep '$1' chex.sh
case "$1" in
$ ./chex.sh
./chex.sh: line 111: $1: unbound variable
$ ./chex.sh foo
incorrect/no options passed.. exiting
if set | grep -q '^VARIABLE='
then
echo VARIABLE is set
fi

Resources