Get string from variable lists in Bash - bash

Question

I have a file (file.env) similar to this:
kw_var1='string1'
kw_var2='string 2'
kw_var3='this is string 3'
kw_var4='does not matter'
kw_var5='maybe'
w_var1=qwert_1
w_var2=qwert_2
w_var3=qwert_3
w_var4=qwert_4
and I need to create a string list_of_values which contains the values of all variables that start with kw_, i.e.
$ echo -e $list_of_values
should output:
'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe'
I tried to iterate over them, but cannot get this to work. My code:
list_of_values=$(for param in $(cat $file.env | grep "kw\_"); do echo $(echo -e '$'$param | cut -s -d '=' -f1); done)
but this is what I get:
$kw_var1 $kw_var2 $kw_var3 $kw_var4 $kw_var5
Note that:
the variable values will contain spaces;
list_of_values will be used as an argument to another function
Any ideas with what is wrong?
UPDATE:
When doing the final echo I used:
$ echo -e $list_of_values | tr '\n' ' '
to get everything in one line.

Answer 1

Trying your code
I tried your command and get this as output :
$kw_var1
$kw_var2
$kw_var3
$kw_var4
$kw_var5
You had the wrong output because you chose the first field when you used cut instead of the second.
Fixing cut command
for param in $(cat test.txt | grep "kw\_"); do echo $(echo '$'$param | cut -s -d '=' -f2); done
Returns :
'string1'
'string
'this
'does
'maybe'
Fixing IFS
You used a for in loop but it does not iterate over newlines, it iterates over spaces. You need to change the IFS (Internal Field Separator) variable first :
IFS=$'\n'; for param in $(cat <file> | grep "kw\_"); do echo $(echo $param | cut -s -d '=' -f2); done
Output :
'string1'
'string 2'
'this is string 3'
'does not matter'
'maybe'
Using printf
To get the output on one line, you can use printf instead of echo :
for param in $(cat <file> | grep "kw\_"); do printf "$(echo $param | cut -s -d '=' -f2) "; done; printf "\n"
Output :
'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe'
Using while
You could simplify the command and use a while read statement that iterates directly over lines :
cat <file> | grep "kw\_" | cut -d"=" -f2 | while read line; do printf "${line} "; done; printf "\n"
Using awk
Last but not least, you can use awk which radically simplifies your code:
awk -F"=" '/kw_/{printf "%s ", $2}END{print ""}' <file>
Output :
'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe'
If the extra space at the end of the line is annoying, you can do this :
awk -F"=" '/kw_/{printf "%s%s", delim, $2; delim=" "}END{print ""}' <file>
Awk explained :
# Using = as delimiter
awk -F"=" '
# If line contains kw_
/kw_/{
# Prints second field
printf "%s%s", delim, $2;
delim=" "
}
END{
# Prints newline
print ""
}' <file>
Final code
list_of_values=$(awk -F"=" '/kw_/{printf "%s%s", delim, $2; delim=" "}END{print ""}' $file.env)

Answer 2

$ cat tst.awk
/^kw_/ {
sub(/[^=]+=/,"")
str = str sep $0
sep = " "
}
END {
print str
}
e.g. note that it handles this=that in your desired output string correctly:
$ cat file
kw_var1='string1'
kw_var2='string 2'
kw_var3='this is string 3'
kw_var4='does not matter'
kw_var5='maybe'
kw_var6='this=that'
w_var1=qwert_1
w_var2=qwert_2
w_var3=qwert_3
w_var4=qwert_4
$ awk -f tst.awk file
'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe' 'this=that'
Updated: given what you've now told us in comments, here's how I'd do it assuming you need to access individual values by their tags sometimes, otherwise you could use a regular array instead of associative:
$ cat tst.sh
#!/bin/env bash
declare -A kw
declare -A w
while IFS= read -r line; do
tag="${line%%=*}"
val="${line#*=}"
case "$tag" in
kw* ) kw["$tag"]="$val" ;;
w* ) w["$tag"]="$val" ;;
?* ) printf 'Error: unexpected contents: "%s"\n' "$line"; exit 1;;
esac
done < file.env
printf '\nAll kw indices => values:\n'
for idx in "${!kw[#]}"; do
printf '\t%s => %s\n' "$idx" "${kw[$idx]}"
done
printf '\nAll kw values passed to a function (printf) at once:\n'
printf '\t%s\n' "${kw[#]}"
printf '\nAll w indices => values:\n'
for idx in "${!w[#]}"; do
printf '\t%s => %s\n' "$idx" "${w[$idx]}"
done
printf '\nAll w values passed to a function (printf) at once:\n'
printf '\t%s\n' "${w[#]}"
.
$ ./tst.sh
All kw indices => values:
kw_var4 => does not matter
kw_var5 => maybe
kw_var6 => this=that
kw_var1 => string1
kw_var2 => string 2
kw_var3 => this is string 3
All kw values passed to a function (printf) at once:
does not matter
maybe
this=that
string1
string 2
this is string 3
All w indices => values:
w_var3 => qwert_3
w_var2 => qwert_2
w_var1 => qwert_1
w_var4 => qwert_4
All w values passed to a function (printf) at once:
qwert_3
qwert_2
qwert_1
qwert_4
The above was run on this file.env without the redundant single quotes around the values, otherwise you'd just remove them in the script:
$ cat file.env
kw_var1=string1
kw_var2=string 2
kw_var3=this is string 3
kw_var4=does not matter
kw_var5=maybe
kw_var6=this=that
w_var1=qwert_1
w_var2=qwert_2
w_var3=qwert_3
w_var4=qwert_4
wrt our discussion in the comments and using printf '<%s>\n' in place of fitsort which I don't know and don't have:
$ list[0]='foo bar'; list[1]='etc'
$ printf '<%s>\n' "${list[#]}"
<foo bar>
<etc>
$ printf '<%s>\n' $(printf '%s\n' "${list[#]}")
<foo>
<bar>
<etc>
$ printf '<%s>\n' "$(printf '%s\n' "${list[#]}")"
<foo bar
etc>
See how the first version correctly simply passes the contents of list[] to the fitsort-replacement command while the others pass the strings output by printf to it instead?

Answer 3

Make two arrays of your bunch of variables, then you can easily iterate over them like this
#!/bin/bash
kw=(
'string1'
'string 2'
'this is string 3'
'does not matter'
'maybe'
)
w=(
'qwert_1'
'qwert_2'
'qwert_3'
'qwert_4'
)
for i in {1..5}
do
echo -n "\"${kw[$i]}\" "
done
echo
for i in {1..4}
do
echo -n "\"${w[$i]}\" "
done
echo

Answer 4

I used dynamic references.
$: out="$( . file.env; for r in ${!kw_*}; do printf "'%s' " "${!r}"; done; echo )"
$: echo "$out"
'string1' 'string 2' 'this is string 3' 'does not matter' 'maybe'

Answer 5

Your attempt uses a few practices that aren't recommended, and contains a few syntax errors:
You use $file.env, but it should be just file.env
Don't use for to read lines from a file (see this article)
echo $(cmd) is the same as just cmd plus globbing and word splitting, which often isn't what you want
echo -e '$'$param is going to print a literal $ sign
cut -f1 is selecting the first field, but you want the second one
This is a solution "in the spirit" of what you tried, but using just Bash:
list=$(
while IFS='=' read -r key val; do
[[ $key == kw_* ]] && printf '%s ' "$val"
done < file.env
)
list=${list% } # Remove trailing blank
If you deal with strings containing spaces, though, it's generally advisable to use an array instead. Since file.env is valid Bash, we can source the lines we're interested in and then build an array with the values of all the kw_ variables:
source <(grep '^kw_' file.env)
declare -n var
for var in "${!kw_#}"; do list+=("$var"); done
The array now contains one string per element, without the literal single quotes:
$ printf '%s\n' "${list[#]}"
string1
string 2
this is string 3
does not matter
maybe
declare -n sets var to a nameref: it is treated as if it actually were the variable whose name it holds. This requires Bash 4.3 or newer.