Creating sub variables from a dynamic list of variables in bash? - bash

Given a dynamic list of things, I want to add a counter variable for each thing in the list to be accessed from outside the loop like so - the count for $X would ideally be given by:
for THING in `cat list.txt`
#do stuff with $THING
done
echo $Xcount
Obviously "$THING"count=value doesn't work, nor did a couple iterations of $ and \$ give me what I want.
The command:
echo "$THING"count=value
is what I would like executed, but back ticking it alone yields this error:
echo "$THING"count=value
bash: Xcount=value: command not found
Quoting the value doesn't change results.
Is there a clean way to address this in this fashion?Note that each $THING will hold a string value, not a number and that the contents of the list of things will vary over time. If "W" appears I'll need a counter for it and if X vanishes from the list I'll still need a counter for it as well - so I don't think a pair of arrays (one for items, one for counts) is the way to go, even if it can be made to work by expanding and resorting it every time - I want O(1) access time, memory space is "free" in this example, IO and CPU are my optimization constraints.
Rici's suggestion is serviceable, (loop based example follows for those who are good enough with that solution):
unset COUNT
declare -A COUNT
for ITEM in X Y Z; do
THING=$ITEM;
((COUNT[$THING]+=1));
done
for i in "${!COUNT[#]}"
do
echo -n "key : $i" " ";
echo "value: ${COUNT[$i]}";
done
but my preference would be to actually be able to declare variables Xcount Ycount and Zcount if the list provided were X Y and Z. Is there a way to do that, i.e. to execute this from inside the loop:
echo "$THING"count=value
withOUT getting "bash: Xcount=value: command not found"
?

Here, the let command is useful to allow variable names to be dynamically created. To access the values, you might want indirect variables:
for thing in foo bar baz foo bar foo; do
let "${thing}count++"
done
for thing in foo bar baz; do
varname=${thing}count
echo "$varname = ${!varname}"
done
foocount = 3
barcount = 2
bazcount = 1
I would not actually recommend doing this: associative arrays are conceptually much easier to deal with.
Get out of the habit of using ALL_CAPS_VARNAMES: one day you'll use PATH and wonder you can't find any commands.

Make $COUNT an associative array and set
COUNT[$THING]=value
Silly example for the skeptical:
$ declare -A COUNT
$ THING=X
$ COUNT[$THING]=3
$ THING=Y
$ COUNT[$THING]=42
$ THING=X
$ ((COUNT[$THING]+=7))
$ echo ${COUNT[X]} ${COUNT[Y]}
10 42

Related

How can I create an array whose name includes a variable?

I'm having some trouble writing a command that includes a String of a variable in Bash and wanted to know the correct way to do it.
I want to try and fill the Row arrays with the numbers 1-9 but I'm getting myself stuck when trying to pass a variable Row$Line[$i]=$i.
Row0=()
Row1=()
Row2=()
FillArrays() {
for Line in $(seq 0 2)
do
for i in $(seq 1 9)
do
Row$Line[$i]=$i
done
done
}
I can get the desired result if I echo the command but I assume that is just because it is a String.
I want the for loop to select each row and add the numbers 1-9 in each array.
FillArrays() {
for ((Line=0; Line<8; Line++)); do
declare -g -a "Row$Line" # Ensure that RowN exists as an array
declare -n currRow="Row$Line" # make currRow an alias for that array
for ((i=0; i<9; i++)); do # perform our inner loop...
currRow+=( "$i" ) # ...and populate the target array...
done
unset -n currRow # then clear the alias so it can be reassigned later.
done
}
References:
https://wiki.bash-hackers.org/syntax/ccmd/c_for describes the C-style for loop in bash
BashFAQ #6 discusses indirect reference and assignment in detail, including techniques that precede namevars.
Variable expansion happens too late for an assignment to understand it. You can delay the assignment by using the declare builtin. -g is needed in a function to make the variable global.
Also, you probably don't want to use $Line as the array index, but $i, otherwise you wouldn't populate each line array with numbers 1..9.
#! /bin/bash
Row0=()
Row1=()
Row2=()
FillArrays() {
for Line in $(seq 0 8)
do
for i in $(seq 1 9)
do
declare -g Row$Line[$i]=$i
done
done
}
FillArrays
echo "${Row1[#]}"
But note that using variables as parts of variable names is dangerous. For me, needing this always means I need to switch from the shell to a real programming language.

copy the value of the variable rather than reference bash script

I am a bit new to the bash scripting. So please bear with me. I am trying to create a table and assign the values in for loop like this:
packages=("foo" "bar" "foobar")
packageMap=()
function test() {
i=0;
for package in "${packages[#]}"
do
echo $i
packageMap[$package]=$i
i=$(expr $i + 1)
done
}
test
echo the first value is ${packageMap["foo"]}
The output for this is:
0
1
2
the first value is 2
While my expected output is:
0
1
2
the first value is 0
So basically the variable's reference is being assigned to this rather than the value. How to solve this?
My bash version:
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)
TIA
bash 3.2 only has indexed arrays, so packageMap[$package] only works as intended if $package is an integer, not an arbitrary string.
(What you are observing is $package being evaluated in an arithmetic context, where foo, bar, and foobar are recursively expanded until you get an integer value. Undefined variables expand to 0, so packageMap[foo] is equivalent to packageMap[0].)
If you were using bash 4 or later, you could use an associative array:
packages=("foo" "bar" "foobar")
declare -A packageMap
test () {
i=0
for package in "${packages[#]}"
do
echo $i
packageMap[$package]=$i
i=$(($i + 1))
done
}
Given that i is the same as the index of each element of packages, you could also write
test () {
for i in "${!packages[#]}"; do
package=${packages[i]}
packageMap[$package]=$i
done
}
instead of explicitly incrementing i.
As chep says, there are no associative arrays in bash 3. That said, if you don't mind wasting a bit of CPU, you can use functions to similar effect:
#!/bin/bash
packages=("foo" "bar" "foobar")
function packagemap () {
local i
for i in "${!packages[#]}"; do
[[ ${packages[$i]} = $1 ]] && echo "$i" && return
done
echo "unknown"
}
echo "the first value is $(packagemap "foo")"
The ${!array[#]} construct expands to the set of indices for the array, which for a normally non-associative array consist of incrementing integers starting at 0. But array members can be removed without the indices being renumbered (i.e. unset packages[1]), so it's important to be able to refer to actual indices rather than assuming they're sequential with for loop that simply counts.
And I note that you're using Darwin. Remember that you really need it, you can install bash 4 using Homebrew or MacPorts.

Unable to set second to last command line argument to variable

Regardless of the number of arguments passed to my script, I would like for the second to the last argument to always represent a specific variable in my code.
Executing the program I'd type something like this:
sh myprogram.sh -a arg_a -b arg_b special specific
test=("${3}")
echo $test
The results will show 'special'. So using that same idea if I try this (since I won't know that number of arguments):
secondToLastArg=$(($#-1))
echo $secondToLastArg
The results will show '3'. How do I dynamically assign the second to last argument?
You need a bit of math to get the number you want ($(($#-1))), then use indirection (${!n}) to get the actual argument.
$ set -- a b c
$ echo $#
a b c
$ n=$(($#-1))
$ echo $n
2
$ echo ${!n}
b
$
Indirection (${!n}) tells bash to use the value of n as the name of the variable to use ($2, in this case).
You can use $# as array & array chopping methods:
echo ${#:$(($#-1)):1}
It means, use 1 element starting from $(($#-1))...
If some old versions of shells do not support ${array:start:length} syntax but support only ${array:start} syntax, use below hack:
echo ${#:$(($#-1))} | { read x y ; echo $x; } # OR
read x unused <<< `echo ${#:$(($#-1))}`

Create associative array in bash 3

After thoroughly searching for a way to create an associative array in bash, I found that declare -A array will do the trick. But the problem is, it is only for bash version 4 and the bash version the server has in our system is 3.2.16.
How can I achieve some sort of associative array-like hack in bash 3? The values will be passed to a script like
ARG=array[key];
./script.sh ${ARG}
EDIT: I know that I can do this in awk, or other tools but strict bash is needed for the scenario I am trying to solve.
Bash 3 has no associative arrays, so you're going to have to use some other language feature(s) for your purpose. Note that even under bash 4, the code you wrote doesn't do what you claim it does: ./script.sh ${ARG} does not pass the associative array to the child script, because ${ARG} expands to nothing when ARG is an associative array. You cannot pass an associative array to a child process, you need to encode it anyway.
You need to define some argument passing protocol between the parent script and the child script. A common one is to pass arguments in the form key=value. This assumes that the character = does not appear in keys.
You also need to figure out how to represent the associative array in the parent script and in the child script. They need not use the same representation.
A common method to represent an associative array is to use separate variables for each element, with a common naming prefix. This requires that the key name only consists of ASCII letters (of either case), digits and underscores. For example, instead of ${myarray[key]}, write ${myarray__key}. If the key is determined at run time, you need a round of expansion first: instead of ${myarray[$key]}, write
n=myarray__${key}; echo ${!n}
For an assignment, use printf -v. Note the %s format to printf to use the specified value. Do not write printf -v "myarray__${key}" %s "$value" since that would treat $value as a format and perform printf % expansion on it.
printf -v "myarray__${key}" %s "$value"
If you need to pass an associative array represented like this to a child process with the key=value argument representation, you can use ${!myarray__*} to enumerate over all the variables whose name begins with myarray__.
args=()
for k in ${!myarray__*}; do
n=$k
args+=("$k=${!n}")
done
In the child process, to convert arguments of the form key=value to separate variables with a prefix:
for x; do
if [[ $x != *=* ]]; then echo 1>&2 "KEY=VALUE expected, but got $x"; exit 120; fi
printf -v "myarray__${x%%=*}" %s "${x#*=}"
done
By the way, are you sure that this is what you need? Instead of calling a bash script from another bash script, you might want to run the child script in a subshell instead. That way it would inherit from all the variables of the parent.
Here is another post/explanation on associative arrays in bash 3 and older using parameter expansion:
https://stackoverflow.com/a/4444841
Gilles' method has a nice if statement to catch delimiter issues, sanitize oddball input ...etc. Use that.
If you are somewhat familiar with parameter expansion:
http://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
To use in your scenario [ as stated: sending to script ]:
Script 1:
sending_array.sh
# A pretend Python dictionary with bash 3
ARRAY=( "cow:moo"
"dinosaur:roar"
"bird:chirp"
"bash:rock" )
bash ./receive_arr.sh "${ARRAY[#]}"
Script 2: receive_arr.sh
argAry1=("$#")
function process_arr () {
declare -a hash=("${!1}")
for animal in "${hash[#]}"; do
echo "Key: ${animal%%:*}"
echo "Value: ${animal#*:}"
done
}
process_arr argAry1[#]
exit 0
Method 2, sourcing the second script:
Script 1:
sending_array.sh
source ./receive_arr.sh
# A pretend Python dictionary with bash 3
ARRAY=( "cow:moo"
"dinosaur:roar"
"bird:chirp"
"bash:rock" )
process_arr ARRAY[#]
Script 2: receive_arr.sh
function process_arr () {
declare -a hash=("${!1}")
for animal in "${hash[#]}"; do
echo "Key: ${animal%%:*}"
echo "Value: ${animal#*:}"
done
}
References:
Passing arrays as parameters in bash
If you don't want to handle a lot of variables, or keys are simply invalid variable identifiers, and your array is guaranteed to have less than 256 items, you can abuse function return values. This solution does not require any subshell as the value is readily available as a variable, nor any iteration so that performance screams. Also it's very readable, almost like the Bash 4 version.
Here's the most basic version:
hash_index() {
case $1 in
'foo') return 0;;
'bar') return 1;;
'baz') return 2;;
esac
}
hash_vals=("foo_val"
"bar_val"
"baz_val");
hash_index "foo"
echo ${hash_vals[$?]}
More details and variants in this answer
You can write the key-value pairs to a file and then grep by key. If you use a pattern like
key=value
then you can egrep for ^key= which makes this pretty safe.
To "overwrite" a value, just append the new value at the end of the file and use tail -1 to get just the last result of egrep
Alternatively, you can put this information into a normal array using key=value as value for the array and then iterator over the array to find the value.
This turns out to be ridiculously easy. I had to convert a bash 4 script that used a bunch of associative arrays to bash 3. These two helper functions did it all:
array_exp() {
exp=${#//[/__}
eval "${exp//]}"
}
array_clear() {
unset $(array_exp "echo \${!$1__*}")
}
I'm flabbergasted that this actually works, but that's the beauty of bash.
E.g.
((all[ping_lo] += counts[ping_lo]))
becomes
array_exp '((all[ping_lo] += counts[ping_lo]))'
Or this print statement:
printf "%3d" ${counts[ping_lo]} >> $return
becomes
array_exp 'printf "%3d" ${counts[ping_lo]}' >> $return
The only syntax that changes is clearing. This:
counts=()
becomes
array_clear counts
and you're set. You could easily tell array_exp to recognize expressions like "=()" and handle them by rewriting them as array_clear expressions, but I prefer the simplicity of the above two functions.

Capturing multiple line output into a Bash variable

I've got a script 'myscript' that outputs the following:
abc
def
ghi
in another script, I call:
declare RESULT=$(./myscript)
and $RESULT gets the value
abc def ghi
Is there a way to store the result either with the newlines, or with '\n' character so I can output it with 'echo -e'?
Actually, RESULT contains what you want — to demonstrate:
echo "$RESULT"
What you show is what you get from:
echo $RESULT
As noted in the comments, the difference is that (1) the double-quoted version of the variable (echo "$RESULT") preserves internal spacing of the value exactly as it is represented in the variable — newlines, tabs, multiple blanks and all — whereas (2) the unquoted version (echo $RESULT) replaces each sequence of one or more blanks, tabs and newlines with a single space. Thus (1) preserves the shape of the input variable, whereas (2) creates a potentially very long single line of output with 'words' separated by single spaces (where a 'word' is a sequence of non-whitespace characters; there needn't be any alphanumerics in any of the words).
Another pitfall with this is that command substitution — $() — strips trailing newlines. Probably not always important, but if you really want to preserve exactly what was output, you'll have to use another line and some quoting:
RESULTX="$(./myscript; echo x)"
RESULT="${RESULTX%x}"
This is especially important if you want to handle all possible filenames (to avoid undefined behavior like operating on the wrong file).
In case that you're interested in specific lines, use a result-array:
declare RESULT=($(./myscript)) # (..) = array
echo "First line: ${RESULT[0]}"
echo "Second line: ${RESULT[1]}"
echo "N-th line: ${RESULT[N]}"
In addition to the answer given by #l0b0 I just had the situation where I needed to both keep any trailing newlines output by the script and check the script's return code.
And the problem with l0b0's answer is that the 'echo x' was resetting $? back to zero... so I managed to come up with this very cunning solution:
RESULTX="$(./myscript; echo x$?)"
RETURNCODE=${RESULTX##*x}
RESULT="${RESULTX%x*}"
Parsing multiple output
Introduction
So your myscript output 3 lines, could look like:
myscript() { echo $'abc\ndef\nghi'; }
or
myscript() { local i; for i in abc def ghi ;do echo $i; done ;}
Ok this is a function, not a script (no need of path ./), but output is same
myscript
abc
def
ghi
Considering result code
To check for result code, test function will become:
myscript() { local i;for i in abc def ghi ;do echo $i;done;return $((RANDOM%128));}
1. Storing multiple output in one single variable, showing newlines
Your operation is correct:
RESULT=$(myscript)
About result code, you could add:
RCODE=$?
even in same line:
RESULT=$(myscript) RCODE=$?
Then
echo $RESULT $RCODE
abc def ghi 66
echo "$RESULT"
abc
def
ghi
echo ${RESULT#Q}
$'abc\ndef\nghi'
printf '%q\n' "$RESULT"
$'abc\ndef\nghi'
but for showing variable definition, use declare -p:
declare -p RESULT RCODE
declare -- RESULT="abc
def
ghi"
declare -- RCODE="66"
2. Parsing multiple output in array, using mapfile
Storing answer into myvar variable:
mapfile -t myvar < <(myscript)
echo ${myvar[2]}
ghi
Showing $myvar:
declare -p myvar
declare -a myvar=([0]="abc" [1]="def" [2]="ghi")
Considering result code
In case you have to check for result code, you could:
RESULT=$(myscript) RCODE=$?
mapfile -t myvar <<<"$RESULT"
declare -p myvar RCODE
declare -a myvar=([0]="abc" [1]="def" [2]="ghi")
declare -- RCODE="40"
3. Parsing multiple output by consecutives read in command group
{ read firstline; read secondline; read thirdline;} < <(myscript)
echo $secondline
def
Showing variables:
declare -p firstline secondline thirdline
declare -- firstline="abc"
declare -- secondline="def"
declare -- thirdline="ghi"
I often use:
{ read foo;read foo total use free foo ;} < <(df -k /)
Then
declare -p use free total
declare -- use="843476"
declare -- free="582128"
declare -- total="1515376"
Considering result code
Same prepended step:
RESULT=$(myscript) RCODE=$?
{ read firstline; read secondline; read thirdline;} <<<"$RESULT"
declare -p firstline secondline thirdline RCODE
declare -- firstline="abc"
declare -- secondline="def"
declare -- thirdline="ghi"
declare -- RCODE="50"
After trying most of the solutions here, the easiest thing I found was the obvious - using a temp file. I'm not sure what you want to do with your multiple line output, but you can then deal with it line by line using read. About the only thing you can't really do is easily stick it all in the same variable, but for most practical purposes this is way easier to deal with.
./myscript.sh > /tmp/foo
while read line ; do
echo 'whatever you want to do with $line'
done < /tmp/foo
Quick hack to make it do the requested action:
result=""
./myscript.sh > /tmp/foo
while read line ; do
result="$result$line\n"
done < /tmp/foo
echo -e $result
Note this adds an extra line. If you work on it you can code around it, I'm just too lazy.
EDIT: While this case works perfectly well, people reading this should be aware that you can easily squash your stdin inside the while loop, thus giving you a script that will run one line, clear stdin, and exit. Like ssh will do that I think? I just saw it recently, other code examples here: https://unix.stackexchange.com/questions/24260/reading-lines-from-a-file-with-bash-for-vs-while
One more time! This time with a different filehandle (stdin, stdout, stderr are 0-2, so we can use &3 or higher in bash).
result=""
./test>/tmp/foo
while read line <&3; do
result="$result$line\n"
done 3</tmp/foo
echo -e $result
you can also use mktemp, but this is just a quick code example. Usage for mktemp looks like:
filenamevar=`mktemp /tmp/tempXXXXXX`
./test > $filenamevar
Then use $filenamevar like you would the actual name of a file. Probably doesn't need to be explained here but someone complained in the comments.
How about this, it will read each line to a variable and that can be used subsequently !
say myscript output is redirected to a file called myscript_output
awk '{while ( (getline var < "myscript_output") >0){print var;} close ("myscript_output");}'

Resources