I am using a bash for loop to cycle through a directory and print the file size. My issue though is that I need to assign a unique variable to each of the values so that they can be used later but the array is inputting all the data as one element in the array. I have tried both a double for loop and a if statement with a nested for loop but did not get the right results.
Question: How can I fix the below code to match my needs or is there a better method?
declare -a byte
for b in /home/usr/frames/*
do
byte+=$(wc -c < $b)
done
declare -p byte
With associative array (if available)
#!/usr/bin/env bash
for b in /home/usr/frames/*; do
declare -A byte["$b"]=$(wc -c < "$b")
done
Use Parameter Expansion to extract just the file name.
declare -A byte["${b##*/}"]=$(wc -c < "$b")
Now check the value of byte
declare -p byte
A variation on OPs original code and jetchisel's answer:
unset byte
declare -A byte # -A == associative array
for b in /home/usr/frames/*
do
byte[$(basename "${b}")]=$(wc -c < "${b}")
done
declare -p byte
Using some *.txt files on my host:
$ pwd
/c/temp/cygwin
$ wc -c *.txt
22 file.txt
405 somefile.txt
214 test.txt
With the modified for clause:
unset byte
declare -A byte
for b in /c/temp/cygwin/*.txt
do
byte[$(basename "${b}")]=$(wc -c < "${b}")
done
declare -p byte
Generates:
declare -A byte=([somefile.txt]="405" [test.txt]="214" [file.txt]="22" )
Related
This question already has answers here:
Read values into a shell variable from a pipe
(17 answers)
Closed 12 months ago.
I just discovered mapfile in bash when shellcheck recommended it to my code to capture two shell parameters from a space-delimited string of two words. But, mapfile does not seem to function consistently. For example, this works as expected:
mapfile arr < myfile
The array arr is populated with entries corresponding to the lines of myfile. However, this does not work:
echo -e "hello\nworld" | mapfile arr
The array arr is not populated at all. And, this doesn't work either:
echo "hello world" | mapfile -d ' ' arr
I am not sure why it would make a difference where the standard input for the command comes from. I didn't know it would be possible to distinguish what the input came from, a file or a pipeline.
Any clues?
Note to moderator: It was suggested my question was a duplicate to Read values into a shell variable from a pipe . I do not agree. Nowhere is mapfile mentioned in that question, nor was there any other useful Q/A found in a SO search. In addition, that referenced question did not deal with shell parameter assignments. Therefore, this question and answers are valuable.
Technically the array is being populated; the issue is that mapfile is called in a sub-process which, when it exits (back to the command line) the array assignment is lost, ie, you can't pass assignments 'up' from a sub-process to a parent/calling process.
Try these instead:
$ mapfile -d ' ' arr < <(echo -e "hello\nworld")
$ typeset -p arr
declare -a arr=([0]=$'hello\nworld\n')
$ mapfile -d ' ' arr < <(echo "hello world")
$ typeset -p arr
declare -a arr=([0]="hello " [1]=$'world\n')
While these will populate the array ... you'll have to decide if this is what you were expecting to show up in the array. Perhaps the following are a bit closer to what's desired?
$ mapfile -t arr < <(echo -e "hello\nworld")
$ typeset -p arr
declare -a arr=([0]="hello" [1]="world")
$ mapfile -t arr < <(echo "hello world")
$ typeset -p arr
declare -a arr=([0]="hello world")
On the 2nd command set, if the intention is to parse each word into an array then perhaps switch out mapfile with read?
$ read -ra arr < <(echo "hello world")
$ typeset -p arr
declare -a arr=([0]="hello" [1]="world")
So I know I can use a single IFS in a read statement, but is it possible to use two. For instance if I have the text
variable = 5 + 1;
print variable;
And I have the code to assign every word split to an array, but I also want to split at the ; as well as a space, if it comes up.
Here is the code so far
INPUT="$1"
declare -a raw_parse
while IFS=' ' read -r -a raw_input; do
for raw in "${raw_input[#]}"; do
raw_parse+=("$raw")
done
done < "$INPUT"
What comes out:
declare -a raw_parse=([0]="variable" [1]="=" [2]="5" [3]="+" [4]="1;" [5]="print" [6]="variable;")
What I want:
declare -a raw_parse=([0]="variable" [1]="=" [2]="5" [3]="+" [4]="1" [5]=";" [6]="print" [7]="variable" [8]=";")
A workaround with GNU sed. This inserts a space before every ; and replaces every newline with a space.
read -r -a raw_input < <(sed -z 's/;/ ;/g; s/\n/ /g' "$INPUT")
declare -p raw_input
Output:
declare -a raw_input=([0]="variable" [1]="=" [2]="5" [3]="+" [4]="1" [5]=";" [6]="print" [7]="variable" [8]=";")
A command emits the string: "[abc]=kjlkjkl [def]=yutuiu [ghi]=jljlkj"
I want to load a bash associative array using these key|value pairs, but the result I'm getting is a single row array where the key is formed of the first pair [abc]=kjlkjkl and the value is the whole of the rest of the string, so: declare -p arr returns declare -A arr["[abc]=kjlkjkl"]="[def]=yutuiu [ghi]=jljlkj"
This is what I am doing at the moment. Where am I going wrong please?
declare -A arr=()
while read -r a b; do
arr["$a"]="$b"
done < <(command that outputs the string "[abc]=kjlkjkl [def]=yutuiu [ghi]=jljlkj")
You need to parse it: split the string on spaces, split each key-value pair on the equals sign, and get rid of the brackets.
Here's one way, using tr to replace the spaces with newlines, then tr again to remove all brackets (including any that occur in a value), then IFS="=" to split the key-value pairs. I'm sure this could be done more effectively, like with AWK or Perl, but I don't know how.
declare -A arr=()
while IFS="=" read -r a b; do
arr["$a"]="$b"
done < <(
echo "[abc]=kjlkjkl [def]=yutuiu [ghi]=jljlkj" |
tr ' ' '\n' |
tr -d '[]'
)
echo "${arr[def]}" # -> yutuiu
See Cyrus's answer for another take on this, with the space and equals steps combined.
Append this to your command which outputs the string:
| tr ' =' '\n ' | tr -d '[]'
You can use the "eval declare" trick - but be sure your input is clean.
#! /bin/bash
s='[abc]=kjlkjkl [def]=yutuiu [ghi]=jljlkj'
eval declare -A arr=("$s")
echo ${arr[def]} # yutuiu
If the input is insecure, don't use it. Imagine (don't try) what would happen if
s='); rm -rf / #'
The "proper" good™ solution would be to write your own parser and tokenize the input. For example read the input char by char, handle [ and ] and = and space and optionally quoting. After parsing the string, assign the output to an associative array.
A simple way could be:
echo "[abc]=kjlkjkl [def]=yutuiu [ghi]=jljlkj" |
xargs -n1 |
{
declare -A arr;
while IFS= read -r line; do
if [[ "$line" =~ ^\[([a-z]*)\]=([a-z]*)$ ]]; then
arr[${BASH_REMATCH[1]}]=${BASH_REMATCH[2]}
fi
done
declare -p arr
}
outputs:
declare -A arr=([abc]="kjlkjkl" [ghi]="jljlkj" [def]="yutuiu" )
If I have a file example1.txt containing multiple strings
str1
str2
str3
...
I can read them into a bash array by using
mapfile -t mystrings < example1.txt.
Now say my file example2.txt is formatted as a table
str11 str12 str13
str21 str22 str23
str31 str32 str33
... ... ...
and I want to read each column into a different array. I know I can use other tools such as awk to separate each line into fields. Is there some way to combine this functionality with mapfile? I'm looking for something like
mapfile -t firstcol < $(cat example2.txt | awk '//{printf $1"\n"}')
mapfile -t secondcol < $(cat example2.txt | awk '//{printf $2"\n"}')
(which doesn't work).
Any other suggestion on how to handle a table in bash is also welcome.
Reading each row is simple, so let's build off that. I'll assume you have a proper matrix (i.e., each row has the same number of columns. This will be much easier since you are using bash 4.3.
while read -a row; do
c=0
for value in "${row[#]}"; do
declare -n column=column_$(( c++ ))
column+=( "$value" )
done
done < table.txt
There! Now, did it work?
$ echo "${column_0[#]}"
str11 str21 str31
$ echo "${column_1[#]}"
str12 str22 str32
I think so!
declare -n makes a nameref to an array (implicitly declared by the += on the next line) using a counter that increments as we iterate over each row. Then we simply append the current column value to the array behind the current nameref.
You should be using process substitution like this:
mapfile -t firstcol < <(awk '{print $1}' example2.txt)
mapfile -t secondcol < <(awk '{print $2}' example2.txt)
mapfile -t thirdcol < <(awk '{print $3}' example2.txt)
Hmm. Something like this, perhaps?
readarrays() {
declare -a values
declare idx line=0
while read -a values; do
for idx in "${!values[#]}"; do
[[ ${#:idx+1:1} ]] || break
declare -g "${#:idx+1:1}[$line]=${values[#]:idx:1}"
done
(( ++line ))
done
}
Tested as:
bash4-4.3$ (readarrays one two three <<<$'a b c\nd e f'; declare -p one two three)
declare -a one='([0]="a" [1]="d")'
declare -a two='([0]="b" [1]="e")'
declare -a three='([0]="c" [1]="f")'
The output is
/ ext4
/boot ext2
tank zfs
On each line the delimiter is a space. I need an associative array like:
"/" => "ext4", "/boot" => "ext2", "tank" => "zfs"
How is this done in bash?
If the command output is in file file, then:
$ declare -A arr=(); while read -r a b; do arr["$a"]="$b"; done <file
Or, you can read the data directly from a command cmd into an array as follows:
$ declare -A arr=(); while read -r a b; do arr["$a"]="$b"; done < <(cmd)
The construct <(...) is process substitution. It allows us to read from a command the same as if we were reading from a file. Note that the space between the two < is essential.
You can verify that the data was read correctly using declare -p:
$ declare -p arr
declare -A arr='([tank]="zfs" [/]="ext4" [/boot]="ext2" )'