This question already has answers here:
Convert a JSON array to a bash array of strings
(4 answers)
Closed 3 years ago.
I am getting following string and I am trying to convert that into array.
[ "aaa", "bbb" ]
Would someone let me know if there is a way to convert this into BASH array.
You can use jq to extract the individual strings, and then read them line by line:
myJsonArray='[ "aaa", "bbb", "more \"complex\"\u0020value" ]'
mapfile -t myBashArray < <(jq -r '.[]' <<< "$myJsonArray")
declare -p myBashArray
This outputs declare -a myBashArray=([0]="aaa" [1]="bbb" [2]="more \"complex\" value")
If you additionally want to support elements with linefeeds and such, you can have jq output NUL-separated values with a bit more work:
myJsonArray='[ "multi\nline\ndata", "hostile'\''\"\u0000\n$(rm foo)data" ]'
array=()
while IFS= read -r -d '' line
do
array+=("$line")
done < <(jq -j '(.[] | gsub("\u0000"; "")) + "\u0000"' <<< "$myJsonArray")
declare -p array
This outputs declare -a array=([0]=$'multi\nline\ndata' [1]=$'hostile\'"\n$(rm foo)data')
It makes sure NUL bytes in the data don't interfere, but they will be stripped from the output. This is unlikely to matter since Bash variables can't represent NUL bytes in the first place.
Related
This question already has answers here:
Read values into a shell variable from a pipe
(17 answers)
Closed 12 months ago.
I just discovered mapfile in bash when shellcheck recommended it to my code to capture two shell parameters from a space-delimited string of two words. But, mapfile does not seem to function consistently. For example, this works as expected:
mapfile arr < myfile
The array arr is populated with entries corresponding to the lines of myfile. However, this does not work:
echo -e "hello\nworld" | mapfile arr
The array arr is not populated at all. And, this doesn't work either:
echo "hello world" | mapfile -d ' ' arr
I am not sure why it would make a difference where the standard input for the command comes from. I didn't know it would be possible to distinguish what the input came from, a file or a pipeline.
Any clues?
Note to moderator: It was suggested my question was a duplicate to Read values into a shell variable from a pipe . I do not agree. Nowhere is mapfile mentioned in that question, nor was there any other useful Q/A found in a SO search. In addition, that referenced question did not deal with shell parameter assignments. Therefore, this question and answers are valuable.
Technically the array is being populated; the issue is that mapfile is called in a sub-process which, when it exits (back to the command line) the array assignment is lost, ie, you can't pass assignments 'up' from a sub-process to a parent/calling process.
Try these instead:
$ mapfile -d ' ' arr < <(echo -e "hello\nworld")
$ typeset -p arr
declare -a arr=([0]=$'hello\nworld\n')
$ mapfile -d ' ' arr < <(echo "hello world")
$ typeset -p arr
declare -a arr=([0]="hello " [1]=$'world\n')
While these will populate the array ... you'll have to decide if this is what you were expecting to show up in the array. Perhaps the following are a bit closer to what's desired?
$ mapfile -t arr < <(echo -e "hello\nworld")
$ typeset -p arr
declare -a arr=([0]="hello" [1]="world")
$ mapfile -t arr < <(echo "hello world")
$ typeset -p arr
declare -a arr=([0]="hello world")
On the 2nd command set, if the intention is to parse each word into an array then perhaps switch out mapfile with read?
$ read -ra arr < <(echo "hello world")
$ typeset -p arr
declare -a arr=([0]="hello" [1]="world")
This question already has answers here:
How to sort an array in Bash
(20 answers)
Closed 2 years ago.
In bash, there is an array like this:
arr=(12345_34, 5_32134, 8_123, 13_1234)
And I'd like to sort (decreasing order) this array based on the numbers before the underscore.
So the desired result is the following:
(12345_34, 13_1234, 8_123, 5_32134)
I tried sort -t _-k 2 -g $arr
arr=(12345_34 5_32134 8_123 13_1234)
readarray -t arr_sorted < <(printf '%s\n' "${arr[#]}" | sort -r -t _ -g)
declare -p arr_sorted
...properly emits as output the ordering specified in the question:
declare -a arr_sorted=([0]="12345_34" [1]="13_1234" [2]="8_123" [3]="5_32134")
If you need to target versions of bash too old to have readarray, a while read loop can substitute, with considerable loss of terseness:
# define the input array
arr=(12345_34 5_32134 8_123 13_1234)
# generate a sorted version
arr_sorted=( )
while IFS= read -r item; do
arr_sorted+=( "$item" )
done < <(printf '%s\n' "${arr[#]}" | sort -r -t _ -g)
# print the sorted version to demonstrate that we built it correctly
declare -p arr_sorted
Try this:
sorted=($(printf '%s\n' "${arr[#]}" | sort -nr))
This question already has answers here:
Creating an array from a text file in Bash
(7 answers)
Closed 3 years ago.
I have a problem with one of my bash scripts.
I have a file where is stored a list of email addresses line by line like so :
mail#adress1
mail#adress2
...
what I'd like to do is to actually put each line of the file in an array where each index corresponds to a line in the right order.
For me mapfile was not available, you can also do this with potentially older Bash versions:
set -f
IFS=$'\n'
arr=($(<foo.txt))
To read the lines of a file into an array:
mapfile -t myArray < myFile
or
myArray=()
while IFS= read -r line || [[ "$line" ]] ; do
myArray+=( "$line" )
done < myFile
To read the fields of a line into an array: use read with -a and a "herestring" <<<
# suppose: line="foo,bar,baz"
IFS=, read -r -a fields <<< "$line"
This question already has answers here:
How do I pipe a file line by line into multiple read variables?
(3 answers)
How to split one string into multiple variables in bash shell? [duplicate]
(5 answers)
Read tab-separated file line into array
(3 answers)
Closed 4 years ago.
Suppose I have a string with pipe separator:
str="1|2|3|4"
I want them to be assigned to specific variables.
var_a=1
var_b=2
var_c=3
var_d=4
I am doing it in this way:
var_a="`echo $str | cut -d'|' -f1`"
var_b="`echo $str | cut -d'|' -f2`"
var_c="`echo $str | cut -d'|' -f3`"
var_d="`echo $str | cut -d'|' -f4`"
Can this be done in an efficient way? Please suggest.
It is better to use an array to store individual delimited values:
str="1|2|3|4"
IFS='|' read -ra arr <<< "$str"
#examine array values
declare -p arr
declare -a arr='([0]="1" [1]="2" [2]="3" [3]="4")'
To loop through array, use:
for i in "${arr[#]}"; do echo "$i"; done
1
2
3
4
IFS='|' read -r var_a var_b var_c var_d rest <<<"$str"
rest is the variable that gets further columns after the first four, should any others exist. If you just want to discard them, the conventional name to use for a placeholder variable is _.
This is covered in detail in BashFAQ #1: How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
Here is how to split a tab-split line into an array:
IFS=$'\t' read -a array < <(echo -e "a\tb\tc")
And here is how to read null-terminated lines into an array:
while IFS= read -r -d '' item
do array+=("$item")
done < <(echo -e "a\0b\0c\0")
Now, is it possible to combine both and have a one-liner suitable to read a null-split line into an array?
First, putting -r -d '' together with -a array, and setting IFS to empty, doesn't to work:
IFS= read -r -d '' -a array < <(echo -e "a\0b\0c")
as array will have 1 item: a
Secondly, such while construct reading a null-split line into an array is mildly unsatisfactory. It will drop the last item if the last item isn't followed by null. A workaround is to append the last item to the array after the loop, as suggested by chepner below.
while IFS= read -r -d '' item
do array+=("$item")
done < <(echo -e "a\0b\0c")
array+=("$item")
There isn't, if you don't consider a single while loop a "one"-liner.
while IFS= read -r -d '' item; do array+=("$item"); done < <(echo -e "a\0b\0c\0")
read only reads a line of input at a time, and you are using -d '' to define what a line is. That is, you aren't treating "a\0b\0c" as a single line of input, but as multiple lines separate by null byte. The readarray command, introduced in bash 4, doesn't provide a way to specify what to consider as the line separator.
readarray a.k.a. mapfile supports nul termination with the -d '' flag, at least in bash 5.2.2 where I tested, and 4.4.20 on a test Ubuntu 18.04 container, which is as far back as I cared to go.
TL;DR:
mapfile -d '' ARRAYNAME < <(command that emits null bytes here)
mapfile -d '' ARRAYNAME < file-with-null-bytes
Order is significant, arrayname must come after the -d ''. Using a pipe instead will not work how you expect, because the mapfile will run in a subshell, hence the use of command-redirection above.
To append to an existing array, use
mapfile -d '' -O "${#ARR[#]}" ARR
Demo and explanation
Set up a NUL-terminated stream with one plain entry, one entry with embedded space, and one entry with embedded newline. Use xargs -0 to show it's correct.
$ python3 -c 'print("\x00".join(["1", "2 3", "4\n5"]), end="\x00")'
12 34
5
$ python3 -c 'print("\x00".join(["1", "2 3", "4\n5"]), end="\x00")' \
| xargs -0 -I{} echo "[{}]"
[1]
[2 3]
[4
5]
Python is only used to generate the demo input; use whatever else you prefer.
Now we use mapfile -d '' to read the demo null terminated stream and verify that it was split solely on null bytes:
$ mapfile -d '' ARR < <(python3 -c 'print("\x00".join(["1", "2 3", "4\n5"]), end="\x00")')
$ echo "${#ARR[#]}"
3
$ printf "[%s]" "${ARR[#]}"
[1][2 3][4
5]$
Note:
readarray is just an alias for mapfile
You can write mapfile -d $'\0' ARRAYNAME instead if you want, to make your intent more explicit. $'xxxxx' is a bash escape-string. But bash strings are null-terminated, so $'\0' is the same as the empty-string anyway. mapfile special-cases the zero-length delimiter argument to mean null terminated.
Order of arguments to mapfile is significant. Array name must be the last argument. mapfile ARRAYNAME -d '' is not the same as mapfile -d '' ARRAYNAME. Redirection operators are removed by the calling shell so they don't count.
Invoking mapfile in a pipeline won't export the resulting array to the shell that invokes the pipeline, because the mapfile in the pipeline will run in its own separate shell process. Use process substitution redirection < <(command) instead of a pipe.
So this won't work
$ # DELIBERATELY WRONG #
$ python3 -c 'print("\x00".join(["1", "2 3", "4\n5"]), end="\x00")' | mapfile -d '' ARR2
$ echo "${#ARR2[#]}"
0
$ echo "${ARR2[#]}"
$ # DELIBERATELY WRONG #
I used Python for the demo output because working with null bytes in bash is too painful. Here's an alternative using only sh etc, using the creation of files and then find -print0 to get the desired output:
mkdir omg
cd omg
touch '1' '2 3' $'4\n5'
find . -print0