This question already has answers here:
Creating an array from a text file in Bash
(7 answers)
Closed 3 years ago.
I have a problem with one of my bash scripts.
I have a file where is stored a list of email addresses line by line like so :
mail#adress1
mail#adress2
...
what I'd like to do is to actually put each line of the file in an array where each index corresponds to a line in the right order.
For me mapfile was not available, you can also do this with potentially older Bash versions:
set -f
IFS=$'\n'
arr=($(<foo.txt))
To read the lines of a file into an array:
mapfile -t myArray < myFile
or
myArray=()
while IFS= read -r line || [[ "$line" ]] ; do
myArray+=( "$line" )
done < myFile
To read the fields of a line into an array: use read with -a and a "herestring" <<<
# suppose: line="foo,bar,baz"
IFS=, read -r -a fields <<< "$line"
Related
I have a file that contains parts of file names either as newline (or separated by spaces). Lets take the following example:
cat file.txt
1
4
500
The actual file names are file_1.dat, file_2.dat, file_3.dat, file_4.dat file_500.dat, and so on.
I want to combine only those files whose names (or part of the names) are stored in file.txt.
To do so I am doing the following:
## read the file and assign to an array
array=()
while IFS= read -r line; do
array+=($line)
done < file.txt
## combine the contents of the files
for file in ${array[#]}
do
cat "file_$file.dat"
done > output.dat
Now in this solution what I don't like is the assignment of the array, that I have to run a do loop for this.
I tried to use
mapfile -t array < <(cat file.txt)
I also tried,
array=( $(cat file2.txt) )
The array that is needed finally is
array=(1 4 500)
In some of the answers(in this platform), I see that doing in the above way (the last option) might be harmful. I wanted to have some clarification on what to do for such assignments.
My question is: in this situation what is the best (safe and fast) way to assign the content of a file into an array?
array=( $(cat file2.txt) )
does not necessarily put each line in the array. It puts each word resulting from word-splitting and globbing into the array.
Consider this file
1
2 3
*
mapfile -t array < file.txt will create an array with the elements 1, 2 3, and *.
array=( $(cat file.txt) ) will create an array with the elements 1, 2, and 3, along with an element for each file name in the current directory.
Using mapfile is both safer and makes your intent of storing one line per element clearer.
However, there is no need for an array at all. You can process each file as you pull a line from your input file.
while IFS= read -r line; do
cat "file_$line.dat"
done < file.txt > output.dat
If you don’t want to deduplicate the file name fragments:
readarray -t lines < file.txt
declare -a fragments
for line in "${lines[#]}"; do
fragments+=($line)
done
names=("${fragments[#]/#/file_}")
names=("${names[#]/%/.dat}")
cat "${names[#]}"
If you do want to deduplicate the file name fragments:
readarray -t lines < file.txt
declare -Ai set_of_fragments
for line in "${lines[#]}"; do
for fragment in $line; do
((++set_of_fragments["${fragment}"]))
done
done
fragments=("${!set_of_fragments[#]}")
names=("${fragments[#]/#/file_}")
names=("${names[#]/%/.dat}")
cat "${names[#]}"
This question already has answers here:
Convert a JSON array to a bash array of strings
(4 answers)
Closed 3 years ago.
I am getting following string and I am trying to convert that into array.
[ "aaa", "bbb" ]
Would someone let me know if there is a way to convert this into BASH array.
You can use jq to extract the individual strings, and then read them line by line:
myJsonArray='[ "aaa", "bbb", "more \"complex\"\u0020value" ]'
mapfile -t myBashArray < <(jq -r '.[]' <<< "$myJsonArray")
declare -p myBashArray
This outputs declare -a myBashArray=([0]="aaa" [1]="bbb" [2]="more \"complex\" value")
If you additionally want to support elements with linefeeds and such, you can have jq output NUL-separated values with a bit more work:
myJsonArray='[ "multi\nline\ndata", "hostile'\''\"\u0000\n$(rm foo)data" ]'
array=()
while IFS= read -r -d '' line
do
array+=("$line")
done < <(jq -j '(.[] | gsub("\u0000"; "")) + "\u0000"' <<< "$myJsonArray")
declare -p array
This outputs declare -a array=([0]=$'multi\nline\ndata' [1]=$'hostile\'"\n$(rm foo)data')
It makes sure NUL bytes in the data don't interfere, but they will be stripped from the output. This is unlikely to matter since Bash variables can't represent NUL bytes in the first place.
I have a bash script with following variable:
operators_list=$'andrii,bogdan,eios,fre,kuz,pvm,sebastian,tester,tester2,vincent,ykosogon'
while IFS=, read -r tech_login; do
echo "... $tech_login ..."
done <<< "$operators_list"
I need to read arguments from variable and work with them in loop. But it returns echo only one time with all items:
+ IFS=,
+ read -r tech_login
+ echo '... andrii,bogdan,eios,fre,kuz,pvm,sebastian,tester,tester2,vincent,ykosogon ...'
... andrii,bogdan,eios,fre,kuz,pvm,sebastian,tester,tester2,vincent,ykosogon ...
+ IFS=,
+ read -r tech_login
What am I doing wrong? How to rework script, so it will work only with one item per time?
operators_list=$'andrii,bogdan,eios,fre,kuz,pvm,sebastian,tester,tester2,vincent,ykosogon'
So you have strings separated by ,. You can do that multiple ways:
using bash arrays:
IFS=, read -a operators <<<$operators_list
for op in "${operators[#]}"; do
echo "$op"
done
Using a while loop, like you wanted:
while IFS= read -d, -r op; do
echo "$op"
done <<<$operators_list
Using xargs, because why not:
<<<$operators_list xargs -d, -n1 echo
The thing with IFS and read delimeter is: read reads until delimeter specified with -d. Then after read has read a full string (usually whole line, as default delimeter is newline), then the string is splitted into parts using IFS as delimeter. So you can:
while IFS=: read -d, -r op1 op2; do
echo "$op1" "$op2"
done <<<"op11:op12,op12:op22"
This question already has answers here:
How do I pipe a file line by line into multiple read variables?
(3 answers)
How to split one string into multiple variables in bash shell? [duplicate]
(5 answers)
Read tab-separated file line into array
(3 answers)
Closed 4 years ago.
Suppose I have a string with pipe separator:
str="1|2|3|4"
I want them to be assigned to specific variables.
var_a=1
var_b=2
var_c=3
var_d=4
I am doing it in this way:
var_a="`echo $str | cut -d'|' -f1`"
var_b="`echo $str | cut -d'|' -f2`"
var_c="`echo $str | cut -d'|' -f3`"
var_d="`echo $str | cut -d'|' -f4`"
Can this be done in an efficient way? Please suggest.
It is better to use an array to store individual delimited values:
str="1|2|3|4"
IFS='|' read -ra arr <<< "$str"
#examine array values
declare -p arr
declare -a arr='([0]="1" [1]="2" [2]="3" [3]="4")'
To loop through array, use:
for i in "${arr[#]}"; do echo "$i"; done
1
2
3
4
IFS='|' read -r var_a var_b var_c var_d rest <<<"$str"
rest is the variable that gets further columns after the first four, should any others exist. If you just want to discard them, the conventional name to use for a placeholder variable is _.
This is covered in detail in BashFAQ #1: How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
I am trying to parse a huge text file, say 200mb.
the text file contains some strings
123
1234
12345
12345
so my script looked like
while read line ; do
echo "$line"
done <textfile
however using this above method, my string " 12345" gets truncated to "12345"
I tried using
sed -n "$i"p textfile
but the the throughput is reduced from 27 to 0.2 lines per second, which is inacceptable ;-)
any Idea howto solve this?
You want to echo the lines without a fieldsep:
while IFS="" read line; do
echo "$line"
done <<< " 12345"
When you also want to skip interpretation of special characters, use
while IFS="" read -r line; do
echo "$line"
done <<< " 12345"
You can write the IFS without double quotes:
while IFS= read -r line; do
echo "$line"
done <<< " 12345"
This seems to be what you're looking for:
while IFS= read line; do
echo "$line"
done < textfile
The safest method is to use read -r in comparison to just read which will skip interpretation of special characters (thanks Walter A):
while IFS= read -r line; do
echo "$line"
done < textfile
OPTION 1:
#!/bin/bash
# read whole file into array
readarray -t aMyArray < <(cat textfile)
# echo each line of the array
# this will preserve spaces
for i in "${aMyArray[#]}"; do echo "$i"; done
readarray -- read lines from standard input
-t -- omit trailing newline character
aMyArray -- name of array to store file in
< <() -- execute command; redirect stdout into array
cat textfile -- file you want to store in variable
for i in "${aMyArray[#]}" -- for every element in aMyArray
"" -- needed to maintain spaces in elements
${ [#]} -- reference all elements in array
do echo "$i"; -- for every iteration of "$i" echo it
"" -- to maintain variable spaces
$i -- equals each element of the array aMyArray as it cycles through
done -- close for loop
OPTION 2:
In order to accommodate your larger file you could do this to help alleviate the work and speed up the processing.
#!/bin/bash
sSearchFile=textfile
sSearchStrings="1|2|3|space"
while IFS= read -r line; do
echo "${line}"
done < <(egrep "${sSearchStrings}" "${sSearchFile}")
This will grep the file (faster) before it cycles it through the while command. Let me know how this works for you. Notice you can add multiple search strings to the $sSearchStrings variable.
OPTION 3:
and an all in one solution to have a text file with your search criteria and everything else combined...
#!/bin/bash
# identify file containing search strings
sSearchStrings="searchstrings.file"
while IFS= read -r string; do
# if $sSearchStrings empty read in strings
[[ -z $sSearchStrings ]] && sSearchStrings="${string}"
# if $sSearchStrings not empty read in $sSearchStrings "|" $string
[[ ! -z $sSearchStrings ]] && sSearchStrings="${sSearchStrings}|${string}"
# read search criteria in from file
done <"${sSearchStrings}"
# identify file to be searched
sSearchFile="text.file"
while IFS= read -r line; do
echo "${line}"
done < <(egrep "${sSearchStrings}" "${sSearchFile}")