I need to validate some program output. I know the output contains exactly three lines that I would like to store in distinct variables for easy access. I tried several variations of the following without success.
IFS=$'\n' read line1 line2 line3 <<< $(grep pattern file.log)
IFS='' read line1 line2 line3 <<< $(grep pattern file.log)
Is it possible to combine read and <<< to do what I want? How?
If it is not possible, what is the explanation? What alternative do you suggest?
Thank you.
{
read line1
read line2
read line3
} < <(grep pattern file.log)
I feel that putting a command substitution ($()) after a herestring (<<<)
is an unnecessary contortion, as is trying to trick read into reading more than one line at a time. This is a case where multiple reads in a compound command is a natural solution.
Well I have found how to make it work. It hinged on the use of double quotes.
First I tried this:
data=$(grep pattern file.log)
IFS=$'\n' read -d '' -r line1 line2 line3 <<< "$data"
Then found that the variable was not absolutely necessary if I kept the double quotes:
IFS=$'\n' read -d '' -r line1 line2 line3 <<< "$(grep pattern file.log)"
By default, a newline indicates end of data. To avoid read to stop working after the first line, I override the default with -d ''.
By default, the field separator is <space><tab><newline>. Since I want to read entire lines, I set IFS to $'\n'.
Note that -r is not central to the solution. I added it to avoid stumbling on eventual backslashes in my input.
EDIT
Using read and <<< has one subtle but important drawback: blank lines will disappear entirely.
data="1
3"
IFS=$'\n' read -d '' -r line1 line2 line3 <<< "$data"
echo ${line1:-blank}
echo ${line2:-blank}
echo ${line3:-blank}
output:
1
3
blank
Same if you try storing lines into an array with -a:
IFS=$'\n' read -d '' -r -a line_ar <<< "$data"
echo ${line_ar[0]:-blank}
echo ${line_ar[1]:-blank}
echo ${line_ar[2]:-blank}
output:
1
3
blank
However, you will still obtain all lines with a construct like this:
while read -r line ; do
echo ${line:-blank}
done <<< "$data"
output:
1
blank
3
If the number of lines is very limited, then you're better off using multiple reads as suggestd by kojiro. Then again, it is perfectly legal to use <<<:
{
read -r line1
read -r line2
read -r line3
} <<< "$data"
echo ${line1:-blank}
echo ${line2:-blank}
echo ${line3:-blank}
Remember to enclose your "$var" inside double quotes to have newline expanded.
Set IFS to \n and pass -d '' to read. Also you can use process substitution instead of a here string
while IFS=$'\n' read -d'' -r line1 line2 line3; do :; done < <(grep pattern file.log)
Use an array
read -a array < <(grep pattern file.log)
# Optional
line1=${array[0]}
line2=${array[1]}
line3=${array[2]}
However, I would recommend kojiro's answer.
while read -r arry[x]; do
((x++));
done < <(grep pattern file.log)
This will create an array called arry and the 3 lines will belong to each indices starting from 0.
Related
I want to use nested process subtitution with tee in a while loop.
while read line; do
#process line
echo "--$line"
done < <(cat sample.file | tee >(grep "SPECLINE") | grep "LINESTOPROCESS")
Therefore, I need:
all lines in sample.file that contain "LINETOPROCESS" expression should be passed into the loop, and they will be printed with "--" prefix.
all lines contain "SPECLINE" needs to be printed in tee's first process substitution (in the grep).
I want to avoid cat-ting the sample.file more than once as it is too large and heavy.
With a simple sample.test file:
line1 SPECLINE
line2 LINETOPROCESS
line3 LINETOPROCESS
line4 SPECLINE
line5 I don't need it
line6 also not
line7 also not
line8 SPECLINE
line9 LINETOPROCESS
My result:
# ./test.sh
#
My desired result:
# ./test.sh
line1 SPECLINE
--line2 LINETOPROCESS
--line3 LINETOPROCESS
line4 SPECLINE
line8 SPECLINE
--line9 LINETOPROCESS
Or I can also accept this as output:
# ./test.sh
--line2 LINETOPROCESS
--line3 LINETOPROCESS
--line9 LINETOPROCESS
line1 SPECLINE
line4 SPECLINE
line8 SPECLINE
UPDATE1
greps are for demo only.
I really need those 2 substitutions.
sample.file is a http file.
grep "SPECLINE" would be "hxselect -i -s ';' -c 'div.hour'
grep "LINESTOPROCESS" would be "hxselect -i -s ';' -c 'div.otherclass' | hxpipe
hx programs are not line-oriented. They are reading from stdin and outputting to stdout.
Therefore the tee's first command will select divs with 'hour' class and separate them with ';'. Afterwards, the pipe after tee will select all divs with class 'otherclass' and hxpipe will flatten it for the loop for further processing.
I would use no process substitution at all.
while IFS= read -r line; do
if [[ $line = *SPECLINE* ]]; then
printf '%s\n' "$line"
elif [[ $line = *LINETOPROCESS* ]]; then
printf '--%s\n' "$line"
fi
done < sample.txt
You are already paying the cost of reading an input stream line-by-line in bash; no reason to add the overhead of two separate grep processes to it.
A single awk process would be even better, as it is more efficient than bash's read-one-character-at-a-time approach to reading lines of text.
awk '/SPECLINE/ {print} /LINETOPROCESS/ {print "--"$0}' sample.txt
(which is too simple if a single line could match both SPECLINE and LINETOPROCESS, but I leave that as an exercise to the reader to fix.)
The following just loops through the entire file and just prints the matching lines. All other lines are ignored.
while read line; do
case "$line" in
*SPECLINE*) echo "$line" ;;
*LINETOPROCESS*) echo "--$line" ;;
esac
done < sample.file
When you want the tee, you can make 2 changes.
Your testcode greps LINESTOPROCESS, the input is LINETO..
The output process substition gives problems like https://stackoverflow.com/a/42766913/3220113 explained. You can do this differently.
while IFS= read -r line; do
#process line
echo "--$line"
done < x2 |
tee >(grep "SPECLINE") >(grep "LINETOPROCESS") >/dev/null
I don't know hxselect, but it seems to operate on a complete well-formed XML document, so avoid the grep.
In bash script, how can I read the file line by line and assign to the variable with delimiter?
example.txt file contents:
string1
string2
string3
string4
Expected output:
string1,string2,string3,string4
Thanks in advance
Apparently my answer below leaves a comma at the end of the line. A quick workaround is to use the following builtin in Unix:
paste -sd, example.txt
Where you use the paste program to concatenate all the lines into one and then add the string delimiter ','
Using the builtin commands in unix:
tr '\n' ',' < example.txt
This can be broken down as truncating all Newline widcards and inserting a comma delimiter instead.
Other possible ways, just for fun:
mapfile -t a < example.txt
(IFS=,; echo "${a[*]}")
mapfile -t a < example.txt
foo=$(printf '%s' "${a[#]/%/,}")
echo "${foo%,}"
foo=$(<example.txt)
echo "${foo//$'\n'/,}"
{
IFS= read -r foo
while IFS= read -r line; do
foo+=,$line
done
} < example.txt
echo "$foo"
sed ':a;N;$!ba;s/\n/,/g' example.txt
It should work:
#!/bin/bash
output=''
while IFS='' read -r line || [[ -n "$line" ]]; do
output=$output:",$line"
done < "$1"
echo $output
Give the file as argument
When I use "cat test.file", it will show
1
2
3
4
When I use the Bash file,
cat test.file |
while read data
do
echo "$data"
done
It will show
1
2
3
4
How could I make the result just like the original test file?
IFS=''
cat test.file |
while read data
do
echo "$data"
done
I realize you might have simplified the example from something that really needed a pipeline, but before someone else says it:
IFS=''
while read data; do
echo "$data"
done < test.file
Actually, if you don't supply an argument to the "read" call, read will set a default variable called $REPLY which will preserve whitespace. So you can just do this:
$ cat test.file | while read; do echo "$REPLY"; done
Maybe IFS is the key point as others said. You need to add only IFS= between while and read.
cat test.file |
while IFS= read data
do echo "$data"
done
and do not forget quotations of $data, else echo will trim the spaces.
But as Joshua Davies mentioned, you would prefer to use the predefined variable $REPLY.
Just to complement DigitalRoss's response.
For that case that you want to alter the IFS just for this command, you can use parenthesis. If you do, the value of IFS will be changed only inside the subshell.
Like this:
echo ' word1
word2' | ( IFS='' ; while read line ; do echo "$line" check ; done ; )
The output will be (keeping spaces):
word1 check
word2 check
read data will split the data by IFS, which is typically " \t\n". This will preserve the blanks for you:
var=$(cat test.file)
echo "$var"
Alternatively, use a good file parsing tool, like AWK:
awk '{
# Do your stuff
print
}' file
I am trying to parse a huge text file, say 200mb.
the text file contains some strings
123
1234
12345
12345
so my script looked like
while read line ; do
echo "$line"
done <textfile
however using this above method, my string " 12345" gets truncated to "12345"
I tried using
sed -n "$i"p textfile
but the the throughput is reduced from 27 to 0.2 lines per second, which is inacceptable ;-)
any Idea howto solve this?
You want to echo the lines without a fieldsep:
while IFS="" read line; do
echo "$line"
done <<< " 12345"
When you also want to skip interpretation of special characters, use
while IFS="" read -r line; do
echo "$line"
done <<< " 12345"
You can write the IFS without double quotes:
while IFS= read -r line; do
echo "$line"
done <<< " 12345"
This seems to be what you're looking for:
while IFS= read line; do
echo "$line"
done < textfile
The safest method is to use read -r in comparison to just read which will skip interpretation of special characters (thanks Walter A):
while IFS= read -r line; do
echo "$line"
done < textfile
OPTION 1:
#!/bin/bash
# read whole file into array
readarray -t aMyArray < <(cat textfile)
# echo each line of the array
# this will preserve spaces
for i in "${aMyArray[#]}"; do echo "$i"; done
readarray -- read lines from standard input
-t -- omit trailing newline character
aMyArray -- name of array to store file in
< <() -- execute command; redirect stdout into array
cat textfile -- file you want to store in variable
for i in "${aMyArray[#]}" -- for every element in aMyArray
"" -- needed to maintain spaces in elements
${ [#]} -- reference all elements in array
do echo "$i"; -- for every iteration of "$i" echo it
"" -- to maintain variable spaces
$i -- equals each element of the array aMyArray as it cycles through
done -- close for loop
OPTION 2:
In order to accommodate your larger file you could do this to help alleviate the work and speed up the processing.
#!/bin/bash
sSearchFile=textfile
sSearchStrings="1|2|3|space"
while IFS= read -r line; do
echo "${line}"
done < <(egrep "${sSearchStrings}" "${sSearchFile}")
This will grep the file (faster) before it cycles it through the while command. Let me know how this works for you. Notice you can add multiple search strings to the $sSearchStrings variable.
OPTION 3:
and an all in one solution to have a text file with your search criteria and everything else combined...
#!/bin/bash
# identify file containing search strings
sSearchStrings="searchstrings.file"
while IFS= read -r string; do
# if $sSearchStrings empty read in strings
[[ -z $sSearchStrings ]] && sSearchStrings="${string}"
# if $sSearchStrings not empty read in $sSearchStrings "|" $string
[[ ! -z $sSearchStrings ]] && sSearchStrings="${sSearchStrings}|${string}"
# read search criteria in from file
done <"${sSearchStrings}"
# identify file to be searched
sSearchFile="text.file"
while IFS= read -r line; do
echo "${line}"
done < <(egrep "${sSearchStrings}" "${sSearchFile}")
I have a script variable which is multi-line.
How do i traverse this variable to read it line by line and process each line the way I want?
Consider the following multi-line variable
x=$(echo -e "a\nb\nc d e")
and a simple process for each line: just echo it with a prefix=LINE: and with single quotes around the line. Either of the following codes will satisfy that requirement:
while read line; do echo "LINE: '${line}'"; done <<< "$x"
or
while read line; do echo "LINE: '${line}'"; done < <(echo "$x")
Neither creates a subshell (so you can, e.g., set variables in the loop and access them outside of it), and both output
LINE: 'a'
LINE: 'b'
LINE: 'c d e'
But suppose instead you have
x=$(echo -e "a \n b\nc d e")
# note--------^--^
and that leading and trailing whitespace matter for your application (e.g., parsing Git porcelain). Both the above codes will give exactly the same output for the latter variable/data as for the former, which is not what you want. To preserve leading and trailing whitespace, replace while read line with while IFS= read -r line . I.e., either of the following codes
while IFS= read -r line; do echo "LINE: '${line}'"; done <<< "$x"
or
while IFS= read -r line; do echo "LINE: '${line}'"; done < <(echo "$x")
will produce
LINE: 'a '
LINE: ' b'
LINE: 'c d e'
See Greg Wooledge's excellent Bash FAQ for details.
Although I typically use "while read" for processing multi-line variables, I recently had an instance where it removed the leading space from each line in a file. Using this instead fixed my issue:
printf '%s\n' "$var" | while IFS= read -r line
do
echo "$line"
done
Code taken from this Unix Stack Exchange answer.
Edit: updated to fix last line issue as suggested by Nicolae Iotu