How can I assign expressions to a variable for read line? - bash

Question

I have a bash while read line block reading from a text file specified by $filename:
IFS=''
while read -r line
do
...
done < $filename
Instead of reading the whole file each time, I would like to supply different inputs in the redirect depending on the arguments supplied to the script.
Whole file: done < "$filename"
start at line x: done < <(tail -n +"$x" "$filename")
line x to line y: done < <(tail -n +"$x" "$filename" | head -n "$y")
start to line y: done < <(head -n "$y" "$filename")
How can I assign these inputs to a variable ahead of time to be read by the while loop?
My input file is ~4GB with some 58M lines (all with different lengths), and may grow or shrink from time to time. Reading https://unix.stackexchange.com/questions/47407/cat-line-x-to-line-y-on-a-huge-file it appears that tail | head is the fastest method to read from the middle of a file, so given the file size, I'm deliberately avoiding awk and sed for the most part.

Answer 1

Your data is too big to read in whole. The good news is that the contents of a process substitution is a shell script, so you can write:
while IFS= read -r line; do
...
done < <(
if [[ $x && $y ]]; then tail -n +"$x" "$filename" | head -n "$y"
elif [[ $x ]]; then tail -n +"$x" "$filename"
elif [[ $y ]]; then head -n "$y" "$filename"
else cat "$filename"
fi
)
One thing I don't like about process substitutions is that code follows the loop for which it is input. It would be nice if it appeared first. I think this will work, but is untested:
# set up file descriptor 3
exec 3< <(
if [[ $x && $y ]]; then tail -n +"$x" "$filename" | head -n "$y"
elif [[ $x ]]; then tail -n +"$x" "$filename"
elif [[ $y ]]; then head -n "$y" "$filename"
else cat "$filename"
fi
)
# iterate over lines read from fd 3
while IFS= read -u3 -r line; do
...
done
# close fd 3
exec 3<&-

Answer 2

I might handle all of these as part of the loop condition, with an explicitly maintained line counter.
start=10
end=30
i=0
while ((i <= end )) && IFS= read -r line; do
(( i++ >= start )) || continue
...
done < "$filename"
However, if you might skip a significant number of lines at the beginning, it might be more efficient to use sed
while IFS= read -r line; do
...
done < <(sed -n "$start,$stop p" "$filename")
or awk:
while IFS= read -r line; do
...
done < <(awk -v start "$start" -v end "$end" 'NR >= start && NR <= end' "$filename")
This then raises the question of how much of the body of the while loop can be moved into awk itself.