Splitting large files efficiently (currently using awk) - bash

Question

I have a 4 GB file that I need to do some operations on. I have a Bash script to do this, but it Bash seems ill suited to reading large data files into an array. So I decided to break up my file with awk.
My current script is :
for((i=0; i<100; i++)); do awk -v i=$i 'BEGIN{binsize=60000}{if(binsize*i < NR && NR <= binsize*(i+1)){print}}END{}' my_large_file.txt &> my_large_file_split$i.fastq; done
However the problem with this script is that it will read in and loop through this large file 100 times (which presumably will lead to about 400GB of IO).
QUESTION : Is there better strategy of reading in the the large file once? Perhaps doing the writing to files within awk instead of redirecting its output?

Answer 1

Assuming binsize is the number of lines you want per chunk, you could just maintain and reset a line counter as you step through the file, and setting alternate output files within awk instead of using the shell to redirect.
awk -v binsize=60000 '
BEGIN {
outfile="output_chunk_1.txt"
}
count > binsize {
if (filenum>1) {
close(outfile)
}
filenum++
outfile="output_chunk_" filenum ".txt"
count=0
}
{
count++
print > outfile
}
' my_large_file.txt
I haven't actually tested this code, so if it doesn't work verbatim, at least it should give you an idea of a strategy to use. :-)
The idea is that we'll step through the file, updating a filename in a variable whenever our line count for a chunk exceeds binsize. Note that the close(outfile) isn't strictly necessary, as awk will of course close any open files when it exits, but it may save you a few bytes of memory per open file handle (which will only be significant if you have many many output files).
That said, you could do almost exactly the same thing in bash alone:
#!/usr/bin/env bash
binsize=60000
filenum=1; count=0
while read -r line; do
if [ $count -gt $binsize ]; then
((filenum++))
count=0
fi
((count++))
outfile="output_chunk_${filenum}.txt"
printf '%s\n' "$line" >> $outfile
done < my_large_file.txt
(Also untested.)
And while I'd expect the awk solution to be faster than bash, it might not hurt to do your own benchmarks. :)