Split very large files on record border - bash

Question

I have to split a very large file into N smaller files with the following constraints:
I have to split on record border
Record separator can be any character
the number of records in the resulting N files should be the same (+/- 1 record)
I can just use bash and standard coreutils (I have a working solution in Perl but we're not allowed to install Perl/Python/etc)
This is not a real constraint but - if possible - I'd like to scan the original (large) file just once.
Sort order of the resulting files is not important.
My working solution in Perl reads the original file and writes...
- the 1st record to the first file
- ...
- the Nth record to the Nth file
- the N+1 record back to the first file
- etc
So - at the end - with a single scan of the initial file I do get several smaller files with the same number of records (+/- 1).
For example, assume this is the input file:
1,1,1,1A2,2,2,2A3,
3,3,3A4,4,4,4A5,5,
5,5A6,6,6,6A7,7,7,
7,A8,8,8,8A9,9,9,9
A0,0,0,0
With record separator = 'A' and N = 3 I should get three files:
# First file:
1,1,1,1A2,2,2,2A3,
3,3,3
# Second file
4,4,4,4A5,5,
5,5A6,6,6,6
# Third file:
7,7,7,
7,A8,8,8,8A9,9,9,9
A0,0,0,0
UPDATE
Here you have the perl code. I tried to make it as simple and readable as I can:
#!/usr/bin/perl
use warnings;
use strict;
use locale;
use Getopt::Std;
#-----------------------------------------------------------------------------
# Declaring variables
#-----------------------------------------------------------------------------
my %op = (); # Command line parameters hash
my $line = 0; # Output file line number
my $fnum = 0; # Output file number
my #fout = (); # Output file names array
my #fhnd = (); # Output file handles array
my #ifiles = (); # Input file names
my $i = 0; # Loop variable
#-----------------------------------------------------------------------------
# Handling command line arguments
#-----------------------------------------------------------------------------
getopts("o:n:hvr:", \%op);
die "Usage: lfsplit [-h] -n number_of_files",
" [-o outfile_prefix] [-r rec_sep_decimal] [-v] input_file(s)\n"
if $op{h} ;
if ( #ARGV ) {
#ifiles = #ARGV ;
} else {
die "No input files...\n" ;
}
$/ = chr($op{r}) if $op{r} ;
#-----------------------------------------------------------------------------
# Setting Default values
#-----------------------------------------------------------------------------
$op{o} |= 'out_' ;
#-----------------------------------------------------------------------------
# Body - split in round-robin to $op{n} files
#-----------------------------------------------------------------------------
for ( $i = 0 ; $i < $op{n} ; $i++ ) {
local *OUT ; # Localize file glob
$fout[$i] = sprintf "%s_%04d.out", $op{o}, $i ;
open ( OUT, "> $fout[$i]" ) or
die "[lfsplit] Error writing to $fout[$i]: $!\n";
push ( #fhnd , *OUT ) ;
}
$i = 0 ;
foreach ( #ifiles ) {
print "Now reading $_ ..." if $op{v} ;
open ( IN, "< $_" ) or
die "[lfsplit] Error reading $op{i}: $!\n" ;
while ( <IN> ) {
print { $fhnd[$i] } $_ ;
$i = 0 if ++$i >= $op{n} ;
}
close IN ;
}
for ( $i = 0 ; $i < $op{n} ; $i++ ) {
close $fhnd[$i] ;
}
#-----------------------------------------------------------------------------
# Exit
#-----------------------------------------------------------------------------
exit 0 ;

Answer 1

Just for kicks, a pure bash solution, no external programs and no forking (I think):
#!/bin/bash
input=$1
separator=$2
outputs=$3
i=0
while read -r -d"$separator" record; do
out=$((i % outputs)).txt
if ((i < outputs)); then
: > $out
else
echo -n "$separator" >> $out
fi
echo -n "$record" >> $out
((i++))
done < $input
Sadly this will reopen every file for every output operation. I'm sure it's possible to fix this, using <> to open a file descriptor and keep it open, but using that with non-literal file descriptors is a bit of a pain.