Bash PDF-merging misses files - bash

Question

I'm trying to merge many PDF files in chunks of 3000 or so files. After many tries, this script seemed to do the trick. (of course I was wrong)
#!/bin/bash
basepath='/home/lemonidas/pdfstuff';
datename=`date "+%Y%m%d%H%M.%S"`;
start=`date "+%s"`;
echo "parsing pdf list to file..."
find $basepath/input/ -name "*.pdf" | xargs -I {} ls {} >> $basepath/tmp/biglist$datename.txt
split -l 3000 $basepath/tmp/biglist$datename.txt $basepath/tmp/splitfile
rm $basepath/tmp/biglist$datename.txt
echo "deleting big file..."
echo "done splitting!"
declare -i x
x=1
for f in $basepath/tmp/splitfile*
do
linenum=`cat $f | wc -l`;
echo "Processing $f ($linenum lines)..."
# merge to one big PDF
cat $f | xargs gs -q -sstdout=$basepath/error.log -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=$basepath/output/$x.big.pdf 2>$basepath/error.log
echo "Completed PDF $x"
(( x++ ))
# delete the list file
rm $f
echo "Deleted processed file $f"
done
end=`date "+%s"`;
echo "Started: $start"
echo "Finished: $end"
The problem is, I have 22000 2-page PDFs, each output file (except the last one) should be 6000 pages (since we have 3000 PDFs in each merge list, as verified by the "wc -l" before parsing), and I only get around 658 pages or so.
No errors are reported except this by gs:
Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.Warning: Embedded symbolic TT fonts must contain a cmap for Platform=1 Encoding=0.
This file had errors that were repaired or ignored.
The file was produced by: >>>> Powered By Crystal
Please notify the author of the software that produced this file that it does not conform to Adobe's published PDF specification.
over and over (but not 22000 times though)
When I try it with 300-400 files, it runs smoothly, but when I try the full run, after 2.5 hours, I get much less than half the files merged.
My next thought is to convert each 2-page PDF in .pgm files, but I then have no idea how to remake them as PDF (so that no font embedding issues arise).
Am I missing something? (probably)

Answer 1

You would probably do better to use a tool better suited to the task. pdfwrite (the Ghostscript device for emitting PDF files) is not in my opinion the right tool for this.
In order to 'merge' PDF files, Ghostscript fully interprets the input into marking operations, then rewrite the marking operations as a PDF file. While creating that list of operations a great deal of information needs to be held (fonts, images, other things) and compared against new input to see if we already have a copy. As the input grows larger, it takes longer to scan that list, and of course the memory consumption increases. You may find that Ghostscript is already swapping memory.
Now I'm not sure this is your actual problem, or if you are saying that after you 'merge' the files there are pages missing. That should not happen. You also don't say what version of Ghostscript you are using.
All the same, I would think that a tool like pdftk would be faster at doing this kind of merge, though the final PDF file might well be larger/less efficient than pdfwrite would make.