piping images from simulation into ffmpeg - shell

Question

I'm writing a script that creates a video from images during their creation. A simulation software is writing out images with an unknown time interval (seconds to minutes). I'm doing this because mp4 is taking up much less space. Also, the images will end up in a video anyhow.
Here is my code. I commented the most important parts (as far as I understand). I gathered this from other posts I can't find the link to anymore.
#!/usr/bin/bash
# Create pipe and start piping it to ffmpeg process
mkfifo pipe
cat pipe | ffmpeg -y -f image2pipe -i pipe:0 -c:v libx264 -pix_fmt yuv420p ../output_video.mp4 &
# Open pipe for writing.
exec 3>pipe
while true; do # loop endlessly
sleep .05; # is there a new image? get the oldest of them
if ls -A *.png 1> /dev/null 2>$1; then
NEWFILE=$( ls -At *.png | tail -n 1 );
echo "$NEWFILE"
cat "$NEWFILE" >&3 # add the png to the pipe
# rm out.png; # remove or backup the image
mv "$NEWFILE" "./backup/$NEWFILE";
fi
done
My script is actually quite far already. It works fine for images in a folder and when the pictures are delivered quickly. When the images come one by one with longer intervals ffmpeg just seems to stop listening to the pipe input.
Here is the log from an example - not sure if this is helpful. Files 465 to 540 are already present in the folder, the others are added with ~15s intervals. As we can see ffmpeg first waits for 8-10 images to arrive before starting to compile. Afterwards ffmpeg continues to wait for images and compiling them. At the end I abort the script, now ffmpeg completes the mp4.
But when I write out the frames of the mp4 it includes every frame until image 610. Why did ffmpeg stop to listen to the pipe? The 'chunk too big' error could be a hint, but how do I fix it?
scene_image_00465.png
ffmpeg version 2.6.8 Copyright (c) 2000-2016 the FFmpeg developers
built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-4)
configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --enable-bzlib --disable-crystalhd --enable-gnutls --enable-ladspa --enable-libass --enable-libcdio --enable-libdc1394 --enable-libfaac --enable-nonfree --enable-libfdk-aac --enable-nonfree --disable-indev=jack --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-openal --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libx264 --enable-libx265 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect
libavutil 54. 20.100 / 54. 20.100
libavcodec 56. 26.100 / 56. 26.100
libavformat 56. 25.101 / 56. 25.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 11.102 / 5. 11.102
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 1.100 / 1. 1.100
libpostproc 53. 3.100 / 53. 3.100
scene_image_00470.png
scene_image_00475.png
scene_image_00480.png
scene_image_00485.png
scene_image_00490.png
scene_image_00495.png
scene_image_00500.png
scene_image_00505.png
scene_image_00510.png
Input #0, image2pipe, from 'pipe:0':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: png, rgb24, 1920x1200, 25 fps, 25 tbr, 25 tbn, 25 tbc
[libx264 # 0x19c0980] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 # 0x19c0980] profile High, level 5.0
[libx264 # 0x19c0980] 264 - core 142 r2495 6a301b6 - H.264/MPEG-4 AVC codec - Copyleft 2003-2014 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=18 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '../output_video.mp4':
Metadata:
encoder : Lavf56.25.101
Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 1920x1200, q=-1--1, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder : Lavc56.26.100 libx264
Stream mapping:
Stream #0:0 -> #0:0 (png (native) -> h264 (libx264))
scene_image_00515.png
scene_image_00520.png
scene_image_00525.png
scene_image_00530.png
scene_image_00535.png
scene_image_00540.png
scene_image_00545.png
scene_image_00550.png
scene_image_00555.png=0.0 size= 0kB time=00:00:00.00 bitrate=N/A
scene_image_00560.png
scene_image_00565.png
scene_image_00570.png
scene_image_00575.png
scene_image_00580.png
scene_image_00585.png
scene_image_00590.png=0.0 size= 0kB time=00:00:00.00 bitrate=N/A
scene_image_00595.png
scene_image_00600.png
scene_image_00605.png
scene_image_00610.png
scene_image_00615.png
scene_image_00620.png
scene_image_00625.png
scene_image_00630.png
scene_image_00635.png
scene_image_00640.png
scene_image_00645.png
scene_image_00650.png
scene_image_00655.png
scene_image_00660.png
scene_image_00665.png
scene_image_00670.png
scene_image_00675.png
scene_image_00680.png
scene_image_00685.png
scene_image_00690.png
scene_image_00695.png
scene_image_00700.png
scene_image_00705.png
scene_image_00710.png
scene_image_00715.png
[png # 0x35d0660] chunk too big=0.0 size= 0kB time=00:00:00.00 bitrate=N/A
frame= 30 fps=0.1 q=-1.0 Lsize= 754kB time=00:00:01.12 bitrate=5517.3kbits/s
video:753kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.155077%
[libx264 # 0x19c0980] frame I:1 Avg QP:18.39 size:107795
[libx264 # 0x19c0980] frame P:8 Avg QP:24.90 size: 36026
[libx264 # 0x19c0980] frame B:21 Avg QP:28.37 size: 17835
[libx264 # 0x19c0980] consecutive B-frames: 3.3% 6.7% 10.0% 80.0%
[libx264 # 0x19c0980] mb I I16..4: 32.2% 41.7% 26.0%
[libx264 # 0x19c0980] mb P I16..4: 0.4% 1.2% 1.3% P16..4: 4.9% 6.8% 7.6% 0.0% 0.0% skip:77.8%
[libx264 # 0x19c0980] mb B I16..4: 0.1% 0.4% 0.6% B16..8: 10.1% 5.6% 3.3% direct: 2.0% skip:77.9% L0:42.0% L1:44.5% BI:13.5%
[libx264 # 0x19c0980] 8x8 transform intra:40.7% inter:24.0%
[libx264 # 0x19c0980] coded y,uvDC,uvAC intra: 29.6% 31.1% 29.7% inter: 6.0% 11.0% 3.9%
[libx264 # 0x19c0980] i16 v,h,dc,p: 56% 38% 5% 0%
[libx264 # 0x19c0980] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 44% 19% 31% 1% 1% 1% 1% 0% 1%
[libx264 # 0x19c0980] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 20% 17% 5% 9% 7% 8% 5% 6%
[libx264 # 0x19c0980] i8c dc,h,v,p: 76% 12% 8% 4%
[libx264 # 0x19c0980] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 # 0x19c0980] ref P L0: 57.4% 13.9% 18.1% 10.7%
[libx264 # 0x19c0980] ref B L0: 78.3% 18.6% 3.1%
[libx264 # 0x19c0980] ref B L1: 93.4% 6.6%
[libx264 # 0x19c0980] kb/s:5136.93