Julia program stalls when run from crontab scheduler (Linux) - bash

Question

I have a really specific and tricky bug that I can't figure out how to fix/work around and I can't find a similar case on here.
I have a bash script that invokes a Julia script partway through to generate animation frames, then calls ffmpeg to render the animation. When I run from the terminal everything works great. I wanted to automate the process so I got a fun random simulation once a day, so I added it to my crontab and it runs--but only to a certain point. The animation always stops at a specific frame, then the rest of the script continues and spits out the chopped off animation.
I thought maybe cron was the problem, so I installed jobber and ran the job from there--with jobber the script just stalls at the Julia part. From the resource manager I can see the Julia process still using memory (although well beneath the limit) but it's just gone to sleep.
Another strange thing that I have noticed is that when I invoke the script manually from the command line it runs ~2-4x faster in generating the animation frames than when its running automatically via crontab/jobber.
Is this a weird resource issue? To get the longer animations to render initially I had to modify my ulimit settings, but I changed the config file so they should be set higher for everything? How can I debug this further and/or rectify it?
If you want to see an example of the code being run (both the shell script and julia script being invoked) it's pretty much up to date on my github here. In the threeBodyProb.jl file the I'm pretty sure the hang up is with the frame function in the for looop at the end of the file.
I am running Linux Mint 19.1 Cinnamon. Thanks in advance for the help!
Here is the part of the bash script where it hangs up:
./threeBodyProb.jl
echo animation generated, running ffmpeg >> /home/kirk/Documents/3Body/cron_log.txt
cd tmpPlots
</dev/null ffmpeg -framerate 30 -i "%06d.png" -c:v libx264 -preset slow -coder 1 -movflags +faststart -g 15 -crf 18 -pix_fmt yuv420p -profile:v high -y -bf 2 -fs 15M -vf "scale=720:720,setdar=1/1" "/home/kirk/Documents/3Body/3Body_fps30.mp4"
And here is the for loop that hangs up in Julia:
plotLoadPath="/home/kirk/Documents/3Body/tmpPlots/"
threeBodyAnim=Animation(plotLoadPath,String[])
for i=1:35:length(t)
gr(legendfontcolor = plot_color(:white)) #legendfontcolor=:white plot arg broken right now (at least in this backend)
print("$(#sprintf("%.2f",i/length(t)*100)) % complete\r") #output percent tracker
pos=[plotData[1][i],plotData[2][i],plotData[3][i],plotData[4][i],plotData[5][i],plotData[6][i]] #current pos
limx,limy=getLims(pos./1.5e11,10) #convert to AU, 10 AU padding
p=plot(plotData[1][1:i]./1.5e11,plotData[2][1:i]./1.5e11,label="",linecolor=colors[1]) #plot orbits up to i
p=plot!(plotData[3][1:i]./1.5e11,plotData[4][1:i]./1.5e11,label="",linecolor=colors[2])
p=plot!(plotData[5][1:i]./1.5e11,plotData[6][1:i]./1.5e11,label="",linecolor=colors[3])
p=scatter!(starsX,starsY,markercolor=:white,markersize=:1,label="") #fake background stars
star1=makeCircleVals(rad[1],[plotData[1][i],plotData[2][i]]) #generate circles with appropriate sizes for each star
star2=makeCircleVals(rad[2],[plotData[3][i],plotData[4][i]]) #at current positions
star3=makeCircleVals(rad[3],[plotData[5][i],plotData[6][i]])
p=plot!(star1[1]./1.5e11,star1[2]./1.5e11,label="$(#sprintf("%.1f", m[1]./2e30))",color=colors[1],fill=true) #plot star circles with labels
p=plot!(star2[1]./1.5e11,star2[2]./1.5e11,label="$(#sprintf("%.1f", m[2]./2e30))",color=colors[2],fill=true)
p=plot!(star3[1]./1.5e11,star3[2]./1.5e11,label="$(#sprintf("%.1f", m[3]./2e30))",color=colors[3],fill=true)
p=plot!(background_color=:black,background_color_legend=:transparent,foreground_color_legend=:transparent,
background_color_outside=:white,aspect_ratio=:equal,legendtitlefontcolor=:white) #formatting for plot frame
p=plot!(xlabel="x: AU",ylabel="y: AU",title="Random Three Body Problem\nt: $(#sprintf("%0.2f",t[i]/365/24/3600)) yrs after start",
legend=:best,xaxis=("x: AU",(limx[1],limx[2]),font(9,"Courier")),yaxis=("y: AU",(limy[1],limy[2]),font(9,"Courier")),
grid=false,titlefont=font(14,"Courier"),size=(720,721),legendfontsize=8,legendtitle="Mass (in solar masses)",legendtitlefontsize=8) #add in axes/title/legend with formatting
frame(threeBodyAnim,p) #generate the frame
end
If it helps, when run from cron or jogger it always generates 407 frames and fails at the 408th.
UPDATE: Following #TasosPapastylianou's suggestion below I think the problem may be in differing Julia environments when run directly in the terminal vs from a background process like crontab or jobber. I've added the output when a test script that gets the Julia environment is run both from crontab/jobber and directly from the command line. I’m not sure if this is the problem, and if it is how I should tell cron to work in this environment for this job (tried sourcing .bashrc and .profile in script but that had no effect on env output).
From cron:
SHLVL=1
HOME=/home/kirk
LOGNAME=kirk
_=/home/kirk/bashTest.sh
PATH=/opt/someApp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=en_US.UTF-8
SHELL=/bin/bash
PWD=/home/kirk
OPENBLAS_MAIN_FREE=1
From jobber:
MAIL=/var/mail/kirk
USER=kirk
HOME=/home/kirk
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
LOGNAME=kirk
XDG_SESSION_ID=c4
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
XDG_RUNTIME_DIR=/run/user/1000
LANG=en_US.UTF-8
SHELL=/bin/sh
PWD=/home/kirk
XDG_DATA_DIRS=/home/kirk/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
OPENBLAS_MAIN_FREE=1
And when run manually from command line:
GJS_DEBUG_TOPICS=JS ERROR;JS LOG
LESSOPEN=| /usr/bin/lesspipe %s
PERLBREW_VERSION=0.86
PGPLOT_DIR=/home/kirk/Documents/research/MESA/mesasdk/lib/pgplot
USER=kirk
LANGUAGE=en_US
XDG_SEAT=seat0
SSH_AGENT_PID=1786
XDG_SESSION_TYPE=x11
SHLVL=1
CONDA_SHLVL=0
HOME=/home/kirk
DESKTOP_SESSION=cinnamon
GTK_MODULES=gail:atk-bridge
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
PERLBREW_ROOT=/home/kirk/perl5/perlbrew
PERLBREW_MANPATH=/home/kirk/perl5/perlbrew/perls/perl-5.24.1/man
MESA_DIR=/home/kirk/Documents/research/MESA/mesa-r11701
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
CINNAMON_VERSION=4.0.10
COLORTERM=truecolor
_CE_M=
MANDATORY_PATH=/usr/share/gconf/cinnamon.mandatory.path
QT_QPA_PLATFORMTHEME=qt5ct
HEADAS=/home/kirk/Documents/research/HEASOFT/heasoft-6.26/x86_64-pc-linux-gnu-libc2.27
LOGNAME=kirk
_=./bashTest.sh
DEFAULTS_PATH=/usr/share/gconf/cinnamon.default.path
GIO_EXTRA_MODULES=/usr/lib/x86_64-linux-gnu/gio/modules/
GTK_OVERLAY_SCROLLING=1
XDG_SESSION_ID=c12
TERM=xterm-256color
MESASDK_VERSION=x86_64-linux-20190503
XMM_DIR=/home/kirk/Documents/research/XMM_Newton/xmmsas_20190531_1155
_CE_CONDA=
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
PATH=/home/kirk/Documents/research/MESA/mesasdk/bin:/home/kirk/anaconda3/bin:/home/kirk/anaconda3/condabin:/home/kirk/perl5/perlbrew/bin:/home/kirk/perl5/perlbrew/perls/perl-5.24.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
GDM_LANG=en_US
PERLBREW_HOME=/home/kirk/.perlbrew
SESSION_MANAGER=local/kirk-Inspiron-7352:#/tmp/.ICE-unix/1709,unix/kirk-Inspiron-7352:/tmp/.ICE-unix/1709
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/bc8ec572_68ae_4d79_88a2_3cb33f74d86c
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
DISPLAY=:0
VALGRIND_LIB=/home/kirk/Documents/research/MESA/mesasdk/lib/valgrind
LANG=en_US.UTF-8
XDG_CURRENT_DESKTOP=X-Cinnamon
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
PERLBREW_PATH=/home/kirk/perl5/perlbrew/bin:/home/kirk/perl5/perlbrew/perls/perl-5.24.1/bin
XDG_SESSION_DESKTOP=cinnamon
GNOME_TERMINAL_SERVICE=:1.62
XAUTHORITY=/home/kirk/.Xauthority
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/kirk
MESASDK_ROOT=/home/kirk/Documents/research/MESA/mesasdk
CONDA_PYTHON_EXE=/home/kirk/anaconda3/bin/python
SHELL=/bin/bash
QT_ACCESSIBILITY=1
GDMSESSION=cinnamon
LESSCLOSE=/usr/bin/lesspipe %s %s
QT_LOGGING_RULES=qt5ct.debug=false
PERLBREW_PERL=perl-5.24.1
GJS_DEBUG_OUTPUT=stderr
GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1
XDG_VTNR=7
PWD=/home/kirk
CONDA_EXE=/home/kirk/anaconda3/bin/conda
XDG_DATA_DIRS=/usr/share/cinnamon:/usr/share/gnome:/home/kirk/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
XDG_CONFIG_DIRS=/etc/xdg/xdg-cinnamon:/etc/xdg
OMP_NUM_THREADS=2
PERLBREW_SHELLRC_VERSION=0.82
VTE_VERSION=5202
MANPATH=/home/kirk/Documents/research/MESA/mesasdk/share/man:/home/kirk/perl5/perlbrew/perls/perl-5.24.1/man:/usr/local/man:/usr/local/share/man:/usr/share/man
OPENBLAS_MAIN_FREE=1
UPDATE 2: Again following #TasosPapastylianou's suggestion, after telling the Julia script to log any errors when run from crontab I get the following stacktrace when it attempts to generate frame 408:
ERROR: LoadError: SystemError: opening file "/tmp/juliaFCI2yw.png": No such file or directory
Stacktrace:
[1] #systemerror#43(::Nothing, ::Function, ::String, ::Bool) at ./error.jl:134
[2] systemerror at ./error.jl:134 [inlined]
[3] #open#309(::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Function, ::String) at ./iostream.jl:289
[4] open at ./iostream.jl:281 [inlined]
[5] #open#310(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(Base, Symbol("##274#275")){String}, ::String) at ./iostream.jl:373
[6] open at ./iostream.jl:373 [inlined]
[7] read at ./io.jl:297 [inlined]
[8] _show(::IOStream, ::MIME{Symbol("image/png")}, ::Plots.Plot{Plots.GRBackend}) at /home/kirk/.julia/packages/Plots/h3o4c/src/backends/gr.jl:1603
[9] show(::IOStream, ::MIME{Symbol("image/png")}, ::Plots.Plot{Plots.GRBackend}) at /home/kirk/.julia/packages/Plots/h3o4c/src/output.jl:198
[10] png(::Plots.Plot{Plots.GRBackend}, ::String) at /home/kirk/.julia/packages/Plots/h3o4c/src/output.jl:8
[11] frame(::Animation, ::Plots.Plot{Plots.GRBackend}) at /home/kirk/.julia/packages/Plots/h3o4c/src/animation.jl:20
[12] top-level scope at /home/kirk/Documents/3Body/threeBodyProb.jl:265
[13] include at ./boot.jl:326 [inlined]
[14] include_relative(::Module, ::String) at ./loading.jl:1038
[15] include(::Module, ::String) at ./sysimg.jl:29
[16] exec_options(::Base.JLOptions) at ./client.jl:267
[17] _start() at ./client.jl:436
I'm unsure how to diagnose this--does cron limit the number of files a process can create or something like that? In my bash script I have also manually added the following settings (just in case) but that still resulted in the stacktrace above:
ulimit -n 4096
ulimit -t unlimited

Answer 1

Thanks so much for the help #TasosPapastylianou--that error message eventually led me to this post which fixed my problem (and also significantly sped up the animation rendering process as a nice byproduct).
Ultimately it appears the problem was not with cron or the bash script, but instead with Julia's GR backend. I added the line
GR.inline("png")
To the top of the for loop generating the plots to explicitly tell it I was making png files and apparently that fixes everything--not really sure why this is needed and why it's only needed when running from crontab/jobber so if anyone has further insight I'd love to know, but I'm glad it works!
Thanks again to everyone for their help and insights--this tip should be helpful to anyone making animations in a similar way with Julia due to the dramatic improvement in performance that came from this one line!