I have a Python3 program that I need to execute 2 minutes past every hour.
Unable to get python to execute directly from cron, I’ve written a shell script with the commands in it.
The crontab to execute this shell script is this:
*/2 * * * * bash /home/john/PYTHONS/MASTER.sh
The shell script is simple. I’ve tried various versions, including this:
#!/bin/bash
cd /home/john/PYTHONS/
python3 ~/PYTHONS/COMPLEXPYTHON.py
python3 ~/PYTHONS/SIMPLEPYTHON.py
and including this:
#!/bin/bash
cd /home/john/PYTHONS/GDAXDEV/VPUMP
python3 $HOME/PYTHONS/COMPLEXPYTHON.py
python3 $HOME/PYTHONS/SIMPLEPYTHON.py
Here’s the issue:
If I execute the shell script from the command line BOTH python scripts run just fine.
If I execute the shell script from cron, ONLY THE SECOND python script runs. The first one just doesn’t run at all. And, I can’t seem to find out why, or how to fix it.
Any suggestions or help would be greatly appreciated.
I see questions about this issue all around, and no single one of them solved the issue for me. Several hours of testing every combination of variables lead me to a solution that’s reliable.
I’m no longer using a shell script, but invoking python directly in cron.
First, we did this:
$ which python3
/home/john/[path to python]/bin/python3
So, we added the shebang in every one of the python files in the project:
#!/home/john/[path to python]/bin/python3
And, in cron we used the full path to that python3, as well as the full path to the WORKING.py file we wanted to run. We want the thing to run 5 minutes past every hour so:
5 * * * * /home/john/[path to python]/bin/python3 /home/john/[path to dir]/WORKING.py
However, the log files this produced were being placed in /home/john/, and I wanted them in the specific directory. Also, there are a load of additional python files in there – with the same names as python files in nearby directories – so I wanted to be certain that any call gets the right one.
So, I added a CD command at the beginning of the cron command:
5 * * * * cd /home/john/[path to dir]/ && /home/john/[path to python]/bin/python3 /home/john/path to dir]/WORKING.py
It’s been running reliably ever since.
I’m grateful for the resources at stackoverflow and the great and fast help I’ve received in this community, and I hope this entry helps save someone else a whole afternoon of frustration.
Related
I've created a bash file that queries my database and then updates some tables.
When I run it manually everything goes smoothly but when I run it with a cronjob it runs the first query and then stops before it goes into a loop.
After looking into it on the net I found a few things that may be the issue but from my side everything looks in order.
So what I did:
Checked if #!/bin/bash is included in my bash at the start and it is.
Checked that the path is correct in the cronjob. My cronjob below
0-59/5 * * * * cd /path/path2/bashLocation/; ./bash.sh
The loop is in the format of
for ID in ${IDS//,/ }
do
...do something
done
This works fine tested manually. My IDS are in string format that why I split it with //,/.(Works fine)
I log all outputs in a log file but it doesn't show any error.
Has anyone encountered this issue before or has any ideas how to fix the issue?
If the command you are running in cron has percent signs ('%'), they need to be escaped with a backslash. I've been bitten by this. From the manpage: "Percent-signs (%) in the command, unless escaped with backslash () ..."
The $PATH variable may be different when run from cron. Try putting something like this at the beginning of your script: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Try running bash explicitly, i.e. rather than ./bash.sh in crontab, try /bin/bash bash.sh
I don't know how helpful this may be to some people but I noticed when I printenv shell in my logs it printed that it was bin/sh even if I define it at the top of my script and run it as a bash file.
So what I did was changed all parts of my code that where not supported by shell and my conjob works fine.
So I assume that conjob does not support bash files. (Didn't find anything on the internet about this.)
Why it runs in /bin/sh I don't know.
Hope someone finds this helpful.
I have a cron job to run a sed shell called sedcmd.sh to pre-process some json data. When i am in the proper directory I manually run it with
. ./sedcmd.sh
And it works. The shell itself works fine.
for reference one of the commands inside looks like
sed -i '/^\s*$/d' /home/school/Desktop/Programs/rawjsondata.txt
my cronjob looks like
5 * * * * . ./home/school/Desktop/Programs/sedcmd.sh
I get the error "No such file or directory found". What am I doing wrong. I've triple checked for any random spelling errors. I also can't seem to run sedcmd.sh from any other directory even when i give the entire file path, so its definitely something I'm doing wrong. My thoughts for solutions are either
I should add sedcmd.sh to my $PATH or bashrc so i can call it from anywhere. Which I don't know how to do.
OR
figure out how to call it correctly from crontab. Which I also dont know how to do.
When you are running a script from terminal, you do:
./script_name.sh but to execute the same script from crontab you do something like 5 * * * * /path/to/script/script_name.sh
As Sam has got this answer from the comments,posting the answer as community wiki.
I've looked for answers to this already without avail, so I turn here.
I'm running Ubuntu 12.04LTS, using RVM, running ruby 2.0.0p353.
I've written a script that, in a nutshell, modifies a file. Let's call this file access.txt.
Whenever I run the script by hand, it works beautifully, and modifies the file.
However, when I run this via crontab, it does not update the file. This is currently what I have in crontab.
0 * * * * /path/to/script.rb
I should also note that despite it being bad practice, I am running this as root.
I've also tried specifying it to run within a bash file.
0 * * * * /path/to/bash.sh
This also produces no results.
The cron log shows these jobs as running, and both have executable permissions.
Does anyone have any ideas?
I have a ruby script containing system command like http://gist.github.com/235833, while I ran this script from shell, it works correctly, but when I added it to my cron job list, it doesn't work any more, the cron job is like:
10/* * * * * cd /home/hekin; /usr/bin/ruby my_script.rb
any idea what's going wrong with what i've done? Thank you.
Thank you all for your answers.
It's my mistake.
Since I'm using ssh key forwarding on the local machine, while I executed the script from the shell, the ssh key forwarding related environment variables are all sitting there, but from cron job context, those environment variables are missing.
Try to separate the things that might go wrong. The ones I can think of are:
The cron syntax - is the time value given legal and fitting your shell?
Permissions - execute permissions and read permissions for the relevant directory and file
Quoting - what scope does cron cover? Does it run only the first command?
In order to dissect this, I suggest you first run a really simple cron job, like 'ls'. Next run a single-liner script. Next embed your commands in a shell-script file. Somewhere along these lines you should find the problem.
The problem is your environment. While testing in your shell its fully equipped and boosted by your shell environment. While running under cron its very, very stripped down.
Where is the destination "." for your script? I guess it will be "/" and may not "$HOME" thus your script won't be able to write at that location and fails. Try using an absolut path for the destination.
Is there anything like lint for crontab? I'd like to know that i've got all my spaces and stars sorted out without waiting for something to not work.
There's a Python linter for crons. See chkcrontab project
You can install it via pip:
pip3 install chkcrontab
Example usage:
chkcrontab /etc/cron.d/power-schedule
Checking correctness of /etc/cron.d/power-schedule
E: 15: 0 12 * foo * * root echo hi
e: FIELD_VALUE_ERROR: foo is not valid for field "month" (foo)
e: INVALID_USER: Invalid username "*"
E: There were 2 errors and 0 warnings.
I've found CronWTF to be incredibly helpful when writing crontabs - it translates your stars and commands into something more human friendly, to make it easier to read strange cron jobs.
Better yet, because it's all javascript you can run it locally, and noone need know about your top sekrit cron jobs.
Another alternative if you code ruby is to use the whenever gem - you use a sample ruby file called schedule.rb to parse, and generate crontabs from like so:
every 10.minutes do
command "/usr/bin/my_great_command"
end
Will give you a crontab entry of
0,10,20,30,40,50 * * * * /usr/bin/my_great_command
And this one here:
every 2.days, :at => '4:30am' do
command "/usr/bin/my_great_command"
end
Will give you:
30 4 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31 * * /usr/bin/my_great_command
I don't think you need a lint for crontab. There's 5 fields that are space separated then a space then the command to run and its args finish off the line.
Also, on Ubuntu at least, crontab won't let you save a bum file. I just tried a few things and it barfed on all of them. I guess that means that crontab is its own 'lint for cron'.
It might be a bit off, but an easy way would be to just load it with a graphical crontab editor like kcron or gcrontab. If you need to call it in a script, this question is about how to do it in php.
I'm not sure if this is the sort of thing you're looking for, but it makes writing crontabs really easy by showing you exactly what you're setting the schedule to:
https://crontab.guru/
You can try shell script named 48-verifycron from Wicked Cool Shell Scripts, 2nd Edition,
if you can't access python and pip to use chkcrontab