I've created a bash file that queries my database and then updates some tables.
When I run it manually everything goes smoothly but when I run it with a cronjob it runs the first query and then stops before it goes into a loop.
After looking into it on the net I found a few things that may be the issue but from my side everything looks in order.
So what I did:
Checked if #!/bin/bash is included in my bash at the start and it is.
Checked that the path is correct in the cronjob. My cronjob below
0-59/5 * * * * cd /path/path2/bashLocation/; ./bash.sh
The loop is in the format of
for ID in ${IDS//,/ }
do
...do something
done
This works fine tested manually. My IDS are in string format that why I split it with //,/.(Works fine)
I log all outputs in a log file but it doesn't show any error.
Has anyone encountered this issue before or has any ideas how to fix the issue?
If the command you are running in cron has percent signs ('%'), they need to be escaped with a backslash. I've been bitten by this. From the manpage: "Percent-signs (%) in the command, unless escaped with backslash () ..."
The $PATH variable may be different when run from cron. Try putting something like this at the beginning of your script: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Try running bash explicitly, i.e. rather than ./bash.sh in crontab, try /bin/bash bash.sh
I don't know how helpful this may be to some people but I noticed when I printenv shell in my logs it printed that it was bin/sh even if I define it at the top of my script and run it as a bash file.
So what I did was changed all parts of my code that where not supported by shell and my conjob works fine.
So I assume that conjob does not support bash files. (Didn't find anything on the internet about this.)
Why it runs in /bin/sh I don't know.
Hope someone finds this helpful.
Related
I have a Python3 program that I need to execute 2 minutes past every hour.
Unable to get python to execute directly from cron, I’ve written a shell script with the commands in it.
The crontab to execute this shell script is this:
*/2 * * * * bash /home/john/PYTHONS/MASTER.sh
The shell script is simple. I’ve tried various versions, including this:
#!/bin/bash
cd /home/john/PYTHONS/
python3 ~/PYTHONS/COMPLEXPYTHON.py
python3 ~/PYTHONS/SIMPLEPYTHON.py
and including this:
#!/bin/bash
cd /home/john/PYTHONS/GDAXDEV/VPUMP
python3 $HOME/PYTHONS/COMPLEXPYTHON.py
python3 $HOME/PYTHONS/SIMPLEPYTHON.py
Here’s the issue:
If I execute the shell script from the command line BOTH python scripts run just fine.
If I execute the shell script from cron, ONLY THE SECOND python script runs. The first one just doesn’t run at all. And, I can’t seem to find out why, or how to fix it.
Any suggestions or help would be greatly appreciated.
I see questions about this issue all around, and no single one of them solved the issue for me. Several hours of testing every combination of variables lead me to a solution that’s reliable.
I’m no longer using a shell script, but invoking python directly in cron.
First, we did this:
$ which python3
/home/john/[path to python]/bin/python3
So, we added the shebang in every one of the python files in the project:
#!/home/john/[path to python]/bin/python3
And, in cron we used the full path to that python3, as well as the full path to the WORKING.py file we wanted to run. We want the thing to run 5 minutes past every hour so:
5 * * * * /home/john/[path to python]/bin/python3 /home/john/[path to dir]/WORKING.py
However, the log files this produced were being placed in /home/john/, and I wanted them in the specific directory. Also, there are a load of additional python files in there – with the same names as python files in nearby directories – so I wanted to be certain that any call gets the right one.
So, I added a CD command at the beginning of the cron command:
5 * * * * cd /home/john/[path to dir]/ && /home/john/[path to python]/bin/python3 /home/john/path to dir]/WORKING.py
It’s been running reliably ever since.
I’m grateful for the resources at stackoverflow and the great and fast help I’ve received in this community, and I hope this entry helps save someone else a whole afternoon of frustration.
I have a simple incron task setup to run a command whenever a particular .json file is written-to, then closed.
/var/www/html/api/private/resources/myfile.json IN_CLOSE_WRITE,IN NO LOOP /var/www/html/api/private/resources/run_service.sh
I can see that whenever the file to written to, there is a syslog entry for the event, and the command that was triggered - along the lines of <date> - incrond: CMD (/var/www/html/api/private/resources/run_service.sh).
But nothing seems to happen...
initially I thought this would be caused by an issue with the script, but replacing the script command to something simple such as echo "hello world" > /tmp/mylog.log still yields no output or results. I seem to have hit a brick wall with this one!
Update
Changing the incron command to read "/bin/bash /var/www/html/api/private/resources/run_service.sh" now seems to triggering the script correctly, as I can now get output from the script.
A simple mistake on my part, despite all examples online showing that using the script as the command should run it, for me it only works if I explicitly call bash to execute it
"<my directory/file to watch> <trigger condition> /bin/bash /var/www/html/api/private/resources/run_service.sh
I have this script
#!/bin/sh
curl -4 http://wttr.in/Colorado\ Springs
that I want to automatically execute each morning. I have my crontab entry as
* 7 * * * (path to script)
But either the script doesnt run, or it runs and then immediately closes the shell. I know that my cronjobs are running as I have other scripts for backups that run on an hourly basis but cant figure out what detail I am missing here. I found one suggestion to include $SHELL in the script but that made no difference. Any suggestions?
Usually when I have to keep the terminal open I would exec bash as my last command. I do that when I write installer script which would open terminal; do the job and get lost after that. But if there is an error then I want the terminal to stay there so that I can read the error.
exec is used to replace the current program with argument which we provide to exec.
Actually, I don't know what are you trying to achieve with this call in your crontab. Do you want to see the weather report on your terminal? Do you want to save the weather report in the file? Get it in your emails?
If you do no redirections, you'll get the report in your mail.
If you want to have it in a file, just do:
curl wttr.in/Colorado+Springs > file
If you want to have it on you terminals do
curl wttr.in/Colorado+Springs | wall
Please note that you don't need -4, http:// and you can replace \space with +.
(DISCLAIMER: I'm the author of wttr.in)
I have a ruby script containing system command like http://gist.github.com/235833, while I ran this script from shell, it works correctly, but when I added it to my cron job list, it doesn't work any more, the cron job is like:
10/* * * * * cd /home/hekin; /usr/bin/ruby my_script.rb
any idea what's going wrong with what i've done? Thank you.
Thank you all for your answers.
It's my mistake.
Since I'm using ssh key forwarding on the local machine, while I executed the script from the shell, the ssh key forwarding related environment variables are all sitting there, but from cron job context, those environment variables are missing.
Try to separate the things that might go wrong. The ones I can think of are:
The cron syntax - is the time value given legal and fitting your shell?
Permissions - execute permissions and read permissions for the relevant directory and file
Quoting - what scope does cron cover? Does it run only the first command?
In order to dissect this, I suggest you first run a really simple cron job, like 'ls'. Next run a single-liner script. Next embed your commands in a shell-script file. Somewhere along these lines you should find the problem.
The problem is your environment. While testing in your shell its fully equipped and boosted by your shell environment. While running under cron its very, very stripped down.
Where is the destination "." for your script? I guess it will be "/" and may not "$HOME" thus your script won't be able to write at that location and fails. Try using an absolut path for the destination.
This keeps happening to me all the time:
1) I write a script(ruby, shell, etc).
2) run it, it works.
3) put it in crontab so it runs in a few minutes so I know it runs from there.
4) It doesnt, no error trace, back to step 2 or 3 a 1000 times.
When I ruby script fails in crontab, I can't really know why it fails cause when I pipe output like this:
ruby script.rb >& /path/to/output
I sorta get the output of the script, but I don't get any of the errors from it and I don't get the errors coming from bash (like if ruby is not found or file isn't there)
I have no idea what environmental variables are set and whether or not it's a problem. Turns out that to run a ruby script from crontab you have to export a ton of environment variables.
Is there a way for me to just have crontab run a script as if I ran it myself from my terminal?
When debugging, I have to reset the timer and go back to waiting. Very time consuming.
How to test things in crontab better or avoid these problems?
"Is there a way for me to just have crontab run a script as if I ran it myself from my terminal?"
Yes:
bash -li -c /path/to/script
From the man page:
[vindaloo:pgl]:~/p/test $ man bash | grep -A2 -m1 -- -i
-i If the -i option is present, the shell is interactive.
-l Make bash act as if it had been invoked as a login shell (see
INVOCATION below).
G'day,
One of the basic problems with cron is that you get a minimal environment being set by cron. In fact, you only get four env. var's set and they are:
SHELL - set to /bin/sh
LOGNAME - set to your userid as found in /etc/passwd
HOME - set to your home dir. as found in /etc/passwd
PATH - set to "/usr/bin:/bin"
That's it.
However, what you can do is take a snapshot of the environment you want and save that to a file.
Now make your cronjob source a trivial shell script that sources this env. file and then executes your Ruby script.
BTW Having a wrapper source a common env. file is an excellent way to enforce a consistent environment for multiple cronjobs. This also enforces the DRY principle because it gives you just one point to update things as required, instead of having to search through a bunch of scripts and search for a specific string if, say, a logging location is changed or a different utility is now being used, e.g. gnutar instead of vanilla tar.
Actually, this technique is used very successfully with The Build Monkey which is used to implement Continuous Integration for a major software project that is common to several major world airlines. 3,500kSLOC being checked out and built several times a day and over 8,000 regression tests run once a day.
HTH
'Avahappy,
Run a 'set' command from inside of the ruby script, fire it from crontab, and you'll see exactly what's set and what's not.
To find out the environment in which cron runs jobs, add this cron job:
{ echo "\nenv\n" && env|sort ; echo "\nset\n" && set; } | /usr/bin/mailx -s 'my env' you#example.com
Or send the output to a file instead of email.
You could write a wrapper script, called for example rbcron, which looks something like:
#!/bin/bash
RUBY=ruby
export VAR1=foo
export VAR2=bar
export VAR3=baz
$RUBY "$*" 2>&1
This will redirect standard error from ruby to the standard output. Then you run rbcron in your cron job, and the standard output contains out+err of ruby, but also the "bash" errors existing from rbcron itself. In your cron entry, redirect 2>&1 > /path/to/output to get output+error messages to go to /path/to/output.
If you really want to run it as yourself, you may want to invoke ruby from a shell script that sources your .profile/.bashrc etc. That way it'll pull in your environment.
However, the downside is that it's not isolated from your environment, and if you change that, you may find your cron jobs suddenly stop working.