Each morning at 10:00 crontab is setup to run this command in a bash script (the below dates are an example, they are calculated using the linux date command in the script):
hg log -R some-repo -b some-branch --date "2012-11-28 10:00 to 2012-11-29 10:00"
The above command is run on the server with the remote repo where all developers push their code. The output is stored in a string and send to me via email. But its always empty! If I run the script manually a few minutes later I get the expected output.
I have compared the history before the script is run by crontab and verified that something has indeed been committed/pushed to the remote repo.
Any ideas why running the above script AFTER 10:00 gives the correct output and not when run at 10:00?
In the script I do this:
logString=$(hg log -R "$path-to-repo" -b $branch --date "$YESTERDAY to $TODAY")
if [ -z "$logString" ]; then
logString="Nothing"
fi
EDIT: The hg log is ONLY empty when the script is run by crontab.
SOLVED: I needed to specify the full path to hg (/usr/local/bin/hg) for cron to see it. A bit strange since it does not need have the full path to svn and it has worked fine previously without the full path to hg.
How do I see what cron has in its PATH?
crontab has a very limited environment. If you are assuming that certain variables are going to be available from your .profile (or equivalent) it's most likely not going to be there. If you need it you will have to source it into your script.
It also looks like you you have included a snippet of a longer script. Make the simplest script that should work and add it to the cron. That will make it easier to debug. For example, if you hard-code the dates in the script, you should be able to get it to work. Once you have that working you can gradually add complexity until you figure out what's not working.
Related
I've created a bash file that queries my database and then updates some tables.
When I run it manually everything goes smoothly but when I run it with a cronjob it runs the first query and then stops before it goes into a loop.
After looking into it on the net I found a few things that may be the issue but from my side everything looks in order.
So what I did:
Checked if #!/bin/bash is included in my bash at the start and it is.
Checked that the path is correct in the cronjob. My cronjob below
0-59/5 * * * * cd /path/path2/bashLocation/; ./bash.sh
The loop is in the format of
for ID in ${IDS//,/ }
do
...do something
done
This works fine tested manually. My IDS are in string format that why I split it with //,/.(Works fine)
I log all outputs in a log file but it doesn't show any error.
Has anyone encountered this issue before or has any ideas how to fix the issue?
If the command you are running in cron has percent signs ('%'), they need to be escaped with a backslash. I've been bitten by this. From the manpage: "Percent-signs (%) in the command, unless escaped with backslash () ..."
The $PATH variable may be different when run from cron. Try putting something like this at the beginning of your script: export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Try running bash explicitly, i.e. rather than ./bash.sh in crontab, try /bin/bash bash.sh
I don't know how helpful this may be to some people but I noticed when I printenv shell in my logs it printed that it was bin/sh even if I define it at the top of my script and run it as a bash file.
So what I did was changed all parts of my code that where not supported by shell and my conjob works fine.
So I assume that conjob does not support bash files. (Didn't find anything on the internet about this.)
Why it runs in /bin/sh I don't know.
Hope someone finds this helpful.
I am using an EC2 instance, crontab, and slack-cleaner to delete all Slack messages older than 48 hours. To do this, I created delete_slack.sh (I've deleted my slack api token):
for CHANNEL in random general
do
slack-cleaner --token <MY TOKEN> --message --channel $CHANNEL --user "*" --before $(date -d '48 hour ago' "+%Y%m%d") --perform
done
Then I created a crontab line to run it every minute (once it works I'll change the timing to once a day) and had cron spit out the results to a log file:
* * * * * /home/ubuntu/delete_slack/delete_slack.sh >> /var/log/delete_slack.log 2>&1
To test, I ran sh /home/ubuntu/delete_slack/delete_slack.sh >> /var/log/delete_slack.log 2>&1 in the shell and it works fine. However, when I let the crontab run I get an error in the log file:
/home/ubuntu/delete_slack/delete_slack.sh: 3: /home/ubuntu/delete_slack/delete_slack.sh: slack-cleaner: not found
Any ideas? I've been banging my head against this all afternoon.
Sounds like the PATH you get via cron and the PATH you get through your login are different.
Either set the PATH in your script or use the absolute path to slack-cleaner
The PATH tells the shell which directories to search for executables (including scripts). You can echo $PATH to compare your path to the one cron gives and confirm that this is the issue.
If using the absolute path works, that is simplest, but if slack-cleaner uses other exes itself, setting the path may be better.
If you want to go the "modify PATH" method then you want to append the correct path to existing PATH and not completely overwrite it. i.e. export PATH=$PATH:/path/to/slack-cleaner-dir. You can always use which slack-cleaner to find out the correct path. NOTE: you want the directory without "slack-cleaner" appended to the end.
ALWAYS use full path in crons and you'll save a lot of time.
If you don't like export PATH=... then just use /path/to/slack-cleaner-dir instead.
Just load your profile before running the command to be in the exact same situation as when you launch it from your shell :
* * * * * . ~/.profile;/home/ubuntu/delete_slack/delete_slack.sh >> /var/log/delete_slack.log 2>&1
As I read that you're a bit new to this, here are just some more explanations about the profile :
The profile is a file loaded automatically when you connect in shell with your user.
The file is hidden in your home directory, to see it, you can launch :
ls -la ~
If you're in bash, the file will be named .bash_profile, if you're in shell or ksh, it will be named .profile
Hope it helped !
I have this script
#!/bin/sh
curl -4 http://wttr.in/Colorado\ Springs
that I want to automatically execute each morning. I have my crontab entry as
* 7 * * * (path to script)
But either the script doesnt run, or it runs and then immediately closes the shell. I know that my cronjobs are running as I have other scripts for backups that run on an hourly basis but cant figure out what detail I am missing here. I found one suggestion to include $SHELL in the script but that made no difference. Any suggestions?
Usually when I have to keep the terminal open I would exec bash as my last command. I do that when I write installer script which would open terminal; do the job and get lost after that. But if there is an error then I want the terminal to stay there so that I can read the error.
exec is used to replace the current program with argument which we provide to exec.
Actually, I don't know what are you trying to achieve with this call in your crontab. Do you want to see the weather report on your terminal? Do you want to save the weather report in the file? Get it in your emails?
If you do no redirections, you'll get the report in your mail.
If you want to have it in a file, just do:
curl wttr.in/Colorado+Springs > file
If you want to have it on you terminals do
curl wttr.in/Colorado+Springs | wall
Please note that you don't need -4, http:// and you can replace \space with +.
(DISCLAIMER: I'm the author of wttr.in)
I have a bash script that runs every five minutes. Among other things it runs php scripts reading on existing files, and at the end it sends an email. When run manually, it does the job. When the cronjob runs, it partially completes the task. The code below:
DIR="/somedir/"
php ${DIR}client.php $DIR
cat ${DIR}alert_list.txt | uniq | while read alert;
do
if [ -s ${DIR}alerts/$alert.txt ]; then
# send the email.
echo "Sending email for..."$alert >> ${DIR}email.txt
DETAILFILE="tools/"$alert
DETAILFILEP=${DETAILFILE}".txt"
php ${DIR}email.php $alert
fi
done
echo 'search completed.'
in 'cronjob mode' it never gets to the 'do' statement. In manual mode it does everything.
Any thoughts?
Thanks a lot!
I found the issue within the PHP scripts. Relative calls to files which are located in other paths get missed when it runs automatically. Apparently, it runs from somewhere else, so it could not progress because of the missing input files created by initial php script.
Thanks.
The difference between the manual run and the cron run in bash is that in case of a cron the .bash* files are not sourced initially, and hence it might happen some of the required settings (Eg: PATH) are different.
And also, (replying to your previous comment) The PWD in case of a cron is $HOME, so the needed files as you mentioned are not picked, whereas in case of manual run it picks from the path you run.
Hope this helps.
This keeps happening to me all the time:
1) I write a script(ruby, shell, etc).
2) run it, it works.
3) put it in crontab so it runs in a few minutes so I know it runs from there.
4) It doesnt, no error trace, back to step 2 or 3 a 1000 times.
When I ruby script fails in crontab, I can't really know why it fails cause when I pipe output like this:
ruby script.rb >& /path/to/output
I sorta get the output of the script, but I don't get any of the errors from it and I don't get the errors coming from bash (like if ruby is not found or file isn't there)
I have no idea what environmental variables are set and whether or not it's a problem. Turns out that to run a ruby script from crontab you have to export a ton of environment variables.
Is there a way for me to just have crontab run a script as if I ran it myself from my terminal?
When debugging, I have to reset the timer and go back to waiting. Very time consuming.
How to test things in crontab better or avoid these problems?
"Is there a way for me to just have crontab run a script as if I ran it myself from my terminal?"
Yes:
bash -li -c /path/to/script
From the man page:
[vindaloo:pgl]:~/p/test $ man bash | grep -A2 -m1 -- -i
-i If the -i option is present, the shell is interactive.
-l Make bash act as if it had been invoked as a login shell (see
INVOCATION below).
G'day,
One of the basic problems with cron is that you get a minimal environment being set by cron. In fact, you only get four env. var's set and they are:
SHELL - set to /bin/sh
LOGNAME - set to your userid as found in /etc/passwd
HOME - set to your home dir. as found in /etc/passwd
PATH - set to "/usr/bin:/bin"
That's it.
However, what you can do is take a snapshot of the environment you want and save that to a file.
Now make your cronjob source a trivial shell script that sources this env. file and then executes your Ruby script.
BTW Having a wrapper source a common env. file is an excellent way to enforce a consistent environment for multiple cronjobs. This also enforces the DRY principle because it gives you just one point to update things as required, instead of having to search through a bunch of scripts and search for a specific string if, say, a logging location is changed or a different utility is now being used, e.g. gnutar instead of vanilla tar.
Actually, this technique is used very successfully with The Build Monkey which is used to implement Continuous Integration for a major software project that is common to several major world airlines. 3,500kSLOC being checked out and built several times a day and over 8,000 regression tests run once a day.
HTH
'Avahappy,
Run a 'set' command from inside of the ruby script, fire it from crontab, and you'll see exactly what's set and what's not.
To find out the environment in which cron runs jobs, add this cron job:
{ echo "\nenv\n" && env|sort ; echo "\nset\n" && set; } | /usr/bin/mailx -s 'my env' you#example.com
Or send the output to a file instead of email.
You could write a wrapper script, called for example rbcron, which looks something like:
#!/bin/bash
RUBY=ruby
export VAR1=foo
export VAR2=bar
export VAR3=baz
$RUBY "$*" 2>&1
This will redirect standard error from ruby to the standard output. Then you run rbcron in your cron job, and the standard output contains out+err of ruby, but also the "bash" errors existing from rbcron itself. In your cron entry, redirect 2>&1 > /path/to/output to get output+error messages to go to /path/to/output.
If you really want to run it as yourself, you may want to invoke ruby from a shell script that sources your .profile/.bashrc etc. That way it'll pull in your environment.
However, the downside is that it's not isolated from your environment, and if you change that, you may find your cron jobs suddenly stop working.