Tried solutions found online but none worked so far
How can I further debug this ? Thank you.
I am using a M1 mac with Monterey 12.0.1, I changed from bash to zsh as I thought this might help. I touched .zprofile, copied everthing from .bash_profile
PATH is copied from echo $PATH
The command works executed in terminal but does not work via crontab
inside crontab -e
SHELL=/bin/zsh
PATH=/Users/username/opt/anaconda3/bin:/Users/username/opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin
* * * * * /bin/zsh /Users/username/path/to/folder/script.sh > /tmp/result.log
/tmp/result.log is empty
script.sh
#!/bin/zsh
search_dir=/Users/username/path/to/folder
for i in {1..3}
do
echo "Number: $i"
touch $search_dir/$i.txt
done
I'm at my wits end with this. I have the following script:
#!/usr/bin/env bash
declare -a nriArray=(newrelic-cli node-newrelic nri-flex helm-charts infrastructure-agent opentelemetry-exporter-java opentelemetry-exporter-go newrelic-lambda-cli newrelic-node-apollo-server-plugin nri-kubernetes nri-prometheus nri-redis infrastructure-bundle newrelic-lambda-layers nri-jmx newrelic-winston-logenricher-node nri-cassandra micrometer-registry-newrelic newrelic-fluent-bit-output nri-kafka node-newrelic-aws-sdk nri-elasticsearch nri-mysql nri-nagios nri-snmp node-newrelic-superagent nri-kube-events nri-mssql newrelic-module-util-java newrelic-logenricher-dotnet aws-log-ingestion newrelic-lambda-tracer-java nri-docker nri-oracledb k8s-metadata-injection nri-discovery-kubernetes nri-haproxy nri-postgresql node-native-metrics nri-consul nri-rabbitmq nri-vsphere nri-winservices nri-ecs newrelic-airflow-plugin newrelic-monolog-logenricher-php node-newrelic-koa nri-f5 aws_s3_log_ingestion_lambda dropwizard-metrics-newrelic k8s-webhook-cert-manager newrelic-fluentd-output nri-couchbase nri-memcached nri-mongodb java-aws-lambda logstash-output-plugin nri-apache nri-varnish java-log-extensions nri-nginx nri-statsd newrelic-opencensus-exporter-go newrelic-opencensus-exporter-python python-agent-extension)
cd /Users/aschneider/NewRelic-OpenSource/
counter=0
while [[ "$counter" -lt "${#nriArray[#]}" ]]; do
cd "${nriArray[$counter]}"
git pull &>> /Users/aschneider/NewRelic-OpenSource/cronLog.log
gsed -i '/Already up to date./d' /Users/aschneider/NewRelic-OpenSource/cronLog.log
git clean -q -d -f
cd ..
((counter=counter+1))
done
printf "\n\n" >> /Users/aschneider/NewRelic-OpenSource/cronLog.log
holdingVar=$(cat -s /Users/aschneider/NewRelic-OpenSource/cronLog.log) && echo "$holdingVar" > /Users/aschneider/NewRelic-OpenSource/cronLog.log
/Applications/Utilities/terminal-notifier.app/Contents/MacOS/terminal-notifier -message "~/NewRelic-OpenSource/ repositories were updated."
The script recursively goes through an array of GitHub repos, pulls in any updates to them, and then moves onto the next folder. It removes any lines stating that the folder is up to date in order to only keep changes in the log file, and then it uses Terminal Notifier to alert me after each update. It works perfectly when I run it with bash ~/NewRelic-OpenSource/updateOpenSource.sh, but it doesn't even seem to execute when I put the script in crontab. My crontab -l output is below:
SHELL=/usr/local/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
*/30 8-17 * * 1-5 /usr/local/bin/bash /Users/aschneider/NewRelic-OpenSource/updateOpenSources.sh
I have it running every 30 minutes, from 8am to 5pm, on weekdays. Even if I switch to * * * * * [command], it doesn't do anything, so I don't think it's a scheduling syntax issue. I'm currently running it all on my MacBook, which uses GNU bash, version 5.1.4(1)-release (x86_64-apple-darwin20.2.0). I've tried adding the PATH definition to the crontab, as well as to the script itself. I've also verified the script is executable with chmod +x {script}. Any ideas on what I'm doing wrong here?
Cronjobs run as root by default. You may want to run as the user you would normally run the script as, if you were running the script manually. For example,
*/5 * * * * (add the username here) sample_executable
I have also read, but I may be wrong they do not use your PATH variables. So all executables should be fully refrenced before running the script.
Also check the permissions to run the script in crontab.
I read the other related topics but they didn't help me.
I have a shell script which checks if my python script is not running,it will run it. Otherwise it will just skip and do nothing.
It totally works when I use:
bash myshellscrip.sh
And I get the result that I want which is doing some tasks and sending emails to some correspondents. However, when I try to run this particular shell script on crontab, it doesn't send out the emails and doesn't do the other tasks.
I tried the following on crontab and none of them worked.
* * * * * /bin/bash /path/to/my/script/myshellscrip.sh
* * * * * /bin/bash /path/to/my/script/myshellscrip.sh >> /some/other/path/output.txt
When I save the changes into 'output.txt' file, it creates the file but it doesn't send the emails or doing other tasks.
I also tried the option of reboot because I need this program to run at start up too, and this didn't work:
#reboot /bin/bash /path/to/my/script/myshellscrip.sh
Does anyone know how to fix it?
EDIT:
As I was checking with the simplest shell scrip like:
#!/bin/sh
/usr/bin/python /home/pi/DCA/code.py
My crontab wouldn't have any output in my output.txt file although my code.py have something printing out, too.
However, when I use a very simple python code for example only a 'print' statement it will run and save the output into output.txt.
Seems like your shell script crashes / stops before it can do something (possibly due to the environment being different or permission issues). You can check /var/log/syslog to find out.
You could try removing /bin/bash, I don't think that's necessary?
Run the cron job in debug mode. for that, Add -x to the bash command on the cronjob and save their output in the file.
bash -x /path/to/script.sh >> /path/to/the/output.txt
You can find the problem.
Apparently crontab was running my script several times. So I tried to use different locking mechanisms to put a lock around my scrip but only using flock worked for me. In my crontab I added this line:
* * * * * /usr/bin/flock -n /tmp/ms.lockfile /bin/bash /path/to/my/script/myShellScript.sh
I have a bashscript that I'm running with crontab. Unfortunately, a script that works fine when run manually fails with the error:
Syntax error: "(" unexpected (expecting "}")
Where the line in question is line 22 which is:
declare -a PREV_TOTAL=( $(for i in ${range[#]}; do echo 0; done) )
In the larger context:
TOTAL_CPU_USAGE=0
TOTAL_CPU=$(grep -c ^processor /proc/cpuinfo) #set number of CPUs to check for
declare -a 'range=({'"0..$TOTAL_CPU"'})'
let "TOTAL_CPU=$TOTAL_CPU - 1"
#declare array of size TOTAL_CPU to store values (eg. 8 cpus makes arrays of size 8)
declare -a PREV_TOTAL=( $(for i in ${range[#]}; do echo 0; done) )
declare -a PREV_IDLE=( $(for i in ${range[#]}; do echo 0; done) )
This works when manually just fine, but I don't understand what I'm doing wrong that causes cron to give this error? If you know I'd be very appreciative. Thanks.
EDIT: My crontab looks like this:
# m h dom mon dow command
SHELL=/bin/bash
#reboot cd /home/ubuntu/waste-cloud-computing/probe && probe.sh >> /var/log/somelogfile.log 2>&1
And I access it with sudo crontab -e. I'm still getting the issue while providing the SHELL variable.
EDIT 1: Thanks to some help I got past the syntax issues by ensuring the shell was using bash. Now I get the error, /bin/bash: probe.bash: command not found. I assume its some kind of PATH issue, but which bash returns /bin/bash so it seems normal to me. Maybe someone knows what's up?
cron jobs are run by sh by default, not bash. If you are using ubuntu/vixiecron, you can set the SHELL env variable at the top of the crontab to make cron run the commands in your crontab with bash.
SHELL=/bin/bash
If the script you want to be run is a bash script, make sure you have a shebang at the first line:
#!/bin/bash
Also note that there will be other potential troubleshooting steps if your scripts depend on a particular user's profile, env vars, etc. depending on which crontab you are editing.
Thanks to the help of the people here I found my issue was not syntax but rather the use of sh over bash. This was fixed by setting the crontab this way so future users can see:
# m h dom mon dow command
SHELL=/bin/bash
#reboot cd /home/ubuntu/waste-cloud-computing/probe && ./probe.sh >> /var/log/somelogfile.log 2>&1
The key points are the SHELL variable being set and the ./ before running the script.
I normally have several problems with how cron executes scripts as they normally don't have my environment setup. Is there a way to invoke bash(?) in the same way cron does so I could test scripts before installing them?
Add this to your crontab (temporarily):
* * * * * env > ~/cronenv
After it runs, do this:
env - `cat ~/cronenv` /bin/sh
This assumes that your cron runs /bin/sh, which is the default regardless of the user's default shell.
Footnote: if env contains more advanced config, eg PS1=$(__git_ps1 " (%s)")$, it will error cryptically env: ": No such file or directory.
Cron provides only this environment by default :
HOME user's home directory
LOGNAME user's login
PATH=/usr/bin:/usr/sbin
SHELL=/usr/bin/sh
If you need more you can source a script where you define your environment before the scheduling table in the crontab.
Couple of approaches:
Export cron env and source it:
Add
* * * * * env > ~/cronenv
to your crontab, let it run once, turn it back off, then run
env - `cat ~/cronenv` /bin/sh
And you are now inside a sh session which has cron's environment
Bring your environment to cron
You could skip above exercise and just do a . ~/.profile in front of your cron job, e.g.
* * * * * . ~/.profile; your_command
Use screen
Above two solutions still fail in that they provide an environment connected to a running X session, with access to dbus etc. For example, on Ubuntu, nmcli (Network Manager) will work in above two approaches, but still fail in cron.
* * * * * /usr/bin/screen -dm
Add above line to cron, let it run once, turn it back off. Connect to your screen session (screen -r). If you are checking the screen session has been created (with ps) be aware that they are sometimes in capitals (e.g. ps | grep SCREEN)
Now even nmcli and similar will fail.
You can run:
env - your_command arguments
This will run your_command with empty environment.
Depending on the shell of the account
sudo su
env -i /bin/sh
or
sudo su
env -i /bin/bash --noprofile --norc
From http://matthew.mceachen.us/blog/howto-simulate-the-cron-environment-1018.html
Answering six years later: the environment mismatch problem is one of the problems solved by systemd "timers" as a cron replacement. Whether you run the systemd "service" from the CLI or via cron, it receives exactly the same environment, avoiding the environment mismatch problem.
The most common issue to cause cron jobs to fail when they pass manually is the restrictive default $PATH set by cron, which is this on Ubuntu 16.04:
"/usr/bin:/bin"
By contrast, the default $PATH set by systemd on Ubuntu 16.04 is:
"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
So there's already a better chance that a systemd timer is going to find a binary without further hassle.
The downside with systemd timers, is there's a slightly more time to set them up. You first create a "service" file to define what you want to run and a "timer" file to define the schedule to run it on and finally "enable" the timer to activate it.
Create a cron job that runs env and redirects stdout to a file.
Use the file alongside "env -" to create the same environment as a cron job.
Don't forget that since cron's parent is init, it runs programs without a controlling terminal. You can simulate that with a tool like this:
http://libslack.org/daemon/
By default, cron executes its jobs using whatever your system's idea of sh is. This could be the actual Bourne shell or dash, ash, ksh or bash (or another one) symlinked to sh (and as a result running in POSIX mode).
The best thing to do is make sure your scripts have what they need and to assume nothing is provided for them. Therefore, you should use full directory specifications and set environment variables such as $PATH yourself.
The accepted answer does give a way to run a script with the environment cron would use. As others pointed out, this is not the only needed criteria for debugging cron jobs.
Indeed, cron also uses a non-interactive terminal, without an attached input, etc.
If that helps, I have written a script that enables painlessly running a command/script as it would be run by cron. Invoke it with your command/script as first argument and you're good.
This script is also hosted (and possibly updated) on Github.
#!/bin/bash
# Run as if it was called from cron, that is to say:
# * with a modified environment
# * with a specific shell, which may or may not be bash
# * without an attached input terminal
# * in a non-interactive shell
function usage(){
echo "$0 - Run a script or a command as it would be in a cron job, then display its output"
echo "Usage:"
echo " $0 [command | script]"
}
if [ "$1" == "-h" -o "$1" == "--help" ]; then
usage
exit 0
fi
if [ $(whoami) != "root" ]; then
echo "Only root is supported at the moment"
exit 1
fi
# This file should contain the cron environment.
cron_env="/root/cron-env"
if [ ! -f "$cron_env" ]; then
echo "Unable to find $cron_env"
echo "To generate it, run \"/usr/bin/env > /root/cron-env\" as a cron job"
exit 0
fi
# It will be a nightmare to expand "$#" inside a shell -c argument.
# Let's rather generate a string where we manually expand-and-quote the arguments
env_string="/usr/bin/env -i "
for envi in $(cat "$cron_env"); do
env_string="${env_string} $envi "
done
cmd_string=""
for arg in "$#"; do
cmd_string="${cmd_string} \"${arg}\" "
done
# Which shell should we use?
the_shell=$(grep -E "^SHELL=" /root/cron-env | sed 's/SHELL=//')
echo "Running with $the_shell the following command: $cmd_string"
# Let's route the output in a file
# and do not provide any input (so that the command is executed without an attached terminal)
so=$(mktemp "/tmp/fakecron.out.XXXX")
se=$(mktemp "/tmp/fakecron.err.XXXX")
"$the_shell" -c "$env_string $cmd_string" >"$so" 2>"$se" < /dev/null
echo -e "Done. Here is \033[1mstdout\033[0m:"
cat "$so"
echo -e "Done. Here is \033[1mstderr\033[0m:"
cat "$se"
rm "$so" "$se"
Another simple way I've found (but may be error prone, I'm still testing) is to source your user's profile files before your command.
Editing a /etc/cron.d/ script:
* * * * * user1 comand-that-needs-env-vars
Would turn into:
* * * * * user1 source ~/.bash_profile; source ~/.bashrc; comand-that-needs-env-vars
Dirty, but it got the job done for me. Is there a way to simulate a login? Just a command you could run? bash --login didn't work. It sounds like that would be the better way to go though.
EDIT: This seems to be a solid solution: http://www.epicserve.com/blog/2012/feb/7/my-notes-cron-directory-etccrond-ubuntu-1110/
* * * * * root su --session-command="comand-that-needs-env-vars" user1 -l
Answer https://stackoverflow.com/a/2546509/5593430 shows how to obtain the cron environment and use it for your script. But be aware that the environment can differ depending on the crontab file you use. I created three different cron entries to save the environment via env > log. These are the results on an Amazon Linux 4.4.35-33.55.amzn1.x86_64.
1. Global /etc/crontab with root user
MAILTO=root
SHELL=/bin/bash
USER=root
PATH=/sbin:/bin:/usr/sbin:/usr/bin
PWD=/
LANG=en_US.UTF-8
SHLVL=1
HOME=/
LOGNAME=root
_=/bin/env
2. User crontab of root (crontab -e)
SHELL=/bin/sh
USER=root
PATH=/usr/bin:/bin
PWD=/root
LANG=en_US.UTF-8
SHLVL=1
HOME=/root
LOGNAME=root
_=/usr/bin/env
3. Script in /etc/cron.hourly/
MAILTO=root
SHELL=/bin/bash
USER=root
PATH=/sbin:/bin:/usr/sbin:/usr/bin
_=/bin/env
PWD=/
LANG=en_US.UTF-8
SHLVL=3
HOME=/
LOGNAME=root
Most importantly PATH, PWD and HOME differ. Make sure to set these in your cron scripts to rely on a stable environment.
In my case, cron was executing my script using sh, which fail to execute some bash syntax.
In my script I added the env variable SHELL:
#!/bin/bash
SHELL=/bin/bash
I don't believe that there is; the only way I know to test a cron job is to set it up to run a minute or two in the future and then wait.