I pass the file path, containing variables to be sourced, as an argument to my Bash script.
The file is created on Windows, in case that makes any difference.
The following check is performed:
CONFIG_FILE=$1
if [[ -f ${CONFIG_FILE} ]]; then
echo "Is a file"
. ${CONFIG_FILE}
else
echo "Not a file"
fi
When I run the script manually, from the command line, the check is fine and the variables get sourced.
However, when I set up a Cron job using
*/1 * * * * /full/path/to/script.sh /full/path/to/configfile
I get "Not a file" printed out.
I attempted every single setup I found online to solve this:
setting up environment variables both in crontab and script itself (PATH & SHELL)
sourcing the profile (both . /etc/profile and . /home/user/.bash_profile) both in crontab (before executing the script) and in the script itself.
trying to run crontab with the -u user parameter, but don't have permissions for this (and it doesn't make sense, as I am already logged in as the user who should setup the crontab)
I am setting up the crontab with the proper user under whom the script should be run. The user has access rights to the location of the files (as can be observed through running the script from the command line).
Looking for further advice on what can be attempted next.
What you're doing here is (I think) making sure that there is a separate argument behind your /path/config-file. Your original problem seems to be that on Unix your config file was stated as /path/config-file\r (note the trailing \r). You are doing it by adding an argument -q\r so that the config file itself is "clean" of the carriage return. You could add blabla\r for that matter instead of -q\r. Your script never interprets that extra argument; but if you put it on the cron line then your config file argument is "protected", because there's stuff following it, that's all.
What you also could do, is make sure that your cron defintion is Unix-styled (\n terminted lines) instead of DOS styled (\r\n terminated lines). There's probably a utility dos2unix on your Unix box to accomplish that.
Or you could remove the crontab on Unix using crontab -r and then re-create the crontab using crontab -e. Just don't upload files that were created on MS-DOS (or derived).
Found another attempt and it worked.
I added the -q flag in the cronjob line.
*/1 * * * * /path/script.sh /path/config-file -q
Source: Cron Job error "Could not open input file"
Can someone please explain to me what does it do?
I am not so literate in bash.
Related
I have my php script file in /var/www/html/dbsync/index.php. When cd /var/www/html/dbsync/ and run php index.php it works perfectly.
I want to call PHP file through sh file, the location of SH file is as below
/var/www/html/dbsync/dbsync.sh
This is the content of the dbsync.sh file is:
/usr/bin/php /var/www/html/dbsync/index.php >> /var/www/html/dbsync/myscript.log 2>&1 -q -f
When I cd /var/www/html/dbsync/ and run ./dbsync.sh it works perfectly as well.
Now if I set up crontab as below:
1 * * * * /var/www/html/dbsync/dbsync.sh /var/www/html/dbsync
However, this crontab is not working as expected.
What can be wrong?
As seen in comments, the problem is that you are not defining what program should be used to execute the script. Take into account that a cronjob is executed in a tiny environment; there, not much can be assumed. This is why we define full paths, etc.
So you need to say something like:
1 * * * * /bin/sh /var/www/html/dbsync/dbsync.sh /var/www/html/dbsync
# ^^^^^^^
/bin/sh being the binary you want to use to execute the script.
Otherwise, you can set execution permissions to the script and add a shell-script header telling it what interpreter to use:
#!/bin/sh
If you do this, adding the path of the binary is not necessary.
From Troubleshooting common issues with cron jobs:
Using relative paths. If your cron job is executing a script of some
kind, you must be sure to use only absolute paths inside that script.
For example, if your script is located at /path/to/script.phpand
you're trying to open a file called file.php in the same directory,
you cannot use a relative path such as fopen(file.php). The file must
be called from its absolute path, like this: fopen(/path/to/file.php).
This is because cron jobs do not necessarily run from the directory in
which the script is located, so all paths must be called specifically.
Also, I understand you want to run this every minute. If so, 1 * * * * won't do. Intead, it will run at every 1st minute past every hour. So if you want to run it every minute, say * * * * *.
It is important to understand "login shell" and "interactive shell" what they means.
login shell: is briefly when you sign in with ssh session and get a terminal window where you can enter shell commands. After login the system executes some files(.bashrc) and sets some environment variables such as the PATH variable for you.
interactive shell :After login on a system, you can startup manually shell terminal(s). The system executes some profile file assigned to your account (.bash_profile, .bash_login,.profile). This files also sets some environment variables and initialize PATH variable for your manually opened shell session.
By OS started shell scripts and cron jobs does not fit in above mentioned way for starting a shell. Therefore no any system scripts(.bashrc) or user profiles are executed. This means our PATH variable is not initialized. Shell commands could not found because PATH variable does not point to right places.
This explains why your script runs successfully if you start it manually but fails when you start it via crontab.
Solution-1:
Use absolute path of every shell command instead of only the command name used in your script file(s).
instead of "awk" use "/usr/bin/awk"
instead of "sed" use "/bin/sed"
Solution-2: Initialize environment variables and especially the PATH variable before executing shell scripts!
method 1, add this header in your dbsync.sh:
#!/bin/bash -l
method 2, add bash -l in your cron file:
1 * * * * bash -l /var/www/html/dbsync/dbsync.sh /var/www/html/dbsync
I’m having some serious issues with trying to get the proper format for my bash script to be able to run successfully in crontab. The bash script runs successfully when manually prompted from the command line.
Here is the bash script in question (the actual parameters themselves [$1 & $2] have been manually placed in the script):
#!/bin/bash
# Usage: ./s3DeleteByDateVirginia "bucketname" "file type"
past=$(date +"%F" -d "60 days ago")
aws s3api list-objects --bucket $1 --query 'Contents[?LastModified<=`'$past'`][].{Key:Key}' | grep $2 | while read -r line
do
fileName=`echo $line`
aws s3api delete-object --bucket $1 --key "$fileName"
done;
The script is in this bash file: /home/ubuntu/s3DeleteByDateVirginiaSoco1
To set up the script I use: sudo crontab –e
Now I see people online saying you need to give it the proper path which doesn’t make any sense to me especially when it comes to putting it in the right location because I’m seeing a number of various modifications of this online but it consists of this format: SHELL=/bin/sh sPATH=/bin:/sbin:/usr/bin:/usr/sbin but I don't know where to put it.
According to the syslog the cron functionalities that parts working but the script itself doesn’t execute:
In addition to this the script has all of the proper permissions to run.
All in all, I’m more confused that when I started and I’m not seeing that much documentation on how crontab works.
Crontab in question:
Additional Edits based on user's suggestions:
Here's my polished script:
Here's the crontab line:
# m h dom mon dow command
PATH=/usr/local/bin:/usr/bin:/bin:/root/.local/bin/aws
33 20 * * * /home/ubuntu/s3DeleteByDateSoco1
Updated syslog:
Ok, I see several problems here. First, you need to put this in the crontab file for the user you want the script to run as. If you want to run it under your user account, do not use just crontab -e instead of sudo crontab -e (with sudo, it edits the root user's crontab file).
Second, you need to use the correct path & name for the script; it looks like it's /home/ubuntu/s3DeleteByDateVirginiaSoco1, so that's what should be in the crontab entry. Don't add ".sh" if it's not actually part of the filename. It also looks like you tried adding "root" in front of the path; don't do that either, since crontab will try to execute "root" as a command, and it'll fail. bash -c doesn't hurt, but it doesn't help at all either, so don't use it.
Third, the PATH needs to be set appropriately for the executables you use in the script. By default, cron jobs execute with a PATH of just "/usr/bin:/bin", so when you use a command like aws, it'll look for it as /usr/bin/aws, not find it, look for it as /usr/aws, not find it, and give the error "aws: command not found" that you see in the last log entry. First, you need to find out where aws (and any other programs your script depends on) are; you can use which aws in your regular shell to find this out. Suppose it's /usr/local/bin/aws. Then you can either:
Add a line like PATH=/usr/local/bin:/usr/bin:/bin (with maybe any other directories you think are appropriate) to the crontab file, before the line that says to run your script.
Add a line like PATH=/usr/local/bin:/usr/bin:/bin (with maybe any other directories you think are appropriate) to the your script file, before the lines that use aws.
In your script, use an explicit path every time you want to run aws (something like /usr/local/bin/aws s3api list-objects ...)
You can use any (or all) of the above, but you must use at least one or it won't be able to find the aws command (or anything else that isn't in the set of core commands that come with the OS).
Fourth, I don't see where $1 and $2 are supplied. You say they've been manually placed in the script, but I don't know what you mean by that. Since the script expects them as parameters, you need to specify them in the crontab file (i.e. the command in crontab should be something like /home/ubuntu/s3DeleteByDateVirginiaSoco1 bucketname pattern).
Fifth, the script itself doesn't follow good quoting conventions. In general, all variable references should be in double-quotes. For example, use grep "$2" instead of grep $2. Without the double-quotes, variables that contain spaces or certain shell metacharacters can cause weird parsing problems.
Finally, why do you do fileName=echo $line (with backquotes I can't replicate here)? This mostly just copies the value of $line into the variable fileName, but can have those weird parsing problems I mentioned in the last point. If you want to copy a variable reliably, just use fileName="$line" (or fileName=$line -- this is one of the few cases where it's safe to leave the double-quotes off).
BTW, shellcheck.net is good at spotting common problems like bad quoting; I recommend running your scripts through it to see what it finds.
The following script is running git pull and then write the status to file.
#!/bin/bash
git pull
git log -7 > /var/www/domain.com/v.txt
When I run the file manually, the file v.txt is created and the content inside as expected.
bash /var/www/domain.io/update.sh
When I run it via cron the file is created but the content is empty
* * * * * bash /var/www/domain.io/update.sh
Whats wrong?
If I remember correctly the runtime environment of the interactive shell vs cron are different. Specifically the setting of environment variables like PATH can be different. This can effect how scripts run via cron.
As a test you could try the following. Create a small bash shell script that dumps your environment to a file and run this script from both the command line and cron and look for differences.
Eg. file named: r1.sh
#! /bin/bash
env
From the command line:
$ r1.sh > cmd-line.out
From cron
* * * * * /home/your-user-name/r1.sh > /home/your-user-name/cron.out
NOTE: use full path names for shell script and the output file.
When I did this test on my system, there where a number of difference between the two, for example the PATH variable was much shorter.
If this is the case on your system maybe "git" is not available in the cron env.
Anyway, just an idea to try.
I hope this helps.
I have a script to invalidate Amazon CloudFront in a .sh file and it works fine when I run it with bash (bash /../filename.sh). I need to invalidate my distribution every thursday, so I wrote a cron job, but it is giving the error "aws: command not found".
This is my cron job
45 10 * * 2 /usr/bin/bash /var/www/cms/file.sh
What I am missing? Why the cron job is failing when bash could run the script?
This is because your system depends on an environment variable for places where to search for executable files and that variable is not set in the cron sessions. This variable is named $PATH. You can see it's contents in your current session by just typing echo $PATH in your terminal.
When a cron session is started for a job to be executed, this variable is not set.
To solve this issue, there are a few ways:
Method 1:
Use full path names in your shell script and do not depend on the PATH variable for finding executables.
Method 2:
Add the following at the beginning in your script (adjust if needed):
PATH=/sbin:/bin:/usr/sbin:/usr/bin
Method 3:
(This method is generally discouraged)
Add the following line in your crontab (adjust if needed):
PATH=/sbin:/bin:/usr/sbin:/usr/bin
You can do echo $PATH in your terminal and copy the output in the above variable in the crontab.
Thank you Sakis your answer worked perfectly for me.
I was baffled why my scripts would run just fine from the command line but error out when run from cron. The reason was i was trying to use an executable from within bash that was not a recognized command due to lack of reference to a path. Now it's clear.
Did exactly this - no root or other sudo required.
From your own shell, echo $PATH. Copy this line and enter in your own cron (affects all jobs in that cron) or at the top of your script.
PATH=/home/myuser/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
And voila !
I am trying to run this comamand from bash script through crontab, it just gets SystemOut.log, though I expect to get SystemOut_* as well.
/app/hdup/get_logs SystemOut*
But when i tried to run this above command from terminal, it worked properly and got both SystemOut.log and SystemOut_*
Any idea what could have gone wrong?
As pointed out in the comments already: The cron is using a different shell than you do. You can set the shell to be used by cron via a variable at the top of your crontab to have the same result as in your terminal:
SHELL=/bin/bash
Details can be found at https://serverfault.com/a/678414
cron is executed by /bin/sh which is the key point why many special string like * , {},etc not work.
put SHELL=/bin/bash on the first line of you cron file after "crontab -e"