I am trying execute some gcloud commands in bash script from crontab. The script execute sucessfully from command shell but not from the cron job.
I have tried with:
Settng the full path to gcloud like:
/etc/bash_completion.d/gcloud
/home/Arturo/.config/gcloud
/usr/bin/gcloud
/usr/lib/google-cloud-sdk/bin/gcloud
Setting in the begin the script:
/bin/bash -l
Setting in the crontab:
51 21 30 5 6 CLOUDSDK_PYTHON=/usr/bin/python2.7;
/home/myuser/folder1/myscript.sh param1 param2 param3 -f >>
/home/myuser/folder1/mylog.txt`
Setting inside the script:
export CLOUDSDK_PYTHON=/usr/bin/python2.7
Setting inside the script:
sudo ln -s /home/myuser/google-cloud-sdk/bin/gcloud /usr/bin/gcloud
Version Ubuntu 18.04.3 LTS
command to execute: gcloud config set project myproject
but nothing is working, maybe I am doing something wrongly. I hope you can help me.
You need to set your user in your crontab, for it to run the gcloud command. As well explained in this other post here, you need to modify your crontab to fetch the data in your Cloud SDK, for the execution to occur properly - it doesn't seem that you have made this configuration.
Another option that I would recommend you to try out, it's using a Cloud Scheduler to run your gcloud commands. This way, you can use gcloud for your cron jobs in a more integrated and easy way. You can verify more information about this option here: Creating and configuring cron jobs
Let me know if the information helped you!
I found my error, the problem here was only in the command: "gcloud dns record-sets transaction start", the others command was executing sucesfully but only no logging nothing, by that I though that was not executng the other commands. This Command create a temp file ex. transaction.yaml and that file could not be created in the default path for gcloud(snap/bin), but the log simply dont write any thing!. I had to specify the path and name for that file with the flag --transaction-file=mytransaction.yaml. Thanks for your supprot and ideas
I have run into the same issue before. I fixed it by forcing the profile to load in my script.sh,loading the gcloud environment variables with it. Example below:
#!/bin/bash
source /etc/profile
gcloud config set project myprojectecho
echo "Project set to myprojectecho."
I hope this can help others in the future with similar issues, as this also helped me when trying to set GKE nodes from 0-4 on a schedule.
Adding the below line to the shell script fixed my issue
#Execute user profile
source /root/.bash_profile
Related
Good morning,
I am working on a link between my Laravel file server and a sinology backup. The command I am using uses a sudo command to create and then disconnect the link. I want to know if I would be able to run this command from the scheduler.
Thanks
You can use for example (to run at midnight, every day):
0 0 * * * /path/to/your/command
This is a record you can add in cron of the user you use to run the command. Be aware cron have different environment so you should set all the variables you need.
You may need to create special shell script to include there your environment variables:
. ~/.bash_profile
/path/to/your/command
I am running a bash script with sudo and have tried the below but am getting the error below using aws cp. I think the problem is that the script is looking for the config in /root which does not exist. However doesn't the -E preserve the original location? Is there an option that can be used with aws cp to pass the location of the config. Thank you :).
sudo -E bash /path/to/.sh
- inside of this script is `aws cp`
Error
The config profile (name) could not be found
I have also tried `export` the name profile and `source` the path to the `config`
You can use the original user like :
sudo -u $SUDO_USER aws cp ...
You could also run the script using source instead of bash -- using source will cause the script to run in the same shell as your open terminal window, which will keep the same env together (such as user) - though honestly, #Philippe answer is the better, more correct one.
I have my crontab set up as follows (this is inside a docker container).
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/bash
5 * * * * bash /usr/local/bin/process-logs > /proc/1/fd/1 2>/proc/1/fd/
The /usr/local/bin/process-logs is designed to expose some MongoDB logs using mtools to a simple web server.
The problematic part of the script is fairly simple. raw_name is archive_name without the file extension.
aws s3 cp "s3://${s3_bucket}/${file_name}" "${archive_name}"
gunzip "${archive_name}"
mlogvis --no-browser "${raw_name}"
If I manually run the command as specified in the crontab config above
bash /usr/local/bin/process-logs > /proc/1/fd/1 2>/proc/1/fd/2
It all works as expected (this is the expected output from mlogvis)
...
copying /usr/local/lib/python3.5/dist-packages/mtools/data/index.html to /some/path/mongod.log-20190313-1552456862.html
...
When the script gets triggered via crontab it throws the following error
usage: mlogvis [-h] [--version] [--no-progressbar] [--no-browser] [--out OUT]
[--line-max LINE_MAX]
mlogvis: error: unrecognized arguments: mongod.log-20190313-1552460462
The mlogvis command that caused the following error (actual values not parameters)
mlogvis --no-browser "mongod.log-20190313-1552460462"
Again if I run this command myself it all works as expected.
mlogvis: http://blog.rueckstiess.com/mtools/mlogvis.html
I don't believe this to be an issue with the file not having correct permissions or not existing as mlogvis produces a different error in these conditions. I've also tested with removing '-' from the file name thinking it might be trying to parse these as arguments but it made no difference.
I know cron execution doesn't have the same execution environment as the user I tested the script as. I've set the PATH to be the same as the user and when the container starts up I execute env >> /etc/environment so all the environment vars and properly set.
Does anyone know of a way to debug this or has anyone encountered similar? All other components of the script are functioning except mlogvis which is core to the purpose of this job.
Summary of what I've tried as a fix:
Set environment and PATH for cron execution to be the same as the user I tested the script as
Replace - in file name(s) to see if it was parsing the parts as arguments
hardcode a filename with full permissions to see if it was permissions related
Manually run the script -> this works
Manually run the mlogvis command in isolation -> this works
try to load /home/user/.bash_profile before executing script and try again. I suspect that you have missing PATH or other environment variable which is not set.
source /home/user/.bash_profile
Please post your complete script, because while executing via crontab,
you have to be sure your raw_name variable was properly created. As
it seems to depend on archive_name, posting some more context can
help us to help you.
In any case, if you are using bash, you can try something like :
aws s3 cp "s3://${s3_bucket}/${file_name}" "${archive_name}"
gunzip "${archive_name}"
# here you have to be sure that archive_name is correct
raw_name_2=${archive_name%%.*}
mlogvis --no-browser "${raw_name_2}"
It is not going to solve your issue, but probably will take you closer to the right path.
I have a build server. I'm using the Azure Build Agent script. It's a shell script that will run continuously while the server is up. Problem is that I cannot seem to get it to run on startup. I've tried /etc/init.d and /etc/rc.local and the agent is not being run. Nothing concerning the build agent in the boot logs.
For /etc/init.d I created the script agent.sh which contains:
#!/bin/bash
sh ~/agent/run.sh
Gave it the proper permissions chmod 755 agent.shand moved it to /etc/init.d.
and for /etc/rc.local, I just appended the following
sh ~/agent/run.sh &
before exit 0.
What am I doing wrong?
EDIT: added examples.
EDIT 2: Just noticed that the init.d README says that shell scripts need to start with #!/bin/sh and not #!/bin/bash. Also used absolute path, but no change.
FINAL EDIT: As #ewrammer suggested, I used cron and it worked. crontab -e and then #reboot /home/user/agent/run.sh.
It is hard to see what is wrong if you are not posting what you have done, but why not add it as a cron job with #reboot as pattern? Then cron will run the script every time the computer starts.
Just in case, using a supervisor could be a good idea, In Ubuntu 14 you don't have systemd but you can choose from others https://en.wikipedia.org/wiki/Process_supervision.
If using immortal, after installing it, you just need to create a run.yml file in /etc/immortal with something like:
cmd: /path/to/command
log:
file: /var/log/command.log
This will start your script/command on every start, besides ensuring your script/app is always up and running.
I have a ruby script that connects to an Amazon S3 bucket and downloads the latest production backup. I have tested the script (which is very simple) and it works fine.
However, when I schedule this script to be run as a cron job it seems to fail when it loads the Amazon (aws-s3) gem.
The first few lines of my script looks like this:
#!/usr/bin/env ruby
require 'aws/s3'
As I said, when I run this script manually, it works fine. When I run it via a scheduled cron job, it fails when it tries to load the gem:
`require': no such file to load -- aws/s3 (LoadError)
The crontab for this script looks like this:
0 3 * * * ~/Downloader/download.rb > ~/Downloader/output.log 2>&1
I originally thought it might be because cron is running as a different user, but when I do a 'whoami' at the start of my ruby script it tells me it's running as the same user I always use.
I have also done a bundle init and added the gem to my gemfile, but this doesn't seem to have any affect.
Why does cron fail to load the gem? I am running Ubuntu.
As mentioned here https://coderwall.com/p/vhv8aw you can simply try
rvm cron setup # let RMV do your cron settings
Make sure that you make copy of your crontab before running this command
If you're running it manually and it works you're probably in a different shell environment than cron is executing in. Since you mention you're on Ubuntu, the cron jobs probably execute under /bin/sh, and you're manually running them under /bin/bash if you haven't changed anything.
You can debug your environment problems or you can change the shell that your job runs under.
To debug, There are several ways to figure out what shell your cron jobs are using. It can be defined in
/etc/crontab
or you can make a cron job to dump your shell and environment information, as has been mentioned in this SO answer: How to simulate the environment cron executes a script with?
To switch to that shell and see the actual errors causing your job to fail, do
sudo su
env -i <path to shell> (e.g. /bin/sh)
Then running your script you should see what the errors are and be able to fix them (rubygems?).
Option 2 is to switch shells. You can always try something like:
0 3 * * * /bin/bash -c '~/Downloader/download.rb > ~/Downloader/output.log 2>&1'
To force your job into bash. That might also clear things up.
You may also explicitly set your Gem path:
GEM_HOME="/usr/local/rvm/gems/ruby-1.9.2-p290#my-special-gemset"
in a non cron environment execute echo $PATH, copy the path and paste it into your crontab, before your command:
echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin
and inside crontab:
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin
0 3 * * * ~/Downloader/download.rb > ~/Downloader/output.log 2>&1
Add this at the beginning of your cron
PATH="/home/user/.rvm/gems/ruby-2.1.4/bin:/home/user/.rvm/gems/ruby-2.1.4#global/bin:/home/user/.rvm/rubies/ruby-2.1.4/bin:/home/user/.rvm/gems/ruby-2.1.4/bin:/home/user/.rvm/gems/ruby-2.1.4#global/bin:/home/user/.rvm/rubies/ruby-2.1.4/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/home/user/.rvm/bin:/usr/local/sbin:/usr/sbin:/home/user/.rvm/bin:/home/user/.local/bin:/home/user/bin"
GEM_HOME='/home/user/.rvm/gems/ruby-2.1.4'
GEM_PATH='/home/user/.rvm/gems/ruby-2.1.4:/home/user/.rvm/gems/ruby-2.1.4#global'
MY_RUBY_HOME='/home/user/.rvm/rubies/ruby-2.1.4'
IRBRC='/home/user/.rvm/rubies/ruby-2.1.4/.irbrc'
RUBY_VERSION='ruby-2.1.4'
I've tried all the solution above, none of them worked until I tried;
0 12 * * * /bin/bash -l -c 'ruby /Users/simon/Desktop/script.rb'