Can we Schedule Shell Script to run in Cloud Scheduler?
In the Documentation i am seeing 3 targets HTTP,Pub/Sub,App Engine HTTP. Are there any updates in this regard?
Thanks in Adnavce.
It's not possible to trigger shell script as-is with Cloud Scheduler. At Google (and Google Cloud) all is API, and Cloud Scheduler can only call an API.
Therefore, I contributed on this opensource project months ago, and I discovered a way to plug a shell script on a webserver and then to deploy it on Cloud Run.
You can get inspiration from it, and then create a cloud scheduler, which call your Cloud Run with your wrapped shell script
Related
I am trying to automatize deployment of three modules: Cloud Function which is invoked via PubSub subscription from Cloud Scheduler. Currently I have a following script, which uses gcloud command:
gcloud beta pubsub topics create $SCHEDULE_NAME || echo "Topic $SCHEDULE_NAME already created."
gcloud beta functions deploy $SCHEDULE_NAME
--region $CLOUD_REGION
--memory 128MB
--runtime nodejs10
--entry-point $ENTRY_POINT
--trigger-topic $SCHEDULE_NAME
--vpc-connector cloud-function-connector
# gcloud scheduler jobs delete $JOB_NAME # does not work as it needs YES non-interactively
gcloud scheduler jobs create pubsub $SCHEDULE_NAME --message-body='RUN' --topic=$SCHEDULE_NAME --schedule='27 2 * * *' --time-zone='Europe/London' || true
This works, however I am not sure whether this is the most correct way to do this. For instance, there is no way to just update the job if it already exists. I was considering terraform, but I am not sure it is useful just for deploying these three small modules. I discovered also serverless tool, however it seems it can only deploy cloud function, but not schedulers and pubsub topics.
I think your approach is straightforward and fine.
Does Terraform provide the job update capability? If so, you'll likely find that it simply deletes and then (re)creates the job. I think this approach (delete-then-recreate) to updating jobs is fine too and seems to provide more control; you can check whether the schedule is about to fire before|after updating it.
Google provides Deployment Manager as a Google-Cloud-specific deployment tool. In my experience, it's primary benefit is that it's server-side but, ultimately, you're just automating the same APIs that you're using with gcloud.
If you want to learn a tool to manage your infrastructure as code, I'd recommend Terraform over Deployment Manager.
Update
The Scheduler API supports 'patching' jobs:
https://cloud.google.com/scheduler/docs/reference/rest/v1beta1/projects.locations.jobs/patch
And this mechanism is supported by gcloud:
gcloud alpha scheduler jobs update
Basically, I need to run a set of custom shell scripts on ec2 instances to provision some software. Is there any workflow manager like oozie or airflow with api access to schedule the same. I am asking for alternatives like oozie and airflow, as those are that of hadoop environment schedulers and my environment is not. I can ensure that there can be ssh access from the source machine that will run the workflow manager and the ec2 instance where want to install the software. Is there any such open source workflow schedulers?
I would recommend using Cadence Workflow for your use case. There are multiple provisioning solutions built on top of it. For example Banzai Cloud Pipeline Platform
See the presentation that goes over Cadence programming model.
I wrote talend etl job and packaged it into an executable Jar file.
i want schedule this jar file to run every week once.
I am using Amazon AWS. Can it achieve what I wanted? If yes, what are the steps I should proceed with? If not what are other alternatives?
If you are using Linux, I recommend you to use the traditional cron service served by unix system.
step
Edit /etc/crontab
Write schedule and shell script to kick Talend job
Please be careful timezone setting.
alternatives
Wikipedia - List of job scheduler software
I would personally recommend putting your jar in a docker container and putting it into an EC2 instance on AWS. Cron job will work, however, but what happens if you are not signed in when your task is scheduled to execute? If you put the jar in the cloud it will always execute.
As part of my CD pipeline in snap-ci.com, I want to start the instances in my AWS opsworks stack before deploying the application.
As starting hosts takes a certain amount of time (after the command has already returned), I need to poll for the instances to be running (using the describe-instances command in AWS CLI). This command does return a full JSON response of which one of the fields contains the status of the instance (e.g. "running").
I am new to shell scripting and AWS CLI and would appreciate some pointers. I am aware that I can also use the AWS SDK's to program it in java, but that would require to deploy that program to the snap-ci hosts first which sounds complex as well.
AWS CLI has support for wait commands, those will block and wait for the condition you specify, such as waiting for an instance to be ready.
The Advanced Usage of the AWS CLI talk from Re:Invent 2014 shows how to use waiters (18:55), queries, profiles and other tips for using CLI.
I have a script that I need to run once a day that requires a lot of memory. I would like to run it on a dedicated amazon box.
Is there some automated way to build a box, download all required software (like ruby) and then run my script. After the script is ran, I would like to shutdown the box.
The two options I can think of are:
I am thinking about hacking EMR to do this. (My script is a mapper against an empty directory)
Chef - This seemed like too much for one simple script.
You can accomplish setting up a new EC2 instance on startup using the official Ubuntu AMIs, the official Amazon Linux AMIs, and any other AMI that supports the concept of a user-data script.
Create a script (bash, Perl, Python,
whatever) that starts with #!
Pass this script as the user-data when running the EC2 instance.
The script will automatically be run as root on the first boot.
Here's the article where I introduced the concept of a user-data script:
Automate EC2 Instance Setup with user-data Scripts
http://alestic.com/2009/06/ec2-user-data-scripts
Your user-data script can install the required software, configure it, install your work script, and set up a cron job that runs the work script once a day.
ENHANCEMENT:
If the installation script don't take a long time to run (e.g., under an hour or few) then you don't even have to run a single dedicated instance 24 hours a day. You can instead use an approach that lets AWS start an instance for you on a regular schedule.
Here's an article I wrote that provides details on this approach with sample commands:
Running EC2 Instances on a Recurring Schedule with Auto Scaling
http://alestic.com/2011/11/ec2-schedule-instance
The general approach is to use Auto Scaling to start an instance with your user-data script on a regular schedule. Your job will terminate the instance when it has completed. They key is to suspend Auto Scaling's normal desire to re-start instances that terminate so that you don't pay for a running instance until the next time your job starts.