I am trying to automatize deployment of three modules: Cloud Function which is invoked via PubSub subscription from Cloud Scheduler. Currently I have a following script, which uses gcloud command:
gcloud beta pubsub topics create $SCHEDULE_NAME || echo "Topic $SCHEDULE_NAME already created."
gcloud beta functions deploy $SCHEDULE_NAME
--region $CLOUD_REGION
--memory 128MB
--runtime nodejs10
--entry-point $ENTRY_POINT
--trigger-topic $SCHEDULE_NAME
--vpc-connector cloud-function-connector
# gcloud scheduler jobs delete $JOB_NAME # does not work as it needs YES non-interactively
gcloud scheduler jobs create pubsub $SCHEDULE_NAME --message-body='RUN' --topic=$SCHEDULE_NAME --schedule='27 2 * * *' --time-zone='Europe/London' || true
This works, however I am not sure whether this is the most correct way to do this. For instance, there is no way to just update the job if it already exists. I was considering terraform, but I am not sure it is useful just for deploying these three small modules. I discovered also serverless tool, however it seems it can only deploy cloud function, but not schedulers and pubsub topics.
I think your approach is straightforward and fine.
Does Terraform provide the job update capability? If so, you'll likely find that it simply deletes and then (re)creates the job. I think this approach (delete-then-recreate) to updating jobs is fine too and seems to provide more control; you can check whether the schedule is about to fire before|after updating it.
Google provides Deployment Manager as a Google-Cloud-specific deployment tool. In my experience, it's primary benefit is that it's server-side but, ultimately, you're just automating the same APIs that you're using with gcloud.
If you want to learn a tool to manage your infrastructure as code, I'd recommend Terraform over Deployment Manager.
Update
The Scheduler API supports 'patching' jobs:
https://cloud.google.com/scheduler/docs/reference/rest/v1beta1/projects.locations.jobs/patch
And this mechanism is supported by gcloud:
gcloud alpha scheduler jobs update
Related
We have multiple GKE clusters in different GCP projects.
For each cluster in each project, I essentially want to run the following via Go:
kubectl get deployment [deployment_name] -o json | jq '.metadata.labels'
I already have 2 functions to get the GCP projects and the cluster names per project, the last of which uses:
"cloud.google.com/go/container/apiv1"
"google.golang.org/genproto/googleapis/container/v1"
Is it possible to use these APIs to get the deployment information for the GKE clusters?
Or would I need to use kubernetes/client-go?
I couldn't seem to find a function that suited my needs under: https://cloud.google.com/go/docs/reference/cloud.google.com/go/container/latest/apiv1
And the kubernetes Go client seems to rely on a .kubeconfig file, whereas I'd just like to use $GOOGLE_APPLICATION_CREDENTIALS, if possible:
https://github.com/kubernetes/client-go/blob/master/examples/out-of-cluster-client-configuration/main.go
Any help with this would be most appreciated.
I've bitbucket repository, bitbucket pipeline there and EC2 instance. EC2 have access to the repository ( can perform pull and docker build/run)
So it seems I only need to upload to EC2 some bash scripts and call it from bitbucket pipeline. How can I call it? Usually ssh connection is used to perform scripts on EC2, is it applicable from bitbucket pipeline? Is it a good solution?
two ways to solve this problem, I will leave it up to you.
I see you are using AWS, and AWS has a nice service called CodeDeploy. you can use that and create a few deployment scripts and then integrate it with your pipeline. Problem with it is that it is an agent that needs to be installed. so it will consume some resource not much but if u are looking at an agentless design then this solution wont work. you can check the example in the following answer https://stackoverflow.com/a/68933031/8248700
You can use something like Python Fabric (its a small gun) or Ansible (its a big canon) to achieve this. it is an agentless design works purely on SSH.
I'm using both the approaches for different scenarios. For AWS I use CodeDeploy and for any other cloud vendor I use Python Fabric. (We can use CodeDeploy on other than AWS but then it comes under on-premise pricing for which it charges for per deployment)
I hope this brings some clarity.
Can we Schedule Shell Script to run in Cloud Scheduler?
In the Documentation i am seeing 3 targets HTTP,Pub/Sub,App Engine HTTP. Are there any updates in this regard?
Thanks in Adnavce.
It's not possible to trigger shell script as-is with Cloud Scheduler. At Google (and Google Cloud) all is API, and Cloud Scheduler can only call an API.
Therefore, I contributed on this opensource project months ago, and I discovered a way to plug a shell script on a webserver and then to deploy it on Cloud Run.
You can get inspiration from it, and then create a cloud scheduler, which call your Cloud Run with your wrapped shell script
It is possible to run local activities that don't require a connection to the cadence server. Is there a proper way to run workflows locally, too, in case of a cadence outage?
I'm using the Go client.
The service connection is required to make any progress in the workflow execution including scheduling activities.
To run workflows locally you can use local version of the Cadence service. Such version can be easily installed through docker compose.
If you need high availability setup you can use multi-cluster Cadence. So a single cluster outage is not going to cause workflow execution outager.
Basically, I need to run a set of custom shell scripts on ec2 instances to provision some software. Is there any workflow manager like oozie or airflow with api access to schedule the same. I am asking for alternatives like oozie and airflow, as those are that of hadoop environment schedulers and my environment is not. I can ensure that there can be ssh access from the source machine that will run the workflow manager and the ec2 instance where want to install the software. Is there any such open source workflow schedulers?
I would recommend using Cadence Workflow for your use case. There are multiple provisioning solutions built on top of it. For example Banzai Cloud Pipeline Platform
See the presentation that goes over Cadence programming model.