I have a standalone cloud source repository, (not cloned from Github).
I am using this to automate deploying of ETL pipelines . So I am folowing Google recommended guidelines, i.e committing the ETL pipeline as a .py file.
The cloud build trigger associated with the Cloud source repository will run the code as mentioned in the cloudbuild.yaml file and put the resultant .py file on the composer DAG bucket.
Composer will pick up this DAG and run it .
Now my question is, how do I orchestrate the CICD in dev and prod? I did not find any proper documentation to do this. So as of now I am following manual approach. If my code passes in dev, I am committing the same to the prod repo. Is there a way to do this in a better way?
Cloud Build Triggers allow you to conditionally execute a cloudbuily.yaml file on various ways. Have you tried setting up a trigger that fires only on changes to a dev branch?
Further, you can add substitutions to your trigger and use them in the cloudbuild.yaml file to, for example, name the generated artifacts based on some aspect of the input event.
See: https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values and https://cloud.google.com/build/docs/configuring-builds/use-bash-and-bindings-in-substitutions
Related
I am deploying a series of Cloud Build Triggers through Terraform, but I also want Terraform to trigger once every deployed Cloud Build so that it can do the initial deployment.
The Cloud Build Triggers are used to deploy Cloud Functions (and also Cloud Run and maybe Workflows). We could deploy the functions in the Terraform but we want to keep the command easy to modify so we don't want to duplicate it on both Terraform and the Cloud Build config.
It's important for the clarity and the evolutivity/maintainability of your pipeline to separate clearly the concern of each step.
You have a (set of) step to deploy the infrastructure of your project (here, your terraform)
You have a (set of) step that run process on your project (can be an Ansible script on VM, trigger Cloud Functions, Cloud Run, or a Cloud Build trigger).
I'm pretty sure that you can add this trigger in Terraform, but I strongly don't recommend you to do this.
Edit 1
I wasn't clear. You have to run your trigger by API after the terraform deployment, in your main pipeline. Then, the subsequent trigger will be done by Push to the Git repository.
I am using gitlab as repository and want to push my code on ec2 whenever any commit is done on gitlab. The gitlab CD/CI documentation states that I have to add a file .gitlab-ci.yml at the root directory of my repo. This is actually a problem for me because, I want project repo to have only code and not any configuration related info like build and deploy etc. Also when anybody clones the repo, they would have access to location where my code is pushed/deployed on ec2. Is there any work around for this problem ?
You'll need to use a gitlab-ci.yml filke to deploy your application. The file provides instructions and a pipeline "infrastructure" which, if properly configured, will build, test and automatically deploy your code.
If you are worried about leaking credentials, you should use the built-in instance variables to mask your important bits, like a "$SERVERNAME" or "$DB_PASSWORD" for instance.
Lastly, you can use the power of gitignore, in order to not publish all of your credentials or sensitive bits to your projects' servers or instances.
So we are currently just deploying master but are running into issues where we want to deploy the commit/build in which all our testing was ran on. This is normally a snapshop of master at 4:30pm. We run our build configuation for all tests automatically at 4:30pm (lets call this build config ALLTESTS), so we can control how this commit/build is marked in the ALLTESTS config.
We separate testing and deploy, so when a deploy is executed (either manually or automatically) it should only pick a branch/tag/commit/build that has been marked. Adding the tests to our deploy build config is not a viable solution.
Originally I had planned on using Git tags. A tag called deploy would be deleted and added to certain commits and then when the deployment is triggered that commit would be deployed.
The issue I ran into here is that there isnt an easy way to manually add git tags in a build step. Should I just write command-line build step that uses git commands remove the tag deploy from whatever commit has it and to add it to commit that is running?
Is there a better teamcity way to do this? I have successfully got teamcity tags to work via REST API but I am not sure if those fit the need either.
I suppose I could write powershell to parse the rest API to get the build id that was last successful in ALLTESTS and then feed that into the deploy somehow. How would I go about getting a build number and using that as the basis of deploy?
Should I just write command-line build step that uses git commands remove the tag deploy from whatever commit has it and to add it to commit that is running?
Quick answer is no. You can actually use something like this:
git tag -f deploy <commit-sha>
And have your tag updated to the given commit.
Cheers.
I have a Golang app running on Google Cloud App Engine that I can update manually with "gcloud app deploy" but I cannot figure out how to schedule automatic redeployments. I'm assuming I have to use cron.yaml, but then I'm confused about what url to use. Basically it's just a web app with one main index.html page with changing content, and I would like to schedule automatic redeployments... how do I have to go about that?
If you want to automatically re-deploy your app when the code changes, you need what's called CI/CD (Continuous integration/deployment). What a CI does is, for each new commit to your repository, check out the new code and run a test script. If all the tests pass (or if you don't have any tests at all), the CI server can then deploy your code to App Engine, all automatically.
One free (for open-source projects) CI provider is Travis CI. To configure it, you need to make an account with Travis, and a file called .travis.yml in the root of your repository. To set up App Engine deploys, you can follow this guide to set up a service account and add the encrypted file to your repo. It will run a gcloud app deploy from a container on their servers, whenever you push code to a certain branch (master by default) in your repo.
Another option, which avoids setting up CI at all, is to simply change your app to generate the dynamic parts of the page when it gets requested. Reading the documentation for html/template would point you in the right direction.
I am trying to use bamboo to manage my release procedure and just wondering if this is feasible:
The developer finished the integration test at local environment
The developer create a specific tag, e.g. "UAT_1.0.0" and pushed the tag to bitbucket
Bamboo sensed that there is an new tag "UAT_1.0.0" has been created on bitbucket and then start the building process; after that it will deploy the war file to the UAT server
Tester signed off the UAT and created a tag "REL_1.0.0" and pushed the tag to bitbucket
Bamboo sensed the new tag "REL_1.0.0" and start the building process. After build finished, it deploy the war file to the PROD server
It looks like the "Repository triggers the build when changes are committed" is to best way to implement the process. But I can't find out the way to move any further. Any idea?
Yes you can do this (Only if you can deploy custom plugins to it).
You need to build custom triggers as plugins for bamboo.
Get/Install the SDK
Create a plugin. See here.
If you have access to bamboo's source code I suggest you look into classes DependencyTriggerReason, InitialbuildTriggerReason, ScheduledTriggerReason. You need to create a class implementing TriggerReason. You should start with this tutorial if you are new to developing bamboo plugins.
Deploy it to bamboo.
A bit late but... I found a solution. You need to put a "negative" regular expression in the "Exclude changesets" section in the repository configuration, that include the word that you want.
The regular expression is like :
^(?!.test).$
like that: repository configuration
with this, bamboo will only build commits with this word in the message.
Now, if your commit is like "commit for test", bamboo will build it