I'm trying to get my head around if replacing our current CI/CD tool with Github Actions is a pro or con.
My team is currently using GitHub and TeamCity for CI/CD. I find this limiting as it's a separate team that configures the different stages. We use Airflow to schedule our DBT production, so we don't use DBT Cloud.
I'm wondering if there's any benefit to replacing TeamCity with Github Actions as we can write the configuration of the stages (compile, run, test, lint) ourselves. I'm interested in implementing Slim CI , which I don't believe is possible with TeamCity.
Would anyone be able to with the pros and cons of using GitHub Actions alongside Airflow? Is it possible? I'm struggling to find any documentation on using both Actions whilst running DBT production on Airflow.
Right now, my source of information comes from this: https://towardsdatascience.com/how-to-deploy-dbt-to-production-using-github-action-778bf6a1dff6
It mentions "Well, it depends. If you don’t have Airflow running in productions already, you will probably not need it now. There are more simple/elegant solutions than this (dbt Cloud, GitHub Actions, GitLab CI). Also, this approach shares many disadvantages with using a compute instance, such as waste of resources and no easy way for CI/CD."
My main goal is to get rid of TeamCity and implement Slim CI with Github Actions.
Thank you in advance!
Related
Recently research different build tools , GCP CloudBuild is one of the selection
https://github.com/GoogleCloudPlatform/cloud-builders
one of require is work loop an array list and write the function only once and run in parallel
however i did not find Cloudbuild mention any about matrix build
Function which provided by Jenkins Plugin https://www.jenkins.io/blog/2019/11/22/welcome-to-the-matrix/
or Github Action https://docs.github.com/en/free-pro-team#latest/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
Cloud Run is not designed to work with Jenkins out of the box and the links you included do not mention how to do this.
As indicated [1] the best product to integrate Jenkins inside Google Cloud is to use Google Kubernetes Engine (GKE).
[1] https://cloud.google.com/jenkins
I have a single repository that hosts my lambda functions on github. I would like to be able to deploy the new versions whenever new logic is pushed to master.
I did a lot of reasearch and found a few different approaches, but nothing really clear. Would like to know what others feel would be the best way to go about this, and maybe some detail (if possible) into how that pipeline is setup.
Thanks
Welcome to StackOverflow. You can improve your question by reading this page.
You can setup a CI/CD pipeline using CircleCI with its GitHub integration (which is an online Service, so you don't need to maintain anything, like a Jenkins server, for example)
Upon every commit to your repository, a CircleCI build will be triggered. Once the build process is over, you can declare sls deploy, sam deploy, use Terraform or even create a script to upload the .zip file from your GitHub repo to an S3 Bucket and then, within your script, invoke the create-function command. There's an example how to deploy Serverless applications using CircleCI along with the Serverless Framework here
Other options include TravisCI, AWS Code Deploy or even maintain your own CI/CD Server. The same logic applies to all of these tools though: commit -> build -> deploy (using one of the tools you've chosen).
EDIT: After #Matt's answer, it clicked that the OP never mentioned the Serverless Framework (I, somehow, thought he was already using it, so I pointed the OP to tutorials using the Serverless Framework already). I then decided to update my answer with a few other options for serverless deployment
I know that this isn't exactly what you asked for but I use Serverless Framework (https://serverless.com) for deployment and I love it. I don't do my deployments when I push to my repo. Instead I push to my repo after I've deployed. I like this flow because a deployment can fail due to so many things and pushing to GitHub is much less likely to fail. I this way, I prevent pushing code that failed to deploy to my master branch.
I don't know if you're familiar with the framework but it is super simple. The website describes the simple steps to creating and deploy a function like this.
1 # Step 1. Install serverless globally
2 $ npm install serverless -g
3
4 # Step 2. Create a serverless function
5 $ serverless create --template hello-world
6
7 # Step 3. deploy to cloud provider
8 $ serverless deploy
9
10 # Your function is deployed!
11 $ http://xyz.amazonaws.com/hello-world
There are also a number of plugins you can use to integrate easily with custom domains on APIGateway, prune older versions of lambda functions that might be filling up your limits, etc...
Overall, I've found it to be the easiest way to manage and deploy my lambdas. Hope it helps!
Given that you're using AWS Lambda, you may want to consider CodePipeline to automate your release process. [SAM(https://docs.aws.amazon.com/lambda/latest/dg/serverless_app.html) may also be interesting.
I too had the same problem. I wanted to manage 12 lambdas with 1 git repository. I solved it by introducing travis-ci. travis-ci saved the time and really useful in many ways. We can check the logs whenever we want and you can share the logs to anyone by sharing the URL. The sample documentation of all steps can be found here. You can go through it. 👍
I have a bunch of marathon docker tasks that are run on our test deployment machines.
There is a Jenkins CI job, that triggers the deployment a whole bunch of docker containers that are run on marathon-mesos cluster. (3 mesos-slaves, 1 master and 1 marathon.)
There is another downstream jenkins job (a automated test suite) that is triggered after above job. Presently, we wait for sufficient time, so that deployment gets completed then only we proceed with this automation testsuite. I want to change this behavior. I know marathon exposes rest APIs using which I can determine if I am good to go - after all the containers are deployed and all health checks are passing - for running the automation test suite.
Question is: Is there any library already out there for marathon, that I can reuse to accomplish above task ? I do not want to reinvent the wheel.
When I posted this question, I had java library actually in mind, but forgot to mention that. I find #michael 's libraries are also very good. But this is what I settled upon. Marathon-client. I think I saw this, while browsing through mesosphere repositories but somehow missed it.
This is the library: marathon-client
I've successfully been using the two following libs:
Go: gambol99/go-marathon
Python: thefactory/marathon-python
I have a private bitbucket php project repository, codeship account and a vidahost cloud hosting. How to implement continuous integration and deployment? What is the workflow to achieve this? I did googling but could not find complete tutorial.
Ahoy Dipendra,
Marko from the Codeship crew here. Could you send me an in app message with a link to your project, so I can take a look at your current setup and help get you started?
As for deploying to Vidahost, you have multiple options. You could use a tool like Capistrano (or Mina, or something similar in your preferred programming language) or you could use SCP or FTP directly to copy your files. For the latter option, please see our documentation at https://codeship.com/documentation/continuous-deployment/deployment-with-ftp-sftp-scp/. (You'll find documentation on Capistrano and a bunch of other deployment options via this link as well.)
I'll update this answer with more specific details, once I know more about your setup!
Cheers,
Marko
We are running a Jenkins instance with various test jobs as part of the build chain.
I would like to integrate the build, test trend and general CI status information into graphs/widgets using the Shopify Dashing dashboard http://shopify.github.io/dashing/.
In other words, to have a standalone dashboard that indicates key info from the Jenkins instance.
Does anyone have experience with doing this kind of thing? Can you point me in the direction of any good resources/tutorials etc?
Many thanks
Jenkins' url structure is pretty rigid. For example, the build time trend graph for a project would be available at:
http://<jenkinshost>:<port>/job/<jobname>/buildTimeGraph/png
etc. You should be able to just statically link to these using tags in your HTML / ERB / templating language of your choice.
Other information is available via the remote access rest api, for e.g.
http://<jenkinshost>:<port>/job/<jobname>/lastSuccessfulBuild/api/json?pretty=true
would give you a structure containing a lot of project metadata
Have a look at the documentation on the remote access API: https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API