Recently research different build tools , GCP CloudBuild is one of the selection
https://github.com/GoogleCloudPlatform/cloud-builders
one of require is work loop an array list and write the function only once and run in parallel
however i did not find Cloudbuild mention any about matrix build
Function which provided by Jenkins Plugin https://www.jenkins.io/blog/2019/11/22/welcome-to-the-matrix/
or Github Action https://docs.github.com/en/free-pro-team#latest/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix
Cloud Run is not designed to work with Jenkins out of the box and the links you included do not mention how to do this.
As indicated [1] the best product to integrate Jenkins inside Google Cloud is to use Google Kubernetes Engine (GKE).
[1] https://cloud.google.com/jenkins
Related
I'm trying to get my head around if replacing our current CI/CD tool with Github Actions is a pro or con.
My team is currently using GitHub and TeamCity for CI/CD. I find this limiting as it's a separate team that configures the different stages. We use Airflow to schedule our DBT production, so we don't use DBT Cloud.
I'm wondering if there's any benefit to replacing TeamCity with Github Actions as we can write the configuration of the stages (compile, run, test, lint) ourselves. I'm interested in implementing Slim CI , which I don't believe is possible with TeamCity.
Would anyone be able to with the pros and cons of using GitHub Actions alongside Airflow? Is it possible? I'm struggling to find any documentation on using both Actions whilst running DBT production on Airflow.
Right now, my source of information comes from this: https://towardsdatascience.com/how-to-deploy-dbt-to-production-using-github-action-778bf6a1dff6
It mentions "Well, it depends. If you don’t have Airflow running in productions already, you will probably not need it now. There are more simple/elegant solutions than this (dbt Cloud, GitHub Actions, GitLab CI). Also, this approach shares many disadvantages with using a compute instance, such as waste of resources and no easy way for CI/CD."
My main goal is to get rid of TeamCity and implement Slim CI with Github Actions.
Thank you in advance!
Does anyone have any information on how to run AWS lambda scripts from rundeck? I was looking into doing this to have a central place that certain uses can log into run deck and run the scripts that are relevant to them, as not everyone has aws access.
I found this: https://www.slideshare.net/tetutaro/lambda-and-rundeck-58884982
But I was hoping there might be something more official somewhere and in English :)
A good way to integrate with Lambda is to use AWS CLI on the Rundeck server and call functions using script step or command step on your workflow. Take a look at this.
Also, and similar to this answer, another good way to interact with Lamda is to access it using API (you have two options: using HTTP Workflow Step plugin or via script step on your workflow).
Finally, maybe is a good opportunity to develop some custom plugin focused on AWS Lambda.
I have a single repository that hosts my lambda functions on github. I would like to be able to deploy the new versions whenever new logic is pushed to master.
I did a lot of reasearch and found a few different approaches, but nothing really clear. Would like to know what others feel would be the best way to go about this, and maybe some detail (if possible) into how that pipeline is setup.
Thanks
Welcome to StackOverflow. You can improve your question by reading this page.
You can setup a CI/CD pipeline using CircleCI with its GitHub integration (which is an online Service, so you don't need to maintain anything, like a Jenkins server, for example)
Upon every commit to your repository, a CircleCI build will be triggered. Once the build process is over, you can declare sls deploy, sam deploy, use Terraform or even create a script to upload the .zip file from your GitHub repo to an S3 Bucket and then, within your script, invoke the create-function command. There's an example how to deploy Serverless applications using CircleCI along with the Serverless Framework here
Other options include TravisCI, AWS Code Deploy or even maintain your own CI/CD Server. The same logic applies to all of these tools though: commit -> build -> deploy (using one of the tools you've chosen).
EDIT: After #Matt's answer, it clicked that the OP never mentioned the Serverless Framework (I, somehow, thought he was already using it, so I pointed the OP to tutorials using the Serverless Framework already). I then decided to update my answer with a few other options for serverless deployment
I know that this isn't exactly what you asked for but I use Serverless Framework (https://serverless.com) for deployment and I love it. I don't do my deployments when I push to my repo. Instead I push to my repo after I've deployed. I like this flow because a deployment can fail due to so many things and pushing to GitHub is much less likely to fail. I this way, I prevent pushing code that failed to deploy to my master branch.
I don't know if you're familiar with the framework but it is super simple. The website describes the simple steps to creating and deploy a function like this.
1 # Step 1. Install serverless globally
2 $ npm install serverless -g
3
4 # Step 2. Create a serverless function
5 $ serverless create --template hello-world
6
7 # Step 3. deploy to cloud provider
8 $ serverless deploy
9
10 # Your function is deployed!
11 $ http://xyz.amazonaws.com/hello-world
There are also a number of plugins you can use to integrate easily with custom domains on APIGateway, prune older versions of lambda functions that might be filling up your limits, etc...
Overall, I've found it to be the easiest way to manage and deploy my lambdas. Hope it helps!
Given that you're using AWS Lambda, you may want to consider CodePipeline to automate your release process. [SAM(https://docs.aws.amazon.com/lambda/latest/dg/serverless_app.html) may also be interesting.
I too had the same problem. I wanted to manage 12 lambdas with 1 git repository. I solved it by introducing travis-ci. travis-ci saved the time and really useful in many ways. We can check the logs whenever we want and you can share the logs to anyone by sharing the URL. The sample documentation of all steps can be found here. You can go through it. 👍
I have a bunch of marathon docker tasks that are run on our test deployment machines.
There is a Jenkins CI job, that triggers the deployment a whole bunch of docker containers that are run on marathon-mesos cluster. (3 mesos-slaves, 1 master and 1 marathon.)
There is another downstream jenkins job (a automated test suite) that is triggered after above job. Presently, we wait for sufficient time, so that deployment gets completed then only we proceed with this automation testsuite. I want to change this behavior. I know marathon exposes rest APIs using which I can determine if I am good to go - after all the containers are deployed and all health checks are passing - for running the automation test suite.
Question is: Is there any library already out there for marathon, that I can reuse to accomplish above task ? I do not want to reinvent the wheel.
When I posted this question, I had java library actually in mind, but forgot to mention that. I find #michael 's libraries are also very good. But this is what I settled upon. Marathon-client. I think I saw this, while browsing through mesosphere repositories but somehow missed it.
This is the library: marathon-client
I've successfully been using the two following libs:
Go: gambol99/go-marathon
Python: thefactory/marathon-python
We are running a Jenkins instance with various test jobs as part of the build chain.
I would like to integrate the build, test trend and general CI status information into graphs/widgets using the Shopify Dashing dashboard http://shopify.github.io/dashing/.
In other words, to have a standalone dashboard that indicates key info from the Jenkins instance.
Does anyone have experience with doing this kind of thing? Can you point me in the direction of any good resources/tutorials etc?
Many thanks
Jenkins' url structure is pretty rigid. For example, the build time trend graph for a project would be available at:
http://<jenkinshost>:<port>/job/<jobname>/buildTimeGraph/png
etc. You should be able to just statically link to these using tags in your HTML / ERB / templating language of your choice.
Other information is available via the remote access rest api, for e.g.
http://<jenkinshost>:<port>/job/<jobname>/lastSuccessfulBuild/api/json?pretty=true
would give you a structure containing a lot of project metadata
Have a look at the documentation on the remote access API: https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API