Context
I'm developing an application with the following services
Frontend
IAM
Backend
Database
Each service has it own repository contains the source code and the Dockerfile to build the image.
Currently I've created one helm chart that includes all those services. The values.yaml file looks like this;
frontend:
image:
repository: "myimage"
tag: "1.0.0"
iam:
image:
repository: "myimage"
tag: "1.0.3"
backend:
image:
repository: "myimage"
tag: "1.0.1"
database:
image:
repository: "myimage"
tag: "2.0.0"
Question
I'm now looking into the options I have to create a CI process for all of this...
I feel like im lost in the possibilities. Every guide i read on the internet has an example for a helm chart containing only 1 service. But my chart has several and they depend on each-other. Based on the guides I feel like I need to create one umbrella chart with subcharts for every service. So every service should get it's own chart.
My question now is; how do you setup a proper CI (and CD) for all of this?
Below is the flow that I've came up with (probably not correct). I'd like to automate as much as possible.
// frontend repo | feature branch
* Trigger: open pull request (and commit to that pr)
- Lint
- Unit tests
- Code quality checks
- Build new image (tag = commit sha)
- Push image to registry
- Checkout frontend-chart repo
- Commit the new image tag to a feature branch in frontend-chart repo
- Open PR in frontend-chart repo (if no already opened)
- Wait until build succeeded for chart-frontend repo (feature branch)
- Merge to master
// chart-frontend repo | feature branch
* Trigger: open pr
- Deploy frontend chart
- Deploy all 3 other services. Install latest packaged charts.
- Run e2e tests
- Merge to master
// frontend repo | master branch
* Trigger: commit
- retag image to latest
- push image
- Trigger chart-frontend repo
// chart-frontend repo | master branch
* Trigger: build triggered by frontend repo master branch
- determine new chartversion
- helm package with new chartversion
- Add git tag
- push to chartmuseum
- checkout umbrella chart repo
- commit change, to use new chart version for frontend-chart on, to feature branch.
- Open PR on umbrella repo to merge feature branch to master
// umbrella-chart repo | feature branch
* Trigger: open PR
- Deploy the chart
- Run e2e tests
- Merge to master
// umbrella-chart repo | master branch
* Trigger: merged pr/commit
- Determine new chart version
- package chart with new chart version
- publish chart
- Add git tag
- trigger deployment to acc
// env repo | acc branch
* Trigger: build triggered by umbrella-chart with parameters
- deploy newest dyet chart
- import data from production
- run e2e tests
- merge to production
// env repo | production branch
* Trigger: merged pr/commit
- deploy newest dyet chart
- run e2e tests
- (on failure: rollback)
Related
I a have a Gitlab stage that I only want to run if there are changes to some file. I have tried using only and if: changes... however those not work as I want.
I want that this stage is only triggered if the current commit (no matter if it is a merge request or pushed to main) contains changes to some file
My stage:
cache frontend:
stage: cache
cache: &frontend_cache
key: frontend-packages
paths:
- frontend/node_modules/
- frontend/.yarn/
script:
- cd frontend
- yarn install --immutable
only:
changes:
- frontend/yarn.lock
This is a question for the github workflow action dawidd6/action-download-artifact.
There is no discussion board in https://github.com/dawidd6/action-download-artifact, so asking this question in this forum.
This is how I wish to use this workflow in my GitHub repo:
A pull request is created.
This triggers an workflow – lets call it the “build workflow” - to build the entire repo and uploads the build artifacts.
Then another workflow – lets call it the “test workflow” - should start, that should download the build artifact using action-download-artifact and run some other actions.
Now if I put the trigger for the “test workflow” as pull_request, then how can I make it wait for the corresponding “build workflow” to complete? Do I specify the run_id ?
For now I am using “workflow_run” as the trigger for the run WF. But then when a PR is created, it does not show the “test workflow” as one of the checks for the PR. Can you help me figure out the correct way of using the download-artifact action that would help for my purpose?
You could write two workflows where the first builds when the pull request is opened or edited, and the second executes the test when the pull request is closed and merged. The HEAD commit SHA could be used to identify the artifact name between the two workflows.
I'm going to reword your requirements slightly.
Build everything and upload the artifacts when a pull request is opened or edited (e.g. new commits added).
Download the artifact and test it when a pull request is closed and merged.
Here are two sample workflows that would accomplish that. You will need to create a token to share the artifacts between workflows (see secrets.GITHUB_TOKEN below).
Build.yml
name: Build
on:
pull_request:
jobs:
Build:
steps:
- name: Environment Variables
shell: bash
run: |
ARTIFACTS_SHA=$(git rev-parse HEAD)
BUILD_ARTIFACTS=BuildArtifacts_${ARTIFACTS_SHA}
echo "ARTIFACTS_DIR=$ARTIFACTS_DIR" >> $GITHUB_ENV
- name: Build
run: make
- name: Capture Artifacts
uses: actions/upload-artifact#2
with:
name: Artifacts_${{ env.ARTIFACTS_SHA }}
path: path/to/artifact/
Test.yml
name: Test
on:
pull_request:
types: [closed]
jobs:
Test:
steps:
- name: Environment Variables
shell: bash
run: |
ARTIFACTS_SHA=$(git rev-parse HEAD)
BUILD_ARTIFACTS=BuildArtifacts_${ARTIFACTS_SHA}
echo "ARTIFACTS_DIR=$ARTIFACTS_DIR" >> $GITHUB_ENV
- name: Download Artifacts
uses: dawidd6/action-download-artifact#v2
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
workflow: Build.yml
name: ${{ env.BUILD_ARTIFACTS }}
In Gitlab CI it is possible to include one or many files in a .gitlab-ci.yml file.
It is even possible to nest these includes.
See https://docs.gitlab.com/ee/ci/yaml/includes.html#using-nested-includes.
How can I see the resulting CI file all at once?
Right now, when I debug a CI cycle, I open every single include file and
combine the resulting file structure by myself. There has to be a better way.
Example
Content of https://company.com/autodevops-template.yml:
variables:
POSTGRES_USER: user
POSTGRES_PASSWORD: testing_password
POSTGRES_DB: $CI_ENVIRONMENT_SLUG
production:
stage: production
script:
- install_dependencies
- deploy
environment:
name: production
url: https://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN
only:
- master
Content of .gitlab-ci.yml:
include: 'https://company.com/autodevops-template.yml'
image: alpine:latest
variables:
POSTGRES_USER: root
POSTGRES_PASSWORD: secure_password
stages:
- build
- test
- production
production:
environment:
url: https://example.com
This should result in the following file structure:
image: alpine:latest
variables:
POSTGRES_USER: root
POSTGRES_PASSWORD: secure_password
POSTGRES_DB: $CI_ENVIRONMENT_SLUG
stages:
- build
- test
- production
production:
stage: production
script:
- install_dependencies
- deploy
environment:
name: production
url: https://example.com
only:
- master
→ How can I see this output somewhere?
Environment
Self-Hosted GitLab 13.9.1
As I continued to search for a solution, I read about a “Pipeline Editor“ which was released in GitLab version 13.8 this year. Turns out that the feature I am looking for was added to this editor just some days ago:
Version 13.9.0 (2021-02-22) „View an expanded version of the CI/CD configuration” → see Release Notes for a feature description and introduction video.
To view the fully expanded CI/CD configuration as one combined file,
go to the pipeline editor’s »View merged YAML« tab. This tab displays an
expanded configuration where:
Configuration imported with include is copied into the view.
Jobs that use extends display with the extended configuration merged into the job.
YAML anchors are replaced with the linked configuration
Usage: Open project → Module »CI / CD« → Submodule »Editor« → Tab »View merged YAML«
GitLab 15.1 offers an altervative:
Link to included CI/CD configuration from the pipeline editor
A typical CI/CD configuration uses the include keyword to import configuration stored in other files or CI/CD templates. When editing or troubleshooting your configuration though, it can be difficult to understand how all the configuration works together because the included configuration is not visible in your .gitlab-ci-yml, you only see the include entry.
In this release, we added links to all included configuration files and templates to the pipeline editor.
Now you can easily access and view all the CI/CD configuration your pipeline uses, making it much easier to manage large and complex pipelines.
See Documentation and Issue.
I use GitLab CI for my build action.
I need to build my production env only on master upstream branch and only by a tag.
Now I have something like this:
stages:
- build
- test
- deploy
build_project:
stage: build
script:
- cd ./some-dir
- build-script.sh
only:
- master
- tags
Does someone know how should I change my rules?
There's an example of how to do this in the Gitab docs: https://docs.gitlab.com/ce/ci/yaml/README.html#onlyexcept-basic
job:
only:
- branches#gitlab-org/gitlab
except:
- master#gitlab-org/gitlab
- /^release/.*$/#gitlab-org/gitlab
In the circleci version 1 config, there was the option to specify owner as an option in a deployment. An example from the circleci docs ( https://circleci.com/docs/1.0/configuration/ ) with owner: circleci being the key line:
deployment:
master:
branch: master
owner: circleci
commands:
- ./deploy_master.sh
In version 2 of the config, there is the ability to use filters and tags to specify which branches are built, but I have yet to find (in the docs, or on the interwebs) anything that gives me the same capability.
What I'm trying to achieve is run build and test steps on forks, but only run the deploy steps if the repository owner is the main repo. Quite often people fork using the same branch name - in this case master - so having a build fail due to an inability to deploy is counter-intuitive, especially as I would like to use a protected branch in git and only merge commits based on a successful build in a pull request.
I realise we could move to only running builds based on tags being present, but nothing is stopping somebody with a fork also creating a tag in their fork, which puts us back at square one.
Is anybody aware of how to specify the owner of a repo in the version 2 config?
An example from the version 2 config document ( https://circleci.com/docs/2.0/workflows/ ) in case it helps jog somebodies memory:
workflows:
version: 2
un-tagged-build:
jobs:
- build:
filters:
tags:
ignore: /^v.*/
tagged-build:
jobs:
- build:
filters:
branches:
ignore: /.*/
tags:
only: /^v.*/
disclaimer: Developer Evangelist at CircleCI
That feature is not available on CircleCI 2.0. You can request it here.
As an alternative, you might be able to look for the branch name, say master, as well as the CIRCLE_PR_NUMBER environment variable. If that variable has any value, then it's a fork of master and you shouldn't deploy.