Having Gitlab Projects calling the same gitlab-ci.yml stored in a central location - bash

I have many Gitlab project followed the same CI template. Whenever there is a small change in the CI script, I have to manually modify the CI script in each project. Is there a way you can store your CI script in a central location and have your project called that CI script with some environment variable substitution? For instance,
gitlab-ci.yml in each project
/bin/bash -c "$(curl -fsSL <link_to_the_central_location>.sh)"
gitlab-ci.yml in the central location
stages:
- build
- test
build-code-job:
stage: build
script:
- echo "Check the ruby version, then build some Ruby project files:"
- ruby -v
- rake
test-code-job1:
stage: test
script:
- echo "If the files are built successfully, test some files with one command:"
- rake test1
test-code-job2:
stage: test
script:
- echo "If the files are built successfully, test other files with a different command:"
- rake test2

You do not need curl, actually gitlab supports this via the include directive.
you need a repository, where you store your general yml files. (you can choose if it is a whole ci file, or just parts. For this example lets call this repository CI and assume your gitlab runs at example.com - so the project url would be example.com/ci. we create two files in there just to show the possibilities.
is a whole CI definition, ready to use - lets call the file ci.yml. This approach is not really flexible
stages:
- build
- test
build-code-job:
stage: build
script:
- echo "Check the ruby version, then build some Ruby project files:"
- ruby -v
- rake
test-code-job1:
stage: test
script:
- echo "If the files are built successfully, test some files with one command:"
- rake test1
test-code-job2:
stage: test
script:
- echo "If the files are built successfully, test other files with a different command:"
- rake test2
is a partly CI definition, which is more extendable. lets call the files includes.yml
.build:
stage: build
script:
- echo "Check the ruby version, then build some Ruby project files:"
- ruby -v
- rake
.test:
stage: test
script:
- echo "this script tag will be overwritten"
There is even the option to use template string from yaml. please reference the gitlab documentation but it is similar to 2.
we do have our project which wants to use such definitions. so either
For the whole CI file
include:
- project: 'ci'
ref: master # think about tagging if you need it
file: 'ci.yml'
as you can see now we are referencing one yml file, with all the cahnges.
with partial extends
include:
- project: 'ci'
ref: master # think about tagging if you need it
file: 'includes.yml'
stages:
- build
- test
build-code-job:
extends: .build
job1:
extends: .test
script:
- rake test1
job2:
extends: .test
script:
- rake test2
As you see, you can easily use the includes, to have a way more granular setup. Additionally you could define at job1 and job2 variables, eg for the test target, and move the script block into the includes.yml
Futhermore you can also use anchors for the script parts. Which looks like this
includes.yml
.build-scirpt: &build
- echo "Check the ruby version, then build some Ruby project files:"
- ruby -v
- rake
.build:
stage: build
script:
- *build
and you can use also the script anchor within your configuration
For a deeper explanation you can also take a look at https://docs.gitlab.com/ee/ci/yaml/includes.html

Related

Save gradle command result into a variable in Gitlab CI/CD

I would like to save gradle-based project's version into a variable in gitlab-ci script. In my build.gradle I have:
tasks.register('version') {
doLast {
println(version)
}
}
It reads version from gradle.properties (let's say version=0.1) and returns it.
I execute it as gradlew version -q so I get only result, with no unnecessary output. When using unix-style variable creation of command result, that is: version=$(./gradlew version -q), the runner ends script. Is it possible to save the output into a variable for script?
My .gitlab-ci.yml:
image: gradle:jdk11
cache: &wrapper
paths:
- .gradle/wrapper
- .gradle/caches
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
- chmod a+x gradlew
stages:
- prepare
- build
- deploy
wrapper:
stage: prepare
script:
- gradle wrapper
compile:
stage: build
script:
- ./gradlew assemble
artifacts:
paths:
- build/classes/**
- build/libs/*.jar
cache:
<<: *wrapper
policy: pull
properties:
stage: deploy
script:
- eval version=$(./gradlew version -q)
- echo $version # not even called
I also tried to omit eval to have version=$(./gradlew version -q) in script, but nothing changes.
CI output:
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ chmod a+x gradlew
$ version=$(./gradlew version -q)
Cleaning up file based variables
Ok, I've found the solution. I mustn't use variable, simply. It's needed to directly pass evaluation to another command, using double quotes, like:
.gitlab-ci.yml (part):
properties:
stage: deploy
script:
- echo "$(./gradlew version -q)"
It started to work

Regex based rules clause does not work in GitLab CI

I want my Gitlab CI job to not run when the commit message starts with a particular string: [maven-scm]
So, I have the below configuration in my .gitlab-ci.yaml file:
image: maven:3.6.3-jdk-11-slim
stages:
- test
test:
stage: test
cache:
key: all
paths:
- ./.m2/repository
script:
- mvn clean checkstyle:check test spotbugs:check
rules:
- if: '$CI_COMMIT_MESSAGE !~ /^\[maven-scm\] .*$/'
My commit message is: [maven-scm] I hope the test job does not run
But the test job still runs to my frustration. I went over the GitLab documentation for rules but could not find the reason why the job still runs. I am not sure if I am missing something.
Would be great if someone can point me in the right direction.
Update:
I tried the only/except clause instead of the rules. I modified the yaml file to below:
image: maven:3.6.3-jdk-11-slim
stages:
- test
test:
stage: test
cache:
key: all
paths:
- ./.m2/repository
script:
- mvn clean checkstyle:check test spotbugs:check
except:
variables:
- $CI_COMMIT_MESSAGE =~ /^\[maven-scm\] .*$/
The job still runs when the commit message starts with [maven-scm].
This was a tricky problem, because the issue was not with the rules section. The problem is actually the regex. You only need to specify the desired pattern at the start of the commit message, i.e. don't need the following wildcard. The following works and has been tested:
test-rules:
stage: test
rules:
- if: '$CI_COMMIT_MESSAGE !~ /^\[maven-scm\] /'
script:
- echo "$CI_COMMIT_MESSAGE"
This has been tested with the following commit messages:
This commit message will run the job
This commit message [maven-scm] will run the job
[maven-scm] This commit message will NOT run the job
FYI GitLab documentation specifies that rules is preferred over only/except, so best to stick with rules: if. See onlyexcept-basic.

variables substitution (or overriding) when extends jobs from gitlab templates unsing include

Using gitlab ci, I have a repo where all my templates are created.
For example, I have a sonar scanner job template named .sonar-scanner.yml:
sonar-analysis:
stage: quality
image:
name: sonar-scanner-ci:latest
entrypoint: [""]
script:
- sonar-scanner
-D"sonar.projectKey=${SONAR_PROJECT_NAME}"
-D"sonar.login=${SONAR_LOGIN}"
-D"sonar.host.url=${SONAR_SERVER}"
-D"sonar.projectVersion=${CI_COMMIT_SHORT_SHA}"
-D"sonar.projectBaseDir=${CI_PROJECT_DIR}"
I have include this template as a project like this in main gitlab ci file:
include:
- project: 'organization/group/ci-template'
ref: master
file: '.sonar-scanner.yml'
So as you can understand I have a repo named ci-templates where all my templates are created. And in another repo, I extends using include these templates.
Finally, in a repo, when a new merge request is created, my job for sonar is running under another file in my project test/quality.yml:
sonar:
stage: quality
extends:
- sonar-analysis
allow_failure: true
only:
refs:
- merge_requests
All is working well except the substitution or the overriding of my env var. Indeed of my template. I have many sonar server or project names. I would like to know how to override these variables SONAR_SERVER and SONAR_PROJECT_NAME when I extend the job from a template.
In my main .gitlab-ci.yml file I have a variables section declaration, and when I override these variables in, it works.
But it's not really what I want. Using many stages and many micro service it is possible to reuse the same extending job in a different way. That I really want to do is to override these variables directly in the file test/quality.yml.
This, for example does not work:
sonar:
stage: quality
extends:
- sonar-analysis
variables:
SONAR_PROJECT_NAME: foo
SONAR_SERVER: bar
allow_failure: true
only:
refs:
- merge_requests
This not work too:
variables:
SONAR_PROJECT_NAME: foo
SONAR_SERVER: bar
sonar:
stage: quality
extends:
- sonar-analysis
allow_failure: true
only:
refs:
- merge_requests
What is the best way to make this work ?
Since this question was asked in Feburary 2020, a new MR Use non-predefined variables inside CI include blocks has been merged into Gitlab 14.2, which resolves the issue for the overridden job.
The project that does the include can redefine the variables when extending a job:
include:
- project: 'organization/group/ci-template'
ref: master
file: '.sonar-scanner.yml'
sonar:
stage: quality
extends:
- sonar-analysis
variables:
SONAR_PROJECT_NAME: foo
SONAR_SERVER: bar
allow_failure: true
But in this case you probably want the job in the template to start with a dot .sonar-analysis instead of sonar-analysis to not create a real sonar-analysis job in the template (see hidden jobs).
Or you can also directly set the variables values (to redefine them) of an existing job in the project that does the include:
include:
- project: 'organization/group/ci-template'
ref: master
file: '.sonar-scanner.yml'
sonar-analysis:
variables:
SONAR_PROJECT_NAME: foo
SONAR_SERVER: bar
I verified this with a test project, which includes a template from a peer test project, and when it runs, results in two jobs. This is the job output for the overriding job:
$ echo sonar.projectKey=${SONAR_PROJECT_NAME}
sonar.projectKey=foo
$ echo sonar.login=${SONAR_LOGIN}
sonar.login=bob
$ echo sonar.host.url=${SONAR_SERVER}
sonar.host.url=bar

Gitlab CI allow manual action, when previous stage failed

I'm having a Gitlab CI/CD pipeline, and it works OK generally.
My problem is that my testing takes more than 10 minutes and it not stable (YET..) so occasionally randomly it fails on a minor test that I don't care for.
Generally, after retry, it works, but if I need an urgent deploy I need to wait another 10 minutes.
When we have an urgent bug, another 10 minutes is waaaay too much time, so I am looking for a way to force deploy even when the test failed.
I have the next pseudo ci yaml scenario that I'd failed to find a way to accomplish
stages:
- build
- test
- deploy
setup_and_build:
stage: build
script:
- build.sh
test_branch:
stage: test
script:
- test.sh
deploy:
stage: deploy
script:
- deploy.sh
only:
- master
I'm looking for a way to deploy manually if the test phase failed.
but if I add when: manual to the deploy, then deploy never happens automatically.
so a flag like when: auto_or_manual_on_previous_fail will be great.
currently, there is no such flag in Gitlab ci.
Do you have any idea for a workaround or a way to implement it?
Another approach would be to skip the test in case of an emergency release.
For that, follow "Skipping Tests in GitLab CI" from Andi Scharfstein, and:
add "skip test" in the commit message triggering that emergency release
check a variable on the test stage
That is:
.test-template: &test-template
stage: tests
except:
variables:
- $CI_COMMIT_MESSAGE =~ /\[skip[ _-]tests?\]/i
- $SKIP_TESTS
As you can see above, we also included the variable $SKIP_TESTS in the except block of the template.
This is helpful when triggering pipelines manually from GitLab’s web interface.
Here’s an example:
It's possible to control the job attribute of your deploy job by leveraging parent-child pipelines (gitlab 12.7 and above). This will let you decide if you want the job in the child pipeline to run as manual, or always
Essentially, you will need to have a .gitlab-ci.yml with:
stages:
- build
- test
- child-deploy
child-deploy stage will be used to run the child pipeline, in which the deploy job will run with the desired attribute.
Your test could generate as artifact the code for deploy section. For example, in the after_script section of your test, you can check the value of CI_JOB_STATUS builtin variable to decide if you want to write the child job to run as manual or always:
my_test:
stage: test
script:
- echo "testing, exit 0 on success, exit 1 on failure"
after_script:
- if [ "$CI_JOB_STATUS" == "success" ]; then WHEN=always; else WHEN=manual; fi
- |
cat << 'EOF' > deploy.yml
stages:
- deploy
my_deploy:
stage: deploy
rules:
- when: $WHEN
script:
- echo "deploying"
EOF
artifacts:
when: always
paths:
- deploy.yml
Note that variable expension is disabled in the heredoc section, by the use of single quoted 'EOF'. If you need variable expension, remember to escape the $ of $WHEN.
Finally, you can trigger the child pipeline with deploy.yml
gen_deploy:
stage: child-deploy
when: always
trigger:
include:
- artifact: deploy.yml
job: my_test
strategy: depend

GitLab CI and MsBuild (with tests)

I am in the process of migrating my svn repsitories to git with GitLab.
Now I have seen that there is a continuous integration implementation with GitLab CI and just want to try it out.
I already installed and configured a Runner but Gitlab complains that I don't have a .gitlab-ci.yml file.
I already use TeamCity for continuous integration so I don't want to put too much effort into writing a build script.
Can anybody tell me where I can find a basic example of a gitlab-ci.yml file that basically just builds my Solution and runs all tests (MSTests)?
Apparently there is no simple msbuild example but this should get you started:
variables:
Solution: MySolution.sln
before_script:
- "echo off"
- 'call "%VS120COMNTOOLS%\vsvars32.bat"'
# output environment variables (usefull for debugging, propably not what you want to do if your ci server is public)
- echo.
- set
- echo.
stages:
- build
- test
- deploy
build:
stage: build
script:
- echo building...
- 'msbuild.exe "%Solution%"'
except:
- tags
test:
stage: test
script:
- echo testing...
- 'msbuild.exe "%Solution%"'
- dir /s /b *.Tests.dll | findstr /r Tests\\*\\bin\\ > testcontainers.txt
- 'for /f %%f in (testcontainers.txt) do mstest.exe /testcontainer:"%%f"'
except:
- tags
deploy:
stage: deploy
script:
- echo deploying...
- 'msbuild.exe "%Solution%" /t:publish'
only:
- production
Figuring out which tests to run is a bit tricky. My convention is that every project has a folder tests in which the test projects are named after the schema MyProject.Core.Tests (for a project called MyProject.Core)
Just as a first feedback towards gitlab-ci
I like the simplicity and the source control integration. But I would like to be able to modify the script before execution (especially while changing the script) but I could imaging to rerun a specific commit and inject variables or change the script (I can do that with teamcity). Or even ignore a failed test and rerun the script again (I do that a lot with teamcity). I know gitlab-ci does not know anything about my tests I just have a command line that returns an error code.
As an addendum to Jürgen Steinblock answer, I would like to suggest a simplier alternative for the test stage script:
variables:
SOLUTION_DIR: "MySolution"
BUILD_DIR: "Release"
TESTER: "vstest.console.exe" # or "mstest.exe /testcontainer:"
before_script:
- call "%VS120COMNTOOLS%\vsvars32" # import in path tools like msbuild, mstest, etc using VS script
test:
stage: test
script:
- for /f %%F in ('dir /s /b %SOLUTION_DIR%\%BUILD_DIR%\*Tests.dll') do set dllPath=%%F
- "%TESTER% %dllPath%"
This will launch testing on all test project binaries found which ends by convention with *Tests.dll in the build directory. This has the advantage of not using an intermediate file.
This is what I used eventually. It runs all the *Tests.Dll in a single run.
dir /s /b *.Tests.dll | findstr /r bin\\Debug > testcontainers.txt
for /f %%x in (testcontainers.txt) do set list=!list! %%x
set list=%list:~1%
vstest.console.exe %list%
Since things have changed a bit since this question has been opened (and now MS supports core/linux docker deployments right in VS), I thought I'd share my solution.
# Default image is docker:stable
image: docker:stable
# Define deployment stages
stages:
- Test
- Build
# We use docker-in-docker (dind) for building docker images (build stage)
services:
- docker:dind
# Run unit test on dotnet core sdk image
test:
stage: Test
image: mcr.microsoft.com/dotnet/core/sdk:3.1
script:
- dotnet restore "${CI_PROJECT_DIR}/Project.Tests/Project.Tests.csproj"
- dotnet test "${CI_PROJECT_DIR}/Project.Tests/Project.Tests.csproj"
tags:
- docker
only:
- master
# Only build when testing passes, build using Dockerfile/dind
build:
stage: Build
# Print docker instance details for logging/diagnosing
before_script:
- docker info
- docker login registry.gitlab.com -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD}
script:
- docker build -t ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest .
- docker push ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest
after_script:
- docker logout ${CI_REGISTRY}
tags:
- docker
only:
- master
when: on_success
This should run the MS Unit tests on your solution, and if the tests pass, create an image from them (provided you have a Dockerfile next to the gitlab-ci.yml file). Ignore the build stage if you're only looking to perform unit testing automatically on commits.

Resources