Gitlab CI/CD - saving test coverage % in a regex variable? - bash

Recently I added test coverage to our CI/CD pipeline, so we have the % of coverage outputted after each job. Currently our .gitlab-ci.yml looks like this:
run tests:
stage: test
image: python:3
script:
- pip install pytest pytest-cov
- coverage run -m pytest
- coverage report
- coverage xml
coverage: '/TOTAL.*\s([.\d]+)%/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
Coverage is a gitlab CI keyword and it pulls the total % coverage from coverage report using regex on the total coverage that is outputted to the terminal. I want to save the coverage to a variable so I can compare it to a threshold number and fail the pipeline if it goes below the threshold. I tried:
variables:
coverage_percent: '/TOTAL.*\s([.\d]+)%/'
after_script:
- echo $coverage_percent
But $coverage_percent just gets saved as exact string '/TOTAL.*\s([.\d]+)%/' - it doesn't perform a regex on the terminal like the coverage command does. How can I either a) save the result of coverage: '/TOTAL.*\s([.\d]+)%/' or b) perform regex with a variable in the pipeline?

you could try using the gitlab api to pull the value out and perform a comparison, then based on the result of that fail your pipeline.
https://www.reddit.com/r/gitlab/comments/agqhnt/using_gitlab_api_to_access_code_coverage/ has an example of how to do this.
Depending on how often the pipelines value (the bit after /pipelines/) in the URL changes this might work for you :)

Related

Using different Docker images during a single Gitlab-CI job

Hello there i have a particular question regarding using different images under a single gitlab-ci pipeline job. Initially i am using a maven 3.6.1-jdk-8 because the project runs per those requirements. What i want to do is to use another image maven 3.6.3-jdk-11 before running the script so the sonarqube scanner be able to run properly and does not fails. Both images are required for the job.
sonarqube:
image: maven:3.6.1-jdk-8
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: "0" # Tells git to fetch all the branches of the project, required by the analysis task
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
- mvn verify sonar:sonar -Dsonar.qualitygate.wait=true
allow_failure: true
only:
- master

Ginkgo: how to combine test reports

I'm setting up GitLab CI.
We use Ginkgo tests for BDD.
Ginkgo creates a report per each folder where tests are located.
This create a problem with collecting all reports and publishing it as a single test report file.
Is it possible to configure GinkGo in a such way so I could take all test in a single test report file?
What I understand is your reports lies in each test folder:
Example
testScripts
- test_1_directory (contains test spec and result files)
- test_2_directory (contains test spec and result files)
- test_3_directory (contains test spec and result files)
I'm not sure this might exactly help you but can give it a try
In you job add reports paths as mentioned below:
artifacts:
reports:
junit:
- ./packages/e2e/goProject/testScripts/**/**.xml
Assuming .xml are report generated.
At the end all test will be displayed in pipeline's test section

Upload coverage information to SonarCloud from coverlet for C# project

I'm trying to collect coverage info and publish it to SonarCloud for my C# project, using GitHub Actions as my CI pipeline. The execution is very simple, basically trying to execute tests for all projects, merging all coverage files:
run: |
.\.sonar\scanner\dotnet-sonarscanner begin /k:"LanguageDev_Yoakke" /o:"languagedev" /d:sonar.login="${{ secrets.SONAR_TOKEN }}" /d:sonar.host.url="https://sonarcloud.io" /d:sonar.cs.opencover.reportsPaths="CoverageResults/coverage.opencover.xml" /d:"sonar.verbose=true"
dotnet build
dotnet test /p:CollectCoverage=true /p:CoverletOutput="../CoverageResults/" /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" /maxcpucount:1
.\.sonar\scanner\dotnet-sonarscanner end /d:sonar.login="${{ secrets.SONAR_TOKEN }}"
Note, that I do need to output in both opencover and json formats. coverlet only seems to merge properly is json is among the output formats, so it can then convert that to opencover in the end.
The problem is, this generates no coverage information on the CI - downloading artifacts show no generated folder or files. However, if just pass /p:CoverletOutputFormat=opencover, all coverage information is generated on the CI - but not merged properly because of no json output.
Locally, the command
dotnet test /p:CollectCoverage=true /p:CoverletOutput="../CoverageResults/" /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" /maxcpucount:1
just works and generates the proper coverage XML file.
What could be the problem, why does it not work with both coverage formats specified for the CI? Initially I thought that I do not know about escaping quotes in YAML but this does not seem to be the case here.

Gitlab pipeline test stage to fail AND create artifacts anyway

I have a gitlab pipeline running on a windows machine with Windows 7 and powershell 4.0.
The .yaml has the typical 3 stages: build, test and deploy.
For the second stage I want to perform some simple tests that generate a log file which should be available after the test stage finishes.
Here the script section from the test:
script:
- '$exitCode = (start-process C:\app_versions\app_20181211\bin\app.exe -PassThru -Wait).ExitCode'
- 'cat .\TestLogs\BasicFunctionsTestPlan.log'
- 'exit $exitCode'
artifacts:
paths:
- .\TestLogs
expire_in: 1 year
Here I had one problem, after the test run has finished the stage finishes always successfully even if the test themselves failed. Then I had to force the script exit with an error code in case the application tells me that the tests failed.
This caused the second problem: the artifacts link do not get created even they are available (my test produce it anyway).
Probably if I knew how to tell gitlab that the test failed in a more clean way, the artifacts would be available anyway.
I agree that the log file is not an artifact but I would like to keep that file in order to check how the tests have performed, maybe there is a better way to save this file.
Thanks in advance for your help!
EDIT:
Looks like there were more people having the same issue here, maybe it helps understanding better the problem.
I had the same question, but it's easily solved:
You can use artifacts:when to upload artifacts on job failure or despite the
failure.
artifacts:when
source: Gitlab CI yaml reference: artifacts:when
Introduced in GitLab 8.9 and GitLab Runner v1.3.0.
artifacts:when is used to upload artifacts on job failure or despite the
failure.
artifacts:when can be set to one of the following values:
on_success - upload artifacts only when the job succeeds. This is
the default.
on_failure - upload artifacts only when the job
fails.
always - upload artifacts regardless of the job status.
Example:
To upload artifacts only when job fails:
job:
artifacts:
when: on_failure
allow_failure
BTW: you can tell Gitlab CI to continue to the next job after a job failure with allow_failure: true
source: Gitlab CI yaml Reference: allow_failure
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
So combined it could look something like:
job1:
stage: test
script:
- execute_script_that_will_fail
allow_failure: true
artifacts:
when: always # or 'on_failure'
paths:
- resulting_artifacts

Break Push in GitLab based on SonarQube Analysis Result

I have an application in springboot which uses gradle to build the code.
I have setup https://github.com/gabrie-allaigre/sonar-gitlab-plugin on SonarQube and have integrated gitlab CI
to analyse code on every push/commit. What I want to achieve is to break the push/commit if the analysis fails.
Below is my .gitlab-ci.yml
image: XXXXXX:oraclejdk:1.8.0_121
before_script:
- export GRADLE_USER_HOME=`pwd`/.gradle
sonarqube_master_job:
stage: test
only:
- master
- release2.0
script:
- ./gradlew assemble
- ./gradlew -x test sonarqube -Dsonar.host.url=http://sonarqube.XXX.XXX.XXX:9000/sonarqube -Dsonar.login=xxxxxxxxxxxxxxxxxxxx
sonarqube_preview_feature_job:
stage: test
only:
- /^feature\/*/
- development
script:
- git checkout $CI_COMMIT_REF_NAME
- git merge --no-commit --no-ff
- ./gradlew assemble
- ./gradlew -x test sonarqube -Dsonar.host.url=http://XXXX.XXXXX.com:9000/sonarqube -Dsonar.login=xxxxxxxxxxxxxxxxxxxxx -Dsonar.analysis.mode=preview -Dsonar.gitlab.commit_sha=$CI_COMMIT_REF -Dsonar.gitlab.ref_name=$CI_COMMIT_REF_NAME -Dsonar.gitlab.project_id=$CI_PROJECT_ID --stacktrace
How do I make sure the push fails if the analysis fails? Do I need to use webhooks. Is there a sample CI file?
#jibsonline, You can refer to my answer provided in the below link.
However the script answers only how to break the build on sonar analysis and display the results.
How to integrate Sonar Quality Gates with Gitlab-CI
Since gitlab triggers the build, once the changes were pushed, it is not advisable to set up an automated tool to revert the code changes on your behalf. Whenever a build fails, write script (dependencies) such that the code will not be deployed. Since the code is not deployed, your environment will not be effected. Also,set up an email configuration whenever build fails.

Resources