Upload coverage information to SonarCloud from coverlet for C# project - yaml

I'm trying to collect coverage info and publish it to SonarCloud for my C# project, using GitHub Actions as my CI pipeline. The execution is very simple, basically trying to execute tests for all projects, merging all coverage files:
run: |
.\.sonar\scanner\dotnet-sonarscanner begin /k:"LanguageDev_Yoakke" /o:"languagedev" /d:sonar.login="${{ secrets.SONAR_TOKEN }}" /d:sonar.host.url="https://sonarcloud.io" /d:sonar.cs.opencover.reportsPaths="CoverageResults/coverage.opencover.xml" /d:"sonar.verbose=true"
dotnet build
dotnet test /p:CollectCoverage=true /p:CoverletOutput="../CoverageResults/" /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" /maxcpucount:1
.\.sonar\scanner\dotnet-sonarscanner end /d:sonar.login="${{ secrets.SONAR_TOKEN }}"
Note, that I do need to output in both opencover and json formats. coverlet only seems to merge properly is json is among the output formats, so it can then convert that to opencover in the end.
The problem is, this generates no coverage information on the CI - downloading artifacts show no generated folder or files. However, if just pass /p:CoverletOutputFormat=opencover, all coverage information is generated on the CI - but not merged properly because of no json output.
Locally, the command
dotnet test /p:CollectCoverage=true /p:CoverletOutput="../CoverageResults/" /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" /maxcpucount:1
just works and generates the proper coverage XML file.
What could be the problem, why does it not work with both coverage formats specified for the CI? Initially I thought that I do not know about escaping quotes in YAML but this does not seem to be the case here.

Related

Gitlab CI/CD - saving test coverage % in a regex variable?

Recently I added test coverage to our CI/CD pipeline, so we have the % of coverage outputted after each job. Currently our .gitlab-ci.yml looks like this:
run tests:
stage: test
image: python:3
script:
- pip install pytest pytest-cov
- coverage run -m pytest
- coverage report
- coverage xml
coverage: '/TOTAL.*\s([.\d]+)%/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
Coverage is a gitlab CI keyword and it pulls the total % coverage from coverage report using regex on the total coverage that is outputted to the terminal. I want to save the coverage to a variable so I can compare it to a threshold number and fail the pipeline if it goes below the threshold. I tried:
variables:
coverage_percent: '/TOTAL.*\s([.\d]+)%/'
after_script:
- echo $coverage_percent
But $coverage_percent just gets saved as exact string '/TOTAL.*\s([.\d]+)%/' - it doesn't perform a regex on the terminal like the coverage command does. How can I either a) save the result of coverage: '/TOTAL.*\s([.\d]+)%/' or b) perform regex with a variable in the pipeline?
you could try using the gitlab api to pull the value out and perform a comparison, then based on the result of that fail your pipeline.
https://www.reddit.com/r/gitlab/comments/agqhnt/using_gitlab_api_to_access_code_coverage/ has an example of how to do this.
Depending on how often the pipelines value (the bit after /pipelines/) in the URL changes this might work for you :)

Ginkgo: how to combine test reports

I'm setting up GitLab CI.
We use Ginkgo tests for BDD.
Ginkgo creates a report per each folder where tests are located.
This create a problem with collecting all reports and publishing it as a single test report file.
Is it possible to configure GinkGo in a such way so I could take all test in a single test report file?
What I understand is your reports lies in each test folder:
Example
testScripts
- test_1_directory (contains test spec and result files)
- test_2_directory (contains test spec and result files)
- test_3_directory (contains test spec and result files)
I'm not sure this might exactly help you but can give it a try
In you job add reports paths as mentioned below:
artifacts:
reports:
junit:
- ./packages/e2e/goProject/testScripts/**/**.xml
Assuming .xml are report generated.
At the end all test will be displayed in pipeline's test section

Cannot generate unified Allure report from two separate Jenkins maven builds

I want to generate a unified report by merging the results from two separate Jenkins maven projects (and I've read that Allure does support that feature - to merge results different allure-results folders), but it does not work properly as I'll explain below.
My objective is to have visibility from all tests results I've from all projects. Currently I can only see the test results specific to a project.
This is the context:
I've two Maven projects (for testNG) running and reporting results via allure on JENKINS. As you can see below, the allure results from those two builds are working as expected - automatically generated by Allure Jenkins plugin version 2.26.0
Allure Jenkins plugin working fine for separate maven jobs
Now I've also installed the Allure 2.6.0 and added it on system PATH and properly configure ALLURE_HOME environment var.
So when I ran this command here:
allure serve "C:\Users\tester\workspace\scripts-qa\STF\allure-results" --host "127.0.0.1" --port "8081"
It works perfectly to report the results in my Allure-results from my IntelliJ, as you can see below.
Allure serve results from my local worspace
Now when I ran similar command as above one to generate a unified report by merging both reports I got in Jenkins - just the execution trend is been populated - but the total number of tests and its details are not as you can see below.
This is the command I'm executing to generate a unified report:
allure serve "C:\Users\tester\.jenkins\workspace\Regression2\allure-report" "C:\Users\tester\.jenkins\workspace\Regression1\allure-report" --host "127.0.0.1" --port "8081"
Allure problem when trying to merge reports from two separate jenkins projects
Could you please help me to figure out what is wrong here? Is that possible that allure binary cannot serve results generated by allure Jenkins plugin? If yes, what is the work around to fix that?
Thank you!!
Jean.
I can see that you are trying to generate cumulative report from already generated reports, which is not correct - Allure needs raw allure-results to generate report
As a workaround, you need
Archive allure-results for each test execution job and attach it to build as artifact
Copy allure-results artifacts
Unzip it
Build a report based on allure-results from artifacts
Below are code snippets for Pipeline job
Attach allure-results as artefact:
zip archive: true, dir: "target/allure-results", glob: '', zipFile: "allure-results.zip"
Copy artefacts to another job:
copyArtifacts filter: 'allure-results.zip', optional: true, projectName: "Regression1", selector: lastCompleted(), target: "Regression1"
unzip dir: "Regression1", glob: '', zipFile: "Regression1/allure-results.zip"
...
Build report
allure serve Regression1 Regression2
You can also select reports for aggregation like this:
allure includeProperties: false, jdk: '', results: [
[path: '../name-of-job-1/target/allure-results'],
[path: '../name-of-job-2/target/allure-results'],
[path: '../name-of-job-3/target/allure-results']]
}

How to generate .trx result file after TFS build instead of code coverage?

I recently added xxx.runsettings file to my solution to pass parameters from TFS variables to my solution(url). Now I'm not able to generate .trx result file after running TFS build in Test summary/results page, only code coverage is generated.
[.runsettings code taken from here-https://msdn.microsoft.com/en-us/library/jj635153.aspx
Can anyone here tell me how to edit the runsettings file in order to show .trx result file in my test summary instead of code coverage?
TFS 2015 Update 3,TFS ms build
Can see this window once build is completed.
Run functional test Logs
Publish results log -
Logs
2017-06-21T17:20:49.9138829Z Executing the powershell script: C:\agent\tasks\PublishTestResults\1.0.22\PublishTestResults.ps1
2017-06-21T17:20:50.0628925Z ##[warning]No test result files were found using search pattern 'C:\agent_work\2\s**\TestResults\xyz*.trx'.
Just remove <ResultsDirectory>.\TestResults</ResultsDirectory> section from the runsettings file should fix your issue.
By the way, Publish Test Result task does not work in your scenario since it can only publish the test result files on the build agent while the trx file is usually generated on the test agent with Run Functional Test task.
Since you are running the VStest. Which will not generate the .trx result file.
VStest step is actually using VSTest.Console.exe command, it will use the /logger:TfsPublisher which not generate the .trx file. So if you use the built-in tasks such as Visual Studio Test or Run Functional Tests to run tests, results are automatically published and you do not need a separate publish test results task.
To log results into a Visual Studio Test Results File (TRX) use /Logger:trx. More details please refer this command. To generate a .trx file is not related to runsettings file.
For TFS , the test result is automatically published, you could click test run for more details.

JavaScript Unit Tests not working on SonarQube

I have below problems with SonarRunner.
SonarQube along with Sonar runner unable to pull junit format xml reports
Unit Tests or Test Coverage widget doesn't show up. It says No Data.
I am following instructions described here
I manually created report file in XML format as described, but still no luck.
Below is the XML file - TEST-Firefox_210_Mac_OS.com.company.BarTest.xml,
<testsuite name="Firefox_210_Mac_OS.com.company.BarTest" errors="0" failures="0" tests="3" time="0.0">
<testcase classname="Firefox_210_Mac_OS.com.company.BarTest" name="testfullName" time="0.0"/>
</testsuite>
To pull the Unit Test execution report to show on sonar dashboard. I have used sample git project from here
and below is my sonar-project.properties,
# project metadata (required)
sonar.projectKey=org.codehaus.sonar:javascript-sonar-runner-jstestdriver
sonar.projectName=JavaScript project with Sonar Runner reusing reports generated by JsTestDriver
sonar.projectVersion=1.0
# path to source directories (required)
sonar.sources=C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/sources
# path to tests source directories (required)
sonar.tests=C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/tests
sonar.javascript.jstestdriver.reportsPath=C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/target/TEST-Firefox_210_Mac_OS.com.company.BarTest.xml
sonar.sourceEncoding=UTF-8
below is my jsTestDriver.conf
server: http://localhost:9876
load:
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/sources/*.js
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/sources/com/company/*.js
test:
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/tests/*.js
- C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/tests/com/company/*.js
plugin:
- name: "coverage"
jar: "coverage-1.3.5.jar"
module: "com.google.jstestdriver.coverage.CoverageModule"
My sonarqube is running on port : 9000 and below is the screen shot. As you see SonarRunner and jsTestDriver just doing a code analysis and not showing any unit tests.
SonarQube doc website says jsTestDriver will run the javascript unit
tests and copy the results in target folder in XML format
SonarQube doesnt run your Unit Tests, it just gathers the reports generated from your manuall run or other tools automatic run (like: Jenkins).
From SonarQube doc:
Prior to the SonarQube analysis, execute your unit tests in order to
generate XML report. The JUnit like XML format supported is the one
generated by js-test-driver
Then I manually created report file in XML format as described, but
still no luck.
If you created manually the report files and reports are still not showing, check your paths if ok.Check if sonar is really reading data from C:/Sonar/sonar-runner-dist-2.4/sonar-runner-2.4/projects/ path

Resources