I am integrating jenkins jacoco plugin in the jenkins pipeline at two different stages, one after the unit tests which produced a jacoco.exec file.
I used the jacoco(params) to attach this with the build.
Right after that, I run my integration tests with coverage, and produced jaccoco-it.exec and used jacoco(params) to attach it to the build.
But my build shows two different coverage charts with the merged coverage reports.
I would like to get the unit test coverage and integration test coverage separately on the build. Is this possible at all? I could not find any documentation related to this use case.
Related
Does SonarQube always require an external code coverage tool like jacoco (Java), Coverage (python), gcov (c/c++), in order to show coverage on a sonar server?
SonarQube by itself doesn't do any coverage. Its the job for other tools like jacoco and others.
However SonarQube can gather the "results" relevant to the project quality (of course including coverage as an important code quality metric) of the build and allows tracking of the quality over time.
Usually you run coverage tool first, it "adjusts" the code, then you run the tests in the build. Coverage tool creates some results, and only after that you run sonar plugin that processes the results and sends to the sonar qube server.
So, to answer your question: Yes, without an external code coverage tool, sonar won't produce any coverage results, and no, it doesn't have a "default, built-in" coverage tool
I'm not able to see the Integration Tests in new sonarqube 6.7 version, here is what I'm doing,
Run Sonarqube task with Gradle to generate the report along with code coverage.
Here, Sonarqube is by default invoking test task and using the test.exec file from build/jacoco generated by Jacoco plugin.
However, I've a task depends on the test task to place an integrationTest.exec report in the build/jacoco as the integrations tests are run outside build(for some reason)
Though I place the integrationTest.exec file explicitly while Sonarqube execution it is still not considering the report and due to which I do not see any change in the coverage computed also I do not see Integration Tests section under coverage like I see Unit Tests.
Firstly, I would like to understand that whether is it feasible to export the integration tests result like I did above or not?
Second, is the latest Sonarqube having the capability to show integration tests report in the coverage section?
Your comments would help me.
Thanks in advance!
I'm running a Maven multi-module project, and using Sonar Runner to analyze the project for SonarQube 6.3. This project contains both unit and integration tests in every modules. I succeeded generating reports for UT and IT in target/jacoco-ut.exec and target/jacoco-it.exec.
I think analysis parameters for Sonar Runner are good, as I can see both reports are processed and merged during analysis.
From SonarQube 6.3, there's no difference anymore between unit tests and integration tests, though the only measure reported is "Unit tests", which suggests integration tests are ignored.
When I look at the coverage measures in SonarQube, I'm surprised because the number of tests reported is not the sum of the number of unit tests and the number of integration tests. Integration tests are not listed in the measures. To me, if both unit tests and integration tests were merged in SonarQube, I understand there shall both appear in measures, but that's not the case.
I can't find anything in SonarQube documentation about integration tests inclusion in measures. There are only notes that they are merged during analysis, though I don't see anything about my integration tests in the coverage measures.
How can I see integration tests and unit tests in coverage measures?
All tests are now merged into "Coverage", so those numbers include the sum of UT and IT coverage. However prior to the merger there were no metrics about integrations tests themselves (test count, duration, errors, &etc) so there was nothing there to merge.
In fact, metrics about tests (count, errors...) really aren't seen as relevant in general and remain in the system only because they've been grandfathered.
In sonarQube, how to reference the unit test report residing in the server pertaining to Continuous Integration tool? Please note that the sonarQube and CI tools are in different servers. This is needed to show unit test coverage in sonarQube.
Usually, the build of a project does operations like:
Checkout source code
Build source code
Run unit test and generate coverage reports
Run a SonarQube Scanner to trigger the analysis
All this happens within the CI server. This means that you just need to configure your SonarQube analysis to point to the local coverage report files that have been generate just before.
For example, take a look at the following project:
Sample project that reads coverage reports
, and especially to its sonar-project.properties file, in which you will see that coverage report files are referenced using relative paths.
I have come across a very strange phenomenon:
Our CI system (Jenkins with Maven and Sonar plugin) runs Maven test builds on checkins (works fine) and runs nightly builds with Sonar intgration (ran fine until last week).
Now the nightly build compiles and runs the test (all green) and calls the Sonar plugin.
After the static analysis the Sonar start JaCoCo to measure the until test coverage; when JaCoco runs the unit tests (again), a lot of them run through as expected, but then first one test throws an exception (not thrown in the usual runs) about creating an object that is already there, and some tests later the whole process freezes in the try to rollback a transaction (normally not rolled back).
No component has been updated, and no suspicious code changes have been done around the failing tests.
Does anybody know, how JaCoCo treats the unit tests differently and how to solve this problem?
First read my FAQ
You are saying:
Now the nightly build compiles and runs the test (all green) and calls
the Sonar plugin... When JaCoco runs the unit tests (again)
What does sonar plugin do? Main purpose is to collect test reports, including jacoco and send it to sonar. It can run tests for you and immediately collect data too, but this is not what you expect I think. If you want just to send test coverage to sonar server, run your sonar plugin with sonar.dynamicAnalysis=reuseReports parameter. It won't run your tests again.
Your tests run good without sonar plugin and then sonar plugin breaks them? It could be possible if your tests require additional java parameters (-Dcom.stackoverflow=somevalue) that you don't pass to sonar plugin, or may be you use special maven profile (-Psomeprofile) and don't do that when calling sonar plugin
By the way:
when JaCoco runs the unit tests (again)
Jacoco can not run tests. Your unit tests have to be started with jacoco plugin/agent so jacoco will save *.exec file for you. If you see this file(most commonly in target dir) you have jacoco running correctly at least.