Jacoco coverage report of JUnit tests for a Gradle multiproject - gradle

I am having difficulty combining Jacoco coverage report for a Gradle multi-project. For simplicity, I have created a simplified version that is structured as follows. I have the example project on GitHub.
+ root
+ common-a
+ common-b
+ project-a
+ project-b
The root, project-a, and project-b are Spring Boot applications. For some context, a project is meant to be a micro-service. Thus, the root project is simply running all micro-services in one process.
Not all projects have unit and/or integration tests. common-a subproject does not have any tests; common-b, project-a, and project-b projects have unit and integration tests; the root project has only integration tests.
The Jacoco coverage reports for subprojects are correct. It is able to combine both unit and integration tests into one cohesive report. However, the root project does not produce the cohesive report correctly.
I have tried a number of solutions with any success. Most of the solutions I found assume that the root project is simply a logical container and only the subprojects has test code. The results of the solutions I tried fall into one of the following three categories.
No Jacoco report generated at all; the build/reports/jacoco directory is missing
Partial Jacoco report for only sources in root project
Full Jacoco coverage report for root and subproject but 0% coverage; it is supposed to be 100% coverage
The following is the current build.gradle file.
buildscript {
repositories {
jcenter()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:1.3.5.RELEASE")
}
}
// settings for all projects
allprojects {
apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'io.spring.dependency-management'
apply plugin: 'jacoco'
group 'io.github.chriszhong'
sourceCompatibility = JavaVersion.VERSION_1_8
targetCompatibility = JavaVersion.VERSION_1_8
repositories {
jcenter()
}
dependencyManagement {
imports {
mavenBom 'io.spring.platform:platform-bom:2.0.5.RELEASE'
}
}
sourceSets {
integrationTest {
compileClasspath += main.output + test.output
runtimeClasspath += main.output + test.output
java {
srcDir 'src/test-integration/java'
}
resources {
srcDir 'src/test-integration/resources'
}
}
}
configurations {
integrationTestCompile {
extendsFrom testCompile
}
integrationTestRuntime {
extendsFrom testRuntime
}
}
idea {
module {
testSourceDirs += sourceSets.integrationTest.java.srcDirs
testSourceDirs += sourceSets.integrationTest.resources.srcDirs
}
}
task integrationTest(type: Test) {
check {
dependsOn integrationTest
}
testClassesDir sourceSets.integrationTest.output.classesDir
classpath = sourceSets.integrationTest.runtimeClasspath
}
task testReport(type: TestReport) {
dependsOn check
destinationDir file("${reporting.baseDir}/unified/html")
reportOn test, integrationTest
}
task jacocoMerge(type: JacocoMerge) {
dependsOn check
executionData test, integrationTest
// temporary hack to fix a bug requiring that there be at least one test case in each source set for coverage data to be generated;
// it should be if there is at least one test case in any of the source sets
executionData = files(executionData.findAll({ it.exists() }))
}
tasks.withType(Test) {
reports {
html {
destination file("${reporting.baseDir}/${task.name}/html")
}
}
}
tasks.withType(JacocoReport) {
dependsOn jacocoMerge
executionData jacocoMerge.destinationFile
// temporary hack to fix a bug requiring that there be at least one test case in each source set for coverage data to be generated;
// it should be if there is at least one test case in any of the source sets
// executionData = files(executionData.findAll({ it.exists() }))
}
}
apply plugin: 'spring-boot'
version '1.0-SNAPSHOT'
dependencies {
compile project(':project-a')
compile project(':project-b')
testCompile 'junit:junit'
integrationTestCompile 'org.springframework.boot:spring-boot-starter-test'
}
testReport {
dependsOn check, subprojects.testReport
reportOn test, integrationTest, subprojects.test, subprojects.integrationTest
}
jacocoMerge {
dependsOn check, subprojects.jacocoMerge
executionData test, integrationTest, subprojects.jacocoMerge.destinationFile
// temporary hack to fix a bug requiring that there be at least one test case in each source set for coverage data to be generated;
// it should be if there is at least one test case in any of the source sets
executionData = files(executionData.findAll({ it.exists() }))
}
//jacocoTestReport {
// dependsOn jacocoMerge
// executionData test, integrationTest, subprojects.test, subprojects.integrationTest
// executionData jacocoMerge.destinationFile
// temporary hack to fix a bug requiring that there be at least one test case in each source set for coverage data to be generated;
// it should be if there is at least one test case in any of the source sets
// executionData = files(executionData.findAll({ it.exists() }))
// sourceDirectories = files(sourceSets.main.java.srcDirs, subprojects.sourceSets.main.java.srcDirs)
// classDirectories = files(sourceSets.main.output.classesDir, subprojects.sourceSets.main.output.classesDir)
//}
task wrapper(type: Wrapper) {
gradleVersion '2.14'
distributionUrl "https://services.gradle.org/distributions/gradle-${gradleVersion}-all.zip"
}
I have been banging my head at this problem for the past week and is at my wits end. I greatly appreciate any help I can get in resolving this problem.

Related

How do I get a Jacoco coverage report using gradle plugin when all my tests are in a separate submodule

I'm having trouble setting up jacoco coverage for my java project since I'm new to gradle. My final goal is to connect this to sonarqube. All my tests are in a separate module
structure:
./build.gradle
settings.gradle
./submodule1/build.gradle
./submodule1/src/main/java/prismoskills/Foo.java
./submodule2/build.gradle
./submodule2/src/main/java/com/project/prismoskills/Bar.java
./test/build.gradle
./test/src/test/java/prismoskills/TestFooBar.java
One way I can think of is to set additionalSourceDirs in test module and enable jacoco only in root and test module.
The problem with this approach is that my project has a lot of sub modules(which I haven't shown here) and I am having trouble passing additionalsourcedirs to test module's JacocoReport task in an automated way.
Also it looks like this use case can be handled in maven easily by referring to this
https://prismoskills.appspot.com/lessons/Maven/Chapter_06_-_Jacoco_report_aggregation.jsp
Any leads on how to proceed further with gradle will be appreciated. Thanks in advance
gradle version: 6.4
jacoco gradle plugin version: 0.8.5
I think the following solution should solve your problem. The idea is that:
JaCoCo exec file is generated for every project
at the end one XML report with all data is generated
It does the same for JUnit reports because it is easier to see all tests reports together in the root project instead of navigating between directories.
plugins {
id 'base'
id 'org.sonarqube' version '3.0'
}
allprojects {
apply plugin: 'jacoco'
apply plugin: 'project-report'
// ...
jacoco {
toolVersion = 0.8.5
}
}
subprojects {
// ...
test {
reports.html.enabled = false
useJunitPlatform()
finalizedBy jacocoTestReport
}
jacocoTestReport {
dependsOn test
reports.html.enabled = false
}
}
// ...
task testReport(type: TestReport) {
destinationDir = file("${buildDir}/reports/test")
reportOn subprojects*.test
}
task jacocoTestReport(type: JacocoReport) {
subprojects { subproject ->
subproject.tasks.findAll { it.extensions.findByType(JacocoTaskExtension) }.each { extendedTask ->
configure {
sourceSets subproject.sourceSets.main
if (file("${subproject.buildDir}/jacoco/${extendedTask.name}.exec").exists()) {
executionData(extendedTask)
}
}
}
}
reports.xml.enabled = true
}
rootProject.getTasksByName('test', true).each {
it.finalizedBy(testReport)
it.finalizedBy(jacocoTestReport)
}
This line
if (file("${subproject.buildDir}/jacoco/${extendedTask.name}.exec").exists()) {
is added to prevent build failures when some subprojects don't have tests at all.
Following can be defined in the build.gradle of root project. Jacoco plugin will be applied to all the submodules.
subprojects{
plugins{
id 'java'
id 'jacoco'
}
test.finalizedBy jacocoTestReport
}

Gradle Multi-Project Integration Tests with Junit5

How can I create integrations tests using Gradle 5.2 and JUnit 5.3 in a multi-project build file?
This is my current build file:
buildscript {
repositories {
maven {
url "https://plugins.gradle.org/m2/"
}
}
dependencies {
classpath "gradle.plugin.com.github.spotbugs:spotbugs-gradle-plugin:1.6.10"
classpath 'org.owasp:dependency-check-gradle:5.0.0-M2'
}
}
allprojects {
defaultTasks 'clean', 'build', 'publish', 'installDist'
group = 'coyote'
version = '0.1.0'
}
subprojects {
apply plugin: 'java'
apply plugin: 'jacoco'
apply plugin: "com.github.spotbugs"
apply plugin: 'org.owasp.dependencycheck'
apply plugin: 'maven-publish'
apply plugin: 'application'
apply plugin: 'eclipse'
apply plugin: 'idea'
repositories {
jcenter()
mavenCentral()
}
dependencies {
testImplementation ("org.junit.jupiter:junit-jupiter-api:5.3.2")
testRuntimeOnly ("org.junit.jupiter:junit-jupiter-engine:5.3.2")
spotbugsPlugins ("com.h3xstream.findsecbugs:findsecbugs-plugin:1.8.0")
}
tasks.withType(Test) {
useJUnitPlatform()
}
apply from: "$rootDir/integration-test.gradle"
check.dependsOn jacocoTestCoverageVerification
check.dependsOn dependencyCheckAnalyze
spotbugs {
effort = "max"
reportLevel = "low"
ignoreFailures = true
showProgress = true
}
jacocoTestReport {
reports {
xml.enabled true
html.enabled true
}
}
check.dependsOn jacocoTestReport
tasks.withType(com.github.spotbugs.SpotBugsTask) {
reports {
xml.enabled false
html.enabled true
}
}
}
The integration-test.gradle file I am applying on line 46 contains the following:
sourceSets {
itest {
compileClasspath += sourceSets.main.output + configurations.testCompile
runtimeClasspath += output + compileClasspath + configurations.testRuntime
}
}
idea {
module {
testSourceDirs += sourceSets.itest.java.srcDirs
testResourceDirs += sourceSets.itest.resources.srcDirs
scopes.TEST.plus += [configurations.itestCompile]
}
}
task itest(type: Test) {
description = 'Runs the integration tests.'
group = 'verification'
testClassesDirs = sourceSets.itest.output.classesDirs
classpath = sourceSets.itest.runtimeClasspath
outputs.upToDateWhen { false }
}
Everything seems to work well within Intellij, but JUnit5 is not being added to the classpath, so any tests in the itest directories cannot find the JUnit libraries.
Running gradlew itest fails with similar results with the JUnit classes not being found.
I have tried to add use useJUnitPlatform() directly in the itest task, but with no success. I have also tried placing everything in the build.gradle file with no success.
Eventually, I'd like to use the same pattern to define load and security testing, running them separately as part of a CI/CD pipeline so placing everything neatly in their own directories under their respective projects is preferred to mixing everything in one test directory and using tags.
This is also helping other teams model CI/CD practices, so using Gradle 5 and JUnit 5 as designed is preferred as opposed to work-arounds or hacks to make things work. It's acceptable to learn that these versions don't work together or that there are currently defects\issues preventing this approach. Hopefully this is not the issue and I'm just missing something simple.
The solution is I needed to include the JUnit 5 dependencies myself. The following is the correct integration-test.gradle file:
sourceSets {
itest {
compileClasspath += sourceSets.main.output + configurations.testCompile
runtimeClasspath += output + compileClasspath + configurations.testRuntime
}
}
idea {
module {
testSourceDirs += sourceSets.itest.java.srcDirs
testResourceDirs += sourceSets.itest.resources.srcDirs
scopes.TEST.plus += [configurations.itestCompile]
}
}
dependencies {
itestImplementation ("org.junit.jupiter:junit-jupiter-api:5.3.2")
itestRuntimeOnly ("org.junit.jupiter:junit-jupiter-engine:5.3.2")
}
task itest(type: Test) {
description = 'Runs the integration tests.'
group = 'verification'
testClassesDirs = sourceSets.itest.output.classesDirs
classpath = sourceSets.itest.runtimeClasspath
outputs.upToDateWhen { false }
}

CorDapp JaCoCo Code Coverage

I have a Corda based project with several CorDapp sub projects. I've been looking to add JaCoCo code coverage to this project. I'm looking to have a single code coverage report draw in an aggregate report of all the subproject JaCoCo reports.
To add JaCoCo to a maven project with several maven sub projects, I followed this blog entry https://lkrnac.net/blog/2016/10/aggregate-test-coverage-report/. After we ran the build ./gradlew clean test and got our reports, one of our team members noted that the whitelists weren't being created properly anymore when we ran ./gradlew clean deployNodes.
I've gone back to the base Kotlin CorDapp template found here https://github.com/corda/cordapp-template-kotlin to rule out if it's something we've done wrong with our project structure/gradle. Without JaCoCo added, I see all the whitelist entries I would expect. Once I add the JaCoCo code, I only see the 5 default Corda whitelist entries, and none of my added contract entries.
I'm using JaCoCo version 0.8.1 and coveralls version 2.6.3. The changes I've made are all within the build.gradle file for the root directory cordapp-template-kotlin:
subprojects {
repositories {
mavenCentral()
}
apply plugin: 'jacoco'
apply plugin: 'java'
group = 'net.lkrnac.blog'
version = '1.0-SNAPSHOT'
sourceCompatibility = JavaVersion.VERSION_1_8
targetCompatibility = JavaVersion.VERSION_1_8
dependencies {
testCompile("junit:junit:4.12")
}
jacoco {
toolVersion = jacoco_version
}
//command for generating subproject coverage reports
jacocoTestReport {
reports {
xml.enabled false
csv.enabled false
html.destination file("${buildDir}/jacocoHtml")
}
}
}
def publishedProjects = subprojects.findAll()
task jacocoRootReport(type: JacocoReport, group: 'Coverage reports') {
description = 'Generates an aggregate report from all subprojects'
dependsOn(publishedProjects.test)
additionalSourceDirs = files(publishedProjects.sourceSets.main.allSource.srcDirs)
sourceDirectories = files(publishedProjects.sourceSets.main.allSource.srcDirs)
classDirectories = files(publishedProjects.sourceSets.main.output)
executionData = files(publishedProjects.jacocoTestReport.executionData)
doFirst {
executionData = files(executionData.findAll { it.exists() })
}
reports {
html.enabled = true // human readable
xml.enabled = true // required by coveralls
}
}
coveralls {
sourceDirs = publishedProjects.sourceSets.main.allSource.srcDirs.flatten()
jacocoReportPath = "${buildDir}/reports/jacoco/jacocoRootReport/jacocoRootReport.xml"
}
tasks.coveralls {
dependsOn jacocoRootReport
}
I believe that the problem is coming from simply adding a task where JacocoReport as a parameter. Any thoughts how I could proceed to have both code coverage, along with building my whitelists correctly?
I have managed to find how to fix the coverage/whitelisting issue. I started stripping away what seemed to be unnecessary code within the subprojects spec, and found that removing everything except the apply plugin:, jacoco, and jacocoTestReport commands yielded both the root Jacoco code coverage, along with the necessary whitelisting. I didn't need to change any of the other code above to get the whitelisting to work.
For reference, subprojects now looks like this:
subprojects {
apply plugin: 'jacoco'
apply plugin: 'kotlin'
jacoco {
toolVersion = jacoco_version
}
//command for generating subproject coverage reports
jacocoTestReport {
reports {
xml.enabled false
csv.enabled false
html.destination file("${buildDir}/jacocoHtml")
}
}
}

Allure report is empty when use Allure2+Junit5+Gradle+Selenide

My build.gradle is:
apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'org.junit.platform.gradle.plugin'
apply plugin: 'io.qameta.allure'
defaultTasks 'clean', 'test'
ext.junitJupiterVersion = '5.0.0-M4'
ext.selenideVersion = '4.4.3'
compileTestJava {
sourceCompatibility = 1.8
targetCompatibility = 1.8
options.encoding = 'UTF-8'
options.compilerArgs += "-parameters"
}
compileJava.options.encoding = 'UTF-8'
tasks.withType(JavaCompile) {
options.encoding = 'UTF-8'
}
repositories {
jcenter()
mavenCentral()
}
buildscript {
repositories {
jcenter()
mavenCentral()
}
dependencies {
classpath 'org.junit.platform:junit-platform-gradle-plugin:1.0.0-M4'
classpath 'io.qameta.allure:allure-gradle:2.3'
}
}
allure {
aspectjweaver = true
autoconfigure = true
version = '2.1.1'
}
configurations {
agent
}
dependencies {
// JUnit5
compile("org.junit.jupiter:junit-jupiter-api:${junitJupiterVersion}")
compile("org.junit.jupiter:junit-jupiter-engine:${junitJupiterVersion}")
// Selenide
compile("com.codeborne:selenide:${selenideVersion}") {
exclude group: 'junit'
}
// Allure
agent 'org.aspectj:aspectjweaver:1.8.10'
compile 'ru.yandex.qatools.allure:allure-junit-adaptor:1.4.23'
compile 'io.qameta.allure:allure-junit5:2.0-BETA6'
}
junitPlatform {
platformVersion = "1.0.0-M5"
enableStandardTestTask = true
}
task runJupiter(type: JavaExec) {
jvmArgs '-ea'
jvmArgs "-javaagent:${configurations.agent.singleFile}"
classpath = project.sourceSets.test.runtimeClasspath
main 'org.junit.platform.console.ConsoleLauncher'
args '--scan-class-path'
args "--reports-dir=${buildDir}/allure-results"
finalizedBy 'allureReport'
}
test.dependsOn runJupiter
Tests are finished successfully and three folders are created automatically:
{projectDir}\allure-results with .json file
{projectDir}\build\test-results\junit-platform with TEST-junit-jupiter.xml file
{projectDir}\build\reports\allure-report
I tried to open .json and .xml result locally via allure command line (CLI). The allure report is opened but it is blank:
this is a report view
I suppose my mistake in gradle dependencies. I quite confused which libraries and versions should be used for JUnit5+Allure2+Gradle+Selenide+Java8?
The JUnit Platform Gradle plugin does currently not use the test task (it needs changes in Gradle core in order to do so). Thus, things like test.doFirst {...} are not going to work.
Instead of using the plugin, you should be able to create your own task that runs the ConsoleLauncher and add the JVM agent there. See https://stackoverflow.com/a/43512503/6327046 for an example.

Gradle skipping jacoco coverage plugin

I know there are a lot of similar questions for this on stackexchange but this is driving me crazy and nothing from any other answers have helped me. I've used just about every force flag for gradle I can find.
My Gradle version is 2.2.1
My build.gradle
buildscript {
ext {
springBootVersion = '1.5.3.RELEASE'
}
repositories {
mavenCentral()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
classpath("se.transmode.gradle:gradle-docker:1.2")
}
}
apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'org.springframework.boot'
apply plugin: 'application'
apply plugin: 'docker'
apply plugin: 'jacoco'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
ext {
springCloudVersion = 'Dalston.RELEASE'
}
test {
include 'src/test/java'
finalizedBy jacocoTestReport
}
jacocoTestReport {
reports {
xml.enabled true
csv.enabled false
html.enabled false
xml.destination "${buildDir}/jacoco"
}
}
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
}
}
When I run gradle test or gradle clean test jacocoTestReport or gradle clean --re-run etc etc I get the same results.
gradle test
:compileJava
:processResources
:classes
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test UP-TO-DATE
:jacocoTestReport SKIPPED
I have only 1 test in the src/test/java location.
Why will it not generate the reports and run? I'm at a loss.
Remove the wrong include statement in your test configuration.
That seems to cause the test task to always be up-to-date (not executing any test as the include pattern isn't matched)
The jacocoTestReport will be skipped if there's no execution data available that is needed to generate the report. This execution data should be generated while the test task is executed. In your example we see that the test task is always marked as up-to-date (because of the invalid include statement) causing the build to never produce any input for the jacocoTestReport task.
For Spring 2.5 Users, who got stuck with it for hours -just like myself.
I was not having the exec file generated.
And because of that ,
I found that the jacocoTestReport was simply "skipped".
I got it fixed by adding :
test {
useJUnitPlatform()
finalizedBy jacocoTestReport // report is always generated after tests run
}
That's because I'm using Junit5 with spring boot 2.X
It was a bit tricky to find that JaCoCo by default skips every jacocoTestCoverageVerification task in case if tests are run trough a custom named task (not just test).
For example, tests were run via unitTest in my project, JaCoCo did create build/jacoco/unitTest.exec during their execution, yet it kept searching for test.exec during verification. ¯\_(ツ)_/¯
The workaround is to override executionData path for JaCoCo gradle tasks:
apply plugin: 'jacoco'
jacoco {
toolVersion = "0.8.8"
}
def coverageFile = layout -> layout.buildDirectory.file('jacoco/coverage.exec').get()
tasks.named('jacocoTestCoverageVerification') {
getExecutionData().setFrom(files(coverageFile(layout)))
violationRules {
rule {
limit {
minimum = project.properties['testCoverageThreshold'] ?: 0.5
}
}
}
}
tasks.named('jacocoTestReport') {
getExecutionData().setFrom(files(coverageFile(layout)))
}
tasks.named('unitTest') {
jacoco {
destinationFile = coverageFile(layout).asFile
}
finalizedBy jacocoTestReport, jacocoTestCoverageVerification
}

Resources