failsafe+testng run tests that are not explicitly marked with the group name when I request to only run the group - maven

I marked only 1 test class with "http-integration-test" group:
#Test(groups = "http-integration-test", timeOut = 60_000)
public class MyIT
and then ran "mvn clean integration-test". I see multiple tests are run, not just those from this particular class.
why does failsafe maven plugin + testng combination ignore the request to only run "http-integration-test" group?
my pom.xml :
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
<argLine>-ea -Xmx256m</argLine>
</configuration>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<groups>http-integration-test</groups>
<includes>
<include>**/*IntegrationTest.java</include>
<include>**/*IT.java</include>
<include>**/IT*.java</include>
<include>**/TestServerConfigurator.java</include>
<include>*</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
and here's the parent pom file section:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<!-- the default configuration of surefire executes unit
tests implemented using JUnit. please see http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html
for details on these options -->
<configuration>
<!-- -XX:UseSplitVerifier: use legacy class verification; required until javassist can be updated to >= 3.18.2-GA -->
<argLine>-XX:-UseSplitVerifier -XX:MaxPermSize=512m</argLine>
<!-- disable execution of TestNG in case there are both
JUnit and TestNG tests available for execution -->
<testNGArtifactName>none:none</testNGArtifactName>
<excludes>
<!-- Integration Tests end in *IntegrationTest, by
convention -->
<exclude>**/*IntegrationTest.class</exclude>
<exclude>**/*IntegrationTestSuite.class</exclude>
</excludes>
<includes>
<!-- Unit Tests should end in *Test, by convention -->
<include>**/*Test.java</include>
<!-- Include *TestCase, because there are a large
number of tests named this way -->
<include>**/*TestCase.java</include>
</includes>
</configuration>
</plugin>

Related

Why 0% in Jacoco coverage for integration tests

I have setup maven project to get the integration test coverage in my it profile. But the jacoco report shows 0% for all the classes.
My main POM Jacoco integration setup section in the profile and failsafe plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-failsafe-plugin.version}</version>
<executions>
<!--
Invokes both the integration-test and the verify goals of the
Failsafe Maven plugin
-->
<execution>
<id>integration-tests</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<!--
Skips integration tests if the value of skip.it.tests
property is true
-->
<argLine>#{failsafeArgLine}</argLine>
<skipTests>${skip.it.tests}</skipTests>
<testFailureIgnore>true</testFailureIgnore>
<excludes>
<exclude>none</exclude>
</excludes>
<includes>
<!-- Include your Cucumber tests, as an example -->
<include>**/*CucumberTests.java</include>
<include>**/*IT.java</include>
</includes>
<systemPropertyVariables>
<it.server.port>${random.http.port}</it.server.port>
<it.jmx.port>${random.jmx.port}</it.jmx.port>
</systemPropertyVariables>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
.....
<profiles>
<profile>
<id>it</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<packaging.type>jar</packaging.type>
<run.profiles>it</run.profiles>
<skip.it.tests>true</skip.it.tests>
<skipTests>true</skipTests>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<configuration>
<append>true</append>
<!-- Sets the path to the file which contains the execution data. ;-->
<destFile>${user.dir}/target/jacoco-it.exec</destFile>
<!-- Sets the path to the file which contains the execution data.-->
<dataFile>${user.dir}/target/jacoco-it.exec</dataFile>
<!-- Sets the name of the property containing the settings for JaCoCo runtime agent.-->
<propertyName>failsafeArgLine</propertyName>
<excludes>
<exclude>**/*App*</exclude>
<exclude>**/test/**/*</exclude>
<exclude>**/it/**/*</exclude>
<exclude>**/config/*Config*</exclude>
</excludes>
<!-- Sets the output directory for the code coverage report.-->
<outputDirectory>${user.dir}/target/coverage-reports/jacoco-it</outputDirectory>
</configuration>
<executions>
<execution>
<id>pre-integration-test</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<!--
Ensures that the code coverage report for integration tests after
integration tests have been run.
-->
<execution>
<id>post-integration-test</id>
<phase>post-integration-test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
The profile setting of the module POM which has api tests where the command is executed
<profile>
<id>it</id>
<properties>
<skip.it.tests>false</skip.it.tests>
<spring.profile>it</spring.profile>
<skipTests>true</skipTests> -- This is for unit tests
</properties>
</profile>
However I can see jacoco-it.exec file has been generated and the report file as below.
Version:
<jacoco.plugin.version>0.8.3</jacoco.plugin.version>
<maven-failsafe-plugin.version>3.0.0-M5</maven-failsafe-plugin.version>
Command to execute tests: mvn clean verify -P it. This will spin up a server and run all integration tests (api end point tests) there.
Further when setup for unit tests that work as expected with surefire plugin with showing proper coverage in jacoco reports.

Unit tests are getting executed twice when performing mvn install

Unit tests are getting executed twice.
When i am removing goal report and phase prepare-package from maven plugins in pom, test are getting executed once but then coverage are not getting generated in the console.
But when i am adding goal report and phase prepare-package from maven plugins in pom,i am getting coverage in the console but unit tests are getting executed twice.
I need to have goal report and phase prepare-package in my pom in order to get coverage but need to run test cases only once. What is the way to to get the test case executed only once with coverage as well.
<plugins>
<!-- Configure maven-compiler-plugin to use the desired Java version -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<!-- Use build-helper-maven-plugin to add Scala source and test source
directories -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>add-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>src/main/scala</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-test-source</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/test/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<!-- Use scala-maven-plugin for Scala support -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<!-- Need to specify this explicitly, otherwise plugin won't be called
when doing e.g. mvn compile -->
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- disable surefire -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<plugin>
<groupId>org.scoverage</groupId>
<artifactId>scoverage-maven-plugin</artifactId>
<version>${scoverage.plugin.version}</version>
<configuration>
<aggregate>true</aggregate>
<highlighting>true</highlighting>
<scalacPluginVersion>1.3.0</scalacPluginVersion>
<minimumCoverage>30</minimumCoverage>
<failOnMinimumCoverage>false</failOnMinimumCoverage>
</configuration>
<executions>
<execution>
<goals>
<goal>report</goal> <!-- or integration-check -->
</goals>
<phase>prepare-package</phase> <!-- or any other phase -->
</execution>
</executions>
</plugin>
<!-- enable scalatest -->
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<argLine>
-javaagent:"${settings.localRepository}"/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
</argLine>
<forkMode>once</forkMode>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>WDF TestSuite.txt</filereports>
<!-- The scalatest-maven-plugin seems broken for the spanScaleFactor,
so pass it via system property, instead -->
<!-- <spanScaleFactor>${scalatest.span.scale.factor}</spanScaleFactor> -->
<systemProperties>
<spanScaleFactor>${scalatest.span.scale.factor}</spanScaleFactor>
</systemProperties>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>enter code here
You need to do 2 things:
Add following line in 'scoverage-maven-plugin' plugin's 'configuration' section.
<additionalForkedProjectProperties>skipTests=false</additionalForkedProjectProperties>
Set following property in your root pom.xml
<properties>
<!-- Add other properties here... -->
<!-- skipTests is set to 'true' here to avoid duplicate runs of all test cases. It's set to false in scoverage-maven-plugin below. -->
<skipTests>true</skipTests>
</properties>

Reports are not generated when the build is failed in Maven Cucumber Reports

Reports are generating successfully when the build is successful but when there are any failed cases which cause build failure, reports are not generated.
checkBuildResult is already set to false
pom file plugin
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>3.13.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>Simplify360 Automation Test Report</projectName>
<outputDirectory>${project.build.directory}/site/cucumber-reports</outputDirectory>
<cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>
<!-- <jsonFiles>
<param>${project.build.directory}/cucumber.json</param>
</jsonFiles> -->
<!-- <parallelTesting>false</parallelTesting> -->
<buildNumber>8.4.1.2</buildNumber>
<checkBuildResult>false</checkBuildResult>
</configuration>
</execution>
</executions>
</plugin>
And the runner class is as below,
#RunWith(Cucumber.class)
#CucumberOptions(
features = {"classpath:features"},
plugin = {"pretty","json:target/cucumber.json"},
tags = {"#currentTest"},
glue={"helpers","stepDefinitions"},
monochrome = true
)
public class RunCukesTest{
}
Add the following configuration to the sure fire plugin. It will not stop the maven execution after the failure. Then it will generate the report.
<testFailureIgnore>true</testFailureIgnore>
as given below with your existing configuration.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
There is flag present for same which is as below:
<checkBuildResult>true</checkBuildResult>
<!-- Set true to fail build on test failures -->
<!-- Set false to pass build on test failures -->
You need to set in the configuration tag as below:
<configuration>
<projectName>oasys-confirmations</projectName>
<outputDirectory>${project.build.directory}</outputDirectory>
<cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>
<checkBuildResult>true</checkBuildResult>
</configuration>
New config:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>3.20.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>cucumber-jvm-example</projectName>
<!-- output directory for the generated report -->
<outputDirectory>${project.build.directory}</outputDirectory>
<!-- optional, defaults to outputDirectory if not specified -->
<inputDirectory>${project.build.directory}/</inputDirectory>
<jsonFiles>
<!-- supports wildcard or name pattern -->
<param>**/*.json</param>
</jsonFiles>
<!-- optional, defaults to outputDirectory if not specified -->
<classificationDirectory>${project.build.directory}/</classificationDirectory>
<classificationFiles>
<!-- supports wildcard or name pattern -->
<param>sample.properties</param>
<param>other.properties</param>
</classificationFiles>
<parallelTesting>false</parallelTesting>
<checkBuildResult>true</checkBuildResult>
</configuration>
</execution>
</executions>
</plugin>
Old Config working till 3.16.0 version:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>3.16.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>>cucumber-jvm-example</projectName>
<outputDirectory>${project.build.directory}</outputDirectory>
<cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>
<checkBuildResult>true</checkBuildResult>
</configuration>
</execution>
</executions>
</plugin>
NOTE If you get same problem After this configuration then
1. Run RunnerFile Normally TestNG
2. Run Pom.xml as Maven install.
3. Check target folder there is a TagName.html file open it and view the result.

Include testing test groups for failsafe (integration testing) only but exclude them from surefire

The following config does not work. No test is in scope for the goal integration-test.
In case it is unclear, What should happen is that what I do a mvn integration-test the failsafe plugin should run my test. But the surefire plugin configuration is excluding the test. If I uncomment the surefire config block the test is run during the integration-test goal.
Maven config:
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<groups>spring-container-sanity</groups>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludedGroups>spring-container-sanity</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
A Java Test class
#SpringApplicationConfiguration(TestApplication.class)
#TestPropertySource("/test.properties")
public class SimpleTest extends AbstractTestNGSpringContextTests {
#Test(groups = "spring-container-sanity")
public void isHessianServiceExported() throws Exception {
/*...*/
}
}
I don't know why the tests are run when surefire is disabled. Failsafe has a file naming convention for integration tests, if the tests you want to run do not follow this they won't be in scope and the group rule will have nothing to match. So, in this case to get failsafe to resolve the tests it should run correctly you first need to add an include filter. This build block will work:
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<includes>
<include>**/*.java</include>
</includes>
<groups>spring-container-sanity</groups>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludedGroups>spring-container-sanity</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
When annotating your test, make sure to define an array of groups, even if its only a single one.
#Test(groups = {"spring-container-sanity"})

Multiple test configurations for surefire plugin

I have an issue with my pom:
There are two types of tests running within the project:
Main set of tests using TestNG and JUnit code coverage tests. What I would like is to run JUnit tests on test-compile phase and do not run TestNG tests (which are run on test phase) if previous failed.
As of now I have the following which runs only TestNG:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<argLine>
-javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/1.8.4/aspectjweaver-1.8.4.jar
</argLine>
<properties>
<property>
<name>listener</name>
<value>org.uncommons.reportng.HTMLReporter, org.uncommons.reportng.JUnitXMLReporter</value>
</property>
</properties>
<suiteXmlFiles>
<suiteXmlFile>src/test/_all.xml</suiteXmlFile>
</suiteXmlFiles>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
I tried to play with <execution> tag but I can't provide path to TestNG suite then. Any ideas how to combine these tests using surefire plugin and specify different goals?
Solution was to add maven-failsafe-plugin separately from maven-surefire-plugin.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.17</version>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.17</version>
</dependency>
</dependencies>
<configuration>
<includes>
<include>junitcodecoverage/**Test.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<phase>test-compile</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>

Resources