How rerun failed test cases of cucumber-jvm in jenkins - maven

How to rerun failed test cases of cucumber-jvm in jenkins?
According to answers mentioned in this thread:
How to rerun failed test cases in cucumber-jvm?
There is different maven command to move and run scenarios for rerun.txt. How to execute them in Jenkins with separate maven command for rerun?

I use cucumber-serenity framework, which uses cucumber-jvm in the background to run everything. Here are the relevant parts of my pom.
I have everything in a separate project, not mixed with any other code. If this is not your case, the following might break your build!
I turn off unit tests:
<build>
<plugins>
<!-- no unit tests: skip anything named *Test -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.plugin.version}</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
This does not relate to your question, but I manage all my selenium browser drivers with:
<!-- docs: https://ardesco.lazerycode.com/testing/webdriver/2012/08/12/introducing-the-driver-binary-downloader-maven-plugin-for-selenium.html -->
<plugin>
<groupId>com.lazerycode.selenium</groupId>
<artifactId>driver-binary-downloader-maven-plugin</artifactId>
<version>${driver-binary-downloader.plugin.version}</version>
<configuration>
<rootStandaloneServerDirectory>${project.basedir}/selenium/bin</rootStandaloneServerDirectory>
<downloadedZipFileDirectory>${project.basedir}/selenium/zip</downloadedZipFileDirectory>
<customRepositoryMap>${project.basedir}/RepositoryMap.xml</customRepositoryMap>
<overwriteFilesThatExist>true</overwriteFilesThatExist>
</configuration>
<executions>
<execution>
<phase>pre-integration-test</phase>
<goals>
<goal>selenium</goal>
</goals>
</execution>
</executions>
</plugin>
I use the failsafe-plugin to run my integration tests:
<!-- integration tests: run everything named *IT -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${surefire.plugin.version}</version>
<configuration>
<systemPropertyVariables>
<!-- set by driver-binary-downloader-maven-plugin -->
<webdriver.chrome.driver>${webdriver.chrome.driver}</webdriver.chrome.driver>
<webdriver.gecko.driver>${webdriver.gecko.driver}</webdriver.gecko.driver>
</systemPropertyVariables>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The above will not rerun any failed tests, which is what you probably want when you run stuff locally on your machine.
On Jenkins only, I turn on the rerunning of failed tests:
<profiles>
<profile>
<id>jenkins</id>
<activation>
<property>
<name>env.JENKINS_HOME</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<dependencies>
<!-- docs: https://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html#Re-run_execution_in_Cucumber_JVM -->
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>${surefire.plugin.version}</version>
</dependency>
</dependencies>
<configuration>
<rerunFailingTestsCount>2</rerunFailingTestsCount>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>

I pasted the following in my Pom.xml file and I do not see any failed test case rerunning in Jenkins. Can you explain how to configure this Jenkins please
<profiles>
<profile>
<id>jenkins</id>
<activation>
<property>
<name>env.JENKINS_HOME</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<dependencies>
<!-- docs: https://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html#Re-run_execution_in_Cucumber_JVM -->
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.22.2</version>
</dependency>
</dependencies>
<configuration>
<rerunFailingTestsCount>2</rerunFailingTestsCount>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>

Related

Jacoco's report not generate

In my pom.xml
<profile>
<id>coverage</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>-Xmx4096m ${jacoco.agent.ut.arg}</argLine>
<!-- Specific to generate mapping between tests and covered code -->
<properties>
<property>
<name>listener</name>
<value>org.sonar.java.jacoco.JUnitListener</value>
</property>
</properties>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<argLine>-Xmx4096m ${jacoco.agent.it.arg}</argLine>
<!-- Specific to generate mapping between tests and covered code -->
<properties>
<property>
<name>listener</name>
<value>org.sonar.java.jacoco.JUnitListener</value>
</property>
</properties>
<!-- Let's put failsafe reports with surefire to have access to tests failures/success reports in sonar -->
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
</configuration>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.4.201502262128</version>
<executions>
<!-- Prepares a variable, jacoco.agent.ut.arg, that contains the info to be passed to the JVM hosting the code being tested. -->
<execution>
<id>prepare-ut-agent</id>
<phase>process-test-classes</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${project.build.directory}/jacoco-ut.exec</destFile>
<propertyName>jacoco.agent.ut.arg</propertyName>
<append>true</append>
</configuration>
</execution>
<!-- Prepares a variable, jacoco.agent.it.arg, that contains the info to be passed to the JVM hosting the code being tested. -->
<execution>
<id>prepare-it-agent</id>
<phase>pre-integration-test</phase>
<goals>
<goal>prepare-agent</goal>
</goals>
<configuration>
<destFile>${project.build.directory}/jacoco-it.exec</destFile>
<propertyName>jacoco.agent.it.arg</propertyName>
<append>true</append>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.codehaus.sonar-plugins.java</groupId>
<artifactId>sonar-jacoco-listeners</artifactId>
<version>1.5</version>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
Run
mvn clean verify
But in target folder not exist jacoco's coverage report
Since your profile is not activated by default, you need to specify -Pcoverage to your maven command to run this profile.
mvn clean verify -Pcoverage
You can also activate your coverage profile by adding the following to your pom.xml
<profile>
<id>coverage</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<!-- Rest of the Profile build configuration -->
</profile>

providing cucumber test report name through maven command

In my pom.xml I have created different profile for smoke and regression test using cucumber framework. The using the below maven command to run the tests. Following is an example of smoke test:
mvn clean test -Psmoke-test -Dcucumber.options="--plugin html:target/cucumber/smoke-test-report.html"
The test is running fine, but it is not generating the report even if I provided the -Dcucumber.options I am using a common runner class for cucumber, so I cannot give the file name as part of the #CucumberOptions annotation.
Following is the pom.xml confguration:
<profile>
<id>smoke-test</id>
<properties>
<test>AutomationTestTrigger</test>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<spring.profiles.active>automation-test</spring.profiles.active>
<cucumber.filter.tags>#smokeTest</cucumber.filter.tags>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>regression-test</id>
<properties>
<test>AutomationTestTrigger</test>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<spring.profiles.active>automation-test</spring.profiles.active>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</build>
</profile>
Your POM should be like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
<plugin>
<groupId>net.masterthought</groupId>
<artifactId>maven-cucumber-reporting</artifactId>
<version>5.5.0</version>
<executions>
<execution>
<id>execution</id>
<phase>verify</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<projectName>BDD-Automation</projectName>
<outputDirectory>${project.build.directory}/cucumber-report-html</outputDirectory>
<inputDirectory>${project.build.directory}</inputDirectory>
<jsonFiles>
<param>**/*.json</param>
</jsonFiles>
</configuration>
</execution>
</executions>
</plugin>
And command you can use like this:
mvn test -Dtest=Runner verify

JUnit - run tests in a category only if specific profile is active

I have the following tests:
FirstUnitTest.java
SecondUnitTest.java
FirstIntegrationTest.java
SecondIntegrationTest.java
The unit tests are not marked with a category.
The two integration tests are marked with #Category(IntegrationTests.class).
I want by default to run all tests EXCEPT for the integration tests.
If, however, a profile integration-tests-only is active, i want to run ONLY the integration tests.
I naively thought the following configuration would make this work:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludedGroups>com.example.IntegrationTests</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>integration-tests-only</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<groups>com.example.IntegrationTests</groups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
But while running the tests without a profile does exactly what I want - run only the unit tests, if I activate the integration-tests-only profile no tests run at all.
Any ideas what I'm doing wrong?
I assume that this happens because you include and exclude, and Maven merges the configurations and resolves to run nothing.
Consider this re-write of the config (did not run it so might have some minor issues):
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<executions>
<execution>
<id>default-test</id>
<configuration>
<skip>true</skip>
</configuration>
</execution>
<execution>
<id>unit-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>${skipUnitTests}</skip>
<excludedGroups>com.example.IntegrationTests</excludedGroups>
</configuration>
</execution>
<execution>
<id>integ-tests</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>${skipIntegTests}</skip>
<groups>com.example.IntegrationTests</groups>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>no-tests</id>
<properties>
<skipTests>true</skipTests>
</properties>
</profile>
<profile>
<id>unit-tests</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<skipUnitTests>false</skipUnitTests>
<skipIntegTests>true</skipIntegTests>
</properties>
</profile>
<profile>
<id>integ-tests</id>
<properties>
<skipUnitTests>true</skipUnitTests>
<skipIntegTests>false</skipIntegTests>
</properties>
</profile>
</profiles>

Automatically bumping up pom.xml with Maven release plugin

I am playing with the Maven release plugin and I am trying to bump up the pom.xml version automatically. I noticed when I do:
mvn --batch-mode release:prepare
it will automatically bump up the z version, i.e. if it is x.y.z it will be x.y.(z+1). Is there a way to bump up the y or x version without having to specify before hand what those versions should be?
I stumbled upon some blogposts on how to achieve this, and made this maven-setup which makes it possible to specify if you want to do a major, minor or patch-release:
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>${maven.build.helper.plugin.version}</version>
<executions>
<execution>
<id>parse-versions-for-release</id>
<phase>initialize</phase>
<goals>
<goal>parse-version</goal>
</goals>
<configuration>
<propertyPrefix>parsedVersion</propertyPrefix>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
<profiles>
<profile>
<id>major</id>
<activation>
<property>
<name>major</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<configuration>
<autoVersionSubmodules>true</autoVersionSubmodules>
<tagNameFormat>#{project.artifactId}-#{project.version}</tagNameFormat>
<useReleaseProfile>false</useReleaseProfile>
<releaseVersion>${parsedVersion.nextMajorVersion}.0.0</releaseVersion>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>minor</id>
<activation>
<property>
<name>minor</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<configuration>
<autoVersionSubmodules>true</autoVersionSubmodules>
<tagNameFormat>#{project.artifactId}-#{project.version}</tagNameFormat>
<useReleaseProfile>false</useReleaseProfile>
<releaseVersion>${parsedVersion.majorVersion}.${parsedVersion.nextMinorVersion}.0</releaseVersion>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
Here we use the build-helper-maven-plugin to figure out what the current (and next) version is, and we use maven profiles to specify if we want to do a major, minor or patch (default) release.
Source code can be found here: https://github.com/mortenberg80/maven-release-example
The blog posts I got inspired from were:
https://thihenos.medium.com/maven-release-plugin-a-simple-example-of-package-management-9926506acfb9
Maven release: next development version in batch mode

Starting of the Apache tomcat server before integration test

I've been looking for an solution for the last 4 days and raised this question as a bounty but still not getting my answer.
Where i've succeeded with the help pf pom.xml file:-
a) Starting the tomcat server manually using command i.e mvn tomcat7:run. This command also
help me deploying of my war file to tomcat server and starting the server.
b) Running my integration tests using testng.xml file configuration on eclipse.
Where i'm failed with the help pf pom.xml file:-
a) Automatically starting of tomcat server.
b) Running all the integration tests.
c) Stopping of tomcat server.
This question is posted by me but couldn't find the answer
Starting apache server before integration testing not working
Please help where i'm wrong.
Minimal POM
Here is a minimal POM file that I used to achieve what you want. If it doesn't work for you, please post the output of mvn -X clean verify like #BrennaFlood said. Configurations for tomcat7-maven-plugin and maven-failsafe-plugin taken from http://tomcat.apache.org/maven-plugin-2.2/run-mojo-features.html#Use_it_with_selenium_mojo and http://maven.apache.org/surefire/maven-failsafe-plugin/usage.html, respectively.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>tomcat-with-failsafe</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>war</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<prerequisites>
<maven>2.2.1</maven>
</prerequisites>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.8.8</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<version>2.2</version>
<executions>
<execution>
<id>tomcat7-run</id>
<goals>
<goal>run-war-only</goal>
</goals>
<phase>pre-integration-test</phase>
<configuration>
<fork>true</fork>
</configuration>
</execution>
<execution>
<id>tomcat7-shutdown</id>
<goals>
<goal>shutdown</goal>
</goals>
<phase>post-integration-test</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.17</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
It looks like you have tomcat start and stop bound to pre-integration-test and post-integration-test phases, but the TestNG stuff is being run during the test phase, which comes before the integration-test phases. Like the other responder said - you should be running:
mvn clean verify -X
... so that you're catching all the phases up through post-integration-test (and catching all the debug information for troubleshooting), but you should also bind your TestNG components to the integration-test phase.
I just want to add this for anyone that is looking to use maven + tomcat7 + testng. Basically my scenario is that some of our IT test needs the running application so they can send some REST call, but some of the IT does not require the server, I split the test cases in two different suites one for the IT that requires the server in the ServerRequiredIT.xml and others in NonServerRequiredIT.xml, based on that I create two profiles as it follows.
<profiles>
<profile>
<id>run-its</id>
<properties>
<build.profile.id>integration-test</build.profile.id>
<skip.integration.tests>false</skip.integration.tests>
<skip.unit.tests>true</skip.unit.tests>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<version>2.0</version>
<configuration>
<path>/</path>
</configuration>
<executions>
<execution>
<id>start-tomcat</id>
<phase>pre-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<fork>true</fork>
</configuration>
</execution>
<execution>
<id>stop-tomcat</id>
<phase>post-integration-test</phase>
<goals>
<goal>shutdown</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<argLine>-Xmx2048m</argLine>
<printSummary>true</printSummary>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<systemProperties>
<property>
<name>log4j.configuration</name>
<value>file:${project.build.testOutputDirectory}/resources/log4j-surefire.properties</value>
</property>
</systemProperties>
<suiteXmlFiles>
<suiteXmlFile>src/test/scripts/ServerRequiredIT.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>testNG-tests</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<argLine>-Xmx2048m</argLine>
<printSummary>true</printSummary>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<systemProperties>
<property>
<name>log4j.configuration</name>
<value>file:${project.build.testOutputDirectory}/resources/log4j-surefire.properties</value>
</property>
</systemProperties>
<suiteXmlFiles>
<suiteXmlFile>src/test/scripts/NonServerRequiredIT.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
To run the profiles I use mvn test -P 'nameOfProfile'. Important thing here is what the documentation said
The Failsafe Plugin is designed to run integration tests while the
Surefire Plugin is designed to run unit tests
Hope that helps.

Resources