In my project I have surefire as well as failsafe tests.
If I run with mvn clean install -DskipTests then both kinds of tests are skipped.
If I try to run a single failsafe test using -Dit.test=TestName then first all surefire tests run and then my it test.. but as the surefire tests take a long time this is not good.
I would like to skip the surefire tests in some cases but run the failsafe IT tests.
It's sometimes kind of confusing (annoying?) that, by default, -DskipITs=true will skip Failsafe but -DskipTests=true will skip both Surefire and Failsafe tests.
Anyway, you can change this behaviour by configuring the Surefire plugin to use a different 'skip' parameter. For example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<skip>${skipSurefire}</skip>
</configuration>
</plugin>
This will allow you to skip Surefire tests but not Failsafe tests by invoking:
mvn clean verify -DskipSurefire=true
Related
When release:clean release:prepare release:perform is executed , the tests are executed twice
Once at release:prepare , stage run-preparation-goals , task clean verify
And another time on release:perform ,stage run-perform-goals , task deploy
Is there any configuration option to make tests run only on first one and not being executed on perform?
BR
Yulian Oifa
That's because 2 full builds are run as part of the commands you issue.
The release:prepare section performs lots of checks on the code & repository, and does a full build to ensure all tests pass.
The release:perform section tags the repo, then pulls that tag. It performs a build based on that tag and the releases that build to your artefact store of choice (Nexus/Artifactory/wherever).
These steps are designed this way to ensure you don't pollute your repo with a tag on code that doesn't build, and tests are an integral part of your build. Once maven is happy your code is good to go, it creates the tag, and pulls it so it knows for sure that the code being released is the code referenced by the tag, and nothing else. Building that tag for release requires a full build, including the tests.
You can skip the tests on one or other of the steps, or on both, but you're missing the point of tests, and of the release plugin, if you do that.
You can override the goals parameter and add -DskipTests=true.
This parameter will skip the tests only on the perform part.
Eventually i had to change a logic.
I have executed mvn clean install release:clean release:prepare release:perform -Pmaven-release
and create a maven-release. As result the tests are executed only at install
BR
Yulian Oifa
<profile>
<id>maven-release</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<configuration>
<autoVersionSubmodules>true</autoVersionSubmodules>
<arguments>-Dmaven.test.skip</arguments>
</configuration>
</plugin>
</plugins>
</build>
</profile>
You can skip running the tests (managed by surefire plugin) with an explicitly specified parameter -DskipTests:
Something like this:
mvn release:perform -DskipTests
For intergration tests (managed by failsafe plugin) you can use a similar flag: -DskipITs
If you don't want even compile the tests you can use -Dmaven.test.skip=true (for both unit and integration tests)
How do I pass the test case to geb via maven commandline:
I have a test case class named GebishOrgSpec2.groovy
I want to pass this test case maven and let geb run it over the command line.
I am having successful build but no run of the test case
How do I solve this problem?
This is my pom.xml
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<includes>
<include>${inc.TestCase}</include>
</includes>
<systemPropertyVariables>
<geb.build.reportsDir>target/test-reports/geb</geb.build.reportsDir>
</systemPropertyVariables>
</configuration>
</plugin>
I have tried the following codes and they are all not running the testcase
mvn -Dinc.TestCase=GebishOrgSpec2.groovy test
mvn -Dinclude=GebishOrgSpec2.groovy test
mvn install -Dinc.TestCase=GebishOrgSpec2.groovy
mvn install -Dinclude=GebishOrgSpec2.groovy
I have tried these links
How to run Geb TestCases from windows command line?
Grails geb getting parameters from commandline
Geb exception when running tests in maven
they are all not working
System.getProperty("inc.TestCase") is returning null as well
I want to add that if I add the test case directly into the pom it does run successfully
can you please help
Don't use in-/excludes in your POM for something you want to do from the command line. It is much easier, both for
Surefire (unit tests) via mvn -Dtest=TestCircle test and
Failsafe (integration tests) via mvn -Dit.test=ITCircle verify
This works beautifully for Spock/Geb tests.
In my maven project I have a number of modules. Is it possible to turn off running unit test for some modules via command line options?
My project takes about 15 mins to run through all unit tests. I would like to speed up the overall build by running just the unit tests in the module I am working on. I do not want to go in and edit each individual pom.xml to achieve this.
I have tried a solution outlined here: Can I run a specific testng test group via maven? However the result is a lot of test failures in modules that I want to skip. I suppose 'group' is not the same concept of module?
To toggle unit tests on and off for an entire project use Maven Surefire Plugin's capability of skipping tests. There is a drawback with using skipTests from the command line. In a multi-module build scenario, this would disable all tests across all modules.
If you need more fine grain control of running a subset of tests for a module, look into using the Maven Surefire Plugin's test inclusion and exclusion capabilities.
To allow for command-line overrides, make use of POM properties when configuring the Surefire Plugin. Take for example the following POM segment:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.9</version>
<configuration>
<excludes>
<exclude>${someModule.test.excludes}</exclude>
</excludes>
<includes>
<include>${someModule.test.includes}</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<someModule.skip.tests>false</someModule.skip.tests>
<skipTests>${someModule.skip.tests}</skipTests>
<someModule.test.includes>**/*Test.java</someModule.test.includes>
<someModule.test.excludes>**/*Test.java.bogus</someModule.test.excludes>
</properties>
With a POM like the above you can execute tests in a variety of ways.
Run all tests (the above configuration includes all **/*Test.java test source files)
mvn test
Skip all tests across all modules
mvn -DskipTests=true test
Skip all tests for a particular module
mvn -DsomeModule.skip.tests=true test
Only run certain tests for a particular module (this example includes all **/*IncludeTest.java test source files)
mvn -DsomeModule.test.includes="**/*IncludeTest.java" test
Exclude certain tests for a particular module (this example excludes all **/*ExcludeTest.java source files)
mvn -DsomeModule.test.excludes="**/*ExcludeTest.java" test
Found a way to exclude on command line:
# Exclude one test class, by using the explanation mark (!)
mvn test -Dtest=!LegacyTest
# Exclude one test method
mvn verify -Dtest=!LegacyTest#testFoo
# Exclude two test methods
mvn verify -Dtest=!LegacyTest#testFoo+testBar
# Exclude a package with a wildcard (*)
mvn test -Dtest=!com.mycompany.app.Legacy*
This is from: https://blog.jdriven.com/2017/10/run-one-or-exclude-one-test-with-maven/
…and if you like to pass the parameter to maven release plugin in Hudson/Jenkins you have to use
-Darguments=-DskipTests
to get it work.
If you want to use Maven profiles:
http://maven.apache.org/guides/introduction/introduction-to-profiles.html
you might want to make it work doing something like this:
Skipping tests in some modules in Maven
I don't know if there is a supported command line option that does the same.
You also might try using environment properties directly, something as per this doc page:
http://maven.apache.org/plugins/maven-surefire-plugin/examples/skipping-test.html
i.e. something like:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12</version>
<configuration>
<skipTests>${moduleA.skipTests}</skipTests>
</configuration>
</plugin>
then using mvn -DmoduleA.skipTests=false test to test that one module.
I have the next pom.xml
<project>
...
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<argLine>${failsafeArgLine}</argLine>
<includes>
<include>**/dmg/*IT.java</include>
</includes>
<skipTests>${skipTests}</skipTests>
</configuration>
</execution>
</executions>
</plugin>
...
</project>
The problem is that when I'm taking off verify goal then the build is every time successful even if there was test failures.
And when I'm Taking off integration-test goal the integration tests simply do not run
Why failsafe plugin requires both integration-test and verify goals?
In Maven Failsafe plugin reference you can find simple answer why build is always successful
failsafe:integration-test runs the integration tests of an application.
failsafe:verify verifies that the integration tests of an application passed.
Without verify goal test results are not checked at all(but they are executed), so failsafe plugin requires integration-test goal to run tests, and verify to "verify" their results.
In Maven there are two types of test runner plug-ins, Surefire and Failsafe and they both serve different purposes. (I mention Surefire here because I feel it helps explain and contrast different types of failure in builds.)
Surefire
Surefire is a plug-in designed for your pre-deploy tests, such as Unit and Component tests. Surefire is guaranteed to return a failure (and therefore break your build) if any of the tests fail.
You want this behaviour in pre-deploy tests because you want to fail the build early if your tests fail, and as Unit and Component tests don't (or shouldn't) have any external dependencies it's safe to fail a build there and then.
Failsafe
Failsafe is a plug-in designed for post-deploy tests, such as Functional and Smoke tests.
Failsafe, as the name implies handles failures safely, by always returning a success exit code (as you have experienced in your build).
The reason behind this is for post-deploy tests you don't want test failures to immediately break the build because you may have spun up infrastructure, or seeded some test data into the system, which needs clean-up before the build is failed.
Therefore Failsafe splits the failure checking into the failsafe:verify goal which happens in the verify phase.
If you run Maven without parameters (assuming the normal jar lifecycle) you'll notice a phase between integration-test and verify called, post-integration-test. This is the phase which was designed for the tear down of any infrastructure or test data you may have injected into the system to run the tests.
... pre-integration-test, integration-test, post-integration-test, verify, ...
The phase before integration-test, pre-integration-test is designed for the said seeding and infrastructure creation.
To sum it all up, with pre-deploy tests you want to fail the build as early as possible, and so surefire guarantees this by exiting with a failure.
Whereas with post-deploy tests there are often external dependencies which need clean-up before you fail the build. For example in Kubernetes you may want to issue a rollback of the deploy and so failsafe:integration always returns a build success to guarantee you can perform the clean-up before failing the build by checking the status with failsafe:verify.
(The currently accepted answer just quotes the manual and doesn't explain why there are two commands and there are differences which is why I feel this answer is needed.)
I would like to know if there is a way to execute a goal when there is test failures?
Since maven stops its execution (fail fast mode) after encountering a test failure, is there any options to launch a goal when there is test failures?
Regards.
I've been searching for a way to do this as well, with not much success.
However, there is the following question which might provide some general hints:
Maven reporting plugins do not execute if a unit test failure occurs
The idea is that you would run mvn install (or whatever) first, and then run:
mvn -Dmaven.test.skip=true your-plugin:your-goal
This will let you run the build again without running tests, preserving the results for your perusal. Of course, this is only useful if your plugin is parsing the test results...
Though not recommended, by setting the surefire property testFailureIgnore to true, you can continue maven execution even when there are test failures.
...
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
...
<configuration>
<testFailureIgnore>true</testFailureIgnore>
...
</configuration>
</plugin>
...
Just do mvn clean install -DskipTests
I added this plugin in pom.xml and it worked well.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
If you want to the build to run KNOWING beforehand that there are going to be failures, you can use:
mvn <goal> -Dmaven.test.skip = true