Maven failsafe is not complaining that an inexistent profile was specified - maven

I have two profiles in my pom:
<profiles>
<profile>
<id>functional-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<testSourceDirectory>test/test-functional/java</testSourceDirectory>
<includes>
<include>**/*FT.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>it-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<testSourceDirectory>test/test-it/java</testSourceDirectory>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
...
I can trigger each of these two profiles like this:
mvn failsafe:integration-test -Pfunctional-tests
mvn failsafe:integration-test -Pit-tests
But when I run this:
mvn failsafe:integration-test -PrandomWord
It triggers it-tests profile. I was wondering why and if there is a way to have failsafe plugin output something like unrecognised profile.
Thank you for your help
In case it matters, here is my failsafe-plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>

Answering your question about why it triggers it-tests. In fact, it does not activate any of the profiles, hence default plugin configuration is used which has **/*IT.java in include list. So, it runs all IT tests by default.
This is weird approach to manage plugin executions by profiles. I doubt there is a reasonable way to validate profile names as you describe. I would recommend another approach here.
Approach 1. Use <id> and cli with #
You could just specify two executions of the plugin with id and then you could do this: How to execute maven plugin execution directly from command line?
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>it-tests</id>
<phase>none</phase> <!-- detach this execution from default lifecycle -->
<configuration>
<testSourceDirectory>test/test-it/java</testSourceDirectory>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</execution>
<execution>
<id>functional-tests</id>
<phase>none</phase> <!-- detach this execution from default lifecycle -->
<configuration>
<testSourceDirectory>test/test-ft/java</testSourceDirectory>
<includes>
<include>**/*FT.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
Now you could execute it from command line:
mvn failsafe:integration-test#it-tests
mvn failsafe:integration-test#functional-tests
Update: No need to specify goals as it is relevant to lifecycle only, we type it in command line anyway.
Approach 2. Use <skip> and properties
Keep both executions as a part of lifecycle but control execution by providing skip flags. I.e. define two properties e.g. skip.tests.it=true, skip.tests.ft=true and add <skip>${skip.tests.ft}</skip> to relevant configuration sections. Then you could just do
# run with no tests by default
mvn verify
# run with only FT
mvn verify -Dskip.tests.ft=false
# run with all tests
mvn verify -Dskip.tests.ft=false -Dskip.tests.it=false
to run full lifecycle together with desired tests.

Related

Maven install is running failsafe

I am attempting to have my integration tests separated from the usual lifecycle, meaning that I do not want them to be executed during a mvn install. I have the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<excludes>
<exclude>**/*IT.java</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
<goals>
<goal>integration-test</goal>
</goals>
</plugin>
And my integration tests are all suffixed with IT. I was able to confirm that surefire does exclude the *IT.java tests, however it seems that failsafe is being triggered during mvn install anyway. What am I missing?
Thank you for your help
A typical configuration to handle the issue with running integration tests only if you like is a profile like the following:
<profiles>
<profile>
<id>run-its</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
By using the above configuration you can run
mvn clean verify
which will execute the unit tests. Using the following you can activate the integration tests:
mvn -Prun-its clean verify
Based on the default naming conventions the integration tests which will be executed by maven-failsafe-plugin should be named like *IT.java where as the unit tests can be name like *Test.java (will be executed via maven-surefire-plugin).

Is isolation of failsafe and surefire runs from each other using a skip approach possible?

The value of property skipITs when overridden on the command line appears to be ignored by maven if used in conjunction with -Dit.test=full.Classname. The specified failsafe test will not be run (case one).
As opposed to this specifiing skipITs without the it.test switch leads to running of existing failsafe tests (case two).
Background: Am trying to isolate surefire tests from failsafe tests runs by namely either running both types of them or one of each only. The maven calls are:
mvn -Pintegration -DskipTests=true -DskipITs=false -Dit.test=full.Classname verify
in the first case and:
mvn -Pintegration -DskipTests=true -DskipITs=false verify
in the second.
The relevant configuration (pom.xml, snippets only) being:
<properties>
<skipTests>false</skipTests>
<skipITs>false</skipITs>
</properties>
(those are defaults) and
<profiles>
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>custom<id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skipTests>${skipITs}</skipTests>
<skip>${skipITs}</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
Did you observe that too or have you eventually found a better working approach?
By default, the Surefire plugins runs during the test phase and usually you configure the Failsafe plugin to run during the integration-test and verify phase like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
To run the Surefire tests only, use
mvn clean test
To run both the Surefire and the Failsafe tests, use
mvn clean verify
You can completely skip any plugin by using the <skip> configuration option. If you configure the Surefire plugin like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<skip>${skip.surefire}</skip>
</configuration>
</plugin>
you can run only the Failsafe tests by calling
mvn clean verify -Dskip.surefire=true
If the integration test (it.test) has a non-default name I need to add an include pattern accordingly, like in (pom.xml snippet):
<profiles>
<profile>
<id>dbUnit</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>custom</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skipTests>${skipITs}</skipTests>
<skip>${skipITs}</skip>
<includes>
<include>**/*DbTest.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
But that appears valid only for the original approach. Also observe that the default inclusion patterns might need to be added too in case for your integration tests you followed different naming schemes.

Maven Profile does not run appropriate tests

I have configured a profile for running only integration tests but it is still running all tests.
This is the configuration:
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>test</goal>
</goals>
<phase>test</phase>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
I tried with regex like explained at http://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html but I got this not working.
How do I get only tests running that end with IT.java?
You have not reconfigured the surefire run, but instead added another one to the lifecycle. That way, surefire:test runs twice, once with default execution id (default-test) and once with your new execution (integration-test)
If you simply want to change the tests being run, you could use:
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
</profile>
Which would only reconfigure the existing execution.
Still, as khmarbaise already wrote in a comment, it makes much more sense to simply use the maven-failsafe-plugin. If you really wanted to skip normal unit test (why would you want that, anyway?), you could explicitly disable surefire in your profile (using skipTests-configuration).

How do I specify a separate maven goal for running (Cucumber) acceptance tests?

I have the following project structure:
MyProject
--src
--test
--acceptance
--step_definitions
--features
--unit
I would like to be able to run my cucumber tests (in test/acceptance) separately in Maven from the unit tests declared in test/unit, so that they can be run in different CI build plans etc. I am using cucumber-junit so the 'runners' for each acceptance test are written with JUnit.
Is this possible?
Is this possible?
Yes, it is possible. I believe you should separate your unit from the acceptance/integration tests having:
Slightly modified folders structure for both of these, placing your integration test files in the standard location of src/it:
MyProject/
src/main/java/ (SUT)
src/test/ (unit test code)
java/
resources/
src/it/ (acceptance/integration tests)
java/ (steps definitions)
resources/ (feature files)
Moreover, by design, different Maven plugins are intended for unit and integration tests:
for unit tests: maven-surefire-plugin
for acceptance/integration tests: maven-failsafe-plugin
You must also bind execution of maven-failsafe-pulgin. To run the integration tests separately, you can define a new profile:
<profiles>
<profile>
<id>acceptance-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
You will also need to configure the plugin to search the src/it directory tree for test cases.
The acceptance tests can be run afterwards using:
mvn clean verify -Pacceptance-tests
For complete sample, I'd suggest you to follow http://www.weblogism.com/item/334/integration-tests-with-cucumber-jvm-selenium-and-maven
The other answer suggested modifying your folder structure to have shared folder for integration and acceptance tests, but you can have the original folder structure as well. Also you mentioned in the comment you want to keep all three (including not-mentioned integration tests) separate, which is possible, though hackish.
Since you seem to have test/unit for unit tests and test/acceptance for acceptance test, I'm assuming test/integration for integration tests.
<profiles>
<profile>
<id>acceptance-test</id>
<build>
<plugins>
<plugin>
<!-- to run directly: mvn failsafe:integration-test -P acceptance-test -->
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12</version>
<configuration>
<testSourceDirectory>test/acceptance</testSourceDirectory>
<includes>
<include>**/*Acceptance.java</include>
</includes>
<excludes>
<exclude>**/*IT.java</exclude>
<exclude>**/*Test.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.9.1</version>
<executions>
<execution>
<id>add-test-source</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>test/unit</source>
<source>test/integration</source>
<source>test/acceptance</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12</version>
<configuration>
<testSourceDirectory>test/unit</testSourceDirectory>
</configuration>
</plugin>
<plugin>
<!-- to run directly: mvn failsafe:integration-test -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<testSourceDirectory>test/integration</testSourceDirectory>
</configuration>
<!-- execution below can be used, if tests are needed on
mvn:integration-test -->
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
Note though that separation only applies for sources: compiled files will all go to the same folder, and AFAIK that's something you can't change. This means you need to have naming strategy for your tests to separate them from each other.

How to run UnitTests in maven which is in src/test-integration/java folder

When we say mvn test, usual way is that maven will look for the tests present in src/test/java folder. But I have my tests in some different folder, namely src/integration-test/java. How do I run the tests present in this folder through command line?
Thanks in advance,
Manoj.
First you shouldn't run those integration test via the test life cycle, cause
pre-integration-test, integration-test and post-integration-test life cycle phase exist. Apart from that for integration tests the maven-failsafe-plugin is responsible.
There are several options to handle your situations. First you should follow the naming conventions for integration tests
<includes>
<include>**/IT*.java</include>
<include>**/*IT.java</include>
<include>**/*ITCase.java</include>
</includes>
which means to put the integration tests into the default folder src/test/java. If you have a multi-module build it would be the best having a separate module which contains the integration-tests only or you can go the path you decided to use a separate folder (which is not the best):
First you need to add the folder using the buildhelper-maven-plugin to get those integration tests being compiled like this:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.9.1</version>
<executions>
<execution>
<id>add-test-source</id>
<phase>process-resources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/integration-test/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
and you have to configure the maven-failsafe-plugin like this:
<project>
[...]
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.14.1</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
[...]
</project>
After you have configured you can run your integration tests via:
mvn verify
#khmarbaise is right with his recommendation (so +1 for that) but I want to answer your question without speculating about the reasons why the test source are located somewhere else.
If your tests are located in another directory than the standard src/test/java directory, the most simple solution is to change the default value of the testSourceDirectory configuration parameter which is defined in the Super POM.
e.g. for src/foobar/java use
<build>
<testSourceDirectory>src/foobar/java</testSourceDirectory>
</build>
then you can simply run mvn test to execute the tests.
More complex solution...
If you do not want to change the pom.xml configuration you can specifiy the testSourceDirectory parameter on the command line like this:
mvn -DtestSourceDirectory=src/foobar/java clean test
But be sure that your sources are compiled. Otherwise they will not be found and executed. In the above example the test sources are not placed at a location that gets compiled by default, so we nevertheless have to change the pom and add the directory to the list of test sources by using the buildhelper plugin:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.8</version>
<executions>
<execution>
<id>add-test-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/foobar/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
If you do not want to change the configuration of the default value in the pom and not want to pass the new directory at the commandline you have to configure the path in the maven-buildhelper-plugin and the maven-surefire-plugin in your pom.xml like this:
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.8</version>
<executions>
<execution>
<id>add-test-source</id>
<phase>generate-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>src/foobar/java</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.14.1</version>
<configuration>
<testSourceDirectory>src/foobar/java</testSourceDirectory>
</configuration>
</plugin>
</plugins>
</build>
Now again the simple usage of mvn test will execute the test at the not standard location.

Resources