I have configured a profile for running only integration tests but it is still running all tests.
This is the configuration:
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>test</goal>
</goals>
<phase>test</phase>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
I tried with regex like explained at http://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html but I got this not working.
How do I get only tests running that end with IT.java?
You have not reconfigured the surefire run, but instead added another one to the lifecycle. That way, surefire:test runs twice, once with default execution id (default-test) and once with your new execution (integration-test)
If you simply want to change the tests being run, you could use:
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
</profile>
Which would only reconfigure the existing execution.
Still, as khmarbaise already wrote in a comment, it makes much more sense to simply use the maven-failsafe-plugin. If you really wanted to skip normal unit test (why would you want that, anyway?), you could explicitly disable surefire in your profile (using skipTests-configuration).
Related
I have two profiles in my pom:
<profiles>
<profile>
<id>functional-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<testSourceDirectory>test/test-functional/java</testSourceDirectory>
<includes>
<include>**/*FT.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>it-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<testSourceDirectory>test/test-it/java</testSourceDirectory>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
...
I can trigger each of these two profiles like this:
mvn failsafe:integration-test -Pfunctional-tests
mvn failsafe:integration-test -Pit-tests
But when I run this:
mvn failsafe:integration-test -PrandomWord
It triggers it-tests profile. I was wondering why and if there is a way to have failsafe plugin output something like unrecognised profile.
Thank you for your help
In case it matters, here is my failsafe-plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
</executions>
</plugin>
Answering your question about why it triggers it-tests. In fact, it does not activate any of the profiles, hence default plugin configuration is used which has **/*IT.java in include list. So, it runs all IT tests by default.
This is weird approach to manage plugin executions by profiles. I doubt there is a reasonable way to validate profile names as you describe. I would recommend another approach here.
Approach 1. Use <id> and cli with #
You could just specify two executions of the plugin with id and then you could do this: How to execute maven plugin execution directly from command line?
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>it-tests</id>
<phase>none</phase> <!-- detach this execution from default lifecycle -->
<configuration>
<testSourceDirectory>test/test-it/java</testSourceDirectory>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</execution>
<execution>
<id>functional-tests</id>
<phase>none</phase> <!-- detach this execution from default lifecycle -->
<configuration>
<testSourceDirectory>test/test-ft/java</testSourceDirectory>
<includes>
<include>**/*FT.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
Now you could execute it from command line:
mvn failsafe:integration-test#it-tests
mvn failsafe:integration-test#functional-tests
Update: No need to specify goals as it is relevant to lifecycle only, we type it in command line anyway.
Approach 2. Use <skip> and properties
Keep both executions as a part of lifecycle but control execution by providing skip flags. I.e. define two properties e.g. skip.tests.it=true, skip.tests.ft=true and add <skip>${skip.tests.ft}</skip> to relevant configuration sections. Then you could just do
# run with no tests by default
mvn verify
# run with only FT
mvn verify -Dskip.tests.ft=false
# run with all tests
mvn verify -Dskip.tests.ft=false -Dskip.tests.it=false
to run full lifecycle together with desired tests.
I am attempting to have my integration tests separated from the usual lifecycle, meaning that I do not want them to be executed during a mvn install. I have the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<excludes>
<exclude>**/*IT.java</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
<goals>
<goal>integration-test</goal>
</goals>
</plugin>
And my integration tests are all suffixed with IT. I was able to confirm that surefire does exclude the *IT.java tests, however it seems that failsafe is being triggered during mvn install anyway. What am I missing?
Thank you for your help
A typical configuration to handle the issue with running integration tests only if you like is a profile like the following:
<profiles>
<profile>
<id>run-its</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
By using the above configuration you can run
mvn clean verify
which will execute the unit tests. Using the following you can activate the integration tests:
mvn -Prun-its clean verify
Based on the default naming conventions the integration tests which will be executed by maven-failsafe-plugin should be named like *IT.java where as the unit tests can be name like *Test.java (will be executed via maven-surefire-plugin).
The value of property skipITs when overridden on the command line appears to be ignored by maven if used in conjunction with -Dit.test=full.Classname. The specified failsafe test will not be run (case one).
As opposed to this specifiing skipITs without the it.test switch leads to running of existing failsafe tests (case two).
Background: Am trying to isolate surefire tests from failsafe tests runs by namely either running both types of them or one of each only. The maven calls are:
mvn -Pintegration -DskipTests=true -DskipITs=false -Dit.test=full.Classname verify
in the first case and:
mvn -Pintegration -DskipTests=true -DskipITs=false verify
in the second.
The relevant configuration (pom.xml, snippets only) being:
<properties>
<skipTests>false</skipTests>
<skipITs>false</skipITs>
</properties>
(those are defaults) and
<profiles>
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>custom<id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skipTests>${skipITs}</skipTests>
<skip>${skipITs}</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
Did you observe that too or have you eventually found a better working approach?
By default, the Surefire plugins runs during the test phase and usually you configure the Failsafe plugin to run during the integration-test and verify phase like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
To run the Surefire tests only, use
mvn clean test
To run both the Surefire and the Failsafe tests, use
mvn clean verify
You can completely skip any plugin by using the <skip> configuration option. If you configure the Surefire plugin like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<skip>${skip.surefire}</skip>
</configuration>
</plugin>
you can run only the Failsafe tests by calling
mvn clean verify -Dskip.surefire=true
If the integration test (it.test) has a non-default name I need to add an include pattern accordingly, like in (pom.xml snippet):
<profiles>
<profile>
<id>dbUnit</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>custom</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skipTests>${skipITs}</skipTests>
<skip>${skipITs}</skip>
<includes>
<include>**/*DbTest.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
But that appears valid only for the original approach. Also observe that the default inclusion patterns might need to be added too in case for your integration tests you followed different naming schemes.
Since several days, I tried to see where is my mistake in my configuration to run in parallel my Selenium tests.
I have a Selenium Grid with 2 nodes.
In my pom.xml, I have set surefire to run 2 by 2 the methods of my tests with a particular category then other tests.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<id>default-test</id>
<goals>
<goal>test</goal>
</goals>
<configuration>
<parallel>methods</parallel>
<perCoreThreadCount>false</perCoreThreadCount>
<threadCount>2</threadCount>
<reuseForks>false</reuseForks>
<groups>
com.something.categories.Safe,
com.something.categories.Parallel
</groups>
</configuration>
</execution>
<execution>
<id>no-safe</id>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludedGroups>
com.something.categories.Safe,
com.something.Parallel
</excludedGroups>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
When I launch my test mvn clean test -Dtest='TestAwesome' all the tests contains in TestAwesome are launched in the same time (I see more than 2 browsers opended), and so does not respect my threadCount value.
I'm missing something?
Edition after answer
Here my partial pom.xml to solve my issue
<profiles>
<profile>
<id>selenium-tests</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<parallel>all</parallel>
<threadCount>${threads}</threadCount>
<perCoreThreadCount>false</perCoreThreadCount>
<useUnlimitedThreads>true</useUnlimitedThreads>
<systemProperties>
<browser>${browser}</browser>
<screenshotDirectory>${project.build.directory}/screenshots</screenshotDirectory>
<gridURL>${seleniumGridURL}</gridURL>
<env>${env}</env>
</systemProperties>
<groups>${groups}</groups>
<excludedGroups>${excludedGroups}</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
Since you are using a modern-enough version of surefire, you might wanna try the threadCountMethods parameter instead of threadCount in combination with useUnlimitedThreads = true, even though it seems counter-intuitive.
surefire jUnit examples:
As of Surefire 2.7, no additional dependencies are needed to use the full set of options with parallel. As of Surefire 2.16, new thread-count attributes are introduced, namely threadCountSuites, threadCountClasses and threadCountMethods.
Fork options and parallel execution:
As an example with an unlimited number of threads, there is maximum of three concurrent threads to execute suites: parallel = all, useUnlimitedThreads = true, threadCountSuites = 3.
I have the following project structure:
MyProject
--src
--test
--acceptance
--step_definitions
--features
--unit
I would like to be able to run my cucumber tests (in test/acceptance) separately in Maven from the unit tests declared in test/unit, so that they can be run in different CI build plans etc. I am using cucumber-junit so the 'runners' for each acceptance test are written with JUnit.
Is this possible?
Is this possible?
Yes, it is possible. I believe you should separate your unit from the acceptance/integration tests having:
Slightly modified folders structure for both of these, placing your integration test files in the standard location of src/it:
MyProject/
src/main/java/ (SUT)
src/test/ (unit test code)
java/
resources/
src/it/ (acceptance/integration tests)
java/ (steps definitions)
resources/ (feature files)
Moreover, by design, different Maven plugins are intended for unit and integration tests:
for unit tests: maven-surefire-plugin
for acceptance/integration tests: maven-failsafe-plugin
You must also bind execution of maven-failsafe-pulgin. To run the integration tests separately, you can define a new profile:
<profiles>
<profile>
<id>acceptance-tests</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
You will also need to configure the plugin to search the src/it directory tree for test cases.
The acceptance tests can be run afterwards using:
mvn clean verify -Pacceptance-tests
For complete sample, I'd suggest you to follow http://www.weblogism.com/item/334/integration-tests-with-cucumber-jvm-selenium-and-maven
The other answer suggested modifying your folder structure to have shared folder for integration and acceptance tests, but you can have the original folder structure as well. Also you mentioned in the comment you want to keep all three (including not-mentioned integration tests) separate, which is possible, though hackish.
Since you seem to have test/unit for unit tests and test/acceptance for acceptance test, I'm assuming test/integration for integration tests.
<profiles>
<profile>
<id>acceptance-test</id>
<build>
<plugins>
<plugin>
<!-- to run directly: mvn failsafe:integration-test -P acceptance-test -->
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12</version>
<configuration>
<testSourceDirectory>test/acceptance</testSourceDirectory>
<includes>
<include>**/*Acceptance.java</include>
</includes>
<excludes>
<exclude>**/*IT.java</exclude>
<exclude>**/*Test.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>1.9.1</version>
<executions>
<execution>
<id>add-test-source</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>test/unit</source>
<source>test/integration</source>
<source>test/acceptance</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12</version>
<configuration>
<testSourceDirectory>test/unit</testSourceDirectory>
</configuration>
</plugin>
<plugin>
<!-- to run directly: mvn failsafe:integration-test -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<testSourceDirectory>test/integration</testSourceDirectory>
</configuration>
<!-- execution below can be used, if tests are needed on
mvn:integration-test -->
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
Note though that separation only applies for sources: compiled files will all go to the same folder, and AFAIK that's something you can't change. This means you need to have naming strategy for your tests to separate them from each other.