surefire-plugin does not respect threadCount parameter - maven

Since several days, I tried to see where is my mistake in my configuration to run in parallel my Selenium tests.
I have a Selenium Grid with 2 nodes.
In my pom.xml, I have set surefire to run 2 by 2 the methods of my tests with a particular category then other tests.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<id>default-test</id>
<goals>
<goal>test</goal>
</goals>
<configuration>
<parallel>methods</parallel>
<perCoreThreadCount>false</perCoreThreadCount>
<threadCount>2</threadCount>
<reuseForks>false</reuseForks>
<groups>
com.something.categories.Safe,
com.something.categories.Parallel
</groups>
</configuration>
</execution>
<execution>
<id>no-safe</id>
<goals>
<goal>test</goal>
</goals>
<configuration>
<excludedGroups>
com.something.categories.Safe,
com.something.Parallel
</excludedGroups>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
When I launch my test mvn clean test -Dtest='TestAwesome' all the tests contains in TestAwesome are launched in the same time (I see more than 2 browsers opended), and so does not respect my threadCount value.
I'm missing something?
Edition after answer
Here my partial pom.xml to solve my issue
<profiles>
<profile>
<id>selenium-tests</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<parallel>all</parallel>
<threadCount>${threads}</threadCount>
<perCoreThreadCount>false</perCoreThreadCount>
<useUnlimitedThreads>true</useUnlimitedThreads>
<systemProperties>
<browser>${browser}</browser>
<screenshotDirectory>${project.build.directory}/screenshots</screenshotDirectory>
<gridURL>${seleniumGridURL}</gridURL>
<env>${env}</env>
</systemProperties>
<groups>${groups}</groups>
<excludedGroups>${excludedGroups}</excludedGroups>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>

Since you are using a modern-enough version of surefire, you might wanna try the threadCountMethods parameter instead of threadCount in combination with useUnlimitedThreads = true, even though it seems counter-intuitive.
surefire jUnit examples:
As of Surefire 2.7, no additional dependencies are needed to use the full set of options with parallel. As of Surefire 2.16, new thread-count attributes are introduced, namely threadCountSuites, threadCountClasses and threadCountMethods.
Fork options and parallel execution:
As an example with an unlimited number of threads, there is maximum of three concurrent threads to execute suites: parallel = all, useUnlimitedThreads = true, threadCountSuites = 3.

Related

Maven install is running failsafe

I am attempting to have my integration tests separated from the usual lifecycle, meaning that I do not want them to be executed during a mvn install. I have the following configuration:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<excludes>
<exclude>**/*IT.java</exclude>
</excludes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
<goals>
<goal>integration-test</goal>
</goals>
</plugin>
And my integration tests are all suffixed with IT. I was able to confirm that surefire does exclude the *IT.java tests, however it seems that failsafe is being triggered during mvn install anyway. What am I missing?
Thank you for your help
A typical configuration to handle the issue with running integration tests only if you like is a profile like the following:
<profiles>
<profile>
<id>run-its</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
By using the above configuration you can run
mvn clean verify
which will execute the unit tests. Using the following you can activate the integration tests:
mvn -Prun-its clean verify
Based on the default naming conventions the integration tests which will be executed by maven-failsafe-plugin should be named like *IT.java where as the unit tests can be name like *Test.java (will be executed via maven-surefire-plugin).

Maven wildfly deployment with multiple servers (standalone)

in my environment I have two wildfly server where I want to deploy with the wildfly-maven-plugin.
The servers differ in the name dev01 and dev02 but have the same port 9993 and username and password.
My understanding is that the wildfly-maven-plugin support only single server deployment.
If the problem are not big enough we use a module/submodule structure where the war file will be build in a submodule.
I'm using two profiles wildfly-deploy-dev01 and wildfly-deploy-dev02.
<profiles>
<profile>
<id>wildfly-deploy-dev01</id>
<build>
<plugins>
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
<executions>
<execution>
<phase>install</phase>
<goals>
<goal>deploy</goal>
</goals>
<configuration>
<skip>true</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>wildfly-deploy-dev02</id>
<build>
[...]
<profiles>
In the main module I skipped it.
In the war submodule:
<profiles>
<profile>
<id>wildfly-deploy-dev01</id>
<build>
<finalName>${project.artifactId}-v1.0</finalName>
<plugins>
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<configuration>
<skip>false</skip>
<id>wildfly-credentials<id>
<hostname>dev01.example.com</hostname>
<protocol>remote+https</protocol>
<port>9993</port>
</configuration>
<executions>
<execution>
<phase>install</phase>
<goals>
<goal>deploy</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>wildfly-deploy-dev01</id>
<build>
[same as above for hostname dev02.example.com]
</profiles>
First I was thinking everthing works fine but then I found out that only the last server will be deployed.
mvn wildfly:deploy -P wildfly-deploy-dev01,wildfly-deploy-dev02
I played around by setting the configration after the execution tag without success. It looks that the second profile overwrite the first one.
Futher I hardcoded the finalname because the parsedVersion is not parsed.
<finalName>${project.artifactId}-v${parsedVersion.majorVersion}.${parsedVersion.minorVersion}</finalName>
At the moment I'm lost with Maven. Has anybody an idea how I can deploy with the plugin on two servers?
Thanks,
Markus
Ways which I tried:
https://github.com/tsotzolas/wildfly_maven_plugins_examples/blob/master/deployToMultiplesServer/pom.xml
wildfly-maven-plugin not deploying when multiple profiles selected
Cannot access parsedVersion value in pom properties
You should be able to do this in a single profile with different executions. There shouldn't be a need to multiple profiles.
<profiles>
<profile>
<id>wildfly-deploy</id>
<build>
<plugins>
<plugin>
<groupId>org.wildfly.plugins</groupId>
<artifactId>wildfly-maven-plugin</artifactId>
<configuration>
<skip>false</skip>
<id>wildfly-credentials<id>
<protocol>remote+https</protocol>
<port>9993</port>
</configuration>
<executions>
<execution>
<id>deploy-dev1</id>
<phase>install</phase>
<goals>
<goal>deploy</goal>
</goals>
<configuration>
<hostname>dev01.example.com</hostname>
</configuration>
</execution>
<execution>
<id>deploy-dev2</id>
<phase>install</phase>
<goals>
<goal>deploy</goal>
</goals>
<configuration>
<hostname>dev02.example.com</hostname>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profiles>
With this you'd just have to do mvn clean install -Pwildfly-deploy.

Running two maven profiles

My profiles in pom.xml
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<filesets>
<fileset>
<directory>test-output</directory>
<followSymlinks>false</followSymlinks>
</fileset>
</filesets>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>runWithHead</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<id>execution1</id>
<configuration>
<systemPropertyVariables>
<homepage>${ADDRESS}</homepage>
<head>true</head>
<browsers>${browser}</browsers>
<chromeDriverLocation>${chromeDriver}</chromeDriverLocation>
</systemPropertyVariables>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>runHeadlessly</id>
<properties>
<displayProps>target/selenium/display.properties</displayProps>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>selenium-maven-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<id>xvfb</id>
<phase>test-compile</phase>
<goals>
<goal>xvfb</goal>
</goals>
<configuration>
<browsers>${browser}</browsers>
<displayPropertiesFile>${displayProps}</displayPropertiesFile>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<id>execution2</id>
<configuration>
<systemPropertyVariables>
<homepage>${ADDRESS}</homepage>
<head>false</head>
<display.props>${displayProps}</display.props>
<browsers>${browser}</browsers>
<chromeDriverLocation>${chromeDriver}</chromeDriverLocation>
</systemPropertyVariables>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
So first we have got shared plugins and then two different profiles.
When i try to run them with one command
mvn clean install -P runHeadlessly,runWithHead
Only plugin with xvfb id gets executed, any ideas?
(did not post the part with default property variables)
You have defined two profiles:
First profile, runWithHead, defines an execution of the maven-surefire-plugin without specifying any goal element (within a goals section), hence an empty execution
Second profile, runHeadlessly, defines an execution and goal of the selenium-maven-plugin plugin AND again an empty execution of the maven-surefire-plugin
As such, both profiles are executed, but effectively only the xvfb of the xvfb execution of the selenium-maven-plugin would be performed.
Since the maven-surefire-plugin has only one goal, test, try to add the following to both executions:
<goals>
<goal>test</goal>
<goals>
Also note: you are configuring additional executions of the maven-surefire-plugin on top of the default default-test execution which will also be executed. Hence, enabling both profiles you will end up with three executions of this plugin.

Is isolation of failsafe and surefire runs from each other using a skip approach possible?

The value of property skipITs when overridden on the command line appears to be ignored by maven if used in conjunction with -Dit.test=full.Classname. The specified failsafe test will not be run (case one).
As opposed to this specifiing skipITs without the it.test switch leads to running of existing failsafe tests (case two).
Background: Am trying to isolate surefire tests from failsafe tests runs by namely either running both types of them or one of each only. The maven calls are:
mvn -Pintegration -DskipTests=true -DskipITs=false -Dit.test=full.Classname verify
in the first case and:
mvn -Pintegration -DskipTests=true -DskipITs=false verify
in the second.
The relevant configuration (pom.xml, snippets only) being:
<properties>
<skipTests>false</skipTests>
<skipITs>false</skipITs>
</properties>
(those are defaults) and
<profiles>
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>custom<id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skipTests>${skipITs}</skipTests>
<skip>${skipITs}</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
Did you observe that too or have you eventually found a better working approach?
By default, the Surefire plugins runs during the test phase and usually you configure the Failsafe plugin to run during the integration-test and verify phase like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
To run the Surefire tests only, use
mvn clean test
To run both the Surefire and the Failsafe tests, use
mvn clean verify
You can completely skip any plugin by using the <skip> configuration option. If you configure the Surefire plugin like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<skip>${skip.surefire}</skip>
</configuration>
</plugin>
you can run only the Failsafe tests by calling
mvn clean verify -Dskip.surefire=true
If the integration test (it.test) has a non-default name I need to add an include pattern accordingly, like in (pom.xml snippet):
<profiles>
<profile>
<id>dbUnit</id>
<build>
<plugins>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>custom</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<skipTests>${skipITs}</skipTests>
<skip>${skipITs}</skip>
<includes>
<include>**/*DbTest.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
But that appears valid only for the original approach. Also observe that the default inclusion patterns might need to be added too in case for your integration tests you followed different naming schemes.

Maven Profile does not run appropriate tests

I have configured a profile for running only integration tests but it is still running all tests.
This is the configuration:
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>test</goal>
</goals>
<phase>test</phase>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
I tried with regex like explained at http://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html but I got this not working.
How do I get only tests running that end with IT.java?
You have not reconfigured the surefire run, but instead added another one to the lifecycle. That way, surefire:test runs twice, once with default execution id (default-test) and once with your new execution (integration-test)
If you simply want to change the tests being run, you could use:
<profile>
<id>integration</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
</profile>
Which would only reconfigure the existing execution.
Still, as khmarbaise already wrote in a comment, it makes much more sense to simply use the maven-failsafe-plugin. If you really wanted to skip normal unit test (why would you want that, anyway?), you could explicitly disable surefire in your profile (using skipTests-configuration).

Resources