I'm facing some test time execution issue related to local test execution with Maven. I have around 250 tests which run about 15 min on the IDE test runner and about an hour (55min to be exact) when executing them with maven locally. I tried a lot of configurations to make test execution in parallel but neither of them work for me, the time is still the same...probably I'm doing something wrong. Can anyone help on this? The last maven surefire plugin configuration that I tried is the following:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<parallel>methods</parallel>
<threadCount>10</threadCount>
</configuration>
</plugin>
And for test execution command I've used:
mvn test
mvn surefire:test
Related
Hi I am having a strange scenario.
I have a spring boot java project . I have lots of Junit tests as well.
when i run the tests from the terminal by issuing the command mvn clean test, 2 tests belong to a class fails
however if run the those tests belonging to that class from the eclipse Run as >> Junit tests it passes .
any idea why does this happens ? and how can i fix it ?
the following is my sure fire plugin configuration
<maven-surefire-plugin.version>2.18.1</maven-surefire-plugin.version>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
<configuration>
<!-- Force alphabetical order to have a reproducible build -->
<runOrder>alphabetical</runOrder>
</configuration>
</plugin>
thank you
I am executing my Selenium WebDriver Tests with Junit via Jenkins, the test execution itself runs on Sauce Labs. My tests run on our company website, which is present in many countries, so I created execution profiles for maven to run the tests either on all domains or a single specific one. My POM for surefire and profiles looks like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
<groups>${testcase.groups}</groups>
</configuration>
</plugin>
<profile>
<id>all</id>
<properties>
<testcase.groups>categories.AllDomains</testcase.groups>
</properties>
</profile>
<profile>
<id>germany</id>
<properties>
<testcase.groups>categories.Germany</testcase.groups>
</properties>
</profile>
<profile>
<id>poland</id>
<properties>
<testcase.groups>categories.Poland</testcase.groups>
</properties>
</profile>
All tests are correctly marked with the necessary #Category categories.
If i select a single domain and run the tests, all tests are executed and Total Tests / Failed Tests are correctly shown in Jenkins.
But if I run the tests on all domains, Jenkins shows me different amounts of Total Tests / Failed Tests every time. So the feedback I get from each build is different and totally unrealiable, as also the Failed Tests are not correct in comparison with the execution on Sauce Labs (Sauce Labs also has a constant amount of Total Tests / Failed Tests).
The image from the link shows my Jenkins Test Graph, the marked sections all have the same execution configuration to run on all domains, but have totally different numbers of executed tests.
Thanks to Sauce Labs, I know that all my tests specified by the profile are running, as you can see in the Image the number of Total Tests is consistent.
By now I checked all my Code and Jenkins configurations 10 times and cannot find a reason, why this is happening. So why is Jenkins not showing the actual result?
If I go into the execution data and check each class, I can see that it adds and removes random tests per domain (but still executing them on Sauce Labs), no pattern, no repetition!
PS: the config for profiles and Domain execution hasn't changed in a while, this issue started out of nowhere (no updates to Selenium code, Jenkins Plugins, Jenkins itself)
After some more testing I found out it is an issue with the parallel test execution in Surefire. It seems running with parallel=all started to cut off report XMLs and Jenkins did not receive all the feedback for all tests executed.
As I was using parallel=all to decrease the overall test execution time, I looked into forkCount to keep my time low without parallel=all.
So my new Surefire config in my POM looks like this now:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-version}</version>
<configuration>
<parallel>methods</parallel>
<threadCount>10</threadCount>
<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<parallelTestsTimeoutInSeconds>300</parallelTestsTimeoutInSeconds>
<groups>${testcase.groups}</groups>
</configuration>
</plugin>
Test execution time is still very low, but all reports are complete and Jenkins is showing the correct result!
Need your help as i have been struggling with parallel execution using maven. i am trying to run Testng tests in parallel, maven V3.3.9, maven-surefire-plugin V2.18.1, with the below configurations.
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
<parallel>classes</parallel>
<threadCount>10</threadCount>
and also tried <suitethreadpoolsize>10</suitethreadpoolsize>
But, tests aren't running in parallel. i have been searching extensively for two days but couldn't get it working. Tried all the suggestions available on google!! Need your inputs to find out where i might be wrong.
I would like to know if there is a way to execute a goal when there is test failures?
Since maven stops its execution (fail fast mode) after encountering a test failure, is there any options to launch a goal when there is test failures?
Regards.
I've been searching for a way to do this as well, with not much success.
However, there is the following question which might provide some general hints:
Maven reporting plugins do not execute if a unit test failure occurs
The idea is that you would run mvn install (or whatever) first, and then run:
mvn -Dmaven.test.skip=true your-plugin:your-goal
This will let you run the build again without running tests, preserving the results for your perusal. Of course, this is only useful if your plugin is parsing the test results...
Though not recommended, by setting the surefire property testFailureIgnore to true, you can continue maven execution even when there are test failures.
...
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
...
<configuration>
<testFailureIgnore>true</testFailureIgnore>
...
</configuration>
</plugin>
...
Just do mvn clean install -DskipTests
I added this plugin in pom.xml and it worked well.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
If you want to the build to run KNOWING beforehand that there are going to be failures, you can use:
mvn <goal> -Dmaven.test.skip = true
I'd like Maven to stop trying to run my JUnit Spring tests when it encounters the first error. Is this possible?
My test classes look like the following, and I run them just as a standard Maven target.
#ContextConfiguration(locations = {"classpath:/spring-config/store-persistence.xml","classpath:/spring-config/store-security.xml","classpath:/spring-config/store-service.xml", "classpath:/spring-config/store-servlet.xml" })
#RunWith(SpringJUnit4ClassRunner.class)
#Transactional
public class SkuLicenceServiceIntegrationTest
{
...
If there's an error in the Spring config, then each test will try to restart the Spring context, which takes 20 seconds a go. This means we don't find out for ages that any tests have failed, as it'll try to run the whole lot before concluding that the build was a failure!
Answering ten years later, since I needed exactly the same.
The failsafe plugin can be configure to "skip tests after failure", using the skipAfterFailureCount parameter.
Official documentation:
https://maven.apache.org/surefire/maven-failsafe-plugin/examples/skip-after-failure.html
This is more a remark, than an answer, but still, maybe you'll find it useful.
I'd recommend separating your integration tests into a separate phase, and running them with Failsafe, rather than Surefire. This way you can decide whether you need to run only fast unit tests, or the complete set with long-running integration tests:
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<!-- Uncomment/comment this in order to fail the build if any integration test fail -->
<execution>
<id>verify</id>
<goals><goal>verify</goal></goals>
</execution>
</executions>
</plugin>
</plugins>
A workaround for your problem might be singling out a test into a separate execution and run it first; this way the execution would fail and subsequent surefire/failsafe executions will not be launched. See how to configure the plugin to do it.