Jenkins is not showing all executed Tests / Test Results in Build Overview - maven

I am executing my Selenium WebDriver Tests with Junit via Jenkins, the test execution itself runs on Sauce Labs. My tests run on our company website, which is present in many countries, so I created execution profiles for maven to run the tests either on all domains or a single specific one. My POM for surefire and profiles looks like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
<groups>${testcase.groups}</groups>
</configuration>
</plugin>
<profile>
<id>all</id>
<properties>
<testcase.groups>categories.AllDomains</testcase.groups>
</properties>
</profile>
<profile>
<id>germany</id>
<properties>
<testcase.groups>categories.Germany</testcase.groups>
</properties>
</profile>
<profile>
<id>poland</id>
<properties>
<testcase.groups>categories.Poland</testcase.groups>
</properties>
</profile>
All tests are correctly marked with the necessary #Category categories.
If i select a single domain and run the tests, all tests are executed and Total Tests / Failed Tests are correctly shown in Jenkins.
But if I run the tests on all domains, Jenkins shows me different amounts of Total Tests / Failed Tests every time. So the feedback I get from each build is different and totally unrealiable, as also the Failed Tests are not correct in comparison with the execution on Sauce Labs (Sauce Labs also has a constant amount of Total Tests / Failed Tests).
The image from the link shows my Jenkins Test Graph, the marked sections all have the same execution configuration to run on all domains, but have totally different numbers of executed tests.
Thanks to Sauce Labs, I know that all my tests specified by the profile are running, as you can see in the Image the number of Total Tests is consistent.
By now I checked all my Code and Jenkins configurations 10 times and cannot find a reason, why this is happening. So why is Jenkins not showing the actual result?
If I go into the execution data and check each class, I can see that it adds and removes random tests per domain (but still executing them on Sauce Labs), no pattern, no repetition!
PS: the config for profiles and Domain execution hasn't changed in a while, this issue started out of nowhere (no updates to Selenium code, Jenkins Plugins, Jenkins itself)

After some more testing I found out it is an issue with the parallel test execution in Surefire. It seems running with parallel=all started to cut off report XMLs and Jenkins did not receive all the feedback for all tests executed.
As I was using parallel=all to decrease the overall test execution time, I looked into forkCount to keep my time low without parallel=all.
So my new Surefire config in my POM looks like this now:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-version}</version>
<configuration>
<parallel>methods</parallel>
<threadCount>10</threadCount>
<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<parallelTestsTimeoutInSeconds>300</parallelTestsTimeoutInSeconds>
<groups>${testcase.groups}</groups>
</configuration>
</plugin>
Test execution time is still very low, but all reports are complete and Jenkins is showing the correct result!

Related

Stop the Test Execution if the Smoke Test Fails

I have 2 testng.xml files called testng_smoke.xml and testng_regression.xml. I've configured the pom.xml as below to run these testng files.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng_smoke.xml</suiteXmlFile>
<suiteXmlFile>src/test/resources/testng_regression.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
In here it executes both testng files and give the status of the build (Success or Fail) at the end.
But my requirement is to run the smoke test suite (testng_smoke.xml) first and fail the build without moving to regression test suite (testng_regression.xml) if there any test failure in the smoke test suite. Is there any possibility to achieve this requirement.
AFAIK, there is no out of the box way in testng to do this.
If you are using Jenkins or some CI, then it is better to create two jobs or pipeline so that the second task is dependent on the results of the first one.
If not and you want to use single file then one of the possible solutions may be to maintain a variable either as a file or an environment variable who's value can be set in the suiteListener afterSuite method. The beforeSuite method can then throw a SkipException to ignore the suite based on the variable value.

Tests run slower with Maven than with IDE

I'm facing some test time execution issue related to local test execution with Maven. I have around 250 tests which run about 15 min on the IDE test runner and about an hour (55min to be exact) when executing them with maven locally. I tried a lot of configurations to make test execution in parallel but neither of them work for me, the time is still the same...probably I'm doing something wrong. Can anyone help on this? The last maven surefire plugin configuration that I tried is the following:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<parallel>methods</parallel>
<threadCount>10</threadCount>
</configuration>
</plugin>
And for test execution command I've used:
mvn test
mvn surefire:test

Maven skip specific test [duplicate]

In my maven project I have a number of modules. Is it possible to turn off running unit test for some modules via command line options?
My project takes about 15 mins to run through all unit tests. I would like to speed up the overall build by running just the unit tests in the module I am working on. I do not want to go in and edit each individual pom.xml to achieve this.
I have tried a solution outlined here: Can I run a specific testng test group via maven? However the result is a lot of test failures in modules that I want to skip. I suppose 'group' is not the same concept of module?
To toggle unit tests on and off for an entire project use Maven Surefire Plugin's capability of skipping tests. There is a drawback with using skipTests from the command line. In a multi-module build scenario, this would disable all tests across all modules.
If you need more fine grain control of running a subset of tests for a module, look into using the Maven Surefire Plugin's test inclusion and exclusion capabilities.
To allow for command-line overrides, make use of POM properties when configuring the Surefire Plugin. Take for example the following POM segment:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.9</version>
<configuration>
<excludes>
<exclude>${someModule.test.excludes}</exclude>
</excludes>
<includes>
<include>${someModule.test.includes}</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<someModule.skip.tests>false</someModule.skip.tests>
<skipTests>${someModule.skip.tests}</skipTests>
<someModule.test.includes>**/*Test.java</someModule.test.includes>
<someModule.test.excludes>**/*Test.java.bogus</someModule.test.excludes>
</properties>
With a POM like the above you can execute tests in a variety of ways.
Run all tests (the above configuration includes all **/*Test.java test source files)
mvn test
Skip all tests across all modules
mvn -DskipTests=true test
Skip all tests for a particular module
mvn -DsomeModule.skip.tests=true test
Only run certain tests for a particular module (this example includes all **/*IncludeTest.java test source files)
mvn -DsomeModule.test.includes="**/*IncludeTest.java" test
Exclude certain tests for a particular module (this example excludes all **/*ExcludeTest.java source files)
mvn -DsomeModule.test.excludes="**/*ExcludeTest.java" test
Found a way to exclude on command line:
# Exclude one test class, by using the explanation mark (!)
mvn test -Dtest=!LegacyTest
# Exclude one test method
mvn verify -Dtest=!LegacyTest#testFoo
# Exclude two test methods
mvn verify -Dtest=!LegacyTest#testFoo+testBar
# Exclude a package with a wildcard (*)
mvn test -Dtest=!com.mycompany.app.Legacy*
This is from: https://blog.jdriven.com/2017/10/run-one-or-exclude-one-test-with-maven/
…and if you like to pass the parameter to maven release plugin in Hudson/Jenkins you have to use
-Darguments=-DskipTests
to get it work.
If you want to use Maven profiles:
http://maven.apache.org/guides/introduction/introduction-to-profiles.html
you might want to make it work doing something like this:
Skipping tests in some modules in Maven
I don't know if there is a supported command line option that does the same.
You also might try using environment properties directly, something as per this doc page:
http://maven.apache.org/plugins/maven-surefire-plugin/examples/skipping-test.html
i.e. something like:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12</version>
<configuration>
<skipTests>${moduleA.skipTests}</skipTests>
</configuration>
</plugin>
then using mvn -DmoduleA.skipTests=false test to test that one module.

How do I configure my Maven-war-plugin's useCache feature so that consecutive builds are faster?

I’m using Maven 3.3.0 on Mac Yosemite. I wanted to make use of the maven-war-plugin’s useCache feature, but it isn’t doing anything in my multi-module project. When I run
mvn clean install -DskipTests
my project takes about 1:25 to run with the below configuration
<profile>
<id>prepare-deploy-war-to-jboss</id>
<activation>
<file>
<exists>${basedir}/src/main/webapp</exists>
</file>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.6</version>
<configuration>
<useCache>true</useCache>
<cacheFile>/tmp/${project.artifactId}/war/work</cacheFile>
</configuration>
</plugin>
</plugins>
</build>
</profile>
Then I run the same command again and the project takes the same amount of time. I can see “work” files getting created so the plugin is definitely running but consecutive builds do not seem to be doing anything.
My question here isn’t so much as why isn’t useCache speeding up my build, but how can I configure my plugin differently so that consecutive runs do speed up the build? If there is another plugin I should be using that would speed up builds on back-to-back runs, then that would also suffice here.
Looking at the WAR mojo code (at the time of writing), the cache is mainly used by its web app structure concerning overlays management, so in most of the cases it would not improve build time indeed.
Moreover, as stated by its official documentation, the cache mechanism is an experimental feature, hence disabled by default, which probably doesn't achieve (yet) user expectations.
Regardless of the effectiveness of this cache option, some hints to speed up maven builds could be:
Consider whether you really need to clean at each and every run
Consider building offline (-o option) if everything you need is already on your local cache
Consider using threads during your build (-T option)
Consider going on quite mode (-q option), swithing off build log temporarely and getting only error logs (basically: no news, good news)
In your case, the War Plugin is activated upon the existence of a structure typical of a war packaging, which probably means this profile is part of the aggregator/parent pom and then activated only on the war module. Although it might impact very little, also consider moving the War Plugin configuration to its concerned module and avoid such a triggered configuration
Last but not least, during development time, build time is probably more important than war size, so you could switch off the default mechanism of re-compressing external libraries added to the war file, via the recompressZippedFiles option:
Indicates if zip archives (jar,zip etc) being added to the war should be compressed again. Compressing again can result in smaller archive size, but gives noticeably longer execution time.
Default: true
So a sample configuration would look like:
<properties>
<war.recompress.files>false</war.recompress.files>
</properties>
<build>
<finalName>webapp</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.6</version>
<configuration>
<recompressZippedFiles>${war.recompress.files}</recompressZippedFiles>
</configuration>
</plugin>
</plugins>
</build>
Note: since there is no user property for this configuration entry, I also added a property for it, to switch it on/off on demand via command line (or via profile).
You could then test the different execution times executing the default build (with the configuration above disabling recompression) against the previous configuration (below, switching recompression on for the current execution, on demand):
mvn clean install -Dwar.recompress.files=true
You may then consider to profile it to switch it on/off depending on development phase.

Start multiple selenium test cases using jenkins

I'm a newbie to using jenkins.
I'm using jenkins to build by maven project. After the build process I need to run around 5 selenium2 test cases. Each test case takes around 20 minutes to finish, if I run one test case after the other it takes around 100 minutes complete all the test cases.
I want to know if it is possible to start all the test cases at once so that I can save time.
Currently my pom.xml is as follows -
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<junitArtifactName>junit:junit</junitArtifactName>
<includes>
<include>**/*Test*.java</include>
</includes>
</configuration>
</plugin>
Please let me know on how I can do this?
The best way to run tests in parallel from Jenkins is Selenium Grid.
You can find info on the Grid here: http://selenium-grid.seleniumhq.org/
Jenkins plugin here: https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin

Resources