I'm a newbie to using jenkins.
I'm using jenkins to build by maven project. After the build process I need to run around 5 selenium2 test cases. Each test case takes around 20 minutes to finish, if I run one test case after the other it takes around 100 minutes complete all the test cases.
I want to know if it is possible to start all the test cases at once so that I can save time.
Currently my pom.xml is as follows -
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<junitArtifactName>junit:junit</junitArtifactName>
<includes>
<include>**/*Test*.java</include>
</includes>
</configuration>
</plugin>
Please let me know on how I can do this?
The best way to run tests in parallel from Jenkins is Selenium Grid.
You can find info on the Grid here: http://selenium-grid.seleniumhq.org/
Jenkins plugin here: https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin
Related
I have 2 testng.xml files called testng_smoke.xml and testng_regression.xml. I've configured the pom.xml as below to run these testng files.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng_smoke.xml</suiteXmlFile>
<suiteXmlFile>src/test/resources/testng_regression.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
In here it executes both testng files and give the status of the build (Success or Fail) at the end.
But my requirement is to run the smoke test suite (testng_smoke.xml) first and fail the build without moving to regression test suite (testng_regression.xml) if there any test failure in the smoke test suite. Is there any possibility to achieve this requirement.
AFAIK, there is no out of the box way in testng to do this.
If you are using Jenkins or some CI, then it is better to create two jobs or pipeline so that the second task is dependent on the results of the first one.
If not and you want to use single file then one of the possible solutions may be to maintain a variable either as a file or an environment variable who's value can be set in the suiteListener afterSuite method. The beforeSuite method can then throw a SkipException to ignore the suite based on the variable value.
I'm facing some test time execution issue related to local test execution with Maven. I have around 250 tests which run about 15 min on the IDE test runner and about an hour (55min to be exact) when executing them with maven locally. I tried a lot of configurations to make test execution in parallel but neither of them work for me, the time is still the same...probably I'm doing something wrong. Can anyone help on this? The last maven surefire plugin configuration that I tried is the following:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<parallel>methods</parallel>
<threadCount>10</threadCount>
</configuration>
</plugin>
And for test execution command I've used:
mvn test
mvn surefire:test
I am executing my Selenium WebDriver Tests with Junit via Jenkins, the test execution itself runs on Sauce Labs. My tests run on our company website, which is present in many countries, so I created execution profiles for maven to run the tests either on all domains or a single specific one. My POM for surefire and profiles looks like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<parallel>all</parallel>
<threadCount>10</threadCount>
<groups>${testcase.groups}</groups>
</configuration>
</plugin>
<profile>
<id>all</id>
<properties>
<testcase.groups>categories.AllDomains</testcase.groups>
</properties>
</profile>
<profile>
<id>germany</id>
<properties>
<testcase.groups>categories.Germany</testcase.groups>
</properties>
</profile>
<profile>
<id>poland</id>
<properties>
<testcase.groups>categories.Poland</testcase.groups>
</properties>
</profile>
All tests are correctly marked with the necessary #Category categories.
If i select a single domain and run the tests, all tests are executed and Total Tests / Failed Tests are correctly shown in Jenkins.
But if I run the tests on all domains, Jenkins shows me different amounts of Total Tests / Failed Tests every time. So the feedback I get from each build is different and totally unrealiable, as also the Failed Tests are not correct in comparison with the execution on Sauce Labs (Sauce Labs also has a constant amount of Total Tests / Failed Tests).
The image from the link shows my Jenkins Test Graph, the marked sections all have the same execution configuration to run on all domains, but have totally different numbers of executed tests.
Thanks to Sauce Labs, I know that all my tests specified by the profile are running, as you can see in the Image the number of Total Tests is consistent.
By now I checked all my Code and Jenkins configurations 10 times and cannot find a reason, why this is happening. So why is Jenkins not showing the actual result?
If I go into the execution data and check each class, I can see that it adds and removes random tests per domain (but still executing them on Sauce Labs), no pattern, no repetition!
PS: the config for profiles and Domain execution hasn't changed in a while, this issue started out of nowhere (no updates to Selenium code, Jenkins Plugins, Jenkins itself)
After some more testing I found out it is an issue with the parallel test execution in Surefire. It seems running with parallel=all started to cut off report XMLs and Jenkins did not receive all the feedback for all tests executed.
As I was using parallel=all to decrease the overall test execution time, I looked into forkCount to keep my time low without parallel=all.
So my new Surefire config in my POM looks like this now:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-version}</version>
<configuration>
<parallel>methods</parallel>
<threadCount>10</threadCount>
<forkCount>2</forkCount>
<reuseForks>true</reuseForks>
<parallelTestsTimeoutInSeconds>300</parallelTestsTimeoutInSeconds>
<groups>${testcase.groups}</groups>
</configuration>
</plugin>
Test execution time is still very low, but all reports are complete and Jenkins is showing the correct result!
Problem statement
How to invoke tests within a given test server(i.e. within the process)?
Explanation
We know that there is surefire plugin, but this uses surefire:test goal which runs unit tests in separate process. In my case, i need to run the tests within the existing process (pre-configured test server and required environment) to run test. In surefire plugin, there is configuration that has forkCount and reuseForks. If i use it with forkCount as 0, reuseForks true, still creates separate server and hence process. I simply want to avoid setup part for test.
Update after Joe's comment saying it as duplicate As my application usage SpringBoot, running mvn clean install build the project and execute tests. While executing tests, it creates server for each running test. Even though i used configuration given below, in console it shows same process id, but starting server every time. In my case i don't want to make server up for every test case, rather used pre-existing test server and the test should use that server.
Configuration used
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<forkCount>0</forkCount>
<reuseForks>true</reuseForks>
</configuration>
</plugin>
Need your help as i have been struggling with parallel execution using maven. i am trying to run Testng tests in parallel, maven V3.3.9, maven-surefire-plugin V2.18.1, with the below configurations.
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
<parallel>classes</parallel>
<threadCount>10</threadCount>
and also tried <suitethreadpoolsize>10</suitethreadpoolsize>
But, tests aren't running in parallel. i have been searching extensively for two days but couldn't get it working. Tried all the suggestions available on google!! Need your inputs to find out where i might be wrong.