I have a maven project that is running testNG tests via a Testng xml. The XML has the following listeners
<listener class-name="org.uncommons.reportng.HTMLReporter" />
<listener class-name="org.uncommons.reportng.JUnitXMLReporter" />
I have 24 tests in the xml with all of them running packages e.g
<test name="First Test"
preserve-order="true">
<packages>
<package
name="testProject.first.*" />
</packages>
</test>
For some reason all 24 tests run but in the testng report I only ever see 21 tests. The 21 tests that show are not always consistent so that rules out the possibillity of a problem with the setup of some tests. I am wondering if there is possibly a max number of tests that run and if I need to combine tests? or is there a maximum number of tests parameter somewhere that can be set?
The following is my maven-surefire-plugin setup
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12</version>
<configuration>
<properties>
<property>
<name>usedefaultlisteners</name>
<value>false</value>
</property>
<property>
<name>listener</name>
<value>org.uncommons.reportng.HTMLReporter,org.uncommons.reportng.JUnitXMLReporter</value>
</property>
<systemPropertyVariables>
<environment>${env.browser}</environment>
</systemPropertyVariables>
<systemPropertyVariables>
<environment>${env.environment}</environment>
</systemPropertyVariables>
<systemPropertyVariables>
<environment>${env.testlinkRelease}</environment>
</systemPropertyVariables>
</properties>
<suiteXmlFiles>
<suiteXmlFile>src/testCleanup/${cleanup.suite}</suiteXmlFile>
<suiteXmlFile>src/limelightTests/${test.suite}</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
Thanks for any help.
I found the issue. I had a screenshot listener that was running after #AfterMethod had run. #AfterMethod quit the driver so the screenshot listener ended up throwing an error which was not caught and caused the test to be left out of the report.
Related
I have Maven Project and all the configuration is correct, recently i configured the project with GIT and Jenkins,Created Jenkin job.
Earlier i was able to run the complete project by
Right click on project and Run **--> **Run --> Maven test and test execution use to start but now its not throwing any error but not launching the browser for execution.
But if i run the java file [Single test case] separately using TestNG its working as expected[launching the browser and starts the execution]
As i have configured with Jenkins i cannot run every test case separately.
I will have to run the project with Maven test
I have tried the possible solutions :
Clean Project + delete m2 folder + Maven Run [just to make sure its not pointing to any old jar files] --> Did not work for me
Please find the code snippet that i have used in pom.xml
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>2.7</version>
</plugin>
</plugins>
</build>
Thanks in advance for all the suggestions :)
To trace cause, You may go through by below steps. And you will be found where is actual cause and it gets stuck.
1. Create Simple #Test with Console output:
public class NewTest {
#Test
public void f() {
System.out.println("Hello World");
}
}
2. Create TestNG.XML at root location of the project to execute above class:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
<test thread-count="5" name="Test">
<classes>
<class name="packageName.NewTest"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
3. Run and Check TESTNG.XML output
4. Run and Check output from POM.XML
POM.XML:
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.0</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
create a testNG xml and include all test classes there. This could be done by including the top level package:
<test name="sample-Test" preserve-order="true" verbose="2">
<packages>
<package name="org.sample.something.*"/>
</packages>
</test>
2.1 Update your surefire plugin configuration to make it:
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
2.2 OR (more flexible as you can create different profiles pointing to different xml to run different test suites separately) Create a maven profile that will point to your testNG xml
<profiles>
<profile>
<id>selenium-tests</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
mvn clean test -U -Pselenium-tests OR mvn clean test
source for maven
source for xml
If your testng.xml or xml file with test cases is in root directory of project you can try below command.
This will trigger all test cases present in testng.xml
mvn clean test -Dsurefire.suiteXmlFiles=testng.xml
This will execute single test case
mvn clean test -Dtest=className
Please not that, you need to have surefire plugin into the pom.xml.
This is how my POM.xml looks like.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.github.akiraly.reusable-poms</groupId>
<artifactId>pom-parent-with-spring-context</artifactId>
<version>4</version>
</parent>
<groupId>MvnTestFramework</groupId>
<artifactId>MvnTestFrameworkProject</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.9.6</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<forkMode>always</forkMode>
<systemProperties>
<property>
<name>reporter.debug</name>
<value>false</value>
</property>
</systemProperties>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
<reportsDirectory>reports</reportsDirectory>
<parallel>methods</parallel>
<threadCount>5</threadCount>
<skipTests>false</skipTests>
<suitename>${testng.suitename}</suitename>
<groups>${testng.groups}</groups>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<encoding>utf8</encoding>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
This is how my testng.xml looks like:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="testgroups" verbose="1" parallel="tests" group-by-instances="true" preserve-order="false" thread-count="5">
<test name="testgroups" preserve-order="true" group-by-instances="true">
<groups>
<run>
<include name="group1" />
<include name="group2" />
<include name="group3" />
</run>
</groups>
<packages>
<package name="testpackage.*" />
</packages>
</test>
</suite>
What I want is: run only specific groups of tests. Meaning, either group1, or group2 or group3, but not all of them to run together at once.
For the same reason, I have provided {testng.groups} to run the specific group.
In the JVM arguments when I run the testng.xml I specify:
-Dtestng.groups= group2 to run only the group2 tests. However, it just runs all tests in the group1, group2 and group3.
Am I approaching the problem correctly? How can I make sure only a specific set of group runs. Thank you for the help guys.
You can try leveraging the Beanshell capabilities that TestNG has to offer, to get this done.
That way, irrespective of how you run your test suite :
either via your IDE using the TestNG plugin wherein your pom file has no relevance (or)
via maven wherein maven has relevance
Your suite file's method selector will always be honoured.
In a nutshell, you need to define a method selector that uses Beanshell as shown below
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="false">
<test name="Test">
<method-selectors>
<method-selector>
<script language="beanshell">
<![CDATA[whatGroup = System.getProperty("groupToRun");
groups.containsKey(whatGroup);
]]>
</script>
</method-selector>
</method-selectors>
<classes>
<class name="organized.chaos.GroupsPlayGround" />
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
Once you have a suite xml file like this, you can specify the group names via the JVM argument -DwhatGroup.
You can also spice up the beanshell script by choosing to run everything if no groups were found via the JVM argument.
For more information you can refer to my blog post to get a complete understanding of how to work with the beanshell.
I am using mockito 1.8.3, jacoco 0.72 and maven 3.0.5 surefire plugin (2.12.4) to execute unit test and generating coverage report, it was working fine.
With more and more tests are added, it starts not working. I continuously encounter out of memory error during test execution, and cannot find out a way to figure out what is wrong.
I have about 1800+ test cases with mockito as the mocking tool. It is working fine if I do not run jacoco during maven test with "org.jacoco:jacoco-maven-plugin:prepare-agent " before test phase, but as long as I add jacoco agent, I get OOO issue with PermGen full.
I already added the PermGen to 2GB by modifying MAVEN_OPTS (which should not work since surefire will fork a new process) and surefire argline argument in pom, but it does not help a lot.
I try to get a core dump when OOO occurs by adding parameter to surefire plugin, but never saw a dump file in any folder. I am suspicious that my JVM setting does not work for the surefire plugin, but not sure what is wrong. Anyone could do me a favor? Thanks.
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.version}</version>
<inherited>true</inherited>
<configuration>
<properties>
<property>
<name>argLine</name> <value>-server -ea -XX:-UseSplitVerifier -XX:MaxPermSize=2g -Xmx3g -XX:+HeapDumpOnOutOfMemoryError </value>
</property>
<property>
<name>forkMode</name>
<value>once</value>
</property>
<property>
<name>reportFormat</name>
<value>plain</value>
</property>
<property>
<name>skipTests</name>
<value>${maven.test.skip}</value>
</property>
</properties>
</configuration>
</plugin>
You need to set the memory for maven-surefire-plugin like the following:
<plugins>
[...]
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<forkCount>3</forkCount>
<reuseForks>true</reuseForks>
<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
<systemPropertyVariables>
<databaseSchema>MY_TEST_SCHEMA_${surefire.forkNumber}</databaseSchema>
</systemPropertyVariables>
</configuration>
</plugin>
[...]
</plugins>
In case you have jacoco configured along with maven failsafe plugin, then you will need to pass memory parameters to that one too:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.14.1</version>
<configuration>
<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
</configuration>
</plugin>
I have 2 modules both having there respective pom.xmls,I would like to run pom.xml of module1 and using that run some code of module2 by using the classname to be run in the testng.xml.
Now this never works and i get a ClassNotFoundException.
Kindly help.
Cannot find class in classpath: com.org.Console1
at org.testng.xml.XmlClass.loadClass(XmlClass.java:76)
at org.testng.xml.XmlClass.init(XmlClass.java:68)
at org.testng.xml.XmlClass.<init>(XmlClass.java:54)
at org.testng.xml.TestNGContentHandler.startElement(TestNGContentHandler.java:516)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:506)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1322)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2715)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
at org.testng.xml.SuiteXmlParser.parse(SuiteXmlParser.java:17)
at org.testng.xml.SuiteXmlParser.parse(SuiteXmlParser.java:10)
at org.testng.xml.Parser.parse(Parser.java:170)
at org.testng.TestNG.initializeSuitesAndJarFile(TestNG.java:299)
at org.testng.TestNG.run(TestNG.java:972)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:122)
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:88)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:104)
This is my testng.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?><suite allow-return-values="false" configfailurepolicy="skip" data-provider-thread-count="10" group-by-instances="false" junit="false" name="Suite" parallel="false" preserve-order="true" skipfailedinvocationcounts="false" thread-count="5">
<test allow-return-values="false" group-by-instances="false" junit="false" name="Test" preserve-order="true" skipfailedinvocationcounts="false">
<classes>
<class name="com.org.Console1">
</class>
</classes>
</test> <!-- Test -->
This is my pom.xml
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.8</version>
<configuration>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
You don't need to specify a suite for just testing. The surefire-plugin has to be configured to catch all your java-classes, the default is to look up for "*Test.java", i personally dont like this behaviour.
pom-snippet:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<includes>
<include>**/*.java</include><!-- dont just run *Test.java-files, which is default for surefire -->
</includes>
</configuration>
</plugin>
Please also verify that your test-classes are in the right position:
src/main/test/com/org/Console1.java
It HAS to be there, unless you want to configure it (which i dont recommend)
I'm trying to use a custom reporter for TestNG with the maven surefire plugin. I already have a custom listener and that seems to get picked up. However, it doesn't look like the custom reporter is being used at all:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<testFailureIgnore>false</testFailureIgnore>
<properties>
<property>
<name>listener</name>
<value>com.mystuff.test.TestNGListener</value>
</property>
<property>
<name>reporter</name>
<value>com.mystuff.test.MyEmailableReporter</value>
</property>
</properties>
</configuration>
</plugin>
Any idea why this is happening?
I figured this out. It looked like the following doesn't work at all:
<property>
<name>reporter</name>
<value>com.mystuff.test.MyEmailableReporter</value>
</property>
in spite of documentation to the contrary. In the TestNG class it appears that there is a method called TestNG#addListener(IReporter listener), which contrary to its name, accepts a report which implements IReporter. Maven Surefire (v2.12.1) calls this method to add listeners and reports. However, it doesn't look for reports under a property with name reporter. Instead, you have to add your custom reporter to the listener property:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<testFailureIgnore>false</testFailureIgnore>
<properties>
<property>
<name>listener</name>
<value>com.mystuff.test.TestNGListener,com.mystuff.test.MyEmailableReporter</value>
</property>
</properties>
</configuration>
</plugin>
Not very intuitive.