Actually question related to testng-failed.xml has already been asked many times but my problem is little different. I want to run all the failed test cases together so what i did is in my pom I passed testng-failed.xml.
But the problem I am facing is first my testng.xml runs then testng-failed.xml and then it testng-failed.xml gets overridden. Due to this , suppose if i give a second time fresh run to my testcases, testng.xml runs, then my testng-failed.xml has previously failed test cases so it runs the previously failed cases and then updates testng-failed.xml with this time failed cases.
I dont knoe which listener to add to handle this issue that whenever i run first testng.xml should run , then it should override testng-failed.xml and then testng-failed.xml should run.
I am using Maven, selenium, testng.
I just eneterd testng-failed.xml in my pom as shown below. Please let me know which listner to use
<suiteXmlFiles>
<suiteXmlFile>src/resources/testng/testng.xml</suiteXmlFile>
<suiteXmlFile>test-output/testng-failed.xml</suiteXmlFile>
</suiteXmlFiles>
Create class 'RetryListener' by implementing 'IAnnotationTransformer'.
public class RetryListener implements IAnnotationTransformer {
#Override
public void transform(ITestAnnotation testannotation, Class testClass,
Constructor testConstructor, Method testMethod) {
IRetryAnalyzer retry = testannotation.getRetryAnalyzer();
if (retry == null) {
testannotation.setRetryAnalyzer(Retry.class);
}
}
}
Now Create another class.
public class Retry implements IRetryAnalyzer {
private int retryCount = 0;
private int maxRetryCount = 1;
// Below method returns 'true' if the test method has to be retried
else 'false'
//and it takes the 'Result' as parameter of the test method that just
ran
public boolean retry(ITestResult result) {
if (retryCount < maxRetryCount) {
System.out.println("Retrying test " + result.getName() + " with status "
+ getResultStatusName(result.getStatus()) + " for the " + (retryCount+1) + " time(s).");
retryCount++;
return true;
}
return false;
}
public String getResultStatusName(int status) {
String resultName = null;
if(status==1)
resultName = "SUCCESS";
if(status==2)
resultName = "FAILURE";
if(status==3)
resultName = "SKIP";
return resultName;
}
}
And Now Add below lines in your testNG xml file
<listeners>
<listener class-name="com.pack.test.RetryListener"/>
</listeners>
and Do not pass Xml file in pom.xml
Hope it will works
Thanks
Why are you running the testng xml and failed test xml in the same testng task. You should have to separate build task, first that runs testng xml and generates the failed tests xml and then another task running the failed test xml. It will work.
I implemented run one time and rerun three times only the newly failed tests.
mvn $par1=$pSuiteXmlFile test > $test1log
mvn $par1=$failedRelPath test > $failed1log
mvn $par1=$failedRelPath test > $failed2log
mvn $par1=$failedRelPath test > $failed3log
It works, but with small test-cases-count. I have a suite with 300 tests in it and somehow the testng-failed.xml is not created by surefire/testng after the main (first) run. When the suite is smaller, the testng-failed.xml is created as required.
Related
I was getting cucumber reports generated for my Karate project, however, soon after i ran mvn clean install command from my project terminal, some stuff from the target or resources directory got deleted & since then cucumber reports are not getting generated, even though there are no code changes anywhere. Could you please help me fix this one.
I tried to revert the project state to one before i ran the mvn clean install command, but no luck. I am still unable to generate the report as i did earlier.
Given below is my runner.java file code:
#KarateOptions(tags = {"~#ignore"})
public class ApiRunner {
#Test
public void testParallel() {
Results results = Runner.parallel(getClass(), 5, "target/surefire-
reports");
assertTrue(results.getErrorMessages(), results.getFailCount() == 0);
}
public static void generateReport(String basePath) {
Collection<File> jsonFiles =org.apache.commons.io.FileUtils.listFiles(new
File(basePath), new String[]{"json"}, true);
List<String> jsonPaths = new ArrayList(jsonFiles.size());
jsonFiles.forEach(file -> jsonPaths.add(file.getAbsolutePath()));
Configuration config = new Configuration(new File(basePath), basePath);
ReportBuilder reportBuilder = new ReportBuilder(jsonPaths, config);
reportBuilder.generateReports();
}
}
Consolidated Cucumber html reports for all features should get generated successfully after the runner.java file is run.
You are missing the call to the generateReport function in testParallel.
Add this line before the assertTrue :
generateReport(results.getReportDir());
I want to build netty from source code. I pull 4.1 from GitHub. When i want to run maven, I have some problem in test code.
I can run maven install -Dmaven.test.skip=true to build project. But when I want to run netty-example code, I see some error in test code.
I can not remove all test code, so I can not run code in this project.
This is for a jdk1.8, macos ,idea2019.
#Test
public void testFlushViaDisconnect() {
final AtomicInteger flushCount = new AtomicInteger();
EmbeddedChannel channel = newChannel(flushCount, false);
// Simulate read loop;
channel.pipeline().fireChannelRead(1L);
assertEquals(0, flushCount.get());
assertNull(channel.readOutbound());
channel.disconnect();
assertEquals(1, flushCount.get());
assertEquals(1L, channel.readOutbound()); // error this line.
assertNull(channel.readOutbound());
assertFalse(channel.finish());
}
Error:(150, 9) java: ambiguous method call of assertEquals
org.junit.Assert method assertEquals(long,long) and org.junit.Assert method assertEquals(java.lang.Object,java.lang.Object) is all match
I have a Gradle task that executes a TestNG test suite.
I want to be able to pass a flag to the task in order to use a special TestNG XML suite file (or just use the default suite if the flag isn't set).
gradle test
... should run the default standard suite of tests
gradle test -Pspecial
... should run the special suite of tests
I've been trying something like this:
test {
if (special) {
test(testng_special.xml);
}
else {
test(testng_default.xml);
}
}
But I get a undefined property error. What is the correct way to go about this?
if (project.hasProperty('special'))
should do it.
Note that what you're doing to select a testng suite won't work, AFAIK: the test task doesn't have any test() method. Refer to https://discuss.gradle.org/t/how-to-run-acceptance-tests-with-testng-from-gradle/4107 for a working example:
test {
useTestNG {
suites 'src/main/resources/testng.xml'
}
}
This worked for me:
test {
if (properties.containsKey('special')) {
test(testng_special.xml);
}
else {
test(testng_default.xml);
}
}
Here are 3 solutions for Kotlin DSL (build.gradle.kts):
val prop = project.properties["myPropName"] ?: "myDefaultValue"
val prop = project.properties["myPropName"] ?: error("Property not found")
if (project.hasProperty("special")) {
val prop = project.properties["myPropName"]
}
Note that you can omit the project. prefix as it is implicit in Gradle build files.
From Gradle Documentation:
-P, --project-prop
Sets a project property of the root project, for example -Pmyprop=myvalue
So you should use:
gradle test -Pspecial=true
with a value after the property name
I have the following test cases:
Login - Logout
Login - Search - Logout
Login - Search - Click on result - Logout
All these 3 testcases are considered to be independent and are within a single testNg class.
How do I validate the First test case (i.e. how to check if login is successful) ?
I have tried using assert, but when assert fails the remaining test cases (which are actually independent on the first one) will all be skipped.
How do I make sure that I validate all the test cases and irrespective of whether they pass or fail i should continue to execute all the test cases within that testNG class?
It looks like you are looking for ErrorCollector. Or you can use custom SoftAssert. This is my ErrorCollector realization for JUnit:
#Rule //collects errors without aborting test
public ErrorCollector collector = new ErrorCollector();
#Test
public void SampleTest() throws Exception {
...
try {
// Verify label
Assert.assertEquals("label"
,driver.findElement(By.xpath("some_xpath")).getText());
} catch (AssertionError e) {
System.out.println("Report Error: " + e);
collector.addError(e);
}
...
}
Right now we've got a project that builds in two jobs. 1) Is the standard build with unit tests. 2) is the integration tests. They work like this:
build the whole project, run unit tests, start integration test job
build the whole project, deploy it to the integration server, run client side integration tests against integration server
The problem is step 2) now takes over an hour to run and I'd like to parallelize the integration tests so that they take less time. But I'm not exactly sure how I can/should do this. My first thought is that I could have two step 2)s like this:
build the whole project, run unit tests, start integration test job
build the whole project, deploy it to the integration server1, run client side integration tests against integration server1
build the whole project, deploy it to the integration server2, run client side integration tests against integration server2
But then, how do I run half the integration tests on integration server1, and the other half on integration server2? I am using maven, so I could probably figure out something with failsafe and a complex includes/excludes pattern. But that sounds like something that would take a lot of effort to maintain. EG: when someone adds a new integration test class, how do I ensure that it gets run on one of the two servers? Does the developer have to modify the maven patterns?
I found this great article on how to do this, but it gives a way to do it in Groovy code. I pretty much followed these steps, but I haven't written the code to distribute the tests evenly by duration. But this is still a useful tool so I'll share it.
import junit.framework.JUnit4TestAdapter;
import junit.framework.TestSuite;
import org.junit.Ignore;
import org.junit.extensions.cpsuite.ClassesFinder;
import org.junit.extensions.cpsuite.ClasspathFinderFactory;
import org.junit.extensions.cpsuite.SuiteType;
import org.junit.runner.RunWith;
import org.junit.runners.AllTests;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
#RunWith(AllTests.class)
public class DistributedIntegrationTestRunner {
private static Logger log = LoggerFactory.getLogger(DistributedIntegrationTestRunner.class);
public static TestSuite suite() {
TestSuite suite = new TestSuite();
ClassesFinder classesFinder = new ClasspathFinderFactory().create(true,
new String[]{".*IntegrationTest.*"},
new SuiteType[]{SuiteType.TEST_CLASSES},
new Class[]{Object.class},
new Class[]{},
"java.class.path");
int nodeNumber = systemPropertyInteger("node.number", "0");
int totalNodes = systemPropertyInteger("total.nodes", "1");
List<Class<?>> allTestsSorted = getAllTestsSorted(classesFinder);
allTestsSorted = filterIgnoredTests(allTestsSorted);
List<Class<?>> myTests = getMyTests(allTestsSorted, nodeNumber, totalNodes);
log.info("There are " + allTestsSorted.size() + " tests to choose from and I'm going to run " + myTests.size() + " of them.");
for (Class<?> myTest : myTests) {
log.info("I will run " + myTest.getName());
suite.addTest(new JUnit4TestAdapter(myTest));
}
return suite;
}
private static int systemPropertyInteger(String propertyKey, String defaultValue) {
String slaveNumberString = System.getProperty(propertyKey, defaultValue);
return Integer.parseInt(slaveNumberString);
}
private static List<Class<?>> filterIgnoredTests(List<Class<?>> allTestsSorted) {
ArrayList<Class<?>> filteredTests = new ArrayList<Class<?>>();
for (Class<?> aTest : allTestsSorted) {
if (aTest.getAnnotation(Ignore.class) == null) {
filteredTests.add(aTest);
}
}
return filteredTests;
}
/*
TODO: make this algorithm less naive. Sort each test by run duration as described here: http://blog.tradeshift.com/just-add-servers/
*/
private static List<Class<?>> getAllTestsSorted(ClassesFinder classesFinder) {
List<Class<?>> allTests = classesFinder.find();
Collections.sort(allTests, new Comparator<Class<?>>() {
#Override
public int compare(Class<?> o1, Class<?> o2) {
return o1.getSimpleName().compareTo(o2.getSimpleName());
}
});
return allTests;
}
private static List<Class<?>> getMyTests(List<Class<?>> allTests, int nodeNumber, int totalNodes) {
List<Class<?>> myTests = new ArrayList<Class<?>>();
for (int i = 0; i < allTests.size(); i++) {
Class<?> thisTest = allTests.get(i);
if (i % totalNodes == nodeNumber) {
myTests.add(thisTest);
}
}
return myTests;
}
}
The ClasspathFinderFactory is used to find all test classes that match the .*IntegrationTest pattern.
I make N jobs and they all run this Runner but they all use different values for the node.number system property, so each job runs a different set of tests. This is how the failsafe plugin looks:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.12.4</version>
<executions>
<execution>
<id>integration-tests</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<includes>
<include>**/DistributedIntegrationTestRunner.java</include>
</includes>
<skipITs>${skipITs}</skipITs>
</configuration>
</plugin>
The ClasspathFinderFactory comes from
<dependency>
<groupId>cpsuite</groupId>
<artifactId>cpsuite</artifactId>
<version>1.2.5</version>
<scope>test</scope>
</dependency>
I think there should be some Jenkins plugin for this, but I haven't been able to find one. Something that's close is the Parallel Test Executor, but I don't think this does the same thing I need. It looks like it runs all the tests on a single job/server instead of multiple servers. It doesn't provide an obvious way to say, "run these tests here, and those tests there".
I believe you already found a solution by now, but I'll leave a path for the others who'll open this page asking the same question:
Parallel test executor plugin:
"This plugin adds a new builder that lets you easily execute tests defined in a separate job in parallel. This is achieved by having Jenkins look at the test execution time of the last run, split tests into multiple units of roughly equal size, then execute them in parallel."
https://wiki.jenkins-ci.org/display/JENKINS/Parallel+Test+Executor+Plugin
Yes, Parallel Test Executor is a cool plugin if you've got 2 slave or one slave with 8 executor because the this plugin based on "tests splitting" so e.g: you split your junit tests into 4 different array, these arrays will run on 4 different executor on that slave what you specified. I hope you got it :D, it depends on the number of executors on that slave where you want to run parallel testing or you should decrease split tests count to 2 from 4.