How to re-run failed TestNG tests using Gradle - gradle

Question

I know what TestNG generates testng-failed.xml with information about failed test.
I'd like to use it in Gradle to created new task for running failed tests:
task secondTry(type: Test) {
onlyIf {
file("build/reports/tests/test/testng-failed.xml").exists()
}
println file("build/reports/tests/test/testng-failed.xml").exists()
testClassesDir = sourceSets.test.output.classesDir
classpath = sourceSets.test.runtimeClasspath
useTestNG() {
suites("build/reports/tests/test/testng-failed.xml")
}
}
Aftrer running task secondTry I got true for println file("build/reports/tests/test/testng-failed.xml").exists(), but failed tests are not run and task is skipped: secondTry SKIPPED
Does anyone experience in such cases, how to make tests run?
UPDATE:
I've tried to modify task a bit, so just run testng-failed.xml:
task secondTry(type: Test) {
useTestNG() {
useDefaultListeners = true
reports.html.enabled = false
options.suites("build/reports/tests/test/testng-failed.xml")
}
}
As result, build is successfully executed, including secondTry task, but failed tests still are not run.

Answer 1

This can be achieved by a TestNG's feature called RetryAnalyzer. With few little tweaks, we can tell TestNG that how many times a test should get retried until it gets passed and what kind of exceptions/errors would trigger the retrying of the failed test methods.

Answer 2

The idiomatic way of doing this in TestNG is to implement the retry method in IRetryAnalyzer interface and then annotate the required test methods or class that you expect flaky tests to be in. TestNG will automatically run them again as part of your test suite. You can refer to the TestNG docs for this.
Here is a working example of this in Kotlin language, Here the failed test methods would be run again until they meet the specified criteria.
RerunTests.kt
import framework.core.retry.FailureRunner
import org.testng.Assert
import org.testng.annotations.Test
#Test(groups = ["rerun"], retryAnalyzer = FailureRunner::class)
class RerunTests {
fun foobar() {
Assert.assertFalse(true)
}
fun foo() {
Assert.assertEquals(2, 2)
}
}
And here is a sample implementation of a class that implements the required interface
FailureRunner.kt
import org.testng.IRetryAnalyzer
import org.testng.ITestResult
import org.testng.Reporter
class FailureRunner : IRetryAnalyzer {
private var retryCounter = 0
private var maxRetryCount = System.getProperty("maxFailureRetries").toInt()
override fun retry(result: ITestResult?): Boolean {
val testName = result?.testName
return if (retryCounter < maxRetryCount) {
Reporter.log("Current retry count <$retryCounter> is less than the max no of failure retries <$maxRetryCount>")
Reporter.log("Retrying $testName")
++retryCounter
true
} else {
Reporter.log("Retry count exceeded max limit for $testName.")
false
}
}
}

Answer 3

The Test Retry Gradle plugin is designed to retry failed tests, and works with TestNG, JUnit 4 & 5, and Spock. It will rerun each failed test a certain number of times, with the option of failing the build if too many failures have occurred overall.
plugins {
id 'org.gradle.test-retry' version '1.2.1'
}
test {
retry {
maxRetries = 3
maxFailures = 20 // Optional attribute
}
}