I've got a Maven project that contains (Selenium / Cucumber) test code, run through Maven. These tests create custom ExtentReports HTML reports, which are stored in target/extentreports. This HTML report is always created (unless the project does not compile, of course).
What I'd like to accomplish is that after test execution, the contents of this folder are archived into a zip file (and preferably moved to a different folder). This should happen even if the tests fail. So, essentially, I'm looking to add something that will run no matter whether the tests run through the 'test' goal fail.
Currently, I'm running my tests through
mvn clean test
which runs the Cucumber / Selenium tests using JUnit.
I've tried a number of approaches and combinations of Maven plugins (surefire, failsafe), but I haven't found the right solution yet.
I'm running these tests through Jenkins, so any post-build Jenkins solution is fine too. I'm not sure whether installing Jenkins plugins is desirable though (we're not managing Jenkins ourselves), so solutions that do not require this would be preferred.
A standard approach is to:
Make sure the build doesn't fail because of the tests:
mvn test -Dmaven.test.failure.ignore=true
Add a post-build step Publish JUnit test result report which would color the build into yellow.
Jenkins does allow for post operations
pipeline {
agent any
stages {
stage('Example') {
steps {
echo 'Hello World'
}
}
}
post {
always {
echo 'I will always say Hello again!'
//do your zip step here
}
}
}
If you are not using the declarative, then you can wrap your cucumber tests in a try catch
see https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-catcherror-code-catch-error-and-set-build-result
node {
sh './set-up.sh'
try {
sh 'might fail'
echo 'Succeeded!'
} catch (err) {
echo "Failed: ${err}"
} finally {
sh './tear-down.sh'
}
echo 'Printed whether above succeeded or failed.'
}
Related
I am using Jenkins Ver:2.176.3 for declarative multibranch pipeline.
I have junit test in my application code base and using maven surefire plugin to run unit test using maven command.
Multibranch job page shows 'Test Result' link and 'Test Result Trend' graph also.
I think this is being displayed/published here due to plugin 'test-results-analyzer'.
In declarative pipeline we have two stages as shown in code sample and we use maven commands.
Now my problem is that this test result count same unit test for each stage of pipeline so the count of unit tests are being double on this Test Result of Jenkins job page.
I tried skipping unit test in stage 'package-IT' using maven option -DskipTests and as per log it does skip unit testing but still see the duplicate test results.
if know please suggest
stages{
stage('compile-N-test') {
agent {label 'agent'}
steps {
script {
// mvn clean test
}
}
}
stage('packaging-IT') {
agent {label 'agent'}
steps {
script {
//mvn verify
}
}
}
Finally i got the solution: in pipeline we are using withMaven() which invoke test result and keep adding it so increasing count.
I try to use the rerunFailingTestsCount option to deal with flaky tests. In order to display these in the junit results, I use the flaky test handler plugin which would theoretically deal with displaying flaky tests
In my jenkins file this looks like
pipeline {
stages {
stage('tests') {
steps{
// sh mvn verify here
}
post {
always {
junit testResults: 'target/failsafe-reports/**/*.xml', testDataPublishers: [[$class:
'JUnitFlakyTestDataPublisher']]
}
}
}
}
}
the test run fine, flaky ones are repeated but when it comes to publishing the junit results, i get an
Error when executing always post condition: java.lang.AbstractMethodError: you must override contributeTestData
Google wasn't very helpful, maybe someone here had the same problem and can help me or at least can confirm that this plugin works as pipeline script (there is a pull request regarding pipeline support, so I am not sure...)
As the title says, how can I make gradle not to fail a test task if no tests are found? I ran into this problem when I was using the --tests command line option with a multi-subproject project. For instance, this command below will run all tests in class FooTest from subproject A:
gradle test --tests com.foo.bar.FooTest
However, this command fails because of something like this:
Execution failed for task ':B:test'.
> No tests found for given includes: [com.foo.bar.FooTest]
BTW, I know something like below will succeed. But is it possible to make it succeed even with the test task? It's kind of annoying to type a test task name longer than test.
gradle :A:test --tests com.foo.bar.FooTest
The behavior you described is the current Gradle behavior, there is already a ticket on Gradle forum, see https://discuss.gradle.org/t/multi-module-build-fails-with-tests-filter/25835
Based on the solution described in this ticket, you can do something like that to disable the 'failIfNoTest' default behavior:
In your root project build (or better: in an InitScript in your Gradle USER_HOME dir, to make this behavior available for all your local projects)
gradle.projectsEvaluated {
subprojects {
// TODO: filter projects that does not have test task...
test {
filter {
setFailOnNoMatchingTests(false)
}
}
}
}
Then you can execute the following command without having errors if the given test doesn't exist in all sub-projects:
gradle test --tests com.foo.bar.FooTest
it seems that currently only a workaround like this is possible:
test {
afterSuite { desc, result ->
if (!desc.parent) {
if (result.testCount == 0) {
throw new IllegalStateException("No tests were found. Failing the build")
}
}
}
}
I have filed an issue with Gradle to introduce this as a simple config option: https://github.com/gradle/gradle/issues/7452
You can also run the tests only for the current project with
gradle :test --tests com.foo.bar.FooTest
Note the colon before the test task.
I know that I can use the -x test option to prevent the test task from getting called. I also have something like this in my gradle script to prevent tests from being executed in certain cases:
plugins.withType(JavaPlugin).whenPluginAdded {
test {
doFirst {
if (env.equals('prod')) {
throw new StopExecutionException("DON'T RUN TESTS IN PROD!!!!")
}
}
}
}
but is there a way to configure the java plugin to removed the dependency between build -> test?
build depends on test via check. You probably don't want to remove the dependency on check as it may do other things, so you could try:
check.dependsOn.remove(test)
Do you mind if I ask why you want to do this?
You can skip tasks via the command line with the -x option:
./gradlew assembleDebug -x taskToSkip
I don't know if it is possible to remove such a dependency.
You can however skip the execution of tasks, eg: skipping all test tasks (in production) goes like this.
tasks.withType(Test).each { task ->
task.enabled = !env.equals('prod')
}
I have a pretty standard Gradle build that's building a Java project.
When I run it for the first time, it compiles everything and runs the tests. When I run it a second time without changing any files, it runs the tests again.
According to this thread, Gradle is supposed to be lazy by defaut and not bother running tests if nothing has changed. Has the default behaviour here been changed?
EDIT:
If I run gradle test repeatedly, the tests only run the first time and are subsequently skipped. However, if I run gradle build repeatedly, the tests get re-run every time, even though all other tasks are marked as up-to-date.
the gradle uptodate check logs on info level why a task is not considered to be up-to-date. please rerun the "gradle build -i" to run with info logging at check the logging output.
cheers,
René
OK, so I got the answer thanks to Rene prompting me to look at the '-i' output...
I actually have 2 test tasks: the 'test' one from the Java plugin, and my own 'integrationTest' one. I didn't mention this in the question because I didn't think it was relevant.
It turns out that these tasks are writing their output (reports, etc.) to the same directory, so Gradle's task-based input and output tracking was thinking that something had changed, and re-running the tests.
So the next question (which I will ask separately) becomes: how do I cleanly (and with minimal Groovy/Gradle code) completely separate two instances of the test task.
You need to create test tasks in your build.gradle and then call those specific tasks to run a specific set of tests. Here is an example that will filter out classes so that they don't get run twice (such as when running a suite and then re-running its child classes independently):
tasks.withType(Test) {
jvmArgs '-Xms128m', '-Xmx1024m', '-XX:MaxPermSize=128m'
maxParallelForks = 4 // this runs tests parallel if more than one class
testLogging {
exceptionFormat "full"
events "started", "passed", "skipped", "failed", "standardOut", "standardError"
displayGranularity = 0
}
}
task runAllTests(type: Test) {
include '**/AllTests.class'
testReportDir = file("${reporting.baseDir}/AllTests")
testResultsDir = file("${buildDir}/test-results/AllTests")
}
task runSkipSuite(type: Test) {
include '**/Test*.class'
testReportDir = file("${reporting.baseDir}/Tests")
testResultsDir = file("${buildDir}/test-results/Tests")
}
Also, concerning your build question. The "build" task includes a clean step which is cleaning tests from your build directory. Otherwise the execution thinks the tests have already been ran.