When running serenity tests in jenkins I have a stage that runs mvn clean verify. My problem is that this stage always passes as the build of the tests is successful and the stage passes. At the end of the stage I see:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:39 min
[INFO] Finished at: 2023-01-24T14:50:04Z
[INFO] ------------------------------------------------------------------------
even though there are test failures and errors. One solution to this was to use the catchError method with the following:
catchError(buildResult: 'SUCCESS', stageResult: 'UNSTABLE') {
sh "exit 1"
}
but this always shows unstable as some of my tests check error conditions and they pass when an error is thrown. I'd like the tests to be shown as passed in jenkins when Serenity says the tests have passed. Is there a way to do this?
A
Related
I have a legacy websphere EE7 app I am attempting to convert to run on openliberty. It now seems to run on openliberty which is great.
However the integration tests do not seem to run. They hot compile, but do not run. The application is deployed as an EAR file.
The log is below with no evidence of the tests running when I hit enter.
[INFO] To run tests on demand, press Enter.
[INFO] Waiting up to 30 seconds to find the application start up or update message...
[INFO] CWWKM2010I: Searching for (CWWKZ0001I:|CWWKZ0003I:) in <app name removed>\logs\messages.log. This search will timeout after 30 seconds.
[INFO] CWWKM2015I: Match number: 1 is [19/07/22 18:06:21:660 NZST] 00000037 com.ibm.ws.app.manager.AppMessageHelper
A CWWKZ0001I: Application <my application> started in 17.692 seconds..
[INFO] Running integration tests...
[INFO] Integration tests finished.
[INFO]
[INFO] To run tests on demand, press Enter.
[INFO] Tests compilation was successful.
[INFO] Running integration tests...
[INFO] Integration tests finished.
[INFO]
[INFO] To run tests on demand, press Enter.
[INFO] Running integration tests...
[INFO] Integration tests finished.
[INFO]
[INFO] To run tests on demand, press Enter.
Has anyone got any tips on how I can get the tests to run when I press enter? They must be being called from something, or have their path set by something, or require some maven plugin configuration?
I am running Jenkins version 1.411 and use Maven for building.
Even though the application builds successfully, Jenkins treats it as an unstable build. I have disabled all tests to isolate the problem.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45.389s
[INFO] Finished at: Wed May 11 12:16:57 EST 2011
[INFO] [DocLinks] Skip document Adaptiv generated site ... (No such directory: site)
Final Memory: 27M/543M
[INFO] ------------------------------------------------------------------------
channel stopped
Archiving artifacts
Email was triggered for: Unstable
Sending email for trigger: Unstable
An attempt to send an e-mail to empty list of recipients, ignored.
Finished: SUCCESS
It's some time ago that I used hudson/jenkins but you should have a look at the Jenkins Glossary
Unstable build: A build is unstable if it was built successfully and one or more publishers report it unstable. For example if the JUnit publisher is configured and a test fails then the build will be marked unstable.
Publisher: A publisher is part of the build process other than compilation, for example JUnit test runs. A publisher may report stable or unstable result depending on the result of its processing. For example, if a JUnit test fails, then the whole JUnit publisher may report unstable.
So I suppose you have some other build parts (apart from JUnit) that report an unstable result. Have a look at the whole build process log.
If you have unit tests, make sure that they run when executing your build.
In my case, the unit tests are not run and Jenkins tagged the build as Unstable.
When checking the console output, I found that 0 unit tests were running :
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
The fix for me is will not help resolve this topic but no harm to mention it:
I was Spring boot 2.2 and when I upgraded to v2.5, JUnit 5’s Vintage Engine was removed from spring-boot-starter-test, so I needed to add it to execute tests using JUnit 4.
Is the maven surefire plugin needed to run unit tests when using the mvn clean test command?
I read the documentation and know it says that:
The Surefire Plugin is used during the test phase of the build
lifecycle to execute the unit tests of an application. It generates
reports in two different file formats:
Plain text files (.txt) XML files (.xml) By default, these files are
generated at ${basedir}/target/surefire-reports.
However, I ran the tests without using the surefire plugin and they all passed.
no not needed but it is used by maven per default. if you have a better plugin you can change it. but i would prefer the defaults.
the maven-surfire-plugin is bound to the test phase of the default lifecycle. the maven-surfire-plugins runs all Tests matches the filename pattern Test*.java, *Test.java und *TestCase.java reside in the directory src/test/java.
for more information have a look at http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html
you can verify this on the console output... here is the tailed output of mvn test...:
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # rechnungsverwaltung ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) # rechnungsverwaltung ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.447 s
[INFO] Finished at: 2015-10-17T22:36:59+02:00
[INFO] Final Memory: 18M/304M
[INFO] ------------------------------------------------------------------------
Hava a look at the line. there you can see that the maven-surefire-plugin is used
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) #
I have a maven build which I would like to execute the jar creation for, but mvn package fails due to unit test failures.
I'd like to still build the jar in spite of the fact that some tests fail, I tried
mvn package -Dtest=false -DfailIFNoTests=false
But that fails.
mvn jar:jar seems to work, but I'd rather use the more idiomatic mvn package approach.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.119s
[INFO] Finished at: Mon Mar 11 12:07:16 EDT 2013
[INFO] Final Memory: 9M/234M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.10:test (default-test) on project glusterfs: No tests were executed! (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]
The surefire plugin controls test running and failure reporting.
mvn package -Dmaven.test.failure.ignore=true
should compile and run your tests, but not fail the whole build if some of the tests fail. (I've never used this, going on reading docs only.)
You can skip running unit tests with the following:
-Dmaven.test.skip=true
I am trying to build a multi-module Maven project using Jenkins.
When I build the same project on the command-line using the same environment (variables/settings.xml/user) as Jenkins does, test failures cause the build to fail immediately:
Failed tests:
testSomething(com.package_name.TestSomethingOrTheOther)
Tests run: .., Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] There are test failures.
...
# Build fails immediately
Where-as when building in Jenkins:
Failed tests:
testSomething(com.package_name.TestSomethingOrTheOther)
Tests run: .., Failures: 1, Errors: 0, Skipped: 0
[ERROR] There are test failures.
...
# Build continues to other modules
...
[INFO] Reactor Summary:
[INFO] ------------------------------------------------------------------------
[INFO] parent ................................................ SUCCESS [.....s]
[INFO] module-that-failed .................................... SUCCESS [.....s]
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
The build states that all modules and parent are SUCCESS-ful, when in reality, these should be failures.
How do I get Jenkins to enforce fail-fast in maven builds as maven does in the command-line?
Thanks in advance.
Jenkins ignores test failures by default, and then marks the build as unstable if they were any.
The parameter that controls that is: testFailureIgnore (see surefire plugin doc)
I've never tried that, but I would attempt to override that setting on the jenkins job configuration:
-Dmaven.test.failure.ignore=false
You can specify MAVEN_OPTS if you click on Advanced button in the Build section.