Ive included the surefire-report plugin to generate html reports.
When I run "mvn site", it runs my tests and generates the reports but the build passes even if there are test failures.
On the other hand, if I run "mvn install site", the build fails at the test failure and does not even do the site phase. I tried adding -fae but does not help. How can i prevent this?
Ideally I would like it to run through all tests in all my modules, run site, generate the reports and then fail the build.
Note, -Dmaven.test.failure.ignore=false already but no use. Also, if i remove the surefire-report plugin, everything is fine.
On a side note, any good alternatives to surefire-report plugin to generate html junit test reports?
Related
Currently our pipeline does and this creates a war file and puts in into the server
mvn clean test package
We are now trying to add surefire reports for those tests, everything works well when all tests pass
mvn clean test site package
But if some test fails then site does not get called and we have no report.
Although I know that site has tests inside as well so we tried but even if there was failing tests the site command did not make it to be failing and build succeeded.
mvn clean site package
What is the best approach to get tests and surefire report.
With How to build a jar using maven, ignoring test results?
and https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#testFailureIgnore:
<testFailureIgnore>(element) boolean(type) -((ever) since)
Set this to "true" to ignore a failure during testing. Its use is NOT RECOMMENDED, but quite convenient on occasion.
Default value is: false.
User property is: maven.test.failure.ignore.
Using:
mvn -Dmaven.test.failure.ignore=true clean package site
Will:
clean
package (including tests + test results, unless skipped)
not fail the build, when tests fail
and generate the project report (site)
I'm bit confused with mvn verify phase. I've created a Spring Boot project (a simple project, without any explicit configurations added). I've created a few JUnit unit tests which are run with both the mvn verify and mvn test commands.
There isn't any difference observed in the mvn verify and mvn test command output.
What does mvn verify do different than mvn test?
Also some posts on Stack Overflow mentions that mvn verify runs the integration tests. If this is the case then I have few questions.
How does Maven identify a specific test as a unit test or integration test?
If mvn verify is supposed to run only the integration tests, then why are unit tests executed with it?
First of all, when you run a Maven goal, it will run any previous goal. The order of basic phases is:
Validate
Compile
Test
Package
Verify
Install
Deploy
If you run Test, Maven will execute validate, compile and test. Based on this, the first point is that verify includes test.
Based on official documentation:
TEST - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed
VERIFY - run any checks on results of integration tests to ensure quality criteria are met
To run unit tests, the Surefire plugin is recommended. And Failsafe for integration tests.
The verify command executes each default lifecycle phase in order (validate, compile, package, etc.), before executing verify. In most cases the effect is the same as package. However, in case there are integration tests, these will be executed as well. And during the verify phase some additional checks can be done, e.g. if your code is written according to the predefined checkstyle rules.
Conclusion: if you want to run your integration tests and check it, use verify. If you only want to run unit tests, use test.
My personal advice: if in doubt, use verify.
How does Maven identify a specific test as a unit test or integration test?
Integrations Test always takes a name like IT.java or IT.java or *ITCase.java
When generating a skeleton Maven plugin from archetype, the new project includes a Maven project under the src/it directory. It is an integration it (hinted at by the it dir name) and fresh out-of-the-box it passes when run during Maven's integration-test phase.
There are nearly 10 such IT Maven projects, a subset of which intentionally result in BUILD FAILURE, and attendant verify.groovy scripts that ensure those builds fail for the correct reason. Ideally each IT test sub-build that fails for the correct reason results in that IT test passing, but by including any of these failing IT tests as part of the whole integration test suite causes the overall Maven run to fail as well, which is incorrect in my case.
How do I coax Maven to run those failing Maven sub-builds, ignore their build results, but honor the results of their Groovy verification scripts?
Edit: One IT test (disabled) is committed here.
If you like to write an integration test which is intended to fail as a result
you have to express this via the invoker.properties file like this:
invoker.buildResult=failure
The full description of the file can be found in the documentation.
I am using below commands to build my maven code.
Compile-
-DargLine="-DDB_SERVER=localhost -DDB_PORT=5432 -DDB_NAME=sample -DDB_USER=sample -DDB_PASSWORD=sample -DDB_MAX_POOL=10" -Dcom.sample.redis=false clean compiler:compile
Unit-Test Analysis-
DargLine=-DDB_SERVER=localhost -DDB_PORT=1234 -DDB_NAME=sample -DDB_USER=sample -DDB_PASSWORD=sample -DDB_MAX_POOL=10 -Dcom.sample.redis=false -Dcobertura:cobertura-integration-test -Dcobertura.aggregate=false -Dcobertura.report.format=xml integration-test test
And using below sonar properties to capture xml to publish in sonar.
sonar.projectKey=sample
sonar.projectName=sample
sonar.projectVersion=$PipelineId
sonar.modules=admin,client-api,common,om,serviceproviders
sonar.cobertura.reportPath=target/site/cobertura/coverage.xml
sonar.sources=.
sonar.skipPackageDesign=true
sonar.sourceEncoding=UTF-8
Being Multi module the code coverage is showing only 9.4%. Am I missing anything. I don't see any error logs as well.How can I achieve the same using coverage tool like Jacoco.
SonarQube - Version 5.1.1 - LGPL v3
Maven has a lifecycle Maven Lifecycle where each of the targets includes the ones before it. e.g. "test" includes "compile", "integration-test" includes "test", etc. You generally need to only include the target at the tip of the lifecycle. e.g. "mvn test" means (compile AND run the tests).
I'm thinking you want to run the "mvn verify" goal, which is compile, run tests and integration tests, and then run verifications (coverage checks, etc). Cobertura has a plugin that should integrate with maven and tap into various goals to run its pieces at appropriate times. I'm guessing you are messing up cobertura by having multiple targets and trying to break it into pieces - i.e. overwriting the instrumentation or something.
Similarly, you might find using jacoco easier than cobertura. It hides the instrumentation, and integrates pretty seemlessly with maven.
Good luck.
So I have Jenkins with the Cobertura plugin installed. I have Cobertura and findbugs in the POM and my tests are running twice...
I assume that this is because Cobertura instruments the bytecode and this causes the tests to re-run, which isn't a bad thing I guess, since the instrumented isn't the same as non-instrumented code...but I would really like the tests to be run only once.
I have tried running them locally on commandline using these commands:
mvn cobertura:cobertura -Dcobertura.report.format=xml
mvn findbugs:findbugs -Dfindbugs.onlyAnalyze=true
mvn cobertura:cobertura -Dcobertura.report.format=xml findbugs:findbugs -Dfindbugs.onlyAnalyze=true
but I can't get the tests to run twice locally, where as on Jenkins the are running twice. I am not sure why this is happening and whether I could make it stop.
I am using Cobertura to generate reports for me...I assume that to generate them it needs to re-run the tests? But it doesn't make sense since they are already being run once.
We have faced the same behavior. It seems to be default behavior of Cobertura to rerun test cases for calculating coverage.
We switched to JaCoCo tool, which proved to be better . It does not re-run the test cases for coverage report.
Indeed you have to run tests twice with cobertura-maven-plugin (or use different profiles). This behavior is due to the fact that it runs Cobertura instrumentation in its own lifecycle and uses Cobertura executable instead of Cobertura API.
If you want to generate a Cobertura report while only running your tests once, you can give a try to the qualinsight-mojo-cobertura-core plugin. This plugin uses a different approach to run Cobertura instrumentation.
You'll find a documentation on the project's page: https://github.com/QualInsight/qualinsight-mojo-cobertura .
Note that this plugin has still some limitations, but it may be a viable alternative in your context.
Hope it helps !
Regards.