The failsafe plugin code below is intended to populate a target server with subscriber records during the pre-integration-test phase and delete them again in the post-integration-test phase .
There are JMeter plugins bound to the integration-test phase which carry out performance tests .
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.16</version>
<executions>
<execution>
<id>create-subscribers</id>
<phase>pre-integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<test>AbcIntegrationIT#registerSubscriberGroup</test>
</configuration>
</execution>
<execution>
<id>delete-subscribers</id>
<phase>post-integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
<configuration>
<test>AbcIntegrationIT#deleteSubscriberGroup</test>
</configuration>
</execution>
</executions>
</plugin>
The first execution of this plugin runs OK and the performance test run fine .
However the second execution doesn't as , according to Failsafe
[INFO] --- maven-failsafe-plugin:2.16:integration-test (delete-subscribers) # digihost-integration-test-apm ---
[INFO] Skipping execution of surefire because it has already been run for this configuration
!?
I would argue that yes it has run for the pre-integration-test phase but not for the post-integration-test phase .
It looks as if other plugins do support multi-phase executions for a single goal (viz. cobertura plugin Running maven goal in multiple lifecycles ) , which leads me to wonder if this is a bug in Failsafe
Any Thoughts and Feedback greatly appreciated .
Related
Is there a way to enforce the post-integration phase to always run after the integration phase? By always I mean in the advent of test failures during integration phase.
I am running an Angular / Springboot application. I use protractor to run e2e tests that test the whole Angular + Springboot chain. I managed to integrate this in my Maven build so that I can:
setup the backend Springboot server
setup a DB with initial data
run protractor during the integration phase
with the following plugins:
spring-boot-maven-plugin which starts and stops a test server for integration testing:
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
...
</configuration>
<executions>
<execution>
<id>pre-integration-test</id>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>post-integration-test</id>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
</plugin>
and frontend-maven-plugin which runs my protractor tests during the integration phase:
<plugin>
<groupId>com.github.eirslett</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<configuration>
...
</configuration>
<executions>
<execution>
<id>install node and npm</id>
<goals>
<goal>install-node-and-npm</goal>
</goals>
<phase>generate-resources</phase>
</execution>
<execution>
<id>npm install</id>
<goals>
<goal>npm</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
<arguments>install</arguments>
</configuration>
</execution>
<execution>
<id>npm run build</id>
<goals>
<goal>npm</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
<arguments>run build</arguments>
</configuration>
</execution>
<execution>
<id>npm run integration tests</id>
<goals>
<goal>npm</goal>
</goals>
<phase>integration-test</phase>
<configuration>
<arguments>run e2e</arguments>
<testFailureIgnore>true</testFailureIgnore> // this should probably be deleted
</configuration>
</execution>
</executions>
</plugin>
I added testFailureIgnore = true to the frontend-maven-plugin because if any protractor test fails, it will stop my maven build before it gets to execute the post-integration phase. This causes the test server to keep running with that port. Any subsequent runs will fail since the port is already in use until that server is killed (manually). The testFailureIgnore property allows failed tests to be ignored by the build, effectively letting me continue with the post-integration phase.
The obvious downside is that my build will print SUCCESS even when tests have failed. I am looking for behavior similar to the failsafe plugin where failed tests will fail my build, but will still execute the post-integration phase first to cleanup properly.
I can't seem to find a proper solution for this but surely I can't be the first to encounter this problem. What solutions/alternatives are available for this? I imagine using the exec-maven-plugin instead of the frontend-maven-plugin will cause the same issue.
I didn't manage to find a decent solution to this anywhere so I decided to try and create my own. I extended the frontend-maven-plugin with a parameter that logs integration test failures during the integration-test phase, but only fails the build during the verify phase. This allows the post-integration-test phase to finish.
My solution is available from my repository (version 1.9.1-failsafe). This implementation requires a configuration parameter integrationTestFailureAfterPostIntegration to be added. Unfortunately I did not figure out how to make a Mojo execution trigger another Mojo execution at a later phase without user intervention. Because of this the user needs to have an execution that trigger during the verify phase, even if it doesn't do anything useful functionally (ie. npm -version).
My working example:
<execution>
<id>npm run integration tests</id>
<goals>
<goal>npm</goal>
</goals>
<phase>integration-test</phase>
<configuration>
<arguments>run e2e</arguments>
<integrationTestFailureAfterPostIntegration>true</integrationTestFailureAfterPostIntegration>
</configuration>
</execution>
<execution>
<id>fail any integration tests</id>
<goals>
<goal>npm</goal>
</goals>
<phase>verify</phase>
<configuration>
<arguments>-version</arguments>
</configuration>
</execution>
If any IT tests fail, they will be logged during integration-test phase and fail the build at verify. If all IT tests pass, the build will be successful.
I have an open pull request at the frontend-maven-plugin which might get added to the 1.9.2 version. I will still attempt to improve upon the change by removing the need for the verify execution phase to be added manually. Suggestions or improvements on the pull request are welcome!
UPDATE: I already went ahead and released my own version in case the pull request doesn't come through:
<dependency>
<groupId>io.github.alexandertang</groupId>
<artifactId>frontend-maven-plugin</artifactId>
<version>1.9.1-failsafe</version>
</dependency>
In this version I added a verify mojo which simplifies the second execution to:
<execution>
<id>fail any integration tests</id>
<goals>
<goal>verify</goal>
</goals>
<phase>verify</phase> <!--default phase is verify, so this is optional-->
</execution>
I resolved this puting this instructions on package.json
"scripts": {
...
"e2e": "ng e2e && echo Success > e2e/result.txt || echo Error > e2e/result.txt"
}
This will supress the exit code in error situation, and will record a file called result.txt whith Success or Error in your content.
Then, i add the maven-verifier-plugin on maven to verify the content of the file result.txt.
I have the following configuration for the maven failsafe plugin to run my integration tests (based on the documentation at Failsafe Usage Documentation:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.16</version>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
When I try to run an individual test using:
mvn -Dit.test=MyLovelyTest failsafe:integration-test
It does not run my test. It dies with the message: No tests were executed
If I remove the execution definition for verify, then it executes the test as expected. Since I copied the plugin def from the official usage documentation, I'm wondering if there's a bug in the plugin, or is it something I'm doing wrong?
TL;DR: Using maven, I want to run a plugin goal at the beginning of the test phase, before tests actually run. What would be a clean way to do it?
I want to print a message just before the tests actually run. Hence I want to use the echo goal of the echo plugin at the beginning of the test phase (to tell the user that if every tests fail, he'd better have a look at the README since there's a test environment he should set up first)
Attempt n°1
A simple approach could be to run this plugin in the previous phase, process-test-classes.
It works, but it doesn't seem semantically correct to bind this task to this phase...
Attempt n°2
According to Maven documentation, When multiple executions are given that match a particular phase, they are executed in the order specified in the POM, with inherited executions running first., so I tried to set explicitly the surefire plugin:
...
<plugin>
<groupId>com.soebes.maven.plugins</groupId>
<artifactId>maven-echo-plugin</artifactId>
<version>0.1</version>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>echo</goal>
</goals>
</execution>
</executions>
<configuration>
<echos>
<echo>*** If most tests fail, make sure you've installed the fake wiki. See README for more info ***</echo>
</echos>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
...
But tests run before my message is printed.
So, to put it in a nutshell: is there a way to reach my goal, or should I stick to the "process-test-classes solution" even though it seems a bit "hacky"?
Thanks!
As #khmarbaise said, your solution is still hacky, because whole test looks like Integration Test and should be processed by Failsafe Plugin. Failsafe has nice phase pre-integration-test for testing fake wiki etc :)
Based on Guide to Configuring Default Mojo Executions this works for me:
<plugin>
<groupId>com.soebes.maven.plugins</groupId>
<artifactId>maven-echo-plugin</artifactId>
<version>0.1</version>
<executions>
<execution>
<id>1-test</id>
<phase>test</phase>
<goals>
<goal>echo</goal>
</goals>
</execution>
</executions>
<configuration>
<echos>
<echo>*** If most tests fail, make sure you've installed the fake wiki. See README for more info ***</echo>
</echos>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<executions>
<execution>
<id>default-test</id>
<configuration>
<skip>true</skip>
</configuration>
</execution>
<execution>
<id>2-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
This is very odd for me ;)
I have two plugins with executions bound to generate-sources, one listed first in the list of about 6 plugins and the other listed last. However, the one listed last (which depends on the one listed first) always executes first.
How can I execute several maven plugins within a single phase and set their respective execution order?
I want to use maven-failsafe-plugin to run some integration tests. If any test fails, I want Maven to fail the build and not BUILD SUCCESS.
Tests run: 103, Failures: 1, Errors: 0, Skipped: 26
[INFO] BUILD SUCCESS*
how can I configure it, that build not success is?
My failsafe plugin is configured as:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${failsafe.version}</version>
<configuration>
<systemProperties>
<CI_INTEGRATION_OVERRIDE_PATH>${basedir}/..</CI_INTEGRATION_OVERRIDE_PATH>
</systemProperties>
<includes>
<include>**/integration/**/*.java</include>
</includes>
<excludes>
<exclude>**/integration/**/*TestSuite.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
As Andrew pointed out, the correct solution is to use failsafe as intended. The integration-test goal is specifically designed not to fail the build. If you want to fail the build, call mvn verify or mvn failsafe:verify
For the verify goal to work, it needs to read the results of the integration test which by default are written to ${project.build.directory}/failsafe-reports/failsafe-summary.xml so make sure that is getting generated.
Also, you have to make sure you bind your maven-failsafe-plugin configuration to both the integration-test goal and the verify goal in the executions part.
Failure to add either of those will lead to maven succeeding the build instead of failing it when integration tests fail.
Since you are running mvn clean install both the integration-test and verify phases should be executing. According to the failsafe plugin docs the failsafe:integration-test and failsafe:verify goals are bound to those phases, so I don't believe the extra call to failsafe:integration-test is required.
That said however, I'm not sure I trust the failsafe plugin documentation. I answered a similar question for someone earlier this year. It turned out he had to explicitly bind each goal to the correct phase and then failsafe worked as expected. Might be worth a shot.
I had a similar issue with Jenkins builds.
Running the tests locally resulted in failed tests and thus failed build
Running the tests on Jenkins resulted in failed tests but "BUILD SUCCESS"
Solution: in job configuration set MAVEN_OPTS to -Dmaven.test.failure.ignore=false.
Jenkins by default seems to ignore failed integration tests.
See also https://stackoverflow.com/a/28684048/4412885
solution.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
<executions>
<execution>
<id>unit-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>false</skip>
<excludes>
<exclude>**/*IntegrationTest.java</exclude>
</excludes>
</configuration>
</execution>
<execution>
<id>integration-test</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>false</skip>
<enableAssertions>false</enableAssertions>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
<systemPropertyVariables>
<integration>${integration}</integration>
</systemPropertyVariables>
</configuration>
</execution>
</executions>
</plugin>
I have the findbugs plugin working fine in my maven setup. I've setup findbugs to execute during the compile phase. I noticed however that it runs during the test phase as well because the test phase also calls compile. Because I have an automated build pipeline that runs all my targets, I don't need findbugs to run during the test phase. I've tried to exclude findbugs from the test phase with the following but no luck yet.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>2.4.0</version>
<inherited>true</inherited>
<configuration>
<failOnError>${findbugs.failOnError}</failOnError>
<skip>${findbugs.skip}</skip>
<trace>${findbugs.trace}</trace>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
<execution>
<id>findbugs-test-compile</id>
<phase>test</phase>
<goals>
<goal>check</goal>
</goals>
<configuration>
<skip>true</skip>
</configuration>
</execution>
</executions>
</plugin>
It will not be called based on the running through the life-cylcle via compile it simply is running cause you configured to have two executions one in test and one in compile phase. Findbugs should usually run in the reporting area(site).
Just make a single execution:
<executions>
<execution>
<id>findbugs-test-compile</id>
<phase>test</phase>
<goals>
<goal>check</goal>
</goals>
<configuration>
<skip>true</skip>
</configuration>
</execution>
</executions>
The one you like to have. But i recommend to read the documentation cause it should run in reporting area (via site) only.
UPDATE:
If you like to run findbugs only during the site generation than just remove it from the usual build area and put into the reporting area instead.