mvn test
I am looking to initialize certain resources before all test cases executed using maven test and also to destroy them after all test cases are executed.
I have looked into jUnit #BeforeClass, #AfterClass, #Before and #After, but none of them are helpful.
I have tried to use maven life-cycle phases i.e. pre-integration-test as below, but even in this case the expected Test case (TestPostgresqlEmbedded) doesn't get executed first.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.plugin.version}</version>
<executions>
<execution>
<id>test-init</id>
<configuration>
<runOrder>alphabetical</runOrder>
<includes>
<include>**/TestPostgresqlEmbedded.java</include>
</includes>
</configuration>
<phase>pre-integration-test</phase>
</execution>
<execution>
<id>test-all</id>
<configuration>
<runOrder>alphabetical</runOrder>
</configuration>
</execution>
</executions>
</plugin>
How can I achieve this ?
Use the integration-test phase for tests with databases. Then you have pre-integration-test to set up your database resources, and post-integration-test to destroy them.
Related
When I run mvn test it executes unit tests only but when I run mvn integration-test it executes both unit test and integration test even after configuring the maven-failsafe-plugin and excluding the *Test.java file. Not sure what I am missing here. Also worth mentioning that I have not put in maven-surefire-plugin in my pom.xml. Not sure if that is creating this problem. Please guide.
pom.xml
<!-- Integration tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.18</version>
<configuration>
<includes>
<include>**/*IT.java</include>
</includes>
<excludes>
<exclude>**/*Test.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
Unit Test Class File:
com.study.jenkins.ut.MyUnitTest.java
Integration Test Class File:
com.study.jenkins.it.PageIT.java
The maven-surefire-plugin is part of the lifecycle, which is always bound to the test-phase for Java projects. Calling integration-test means that all lifecycle-phases up to the integration-test phase are executed. So the MyUnitTest will always be executed (which is a good thing).
Your includes/excludes have no effect, these are already the defaults for the maven-failsafe-plugin, see http://maven.apache.org/surefire/maven-failsafe-plugin/integration-test-mojo.html#includes
i'm trying to put some bdd into my eclipse plugin project, but can't figure out how to run my integration tests during maven build fase. To write my tests i'm using SWTBot framework.
I already did the feature generation fase, and setup my tests. How do i setup my pom to run my integration tests?
I use below configuration and run mvn clean verify. If you don't want to run tests in parallel, remove parallel, perCoreThreadCount and threadCountClasses tags.
Make sure to update the regular expression to match your test naming convention <include>**/Run*.java</include>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
<executions>
<execution>
<id>acceptance-test</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<outputEncoding>UTF-8</outputEncoding>
<parallel>classes</parallel>
<perCoreThreadCount>true</perCoreThreadCount>
<threadCountClasses>10</threadCountClasses>
<argLine>-Xmx1024m</argLine>
<argLine>-XX:MaxPermSize=256m</argLine>
<includes>
<include>**/Run*.java</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
I want to use maven-failsafe-plugin to run some integration tests. If any test fails, I want Maven to fail the build and not BUILD SUCCESS.
Tests run: 103, Failures: 1, Errors: 0, Skipped: 26
[INFO] BUILD SUCCESS*
how can I configure it, that build not success is?
My failsafe plugin is configured as:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${failsafe.version}</version>
<configuration>
<systemProperties>
<CI_INTEGRATION_OVERRIDE_PATH>${basedir}/..</CI_INTEGRATION_OVERRIDE_PATH>
</systemProperties>
<includes>
<include>**/integration/**/*.java</include>
</includes>
<excludes>
<exclude>**/integration/**/*TestSuite.java</exclude>
</excludes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
As Andrew pointed out, the correct solution is to use failsafe as intended. The integration-test goal is specifically designed not to fail the build. If you want to fail the build, call mvn verify or mvn failsafe:verify
For the verify goal to work, it needs to read the results of the integration test which by default are written to ${project.build.directory}/failsafe-reports/failsafe-summary.xml so make sure that is getting generated.
Also, you have to make sure you bind your maven-failsafe-plugin configuration to both the integration-test goal and the verify goal in the executions part.
Failure to add either of those will lead to maven succeeding the build instead of failing it when integration tests fail.
Since you are running mvn clean install both the integration-test and verify phases should be executing. According to the failsafe plugin docs the failsafe:integration-test and failsafe:verify goals are bound to those phases, so I don't believe the extra call to failsafe:integration-test is required.
That said however, I'm not sure I trust the failsafe plugin documentation. I answered a similar question for someone earlier this year. It turned out he had to explicitly bind each goal to the correct phase and then failsafe worked as expected. Might be worth a shot.
I had a similar issue with Jenkins builds.
Running the tests locally resulted in failed tests and thus failed build
Running the tests on Jenkins resulted in failed tests but "BUILD SUCCESS"
Solution: in job configuration set MAVEN_OPTS to -Dmaven.test.failure.ignore=false.
Jenkins by default seems to ignore failed integration tests.
See also https://stackoverflow.com/a/28684048/4412885
solution.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
<executions>
<execution>
<id>unit-test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>false</skip>
<excludes>
<exclude>**/*IntegrationTest.java</exclude>
</excludes>
</configuration>
</execution>
<execution>
<id>integration-test</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>false</skip>
<enableAssertions>false</enableAssertions>
<includes>
<include>**/*IntegrationTest.java</include>
</includes>
<systemPropertyVariables>
<integration>${integration}</integration>
</systemPropertyVariables>
</configuration>
</execution>
</executions>
</plugin>
I've got a JUnit 4 test suite that contains a number of test classes in the order they need to be run (our Integration tests need to be run in a certain order).
If I use the maven-failsafe-plugin without any configuration it will run the test but not in the correct order. However, If I set the plugin to run the test suite no tests are run.
Is it possible to run a test suite using the failsafe plugin? if so, where have I gone wrong!!
The code is below:
#RunWith(Suite.class)
#SuiteClasses({
TestCase1.class,
TestCase2.class,
...
TestCaseN.class,
})
public class IntegrationSuite {
//Do Nothing.
}
and from pom.xml:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.9</version>
<configuration>
<includes>
<include>IntegrationSuite.java</include>
</includes>
</configuration>
<executions>
<execution>
<id>integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
Thanks :)
Failsafe plugin supports runOrder (click) parameter since version 2.7 (quite recent). There aren't many options, you cannot specify the order explicitly, but you can set it to "alphabetical" and rename your test classes to reflect the run order.
May I also say on the occasion that the fact that test depend on each other is (test) code smell; it's not good, as it is a short path to developing an unmaintainable set of tests and abandoning it finally when its complexity skyrockets above human comprehension. Plus it may fail to expose bugs, as it is a result of one chosen execution path.
BTW, I prefer to include tests like this, with a double asterisk:
<includes>
<include>**/IntegrationSuite.java</include>
</includes>
maven-surefire-plugin can also be used as below code:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<includes>
<include>**/IntegrationSuite.java</include>
</includes>
</configuration>
</plugin>
AFAIK, the maven failsafe plugin fails safe because it has separate goals for running the tests and failing the build based on the tests. These are designed to be bound to the integration-test and verify goals respectively. This allows post-integration-test bound goals to run (shutting down the build) before the build fails.
My question is, how do I do this with the maven-soapui-plugin? I thought I could simply specify <testFailIgnore>true</testFailIgnore> in my soapui plugin config and then call the failsafe plugin verify goal, but that isn't working. I don't think I'm not sure if I'm getting a summary file out of the soapui plugin or not. I keep getting Expected root element 'failsafe-summary' but found 'testsuite' Here is a snippet of the POM:
<plugin>
<groupId>eviware</groupId>
<artifactId>maven-soapui-plugin</artifactId>
<version>4.0.0</version>
<configuration>
<junitReport>true</junitReport>
<exportAll>true</exportAll>
<outputFolder>${project.build.directory}/surefire-reports</outputFolder>
<testFailIgnore>true</testFailIgnore>
<printReport>true</printReport>
</configuration>
<executions>
<execution>
<id>FailingTest</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<projectFile>${basedir}/testData/soapui-integration-tests.xml</projectFile>
<host>localhost</host>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.9</version>
<executions>
<execution>
<phase>verify</phase>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
<configuration>
<summaryFiles>
<summaryFile>${project.build.directory}/surefire-reports/TEST-TestSuite_1.xml</summaryFile>
</summaryFiles>
</configuration>
</execution>
</executions>
</plugin>
Is there something wrong with my POM or is this a bad approach? Are there any better approaches?
There is an open source extension of the soapui plugin which has a separate test-verify goal for exactly this purpose.
https://github.com/redfish4ktc/maven-soapui-extension-plugin
The following example shows the required configuration:
https://github.com/redfish4ktc/maven-soapui-extension-plugin/blob/master/src/it/test-verify_goal/one_failing_project/pom.xml
AFAIK maven-failsafe-plugin can only verify success status of tests run by maven-failsafe-plugin and not by maven-soapui-plugin. It does that by reading test summary report file (failsafe-summary.xml) which has specific format.
maven-soapui-plugin could be adjusted to separate running tests from verifying test success status, to support running post-integration-test tasks (stop server, undeploy artifacts, etc.) before verification. Create a support ticket for this to soapUI guys.
Maybe even maven-failsafe-plugin, it's verify mojo, could be extended to allow specifying different test report format (JUnit style reports are supported by soapUI) or even an xpath expression which would be used by maven-failsafe-plugin to determine if there were failed tests or not. Create a support ticket for this on maven-failsafe-plugin issue tracker.
Until those extensions are supported, and you need to do tasks on post-integration-test phase you can use soapUI JUnit integration and have maven-failsafe-plugin run those soapUI JUnit tests.
I am trying this solution, and it doesn't work.
But I have found one.
To obtain de tests report of SOAPUI tests in Jenkins, I using the failsafe plugin with this configuration in the pom.xml of my Soap tests folder :
<build>
<plugins>
<plugin>
<groupId>eviware</groupId>
<artifactId>maven-soapui-plugin</artifactId>
<version>2.6.1</version>
<configuration>
<projectFile>${basedir}/soap_project_tests.xml</projectFile>
<outputFolder>${filePath.reports.soap}</outputFolder>
<testFailIgnore>true</testFailIgnore>
<junitReport>true</junitReport>
<exportwAll>true</exportwAll>
<printReport>true</printReport>
</configuration>
<executions>
<execution>
<id>run-soap-integration-test</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.11</version>
<configuration>
<reportsDirectory>${filePath.reports.soap}</reportsDirectory>
<printSummary>true</printSummary>
<argLine>-Xmx512m</argLine>
</configuration>
<executions>
<execution>
<id>soap-integration-test-verify</id>
<phase>post-integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The tests cases status are up to Jenkins.