I am using Maven and the failsafe plugin to execute a set of runners that run integration tests.
One of these runners execute tests that involve altering my configuration. For this reason, I would like this runner to be executed only when the other tests have finished executing. Otherwise, there is the possibility of unrepresentative tests.
How can I go about this?
To expand on #khmarbaise comment, in Testng you have annotations that can help you set and reset config files
http://testng.org/doc/documentation-main.html#annotations
Related
IntelliJ has a nice integration with JUnit that allows me to run some or all of my unit tests and visualize the results of each test and test class with a collapsible/expandable hierarchy that even takes into account JUnit5 nested test classes. I can click on any failed test to see the log output from running just that method. It looks like this:
I have now defined an integration test phase to my project using two popular Maven plugins:
maven-failsafe-plugin separates my JUnit tests into two phases. The test classes ending in "Test" run during the test phase, and those ending in "IT" run during the integration-test phase.
fabric8's docker-maven-plugin which I'm using to spin up a test database in the pre-integration-test phase and shut it down in the post-integration-test phase.
I can tell IntelliJ to run a Maven command (such as maven clean verify) that runs this lifecycle and tells me if the tests have passed. However, a Maven "run configuration" in IntelliJ doesn't produce the same helpful output. All I get is the success or failure status of each phase, and I can navigate to the console output for a phase, but not an individual test. The output looks like this:
So here's my question: Is there any way to configure IntelliJ (or Maven, or both) to give that hierarchical test-by-test output while keeping the Maven lifecycle I've defined, with the temporary use of a docker-container database for my integration tests?
I know that I can manually spin up the test database container, use IntelliJ's JUnit runner for my integration tests, and then manually stop the container. Can I do better?
Unfortunately there is no tool to show test-by-test output while keeping the Maven lifecycle as Maven run configuration is used in this case. Please use Junit run configuration for Class/method output in test runner tab.
I have both unit and acceptance tests in a maven project (spring boot). I would like to run unit tests only when my build runs in TeamCity. I am using cucumber.
When i run the tests via command line, everything works as expected (only unit tests are run)
c:\apache-maven-3.3.9\bin\mvn package -Dcucumber.options="--tags #unit"
However, in teamcity, all tests are being run (unit and acceptance). It seems teamcity ignores my cucumber.options
In addition, when I double click on the 'test' lifecycle in Intellij, all tests are run as well (not just unit tests) So my guess is that TeamCity is doing exactly what the 'test' lifecycle does.
How can i get around this problem (in TeamCity)?. I have tried using a 'Command line' step, which works, however, i lose all the tests reporting as well as test coverage reports.
i have solved the Intellij problem by creating (or changing) a configuration:
Try writing it this way in TeamCity:
"-Dcucumber.options= --tags #unit"
I have a Continuous integration setup using jenkins. I need to run Junit test cases and selenium test cases and commit the jar files in svn, only if all the junit and selenium scripts are passed.
Junit and selenium test cases are separate maven projects so i have used two invoke top level maven targets (One for selenium and other for Junit). I came across following issues,
Selenium scripts will be executed first and if any of the selenium test cases are failed, the jenkins build is marked as Failed and it will start to execute the post-build actions skipping all the build steps including junit. I want to run the Junit test cases also even if selenium scripts are failed. So i have added -Dmaven.test.failure.ignore=true in maven goals to execute the junit test cases even if selenium scripts are failed. This worked perfectly as i expected and continued to execute junit test caes and the build is marked as UNSTABLE.
Now i got another issue, i have a build step to invoke ant at the end (To commit jars in svn only if selenium and junit are passed). After adding -Dmaven.test.failure.ignore=true, the jars are getting commited to svn even if there is any failures.
Can anyone help me to solve this problem?
The build should run both Selenium and Junit, even if there is any failure in any one of project.
Jars should be committed in svn only if everything is successful (This is done by using ant target, so i have placed invoke ant at the end in Build step.
Is there any other way to accomplish this ? Thanks in advance.
In a multi module mvn 3.0 build, I set forkedProcessTimeoutInSeconds to one hour. If tests in module A hang, I hit surefire threshold which will fails the build and skips remaining modules. We use a junit timeouts which should kill hung tests prior to this but some scenarios encounter this problem. I'd like to keep my builds running while I investigate enhancements to our junit runner.
Current Command Line: mvn clean install -Dmaven.test.failure.ignore
How I can change the surefire timeout behavior to move to next module and continue the build?
Does anyone have tips for testing one's junit framework?
I was going to review the surefire project's integration tests for ideas for both items. I plan to have a junitsystest module which activated by a specific profile so I can test various problematic situations outside of my builds.
According to Mastering The Maven Command Line – Managing failures --fail-at-end or --fail-never will provide the desired results. Surefire timeout continues to be reported but doesn't halt the build.
Not recommending in the long run as it lengthens bad build times
mvn clean install -Dmaven.test.failure.ignore --fail-never
I'd like to configure jasmine-maven-plugin to make jenkins unstable if a test fails but the only options appear to be:
set haltOnFailure true and have failures break the build
set haltOnFailure false and have failures reported in the logs but the build succeeds.
Is there a way to check the logs post-test and mark the build unstable?
Sam Hasler's answer only works for freestyle Jenkins jobs. We use Maven jobs and this configuration option of the JUnit Jenkins plugin is unavailable for Maven jobs. So I was looking for a more universal solution.
What we did in order to get it working is to reconfigure the Jasmine Maven plugin so as to
no longer halt the build upon test failures and
write the Jasmine test reports to target/surefire-reports where Jenkins expects to find them. This has the additional advantage that we now also see the failed Jasmine test in the build job alongside our Java tests.
<haltOnFailure>false</haltOnFailure>
<jasmineTargetDir>${project.build.directory}/surefire-reports</jasmineTargetDir>
Now our build jobs are yellow (unstable) as expected, no longer red (failed).
Found the answer myself!
I had to configure jenkins to also look at the jasmine junit report:
under Publish JUnit test result report add **/TEST-jasmine.xml to Test report XMLs, comma separated if there is something there already:
**/TESTS-TestSuites.xml,**/TEST-jasmine.xml