I have a bunch of tests implemented using Spock and that contain the #Stepwise clause, meaning that each test method contained in each class must be executed sequentially (they're dependant on the previous one).
Now I'd like to implement a retry mechanism for flaky tests and I'm trying the one provided by the surefire plugin.
this works, but it only retries the test method that failed, when I want it to rerun the whole class (even the tests that were executed correctly, since they're a pre-condition for the next ones). How can it be configured?
An example of how one of my test classes looks like (I know it's a stupid one, I just wrote it like this way because it is suitable as en example):
#Stepwise
class DummyTest extends loginWithNewUser {
def infoDialogAppears() {
given:
at homePage
when:
infoButton.click()
then:
infoDialog
}
def closeInfoDialog() {
when:
infoDialog.close()
then:
!infoDialog
}
Your test is not a simple Spock test but a Geb test. Anyway, have you considered handling flaky tests with the Spock #Retry annotation available both in Spock 1.3 and Spock 2.0?
I have not tried to do that from Surefire/Failsafe or even tried to do it in Spock at all because I believe in fixing flaky tests and having them red if they do fail instead of hiding the flakiness, eliminating the desire to fix it. But FWIW, Spock has that feature built in and I recommend to use it sparingly and on a per-test basis rather than globally because re-running each single failing integration test (even the non-flaky ones) just makes your test execution slower and keeps developers waiting for feedback.
Related
I am writing integration tests in a SpringBootApp with Junit5. It is a must for me to follow a execution order of REST Api calls. In one test class it is ok to use #TestMethodOrder from jUnit5 or another approach for test execution ordering in a class. My question is regard execution test classes in specific order. I found an open issue on the topic in the Junit repo https://github.com/junit-team/junit5/issues/1948 . It's not resolved. Is there a way to achieve this? I have think of #Categories introduced from Junit5 but it will be too hacky approach, it is not an option.
I found this page that explains how to run Geb/Spock tests at the method level which is what I would like to do with my tests, but I am using gradle. Is there a way to convert this to gradle or is it strictly a maven plugin? I can import the maven-surefire-plugin with gradle just fine, however I can't figure out how to convert the configuration block, or if it is even possible.
I've tried something like below but it doesn't work.
tests {
options {
parallel = "methods"
forkCount = 4
}
}
I can execute the tests at the class (spec) level by using gradle maxParallelForks property, but I'd like to run parallel at the test level.
If you are able to run tests in parallel on the method level depends on what test framework you are using.
As far as I know, only TestNG supports it out of the box.
See here: https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/testing/testng/TestNGOptions.html#setParallel-java.lang.String-
There is way to make it work independently of the test framework, using only Gradle, but this way you can only do it on the class level.
In your Gradle test task, set the maxParallelForks property.
See manual: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:maxParallelForks`
I need to know if testing only restControllers in spring boot application is enough in unit testing or better to test all the classes independently?.
There is no simple and 'right' answer. You should decide it for yourself.
First of all it worth to mention two things:
Framework for testing controllers: Spring MockMvc framework, it is for integration testing of Spring web applications. It can test single controller (with mocked service layer) and entire application (including service and database layers, with embedded databases).
Article answering which tests you should have: Test pyramid conception (in short you should have a lot of unit tests and a little of integration tests).
My personal preference is to cover the 'happy path' by integration tests (covering service and DB layers).
Then I cover by unit tests classes with complex logic (not all classes).
In my practice most of applications are quite simple, and I found that I have a lot of integration tests and not so many unit tests.
Max Fariskov is right, and I can add two remarks.
Unit tests work properly when you launch them really often. After
every compile is often enough :). That's why every unit test must chek one little part of the functionality. Testing the whole controller is a too complex task.
Decouple business logic from a controller,
Put it into service,
Test service with its own unit test
Mock service functionality in your controller - test and check only
controller behavior.
Unit tests are fast but the integration tests are slooow and complex. So it is a good idea to split them into different groups. Execute your unit tests often and your integration tests only when you have to.
I'm doing it with "profiles":
Create two empty interfaces. For example:
public interface UnitTest {}
public interface IntegrationTest {}
Mark your test classes with proper annotation
#Category({IntegrationTest.class})
public class MySlowAndComplexServiceTest {}
#Category({UnitTest.class})
public class MyFastServiceTest {}
create two profiles in you .pom
unitTests
true
maven-surefire-plugin
2.19.1
UnitTest.class
integrationTests
false
maven-surefire-plugin
2.19.1
IntegrationTest.class,UnitTest.class
It's done.
Your unit test will be executed by default and you can switch on "integrationTests" profile to activate you integration tests at any moment.
Well, I would like to have a maven goal execute-custom-tests inside my custom-maven-plugin that consists of running test methods (This tests are not unit tests). Something similar to test goal of soapui-pro-maven-plugin, for example.
Why? Basically the main objectives of the plugin are testing stuff (not unit testing) and the tests in src/test are for unit testing, right?
Being more specific I was thinking about something like this:
#Mojo (name = "run-custom-tests", LifecyclePhase.TEST)
public class TesterMojo extends AbstractMojo {
#Parameter(property = "someParameter")
private String someParameter;
// [...] parameters for test configuration
#Override
public void execute() throws MojoExecutionException, MojoFailureException {
// Piece of code that executes a set of custom tests which procedure I specified.
}
}
When test fail, I would like them to be marked as failed tests not as failed executions. What's the right thing to do here? Show me the light, please.
Maven conventions support two types of testing out of the box: unit tests (via maven-surefire-plugin) and integration tests (via maven-failsafe-plugin).
By default, maven-surefire-plugin only looks for the following files with unit tests:
**/Test*.java
**/*Test.java
**/*TestCase.java
Similarly, default includes for integration tests run by maven-failsafe-plugin are the following:
**/IT*.java
**/*IT.java
**/*ITCase.java
So, as you can see, Maven lets each plugin figure out which tests it should care about. So it's perfectly fine for src/test/java to contain different types of tests, not just unit tests.
Different folder
You can put tests in a different folder too. One example would be if you have non-Java tests, since then src/test/java location doesn't make sense. Standard Maven plugins get project model from Maven to figure out the src/test/java location and some 3rd party plugins use the same mechanism. Depending on the plugin you use, you might want to check out its configuration or use maven-build-helper-plugin to add-test-source in order for some plugins to pick up another test folder automatically.
Different tests on demand
From the Maven perspective the core difference between unit tests and integration tests is the additional requirements for the later: they often need to have your project already packaged and they often need additional setup or teardown. But you yourself can set up multiple test goals during both test and integration-test phases. All major test frameworks support specifying which test suite should be run when (e.g., via groups). If your framework doesn't, you can still use plugin includes/excludes. It is a standard practice to combine this with Maven profiles in order to only run smoke tests by default (during development) and to run full tests on CI environment. You can use the same approach to enable anyone (a tester?) to run extra tests on demand, e.g., to run extra heavy tests when certain important part of the code has changed.
I am fairly new to writing Maven pom files and JUnit tests. I have following in my pom and it is calling my test scripts as expected.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.16</version>
</plugin>
It seems JUnit is more popular than sure fire plugin.
1) How is JUnit similar/different from maven sure fire plugin's default behavior (that is working for me from above plugin configuration). I can imagine JUnit having additional API/library; but what do they give me in addition to sure fire plugin
2)what is the easiest way to change my current tests that are running with sure fire plugin to JUnit. I came across following link which sort of implies that adding few lines to pom would be sufficient (?)
http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit.html
3)about previous bullet, what benefits would I have if I convert sure fire plugin tests to JUnit.
Hopefully, I am unambiguous (with my intro background to maven and JUnit)
maven-surefire-plugin is not itself a test framework: it's a Maven plugin that will run tests written with a test framework, either JUnit or TestNG.
I have following in my pom and it is calling my test scripts as expected.
If this is already running your tests then, as Surefire only knows about those two test frameworks, it means you're already using either JUnit or TestNG. You should be able to tell which from the classes you're importing to write your test classes.
(that is working for me from above plugin configuration)
Unless you have a particular requirement there's little reason to move away from the framework you're already using; it doesn't sound like you need to change anything.
As it says right here:
To get started with JUnit, you need to add the required version of JUnit to your project ... This is the only step that is required to get started - you can now create tests in your test source directory (eg, src/test/java).
Your question is confusing and suggests you haven't done any preliminary research yet. When you say "surefire tests" you may mean Pojo tests. If you know what a JUnit test is, it's pretty common sense thing to convert the Pojo tests to JUnit tests. Just put #Test before the Pojo test methods. You may also want to convert assert into the appropriate JUnit assert methods. In summary, just read a JUnit tutorial and the rest will be straight forward.