I found this page that explains how to run Geb/Spock tests at the method level which is what I would like to do with my tests, but I am using gradle. Is there a way to convert this to gradle or is it strictly a maven plugin? I can import the maven-surefire-plugin with gradle just fine, however I can't figure out how to convert the configuration block, or if it is even possible.
I've tried something like below but it doesn't work.
tests {
options {
parallel = "methods"
forkCount = 4
}
}
I can execute the tests at the class (spec) level by using gradle maxParallelForks property, but I'd like to run parallel at the test level.
If you are able to run tests in parallel on the method level depends on what test framework you are using.
As far as I know, only TestNG supports it out of the box.
See here: https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/testing/testng/TestNGOptions.html#setParallel-java.lang.String-
There is way to make it work independently of the test framework, using only Gradle, but this way you can only do it on the class level.
In your Gradle test task, set the maxParallelForks property.
See manual: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:maxParallelForks`
Related
I have some tests that need to check if the main code did a System.exit(...). This works very nicely with suggestions from https://stackoverflow.com/a/309427/1023341. But when running these tests in Jenkins (in stead of in my IDE Eclipse) and later when trying them on the command-line using Maven-Surefire (as Jenkins does) the tests fail without telling me why. It only tells me: Error occurred in starting fork, check output in log.
When setting a SecurityManager during JUnit (5) using System.setSecurityManager and using Surefire plugin, you should restore the SecurityManager after the test.
SecurityManager origSecurityManager = System.getSecurityManager();
try {
// ... code under test here ...
} finally {
System.setSecurityManager(origSecurityManager);
}
or some other more suitable form. This makes sure that Maven-Surefire-plugin stays happy.
Edit for suggested pre-baked solutions:
There are two pre-baked libraries for this:
junit5-system-exit
system-lambda
As the name suggests: the system-lambda is a Java 8+ solution. Both are JUnit 5 compatible. My personal preference lies with the lambda solution.
More background information
I'm creating a fairly large test suite using gradle, geb, and spock in conjunction. Gradle is obviously building and kicking off geb and spock, but I think that spock is where I can control and specify which Spec to run.
I'm building this based off of this starter.
https://github.com/AutomationSchool/geb-and-spock-automation-examples
How can I set this to run just one Spec?
Gradle's Test task takes a tests option. The supported patterns are documented in the javadoc for TestFilter. So if you want to run spec class called MySpecToRun in the project you linked to then you can do it this way:
./gradlew chromeTest --tests=MySpecToRun
Is there any way to specify the order of the unit tests classes run by a Gradle Test task?
I'd like to get some known longer-running tests at either the front or the back of the list, but don't know if it's possible without splitting my tests' execution between multiple tasks.
Running JUnit 4.12 and Gradle 4.5.
Gradle simply delegates execution to the JUnit runner.
So if you want specific test class ordering, you will need to create a Suite and specify the test classes in the order you want, see the JUnit documentation for this.
Now, given the flexibility of Gradle in terms of having different source roots, I would strongly recommend doing the separation at the Gradle level, by create extra test source roots, test task and the like. This will allow you to effectively control when these long running tests are run in a standard build execution, but also to skip them or run these only when desired. The ordering at the JUnit level will not give that flexibility without much more tweaking. See the Gradle documentation on adding source sets.
Well, I would like to have a maven goal execute-custom-tests inside my custom-maven-plugin that consists of running test methods (This tests are not unit tests). Something similar to test goal of soapui-pro-maven-plugin, for example.
Why? Basically the main objectives of the plugin are testing stuff (not unit testing) and the tests in src/test are for unit testing, right?
Being more specific I was thinking about something like this:
#Mojo (name = "run-custom-tests", LifecyclePhase.TEST)
public class TesterMojo extends AbstractMojo {
#Parameter(property = "someParameter")
private String someParameter;
// [...] parameters for test configuration
#Override
public void execute() throws MojoExecutionException, MojoFailureException {
// Piece of code that executes a set of custom tests which procedure I specified.
}
}
When test fail, I would like them to be marked as failed tests not as failed executions. What's the right thing to do here? Show me the light, please.
Maven conventions support two types of testing out of the box: unit tests (via maven-surefire-plugin) and integration tests (via maven-failsafe-plugin).
By default, maven-surefire-plugin only looks for the following files with unit tests:
**/Test*.java
**/*Test.java
**/*TestCase.java
Similarly, default includes for integration tests run by maven-failsafe-plugin are the following:
**/IT*.java
**/*IT.java
**/*ITCase.java
So, as you can see, Maven lets each plugin figure out which tests it should care about. So it's perfectly fine for src/test/java to contain different types of tests, not just unit tests.
Different folder
You can put tests in a different folder too. One example would be if you have non-Java tests, since then src/test/java location doesn't make sense. Standard Maven plugins get project model from Maven to figure out the src/test/java location and some 3rd party plugins use the same mechanism. Depending on the plugin you use, you might want to check out its configuration or use maven-build-helper-plugin to add-test-source in order for some plugins to pick up another test folder automatically.
Different tests on demand
From the Maven perspective the core difference between unit tests and integration tests is the additional requirements for the later: they often need to have your project already packaged and they often need additional setup or teardown. But you yourself can set up multiple test goals during both test and integration-test phases. All major test frameworks support specifying which test suite should be run when (e.g., via groups). If your framework doesn't, you can still use plugin includes/excludes. It is a standard practice to combine this with Maven profiles in order to only run smoke tests by default (during development) and to run full tests on CI environment. You can use the same approach to enable anyone (a tester?) to run extra tests on demand, e.g., to run extra heavy tests when certain important part of the code has changed.
There is more and more gradle code in my build.gradle.
The question is how to test gradle code from build.gradle. Is there any convention?
I'm not sure if this is convention but what You need to do is to make use of gradle tooling API. Basically it enables to You to load a build.gradle file, execute various tasks and verify the output. This code may be written as a normal test classes. You can find examples of such testing here and here for instance. Tests are written in spock.
Usually, code in build.gradle isn't tested. Instead, reusable/generic parts of the code are factored out into plugins and tasks implemented as classes, which are more amenable to testing.
A potential alternative is to test a task A by adding a task testA that depends on A and checks the outcome of A.