I want to run feature files parallelly in alphabetical order and scenarios of each feature file in serial order.
I am using JUnit 5 configurations to run the test in parallel.
Basically want to use junit 5 built in mechanism to run feature file parallel and not sceanrios and feature file should be executed in alphabetical order like it used to work with junit 4 and surefire plugin
You can not. Scenarios are intended to be executed independently from each other. It was a limitation of JUnit 4 that only features were run in parallel.
Depending on what you are testing, you could turn all scenarios in a feature file into a single scenario.
However you should consider rewriting your scenarios such that they do not depend on the actions in previous scenarios.
Typically this means writing scenarios like:
Scenario: Do a thing
Given a thing
When a thing is used
Then something happens
Scenario: Do another thing next
Given a thing that was used
When a thing is used again
Then something else happens
To implement a thing that was used you can reuse the methods you've already written to implement the steps of the first scenario.
Related
I have test suite where in one of the spec the 5th test-case is dependent on 3rd test-case. While the case is run locally via cypress runner - I do not see any issue in order of running.
But while case is running in CI - I'm seeing 5th is failing randomly [verified that no script errors] & upon analysis I notice that certain data records which are created in 3rd case are not returned for 5th case & hence its failing.
Is there a way to order tests within a spec in Cypress?
Unfortunately it is not possible to run tests within the same spec file in specific order in Cypress currently.
Cypress is basically just scheduling events to happen, with no additional control beyond that, so there is no way to guarantee tests will run in a specific order.
I have a number of feature files in my cucumber scenario test suite.
I run the tests by launching Cucumber using the CLI.
These are the steps which occur when the test process is running:
We create a static instance of a class which manages the lifecycle of testcontainers for my cucumber tests.
This currently involves three containers: (i) Postgres DB (with our schema applied), (ii) Axon Server (event store), (iii) a separate application container.
We use spring's new #DynamicPropertySource to set the values of our data source, event store, etc. so that the cucumber process can connect to the containers.
#Before each scenario we perform some clean up on the testcontainers.
This is so that each scenario has a clean slate.
It involves truncating data in tables (postgres container), resetting all events in our event store (Axon Server container), and some other work for our application (resetting relevant tracking event processors), etc.
Although the tests pass fine, the problem is by default it takes far too long for the test suite to run. So I am looking for a way to increase parallelism to speed it up.
Adding the arguments --threads <n> will not work because the static containers will be in contention (and I have tried this and as expected it fails).
The way I see it there is are different options for parallelism which would work:
Each scenario launches its own spring application context (essentially forking a JVM), gets its own containers deployed and runs tests that way.
Each feature file launches its own spring application context (essetially forking a JVM), gets its own containers deployed and runs each scenario serially (as it would normally).
I think in an ideal world we would go for 1 (see *). But this would require a machine with a lot of memory and CPUs (which I do not have access to). And so option 2 would probably make the most sense for me.
My questions are:
is it possible to configure cucumber to fork JVMs which run assigned feature files (which matches option 2 above?)
what is the best way to parallelise this situation (with testcontainers)?
* Having each scenario deployed and tested independently agrees with the cucumber docs which state: "Each scenario should be independent; you should be able to run them in any order or in parallel without one scenario interfering with another. Each scenario should test exactly one thing so that when it fails, it fails for a clear reason. This means you wouldn’t reuse one scenario inside another scenario."
This isn't really a question for stack overflow. There isn't a single correct answer - mostly it depends. You may want to try https://softwareengineering.stackexchange.com/ in the future.
No. This is not possible. Cucumber does not support forking the JVM. Surefire however does support forking and you may be able to utilize this by creating a runner for each feature file.
However I would reconsider the testing strategy and possibly the application design too.
To execute tests in parallel your system has to support parallel invocations. So I would not consider resetting your database and event store for each test a good practice.
Instead consider writing your tests in such a way that each test uses its own isolated set of resources. So for example if you are testing users, you create randomized users for each test. If these users are part of an organization, you create a random organization, ect.
This isn't always possible. Some applications are designed with implicit singleton resources in the code. In this case you'll have to refactor the application to make these resources explicit.
Alternatively consider pushing your Cucumber tests down the stack. You can test business logic at any abstraction level. It doesn't have to be an integration test. Then you can use JUnit with Surefire instead and use Surefire to create multiple forks.
Parallel execution of cucumber 4 work for me, but I want to execute some actions just once for all tests, It is a possible way to run some hooks in another Thread ?
As per your requirement, you want to execute some actions for all test cases once, is it like before or after all test case execution. If so then adding #BeforeClass from JUnit/TestNG and Similarly you can use #AfterClass in your run cuke class. This piece of code would run once before running your first class and after all test execution completed.
would it may work or adding tagged hooks give you some clue. Like for some specific test cases you can use tagged hooks and run those specific actions inside that hook only.
I'm using JMeter for integration and non-regression testing.
The tests are automated and reports are working.
But since it is scenario testing and not performance testing the report doesn't give real business added value for that kind of tests.
My question: Is there any way to have a scenario (transaction controller based)reporting?
For the moment, to have some more meaningful result, transactions controllers and dummy sampler are used.
What we would like to have is the number of success/failure scenarios of the last test run. And also an history of success/failures per test run (1 by day).
Thank you for your advices.
The easiest way of getting the things done is putting your JMeter test under Jenkins orchestration so it will be automatically executed based on a VCS hook or according to the Schedule
Once done you will be able to utilize Jenkins Performance Plugin which adds test results trends charts and ability to mark build as unstable/failed depending on various criteria.
If I am not wrong, you want to create a suite based on particular test cases. like if single case include execution of more than 1 request in a single execution.
If this is the case, you can simple create a test fragment through jmeter gui, and copy all the samplers in single fragment.
Now to control their execution you can use any controller of your choice, i would suggest you to use module controller for http samplers.
I'm using testcomplete Keyword test for a lot of UI test cases. Quite a lot of them has the same steps.
Is there any Macro functionality which can add multiple preset actions/checkpoints easily?
Sure, you can call another keyword test using the Run Keyword Test operation or a script function using the Run Script Routine operation. Both operations allow specifying parameters for a test. Also, you can use the Run Test operation to run any item that can be treated as a separate test (keyword or script test, network suite job or task, a load test).
Moreover, I think that you will find it useful the Data-Driven Testing functionality of TestComplete that allows running a test for every record in a specified data source. Find more information on this feature in the Data-Driven Testing help topic. Videos demonstrating data-driven approach can be found here and here.