Our test suite takes 5 minutes to run (mostly due to Kafka containers being setup before each test I presume).
When running mvn quarkus:dev and working on a test, I don't know how to re-run only a single test, the one I'm working on.
If my test is broken, and is the only one broken, then it is fine. But as soon as it turns green, quarkus will not run it again if I change the test code.
If I am making big changes, quarkus will run all broken tests and I cannot clearly follow the result of the test I am working on.
I can use mvn verify to run a single test, but the compilation time and application startup times makes it too boring and breaks the mental flow.
How can I tell quarkus to run only some specific test while running?
Related
lets take one spec file in cypress in that i have 5 test cases i have to run sequentially
I am tried with npx cypress run command it will run test cases one by one is it write or wrong? i am confused.
Yes it is totally normal. Here is a link to the documentation of Cypress, which explains what Parallelization is and how to set it up.
You need to activate parallelization first, in order to use it. However it is not recommended to be done on a singular machine. It needs a significant amount of resources.
Cypress Parallelization
If your project has a large number of tests, it can take a long time for tests to complete running serially on one machine. Running tests in parallel across many virtual machines can save your team time and money when running tests in Continuous Integration (CI).
Cypress can run recorded tests in parallel across multiple machines since version 3.1.0. While parallel tests can also technically run on a single machine, we do not recommend it since this machine would require significant resources to run your tests efficiently.
This guide assumes you already have your project running and recording within Continuous Integration. If you have not set up your project yet, check out our Continuous Integration guide. If you are running or planning to run tests across multiple browsers (Firefox, Chrome, or Edge), we also recommend checking out our Cross Browser Testing guide for helpful CI strategies when using parallelization.
Turning on parallelization
Refer to your CI provider's documentation on how to set up multiple machines to run in your CI environment.
Once multiple machines are available within your CI environment, you can pass the --parallel key to cypress run to have your recorded tests parallelized.
cypress run --record --key=abc123 --parallel
Running tests in parallel requires the --record flag be passed. This ensures Cypress can properly collect the data needed to parallelize future runs. This also gives you the full benefit of seeing the results of your parallelized tests in Cypress Cloud. If you have not set up your project to record, check out our setup guide.
Source: Cypress Documentation
I have managed to get Cypress to report results into TestRail using the cypress-testrail-reporter. However, when I execute a test run (only manually at the minute using npx cypress run) the results of each test are logged as separate test runs. They are currently showing like this:
Screenshot from my test runs page on TestRail
I want the results from each time I execute my test suite to appear as 1 run rather than 5 different runs as shown in the screenshot.
I've bumped into the very same issue. So I've started to use Cypress-accumulative-reporter which fixes this issue.
This reporter is not perfect btw from time to time it does not publish the last spec executed, so I've created an empty spec file to run at the end... not clean workaround but it works.
https://www.npmjs.com/package/cypress-testrail-accumulative-reporter
I am not sure what is the best practice for running unit tests.
I suppose every developers should pass the unit tests locally before pushing the code to the GIT repo. And then the CI server (Jenkins) would pick up the new changes and run the tests again.
Why do we want to do it twice? Should we?
If the unit tests take a lot of time to run, do we expect the developer only picks the tests related to the change or runs every tests (even outside the scope of his projects), assuming we have a big maven multi-module POM.
Also consider we usually have a powerful hardware for CI server and relatively less powerful workstation for developers.
If the unit tests take a lot of time to run, do we expect the
developer only picks the tests related to the change or runs every
tests (even outside the scope of his projects), assuming we have a big
maven multi-module POM.
As a developer changes a class, modifies the database structure or makes any change that could have side effects, he/she will not/cannot know all potential side effects on the whole application.
And he/she never has to try to be too clever by saying : "I know that it may have be broken with my changes, so I will run just this test".
Unit testing ensures a some degree of code quality : don't make it less helpful
The unit tests are also non-regression tests. To not play all non regression tests before commit and push is taking the risk to introduce flawed code in the source content management.
You will never do it.
Unit tests have to be fast executed
If units test are so long to be executed as it hurts the developer velocity , it generally means that they are bad designed or maybe that they are not real unit tests. A unit test is designed to be run fast.
If you write tests that are long to be run because they require to start a server, to load/clean data in a database, to load/unload some containers, and so for... it means you didn't write unit test but integration tests. These tests are not designed to be executed regularly and automatically on the local development machine but on a CI tool.
The CI has to run all tests
Do we need to run tests in CI server if every developers run the tests
before push?
As explained, integration tests have to be executed by the CI tool.
About unit testing, sparing their execution in the CI side is not a good idea either.
Of course developers have to run the tests before pushing to the SCM but actually you don't have the guarantee that it will always be done.
Besides, even in a perfect world where developers execute all tests before pushing, you could fall into cases where the tests are successful on the developer machine but fail on the CI or the other developer machines.
For example, the developer could introduce in the base code some absolute paths specific to its machine or he/she could also forget to replicate a modification on the database used in the CI environment.
So running all tests (unit and integration tests) has not to be an option for the CI.
Yes, they shall be run twice. Because if some developers don't, then they run never. Developers should run the tests locally to make sure that their code works correctly.
Your CI system is however your reference, so there's no chance of one person arguing that it "works on my machine", but fails for others. Looking forward to continuous delivery, knowing this state on the CI/CD system becomes even more important.
You might hope that always and forever, every commit has been tested successfully locally (and all workstations are the same and identical with production systems...), but hope is a bad strategy.
I am trying to run multiple feature files through maven goal(command line) but after 2 feature files that run successfully, the other feature files (3rd one onwards) fails in some test cases which when ran independently passes all the test cases.
So f I run each feature file individually I get proper results but running them all together gives wrong results.
We are using serenity framework with cucumber jvm. Please help how can we resolve this issue.
Your failing tests fail to fully setup the context. Some state is leaking from the previous ones. Look for what has changed during the first runs (database/mocks/whatever state) that has to be reset before running the third and following.
I have a big test suite written in TestNG that takes several hours to complete.
I'd like for maven/surefire to create a new suite that is a copy of the first but with just the failed tests in it, a suite that should be much faster to run.
Is it possible to create such a suite?
As a fall back I could create it on my own from a test report that is easy to parse, if there is such report.
Thank you.
On completion of run, testng generates a testng-failed.xml (in the same output folder as your reports), which basically is your initial suite file with the listeners, but the tests part contains only the failed testcases.
In case you are using Jenkins, you might consider adding a postbuild step that triggers another build that works on the same workspace as the current build and uses this failed xml. Or depending on how you are triggering your tests, you might look at writing a script to run the failed xml.