I use jasmine to test my server side code and i need to run tests in serial, not in parallel.
My tests need to make CRUD operation in database. If test are executed in parallel i can't ensure that the database is in a good context for my test
Unless you explicitly choose to create asynchronous tests in Jasmine, everything in Jasmine happens sequentially, in the sense that one test runs only after its preceding test has finished. And if you do write asynchronous tests, then parts of your single test may run in parallel, but you still have the constraint that one test runs only after its preceding test has finished.
However, there are a couple caveats to be aware of:
In an async test if your code exceeds Jasmine's timeout period, you might still have code running when Jasmine decides to give up on that test and proceed to the next. (Thanks to #Gregg for this tip; see this answer.)
"JavaScript is usually considered to have a single thread of execution... however, in reality this isn't quite true, in sneaky nasty ways." I am quoting #bobince from this answer.
Related
I have multiple test cases covering different components and in different specs. Each of them run successfully but when I run them together, some of them randomly fail, some for weird reasons like a css-selector isn't found
let routerElement = contentComponentElement.querySelector("router-outlet");
expect(routerElement).toBeTruthy(); //fails sometimes
Could it be possible that because I am running them together, a test case is picking residue or left-over state of the previous test case? Is it possible to clear all the previous data/html etc. before running a new test case?
The issue was some of the test cases used Observables and I wasn't waiting for the Observables to finish before moving to the next test case. I started calling done for such test cases and now things are in order.
This command will run UI test on Simulator for 2 test cases :
xcodebuild test -project MyLib.xcodeproj -scheme MyAppUITests -destination 'platform=iOS Simulator,OS=11.3,name=iPhone 5s' -only-testing:MyAppUITests/MyAppUITests/testExample -only-testing:MyAppUITests/MyAppUITests/testExample2|xcpretty --test --color
This cause testExample and testExample2 to be executed but the app under test is restarted between the 2 test cases executions, despite the fact that test setup is the same for both.
How could I keep the app started just once ?
If you want to do multiple actions in a row without a restart in between, they need to be done as part of a single test method.
The reason that the app is restarted between each test is to ensure that the outcome of each test is independent from the others. Repeating the setup for each test is essential if you want to ensure that when a test fails, it has failed because there is something wrong with that test case.
If you chain many tests together without resetting the state, and one part of the test fails, the other parts of the test will be affected and you will not have clear test results.
While it is true that the setUp() method runs multiple times as part of this framework, this is necessary to ensure test independence. The power of automated UI tests lies in the automation (no manual effort required for repeated execution of the same tests) and the data they can give you (a clear breakdown of which features are or are not working), not in their speed. Restarting the app between each test and running the setUp() for each test should not increase the time complexity of the execution of your test suite.
For example, say you have four components, A, B, C and D, and B breaks. A, C and D are all still working.
If you only have one large test method, testAAndBAndCAndD(), and it fails, you will be unable to tell whether A, B, C or D are broken, or if all of them are broken.
Whereas if you have four separate tests, testA(), testB(), testC() and testD(), only testB() will fail and you will instantly know from the test results that the problem lies with B.
I hope this illustrates why it is necessary to restart the app between each test.
I have a test suite of rspec tests which are divided into different files.
Every file represents one test scenario with some number of test steps.
Now, on some particular tests, it can happen that specific step fails but it is too time consuming and not needed to run rest of the steps in that scenario.
I know there is an option --fail-fast in rspec but if I'm running tests like: rspec spec/* that will mean that when first step fails in any script, it will abort complete execution.
I'm just looking for mechanism to abort execution of that specific test scenario (test script) when failure happens but to continue execution of other test scenarios.
Thanks for the help,
Bakir
Use the RSpec-instafail gem.
According to its documentation, it:
Show failing specs instantly. Show passing spec as green dots as usual.
Using ruby / cucumber, I know you can explicitly call a fail("message"), but what are your other options?
The reason I ask is that we have 0... I repeat, absolutly NO control over our test data. We have cucumber tests that test edge cases that we may or may not have users for in our database. We (for obvious reasons) do not want to throw away the tests, because they are valuable; however since our data set cannot test that edge case, it fails because the sql statement returns an empty data set. Right now, we just have those tests failing, however I would like to see something along the lines of "no_data" or something like that if the sql statement returns an empty data set. So the output would look like
Scenarios: 100 total (80 passed, 5 no_data, 15 fail)
I am willing to use the already implemented "skipped" if there is a skip("message") function.
What are my options so we can see that with the current data, we just don't have any test data for those tests? making these manual tests is also not an option. They need to be run ever week with our automation, but somehow separate from the failures. Failure means defect, no_data found means it's not a testable condition. It's the difference between a warning: we have not tested this edge case, and Alert: broken code.
You can't invoke 'skipped', but you can certainly call pending with or without an error message. I've used this in a similar situation to yours. Unless you're running in strict mode then having pending scenarios won't cause any failures. The problem I encountered was that occasionally a step would get mis-spelled causing cucumber to mark that as pending, since it was not matching a step definition. That then became lost in the sea of 'legitimate' pending scenarios and was weeks before we discovered it.
I have unit tests defined for my Visual Studio 2008 solution. These tests are defined in multiple methods and in multiple classes across several files.
I've read in a blog article that when using MSTest, it is a mistake to think that you can depend on the order of execution of your tests:
Execution Interleaving: Since each instance of the test class is instantiated separately on a different thread, there are no guarantees
regarding the order of execution of unit tests in a single class, or
across classes. The execution of tests may be interleaved across
classes, and potentially even assemblies, depending on how you chose
to execute your tests. The key thing here is – all tests could be
executed in any order, it is totally undefined.
That said, I have to have a pre-execution step before any of these tests gets to run. That is, I actually want to define an order of execution somehow. For example, 1) first create the database; 2) test that it's created; then 3) run the remaining 50 tests in arbitrary order.
Any ideas on how I can do that?
I wouldn't test that the database is successfully created; I will assume that all subsequent tests will fail if it is not, and it feels in a way that you would be testing the test code.
Regarding a pre-test step to set up the database, you can do that by creating a method and decorating it with the ClassInitialize attribute. That will make the test framework execute that method prior to any other method within the test class:
[ClassInitialize()]
public static void InitializeClass(TestContext testContext)
{
// your init code here
}
Unit tests should all work standalone, and should not have dependencies on each other, otherwise you can't run a single test in isolation.
Every test that needs the database should then just create it on demand (if it's not already been created - you can use a singleton/static class to ensure that if multiple tests are executed in a batch, the database is only actually created once).
Then it won't matter which test executes first; it'll just be created the first time a test needs a database to use.
In theory it is correct that tests should be independent of each other and be able to run standalone. But in practice, there is a difference between theory and practice, and VS2010 gives me a hard time with its fixed order of execution (random order that is always the same).
Here are some examples:
I have a unit test that cross checks the dates between some tables and verifies that everything is in agreement. Obviously it is of no use to run this test on an empty database, so I want to to run SOME TIME AFTER the unit test that inserts data. Sorry VS2010 doesn't let you do this.
OK, cool, then I will add it to the insert unit test as an epilogue. But then I want to cross check other 10 things and instead of having a unit test ("Make sure that entities with various parameters can be inserted without crashes") I end up having a mega-test.
Then another case.
My unit test inserts entities, just insert, to make sure that this part of the logic works ok. Then I have a multi-threaded version of the test, to make sure that there are no deadlocks and stuff. Clearly I need the multi-threaded test to run SOME TIME AFTER the single threaded test, and ONLY if the single threaded test succeeds. Sorry, VS2010 can't do this.
Another case. I have a unit test that deletes ALL entities of a given kind in the database. This should result in a bunch of empty tables and lots of zeros in other tables. Clearly it is useless to run it on an empty database, so the test inserts 10.000 entities if it finds the DB empty. However, if it runs AFTER the multithreaded test, it will find 250.000 entities, and to delete ALL of them takes TIME. Sorry, VS2010 won't let me do anything about it.
The funny thing is that because of this situation my unit tests started slowly turning into mega-tests, that took more than 30 mins to complete (each) and then VS2010 would time them out, cause the default test timeout is 30 mins. OMG please help! :-)