I'm using MSTest with Visual Studio 2012, is it possible to see which test runs first, second third and so forth?
I prefer the tests to run in random orders but I just want to know when tests fail, I like to know the sequence they ran and helps me to figure out why they fail easier.
If you want to know the sequence in which your test ran you can do the following:
A bit of work to do:
Add a breakpoint at the start of your method decorated with attribute 'TestMethod'.
Right click your breakpoint and select 'When Hit...'.
Make sure the checkbox next to 'Print a message' is ticked.
Do this for all your test methods.
Let Visual Studio do the rest:
Debug all your tests by pressing ALT + S, D, A.
Wow! It worked:
Find the sequence in your debug output window (ALT + V, O).
There are two different concepts
Coded UI Test - Mostly automation will be in done in these test
MS Test - Mostly unit tests will be written here
Its not possible to run MSTest in order even if you define .playlist file these will never run in sequence.
However in CodedUI case you will use and create Ordred list.
So the answer is NO. The reasoning behind that is - these are unit test case and all the unit test cases are meant to be executed independently.
Related
When using XCTExpectFailure and run the test cases (Cmd + U) From "Show the Test navigator" it show's grey cross (x) Other test cases shows in green Colour.
What is the Grey Indicator (x) In Test Cases ?
Do need to do anything here?
Xcode has four indicators for the result of a test.
Green Tick – The test passed
Red Cross – The test failed
Gray Cross – The test failed, but it was expected to
Xcode will show you this only if you already declared that a test was expected to fail using the XCTExpectFailure API.
It's Xcode way to tell you "hey this test failed but I'm not going to report it as a failure because you told me it was expected." This is useful to avoid blocking a build in CI if because when there's an expected failure.
I would encourage you to use XCTExpectFailure parsimoniously. It's useful to avoid being blocked by a test failure that is unrelated to your work, but you should always take the time to go back and fix the failure, don't leave the XCTExpectFailure call in your test suite for long.
Gray Arrow – The test was skipped
Xcode will show this when a test was skipped using XCTSkipIf or XCTSkipUnless.
Similarly to an expected failure, you don't need to do anything about this because either you or someone else in your team added the API call to skip the test, meaning the behavior is expected. It's useful to skip tests if they depend on certain runtime capabilities that might not be available on the device or Simulator executing a test run.
Grey sign appears when test is run successfully and u can click on it to check the performance results in left bar
I am trying to make my Xamarin.UITest output clearer and easier to work with. Every so often when Xamarin Forms updates the tree changes in subtle ways that break our UITests. Also, when developing a test it isn't always necessarily clear what the query should look like to get to a view element we want our test to interact with.
To address these, when a test fails with an "Unable to find element" error, I want to capture the app's view tree and output it to the test results.
Currently in these cases we have to modify the test code by adding app.Repl(); (see Working With the REPL), re-run the test, wait for the REPL window to appear, type tree, look at the output, type exit to leave the REPL, make my code changes based on what I saw in the tree command's output, and rinse-repeat until I have a working test. Instead, if the test results contains the outputs of the REPL's tree command, I can start making changes to fix the test code immediately and greatly speed up my testing feedback loop.
How could I most easily achieve this?
app.Print.Tree();
I think this is what you searched for.
If I have different packages and each have a test file (pkg_test.go) is there a way for me to make sure that they run in a particular order ?
Say pkg1_test.go gets executed first and then the rest.
I tried using go channels but it seems to hang.
It isn't obvious, considering a go test ./... triggers test on all packages... but runs in parallel: see "Go: how to run tests for multiple packages?".
go test -p 1 would run the tests sequentially, but not necessarily in the order you would need.
A simple script calling go test on the packages listed in the right expected order would be easier to do.
Update 6 years later: the best practice is to not rely on test order.
So much so issue 28592 advocates for adding -shuffle and -shuffleseed to shuffle tests.
CL 310033 mentions:
This CL adds a new flag to the testing package and the go test command
which randomizes the execution order for tests and benchmarks.
This can be useful for identifying unwanted dependencies
between test or benchmark functions.
The flag is off by default.
If -shuffle is set to on then the system
clock will be used as the seed value.
If -shuffle is set to an integer N, then N will be used as the seed value.
In both cases, the seed will be reported for failed runs so that they can reproduced later on.
Picked up for Go 1.17 (Aug. 2021) in commit cbb3f09.
See more at "Benchmarking with Go".
I found a hack to get around this.
I named my test files as follow:
A_{test_file1}_test.go
B_{test_file2}_test.go
C_{test_file3}_test.go
The A B C will ensure they are run in order.
I would like to know if there is a tool which can help me in seeing which lines of code were executed within two break points.
Example: I have break points set at the beginning and end of a method. Now I want to see which are the lines of code that got called from within this method. Basically, the tool must do a recursive deep debugging and record the file name and line number.
Runtime Flow can show all function calls in a running .NET application, though it doesn't work with debugging - you need a separate application execution.
Too often people write tests that don't clean up after themselves when they mess with state. Often this doesn't matter since objects tend to be torn down and recreated for most tests, but there are some unfortunate cases where there's global state on objects that persist for the entire test run, and when you run tests, that depend on and modify that global state, in a certain order, they fail.
These tests and possibly implementations obviously need to be fixed, but it's a pain to try to figure out what's causing the failure when the tests that affect each other may not be the only things in the full test suite. It's especially difficult when it's not initially clear that the failures are order dependent, and may fail intermittently or on one machine but not another. For example:
rspec test1_spec.rb test2_spec.rb # failures in test2
rspec test2_spec.rb test1_spec.rb # no failures
In RSpec 1 there were some options (--reverse, --loadby) for ordering test runs, but those have disappeared in RSpec 2 and were only minimally helpful in debugging these issues anyway.
I'm not sure of the ordering that either RSpec 1 or RSpec 2 use by default, but one custom designed test suite I used in the past randomly ordered the tests on every run so that these failures came to light more quickly. In the test output the seed that was used to determine ordering was printed with the results so that it was easy to reproduce the failures even if you had to do some work to narrow down the individual tests in the suite that were causing them. There were then options that allowed you to start and stop at any given test file in the order, which allowed you to easily do a binary search to find the problem tests.
I have not found any such utilities in RSpec, so I'm asking here: What are some good ways people have found to debug these types of order dependent test failures?
There is now a --bisect flag that will find the minimum set of tests to run to reproduce the failure. Try:
$ rspec --bisect=verbose
It might also be useful to use the --fail-fast flag with it.
I wouldn't say I have a good answer, and I'd love to here some better solutions than mine. That said...
The only real technique I have for debugging these issues is adding a global (via spec_helper) hook for printing some aspect of database state (my usual culprit) before and after each test (conditioned to check if I care or not). A recent example was adding something like this to my spec_helper.rb.
Spec::Runner.configure do |config|
config.before(:each) do
$label_count = Label.count
end
config.after(:each) do
label_diff = Label.count - $label_count
$label_count = Label.count
puts "#{self.class.description} #{description} altered label count by #{label_diff}" if label_diff != 0
end
end
We have a single test in our Continuous Integration setup that globs the spec/ directory of a Rails app and runs each of them against each other.
Takes a lot of time but we found 5 or 6 dependencies that way.
Here is some quick dirty script I wrote to debug order-dependent failure - https://gist.github.com/biomancer/ddf59bd841dbf0c448f7
It consists of 2 parts.
First one is intended to run rspec suit multiple times with different seed and dump results to rspec_[ok|fail]_[seed].txt files in current directory to gather stats.
The second part is iterating through all these files, extracts test group names and analyzes their position to the affected test to make assumptions about dependencies and forms some 'risk' groups - safe, unsafe, etc. The script output explains other details and group meanings.
This script will work correctly only for simple dependencies and only if the affected test is failing for some seeds and passes for another ones, but I think it's still better than nothing.
In my case it was complex dependency when effect could be cancelled by another test but this script helped me to get directions after running its analyze part multiple times on different sets of dumps, specifically only on the failed ones (I just moved 'ok' dumps out of current directory).
Found my own question 4 years later, and now rspec has a --order flag that lets you set random order, and if you get order dependent failures reproduce the order with --seed 123 where the seed is printed out on every spec run.
https://www.relishapp.com/rspec/rspec-core/v/2-13/docs/command-line/order-new-in-rspec-core-2-8
Its most likely some state persisting between tests so make sure your database and any other data stores (include class var's and globals) are reset after every test. The database_cleaner gem might help.
Rspec Search and Destroy
is meant to help with this problem.
https://github.com/shepmaster/rspec-search-and-destroy