it does sound strange, but sometimes it happens that the status of a test case that was run a couple of days ago is automatically reset to active.
Do you have any idea why, or where to start investigating?
I mention that:
we did press save and close after the run (the test case was shown as passed for a couple of days);
the test was not included in any other test suite;
the test is a manual one;
it did not happen for test belonging to suits that were marked as completed;
We are using MTM 2012.
Thanks!
I only know a similar behavior regarding automated tests.
You will observe this if
you associate your automated test run with a CI or nightly build AND
if you set up a *Retention Policy" for these build AND
if "Build Delete Options" are set like this:
In this case the test runs will be deleted along with the build.
But it only applies to automated tests to my knowledge...
The behaviour you are observing is kind of similar one: your most recent manual test run gets deleted for some reason; that is the only reason I can imagine for a test case being reset to active "automatically".
Try to investigate in this direction; means what in your environment can cause manual test run to be deleted?
Related
When I run tests in test explorer individually they all run without errors.
When I click on "Run All" I get varying test that fail.
My question:
Does anybody know what the minimum requirements regarding tests to be able to run them with "Run All"? Like maybe they all have to in a certain way with a certain command?
Running test manually one by one or run all has nothing to do with your tests being failed. You need to check the failure reason for each test and solve it.
Maybe you can post the error messages here so we can help you better.
I discovered that I had a static variable still containing a value other than that expected. Now the outcome is always the same regardless of how I run it.
Sorry for troubling you all
When I run my XCTests, I'd like to automatically rerun, once, any integration (unit/ui) test that fails. Is this possible?
This would be done in the same test, without having to hit 'run' again on my tests, or any part of my tests. I'm running all my tests from the command line.
I am doing UI tests that use the network to make calls to the server. If there is a significant server problem, the test will/should fail and report an error. However, if it is only a temporary problem with the request, it would be nice to rerun the test automatically and see if it passes. Also, with the current state of Xcode UI testing there are some occasional problems where it will crash for an obscure reason, and it would be nice to rerun the test automatically to see if it passes the second time.
It would be especially nice if it could output what happened, i.e. "The test failed the first time, because of failure getting refreshed snapshot, but passed the second time"
You can now do this in Xcode without any scripts.
Press ⌘ + 6 or select the test assistant view.
Filter for failed tests
Select tests, right click and choose Run x test methods > Run in all classes
(optional) If you want to run the same tests again press ⌃ + ⌥ + ⌘ + G for a quick way to run them right away.
Task list to accomplish this without fastlane and with granular test reruns
create a script to rerun the test and capture the list of failed tests, then rerun the tests using the -only-testing:TEST-IDENTIFIER flag of xcodebuild.
parse the resultBundle or .xcresult to find the failed tests
write code to erase the device or simulator executing the tests
reboot the device if it is a real one to clear any possible dialogs
use XCAttachment to collect things you care about, like app logs
use xchtmlreport to generate a nice webpage from the results bundle with your snapshots and attachments.
Fastlane can make it possible here is the awesome blog post regarding same
Stabilizing the CI By Re-runing Flaky iOS XCUI Tests
I recently run into problems when running all my unit tests at once.
I can debug them and run my tests seperate without problems, but when running them all together, the test-run keeps hanging half way through.
This happens:
"Run all tests in Solution"
The first tests parses without problem (slower then usual though)
At some point it gets stuck. Nothing fails, no exceptions, VS just stops running the pending tests.
When stopping the test-run it gets stuck again, and I need to restart VS to abort the test-run.
Normally I would expect a bug in my code, but I haven't made any changes to the code beeing testet since last succesful test-run. The only thing I did was run Performance Wizard - Cuncurrency profiling.
It always stops the same place, when removing some tests from the run it stops a new place (still without actually entering any leftover tests).
I have no clue what is causing this. But seems like I'm having problem with a VS setting rather then a code Error.
Any suggestions? Do Performance Wizard change any settings that might have influenced the way test should be run?
System details:
Windows 7 Ultimate 64-bit,
Visual Studio 10 Premium
This sounds like a concurrency issue. It seems that one test changes the testenvironment in such a way that another test runs into a deadlock. When you remove some tests, the test run order is changed and some other tests get stuck.
So I would look for a concurrency issue regarding your test environment/externall dependencies.
I can't really explain why this works, but it solved the problem!
I reversed the '.csproj' file to an earlier version, in one of the projects that had been in 'contact' with the Performance Wizard, and now my tests works.
ALSO Be aware of that Performance Wizard can change the solution configurations from 'DEBUG' to 'RELEASE' mode in some cases. This was not the case for me, but have been a pain for some of my colleagues.
Has anyone seen this very strange behaviour before?
I've got a solution whith 70 unit tests. All of them pass on my dev machine.
Whenever I commit my changes, our continuous integration process kicks in and the build box will eventually run the same 70 unit tests.
There is only ONE test in the build box that fails all the time.
The error is in one line that only gets a record from our unit test db. (I know it sucks having unit test to rely on data but please don't focus on this as it's not relevant now)
The most weird thing is when I logon myself to the build box, open up the same visual studio solution and manually kick off the unit tests. Result: ALL PASS!
Has anyone ever had this weird situation? I'm guessing there is some weird thing going on with Cruise Control.NET and MSTest?
Surely your unit test runner produces a good log that shows the exact exception message or error? It's kinda pointless to guess at it but an "access denied" kind of error would be an obvious candidate. Setup whatever dbase engine you use (you forgot to mention that too) to give the user account that runs the tests on the build grunt access to the tables.
As said in another answer, it doesn't make too much sense to guess about it when there are detailed logs around...
But because I had this situation several times, here's a guess anyway:
The account, which is used by the CI server to run the tests, may not have appropriate permissions in the database. This would also explain, why the same test succeeds when you run it manually (then with your user account)...
HTH!
Thomas
thanks for your inputs but it wasn't anything related to credentials at all.
I've found out that other tests that were running before that particular one were leaving my unit test database in an inconsistent state, therefore causing errors to the test in question.
It's not a good practice to have your unit tests relying on data, so unless you are extremely bound to it like myself, this is what a recommend to everyone: DO NOT RELY ON DATA TO DO YOUR UNIT TESTING !!!! Make sure you have all the good stuff in place, specifically a good IOC/dependency injector container so your classes are loosely coupled and you can mock up any interface you may want to unit test easily!
If you have system tests that you want to run on your build server or in general, want to be able to run correctly on any machine, including your own, then you must make sure that their states are independent.
In your case, you should have each test init prepare the DB it uses (either by copying a file-based DB or by emptying/filling a service-based DB). Each test should also attempt to undo its changes (delete file or empty DB) but not assume that other tests have done so successfully.
is there a way? do I have to wait for building every time I start the test? I want to build from visual studio not from test
thanks
Any time your code changes and you run your test it is going to do a build... so technically you can run your test over and over again and they will only build the first time, but once you run your test why would you run them again without making a code change?
Couple of things that I use that make your test run faster are:
Check the box for "Only build startup projects and dependencies on Run", located Options->Projects and Solution->Build and Run.
Learn the short cut keys
a. "Ctrl+R, T" Runs test in current context, so if your cursor is inside a test method it will only run that test, but when you do it inside of a non test class it will run all of your test.
b. "Crtl+R, Ctrl+T" Debug test same except debug.
c. Others can be found here, those are 2008 if you need to reference others you can find them via google.
Make sure your test are not calling the database or other time intensive resources, use mocking and stubbing.
Run only small sets of test, ie if I am working in a service class I run only the service class test.
Edit: Reading your question again if you want to build and not from a test you can just go to the menu and click Build->Build Solution or press F6. Also it would be helpful if you indicated which version of visual studio you are using because 2010 is different in the sense that you have to click refresh. Either way are you able to clarify?
This is an old question, but I keep seeing people ask it and the issue is still true in VS2017, and it's also true of other test frameworks (Xunit, etc) run from within VS.
I don't know how to make VS stop building all the time. But I do know how to circumvent the compile - run your tests from a console runner, not from within VS. If you're using ReSharper, it has one.
If you aren't using ReSharper, for MSTest, you can start here. https://msdn.microsoft.com/en-us/library/ms182489.aspx
If you aren't using ReSharper, for XUnit, you can start here. https://xunit.github.io/docs/getting-started-desktop.html#add-xunit-runner-ref
Any changes to source code cause compilation, because in order to run tests VS needs up to date DLL with tests.
If you have already compiled project then you can run test multiple times without compilation.
PS: I run MSTest using TestDriven.NET as for me it is faster.