I'm using MSTest in Visual Studio 2010 and have the need to restore my database after all tests have run.
What I did was decorate a method with the AssemblyCleanupAttribute attribute.
<AssemblyCleanupAttribute()>
Shared Sub AssemblyCleanup()
' Restore my databases which takes a long time...
End Sub
Problem is the clean up takes a reasonable amount of time, so much so that the timeout is reached.
The only reason I started realizing that a timeout occurred is that in debug mode the Output window reports "...QTAgent32.exe, AgentObject: Cleanup: Timeout reached in cleaning up the agent.". Hence it fails very quietly and I would have loved if MSTest reported a Test Run Error.
What is the best way to detect and report the timeout? My ideal solution would be to report the timeout as a test run error.
In short, you cannot cause MSTest to report an error if AssemblyCleanup times out.
If you are encountering this issue, then at this point you need to consider if this limitation of MSTest is too great for you. There are other, and imho better, test frameworks out there.
If you decide to stick with MSTest and just want to ensure, at least, that the code/script in AssemblyCleanup runs to completion then you can choose to either run the clean up code as a Process. That way even if AssemblyCleanup internally calls a Thread.Abort then your Process runs to completion. It's messy though...
Why not wrap the contents of each test in a transaction, and rollback the transaction at the end of the test? See here for more information: http://msdn.microsoft.com/en-us/library/bb381703(v=vs.80).aspx#dtbunttsttedp_topic7
Related
Visual Studio Version 15.2 (26430.6) Release.
Having recently updated to the above version I am running into continuous issues with VS locking up files when trying to build.
Could not copy "obj\Debug\projHype.dll" to "bin\Debug\projHype.dll". Exceeded retry count of 10. Failed.
Also tried running VS2017 with and with out admin priveledges
I tried the suggestions for older versions of visual studio but to no avail. Any ideas how to get around this?
For anyone encountering this. Updating to version 26430.12 will resolve this. Looks like the previous release contained a bug.
While there may be other causes, testhost.exe and testhost.x86.exe can both result in a lock that prevents the build from completing. The symptoms are bewildering-- the test explorer churns indefinitely, sometimes timeout warnings appear in the build. Sometimes the files cannot be accessed even after VS is shut down.
If you are using nUnit or another test framework, make sure that test discovery does not encounter any infinite loops or crashes in your code. If this happens, it can hang the testhost executable. For example, if you use nUnit TestCase or TestCaseSource, if any of these perform an action that can hang, lock up, or crash, they will be invoked before the tests are executed.
This is the tricky bit-- your tests haven't run yet, but your code can lock up VS! While this may not be your problem, if you have unit test discovery in any way, check that it all completes.
One way to make sure all discovery completes is to use the functions from a test itself, and disable them as a TestCase or TestCaseSource (or equivalent in other test frameworks). If the test hangs or crashes, that's the culprit.
For me it helped to run the program again with accepting "run last successful build". After the run, the locks were gone.
I need to generate some delays to make a load test more realistic. In Visual Studio I have found 2 methods that delay execution of a transaction in a loadtest:
Convert to code
or
Set "Think Time" to first request in the transaction
What would be the best practice here. I am not completely sure that Visual Studio executes these test in sequencional manner, so 2nd option might return false results.
On the other hand, I would loose the ability to change settings from Visual Studios UI, if I generate code from a recorded test. And that increases time spent editing the test quite a bit.
Is there a better way than any of my methods?
I believe that the second way is more safier.
Just have in mind that the Think Time is aplied to each Test Run not to each contained Test. So, if you have a Load Test which contains more than one Tests in the Test Mix the Think Time will apllied at the end of the test run (when all included tests were completed).
Also, if you have more than one virtual user, since it's a load test, Think Time is applied to each user seperately. If, for example, you have 100 user then they will concurently start the tests but they may not finish them at the same time. Each user will wait according to the Think Time before starting the second test without waiting for the others to complete their first test.
I recently run into problems when running all my unit tests at once.
I can debug them and run my tests seperate without problems, but when running them all together, the test-run keeps hanging half way through.
This happens:
"Run all tests in Solution"
The first tests parses without problem (slower then usual though)
At some point it gets stuck. Nothing fails, no exceptions, VS just stops running the pending tests.
When stopping the test-run it gets stuck again, and I need to restart VS to abort the test-run.
Normally I would expect a bug in my code, but I haven't made any changes to the code beeing testet since last succesful test-run. The only thing I did was run Performance Wizard - Cuncurrency profiling.
It always stops the same place, when removing some tests from the run it stops a new place (still without actually entering any leftover tests).
I have no clue what is causing this. But seems like I'm having problem with a VS setting rather then a code Error.
Any suggestions? Do Performance Wizard change any settings that might have influenced the way test should be run?
System details:
Windows 7 Ultimate 64-bit,
Visual Studio 10 Premium
This sounds like a concurrency issue. It seems that one test changes the testenvironment in such a way that another test runs into a deadlock. When you remove some tests, the test run order is changed and some other tests get stuck.
So I would look for a concurrency issue regarding your test environment/externall dependencies.
I can't really explain why this works, but it solved the problem!
I reversed the '.csproj' file to an earlier version, in one of the projects that had been in 'contact' with the Performance Wizard, and now my tests works.
ALSO Be aware of that Performance Wizard can change the solution configurations from 'DEBUG' to 'RELEASE' mode in some cases. This was not the case for me, but have been a pain for some of my colleagues.
Has anyone seen this very strange behaviour before?
I've got a solution whith 70 unit tests. All of them pass on my dev machine.
Whenever I commit my changes, our continuous integration process kicks in and the build box will eventually run the same 70 unit tests.
There is only ONE test in the build box that fails all the time.
The error is in one line that only gets a record from our unit test db. (I know it sucks having unit test to rely on data but please don't focus on this as it's not relevant now)
The most weird thing is when I logon myself to the build box, open up the same visual studio solution and manually kick off the unit tests. Result: ALL PASS!
Has anyone ever had this weird situation? I'm guessing there is some weird thing going on with Cruise Control.NET and MSTest?
Surely your unit test runner produces a good log that shows the exact exception message or error? It's kinda pointless to guess at it but an "access denied" kind of error would be an obvious candidate. Setup whatever dbase engine you use (you forgot to mention that too) to give the user account that runs the tests on the build grunt access to the tables.
As said in another answer, it doesn't make too much sense to guess about it when there are detailed logs around...
But because I had this situation several times, here's a guess anyway:
The account, which is used by the CI server to run the tests, may not have appropriate permissions in the database. This would also explain, why the same test succeeds when you run it manually (then with your user account)...
HTH!
Thomas
thanks for your inputs but it wasn't anything related to credentials at all.
I've found out that other tests that were running before that particular one were leaving my unit test database in an inconsistent state, therefore causing errors to the test in question.
It's not a good practice to have your unit tests relying on data, so unless you are extremely bound to it like myself, this is what a recommend to everyone: DO NOT RELY ON DATA TO DO YOUR UNIT TESTING !!!! Make sure you have all the good stuff in place, specifically a good IOC/dependency injector container so your classes are loosely coupled and you can mock up any interface you may want to unit test easily!
If you have system tests that you want to run on your build server or in general, want to be able to run correctly on any machine, including your own, then you must make sure that their states are independent.
In your case, you should have each test init prepare the DB it uses (either by copying a file-based DB or by emptying/filling a service-based DB). Each test should also attempt to undo its changes (delete file or empty DB) but not assume that other tests have done so successfully.
is there a way? do I have to wait for building every time I start the test? I want to build from visual studio not from test
thanks
Any time your code changes and you run your test it is going to do a build... so technically you can run your test over and over again and they will only build the first time, but once you run your test why would you run them again without making a code change?
Couple of things that I use that make your test run faster are:
Check the box for "Only build startup projects and dependencies on Run", located Options->Projects and Solution->Build and Run.
Learn the short cut keys
a. "Ctrl+R, T" Runs test in current context, so if your cursor is inside a test method it will only run that test, but when you do it inside of a non test class it will run all of your test.
b. "Crtl+R, Ctrl+T" Debug test same except debug.
c. Others can be found here, those are 2008 if you need to reference others you can find them via google.
Make sure your test are not calling the database or other time intensive resources, use mocking and stubbing.
Run only small sets of test, ie if I am working in a service class I run only the service class test.
Edit: Reading your question again if you want to build and not from a test you can just go to the menu and click Build->Build Solution or press F6. Also it would be helpful if you indicated which version of visual studio you are using because 2010 is different in the sense that you have to click refresh. Either way are you able to clarify?
This is an old question, but I keep seeing people ask it and the issue is still true in VS2017, and it's also true of other test frameworks (Xunit, etc) run from within VS.
I don't know how to make VS stop building all the time. But I do know how to circumvent the compile - run your tests from a console runner, not from within VS. If you're using ReSharper, it has one.
If you aren't using ReSharper, for MSTest, you can start here. https://msdn.microsoft.com/en-us/library/ms182489.aspx
If you aren't using ReSharper, for XUnit, you can start here. https://xunit.github.io/docs/getting-started-desktop.html#add-xunit-runner-ref
Any changes to source code cause compilation, because in order to run tests VS needs up to date DLL with tests.
If you have already compiled project then you can run test multiple times without compilation.
PS: I run MSTest using TestDriven.NET as for me it is faster.