We have a legacy code base with some unit tests still coupled by global state (static variables etc.). To find them, I need to know the exact execution order the tests ran when I ran them via VS test explorer.
Is there a way to log execution order in VS test explorer?
I know that vstest.console does output/log the execution order but then I need to narrow down the subset of tests which is very inconvenient with vstest. In VS test runner, i can just run subsets.
I also know that there are other tools (like resharper test runner) but this is also not an option.
Not out of the box that I am aware of, but I guess you can find ways to log it, like have a global counter that you increment in a before method.
But, since the order of the tests are non deterministic by design I am not sure how valuable that information is.
If you know what test are coupled then run those as Ordered Tests or use a different test traits to control how you execute those test. The best option is of course to break their external dependencies.
Related
It is not my decision personally to have lingering failing unit tests.
In the solution I work with at my company, there are a few failing unit tests that have lingered over time. Occasionally, while making changes, another unit test will start to fail, but it won't be clear which one it is. To figure it out, it's sometimes necessary to copy at least the names of the failing unit tests out into a text file and do a comparison.
In VS2017, in the Test Explorer, you can right-click a unit test and select Copy, and it will copy the name and some other meta-information to your clipboard. However, if you select multiple unit tests, that option disappears. Additionally there doesn't appear to be any "Copy All" option available.
So if you're needing to copy the names of all of the unit tests that are failing (other related meta-information is okay), is there a way to do this in Visual Studio 2017 other than manually copying the tests one at a time?
This is not a direct answer to your question, but rather a workaround (though I'd call it a best practice compared to what you seem to be doing):
You seem to have a number of unit tests that produce errors and for whatever reason you decided to not fix them in time. Fixing them would be the obvious solution, but lets assume there were reasons to not do so.
Now everybody that develops a feature after that decision is in trouble, because the unit test result just became unreliable. They might fail and you will never know if it's your error, or maybe that test was supposed to fail because of the former (bad) decision. Failed tests have transformed from a red/green quality signal to a broken traffic light signaling nothing.
You should mark those tests that fail on purpose so that you know which they are. If you are using MSTest (the Visual Studio default) you can do so by annotating them with the [Ignore] attribute. That way, they will not be run, will not count as failed, but will still appear in the list and remind you of the fact that they still need to be fixed.
That way, your tests are reliable again. Anything red is something you broke. Anything red is something you need to fix. Yellow are tests that were broken anyway and green.. well is green.
No need to compare lists of testnames against each other. Use the means available.
This question may be a bit nebulous, so please bear with me.
I am using Visual Studio, and I am new to the entire realm of unit testing. One thing I do a lot though is use Unit Testing as a quick and dirty ad-hoc "administration ui" at times when I need to just TRY things, but don't have time to make an actual admin system.
What I mean is ... sometimes I just want to get some data thrown into my database to see how it looks on a page. So I'll make a dirty unit test...
[Fact]
public void install_some_test_data(){
using(var database = RavenDocumentStore()){
using(var session = database.OpenSession()){
// create some objects
// add some objects
// save some objects
}
}
}
Nowhere in here have I really cared about "testing", I just like the fact that I can right click and say "Run it" and it'll go, without launching up the program, without having to have a lot of interaction, etc. etc.
So my question is this;
Is this okay to do? I realize this isn't how a program should be long term managed, but I was scolded for doing this by another developer simply because I wanted to quickly show them how something they wanted to see worked. Is there really a problem with using these convenient tools as simpler ways of running commands against my database that can give me immediate feedback on whether it passed or failed?
My followup question to that is this; I have had to actually search pretty hard to find a unit test runner that lets me run individual methods instead of running every single unit test in a class. Is this normal? This whole thing kind of confuses me in general. Even when I do use tests for actually testing, why would I always want to run EVERY test every time? Isn't that extremely wasteful of time and resources? How do you control the order that they run in? This seems even more imperative when tests have to depend on some data to exist before they can run appropriately.
At the moment, I am using xunit and then the unit test runner in ReSharper, but out of the box, Visual Studio doesn't seem to let me run unit tests individually, just as huge batches in a single class.
Am I the only person confused by this?
Sure, it's actually very easy to execute a single unit test (without an external test runner required).
Go to TEST -> Windows -> Test Explorer
Now you'll see a window like this
Now you can rightclick on the test you want to execute and select 'run selected methods'.
As to your other question: it's hard to distinguish what you are asking. Are you demonstrating a proof of concept with a unit test? How does this test replace your admin panel?
Just put your cursor on a Test function name. Then go to Test -> Run Selected Scope (or similar - the name changes by context). That test should execute and also create a test list for you automatically.
I am new to a fairly large code base(in my case is the Project Katana source code).
I am studying the unit tests in the project in order to get acquainted with the code base (there are around 554 tests in the solution).
Since there are a considerable number of unit tests, I would like to study/review them in the order they were created.
I cannot seem to find a way in Test Explorer to arrange the unit tests in chronological order. A quick internet search found nothing.
Any suggestions?
EDIT: In the meantime, I will review the unit tests in an alternative order: by using the library and then looking through the test methods corresponding to the method I want to use. From the perspective of a consumer of the project, I believe this may be a more efficient way.
An approach that just occurred to me (although not ideal) would be to open the first commit of the source code and go from there.
Although this does not answer the question of how to arrange them in the chronological order in Test Explorer, it does gives us the opportunity to look at the tests in chronological order. A brute force approach.
Has anyone seen this very strange behaviour before?
I've got a solution whith 70 unit tests. All of them pass on my dev machine.
Whenever I commit my changes, our continuous integration process kicks in and the build box will eventually run the same 70 unit tests.
There is only ONE test in the build box that fails all the time.
The error is in one line that only gets a record from our unit test db. (I know it sucks having unit test to rely on data but please don't focus on this as it's not relevant now)
The most weird thing is when I logon myself to the build box, open up the same visual studio solution and manually kick off the unit tests. Result: ALL PASS!
Has anyone ever had this weird situation? I'm guessing there is some weird thing going on with Cruise Control.NET and MSTest?
Surely your unit test runner produces a good log that shows the exact exception message or error? It's kinda pointless to guess at it but an "access denied" kind of error would be an obvious candidate. Setup whatever dbase engine you use (you forgot to mention that too) to give the user account that runs the tests on the build grunt access to the tables.
As said in another answer, it doesn't make too much sense to guess about it when there are detailed logs around...
But because I had this situation several times, here's a guess anyway:
The account, which is used by the CI server to run the tests, may not have appropriate permissions in the database. This would also explain, why the same test succeeds when you run it manually (then with your user account)...
HTH!
Thomas
thanks for your inputs but it wasn't anything related to credentials at all.
I've found out that other tests that were running before that particular one were leaving my unit test database in an inconsistent state, therefore causing errors to the test in question.
It's not a good practice to have your unit tests relying on data, so unless you are extremely bound to it like myself, this is what a recommend to everyone: DO NOT RELY ON DATA TO DO YOUR UNIT TESTING !!!! Make sure you have all the good stuff in place, specifically a good IOC/dependency injector container so your classes are loosely coupled and you can mock up any interface you may want to unit test easily!
If you have system tests that you want to run on your build server or in general, want to be able to run correctly on any machine, including your own, then you must make sure that their states are independent.
In your case, you should have each test init prepare the DB it uses (either by copying a file-based DB or by emptying/filling a service-based DB). Each test should also attempt to undo its changes (delete file or empty DB) but not assume that other tests have done so successfully.
is there a way? do I have to wait for building every time I start the test? I want to build from visual studio not from test
thanks
Any time your code changes and you run your test it is going to do a build... so technically you can run your test over and over again and they will only build the first time, but once you run your test why would you run them again without making a code change?
Couple of things that I use that make your test run faster are:
Check the box for "Only build startup projects and dependencies on Run", located Options->Projects and Solution->Build and Run.
Learn the short cut keys
a. "Ctrl+R, T" Runs test in current context, so if your cursor is inside a test method it will only run that test, but when you do it inside of a non test class it will run all of your test.
b. "Crtl+R, Ctrl+T" Debug test same except debug.
c. Others can be found here, those are 2008 if you need to reference others you can find them via google.
Make sure your test are not calling the database or other time intensive resources, use mocking and stubbing.
Run only small sets of test, ie if I am working in a service class I run only the service class test.
Edit: Reading your question again if you want to build and not from a test you can just go to the menu and click Build->Build Solution or press F6. Also it would be helpful if you indicated which version of visual studio you are using because 2010 is different in the sense that you have to click refresh. Either way are you able to clarify?
This is an old question, but I keep seeing people ask it and the issue is still true in VS2017, and it's also true of other test frameworks (Xunit, etc) run from within VS.
I don't know how to make VS stop building all the time. But I do know how to circumvent the compile - run your tests from a console runner, not from within VS. If you're using ReSharper, it has one.
If you aren't using ReSharper, for MSTest, you can start here. https://msdn.microsoft.com/en-us/library/ms182489.aspx
If you aren't using ReSharper, for XUnit, you can start here. https://xunit.github.io/docs/getting-started-desktop.html#add-xunit-runner-ref
Any changes to source code cause compilation, because in order to run tests VS needs up to date DLL with tests.
If you have already compiled project then you can run test multiple times without compilation.
PS: I run MSTest using TestDriven.NET as for me it is faster.