Is that possible to use Visual Studio Code Coverage without Unit Testing? I would like to make a coverage analysis within a normal program execution.
UPDATE:
There are no current solutions to use the Visual Studio Coverage tooling for what I want, although dotCover seems to be a nice third part solution for the problem.
Here's a more detailed answer rather than just a link:
To do this for already instrumented files with an IIS Express application:
Get the name of the site from C:\Users\<your user>\Documents\IISExpress\config\applicationhost.config
vsperfcmd /start:coverage /output:run.coverage
launch your app
run your manual tests
then to finish
vsperfcmd /shutdown
from related question https://stackoverflow.com/a/23791306/57883
for a full walk through, here's a link to the blog article I just posted on it:
http://imaginarydevelopment.blogspot.com/2015/02/get-code-coverage-from-vs-without.html
I think you may be misunderstanding what Code Coverage is. Code Coverage indicates how much of your code is exercised by your unit tests. If you have no unit tests, you have zero code coverage. Are you perhaps referring to code profiling (measuring how long it takes for code units to execute?)
UPDATE:
If you're looking for metrics on what code is executed during normal execution, you want to do profiling. There are several profilers out there (eg Red Gate ANTS) or you can use your own homegrown solution. If your app is ASP.NET/ASP.NET MVC, You can also check out the MVC Mini Profiler made by the StackExchange team: http://code.google.com/p/mvc-mini-profiler/
NEW UPDATE
I'm confused by the comments. Code coverage is about unit testing.
From Wikipedia:
In computer science, code coverage is a measure used to describe the
degree to which the source code of a program is tested by a particular
test suite. A program with high code coverage has been more thoroughly
tested and has a lower chance of containing software bugs than a
program with low code coverage.
The title of the question asks how to "use Visual Studio Code Coverage without Unit Testing." There is no point in measuring code coverage if there are no unit tests.
Related
We write code, and we write tests. When you write tests, you're thinking about both the expected usage of the thing you're testing - and you're also thinking about the internal implementation of that thing, writing your test to try exposing bad behaviors.
Whenever you change the implementation of the thing, you're adding new lines of code, and sometimes it's difficult to be sure that your test actually exercises all the code in the thing. One thing you can do is set a breakpoint, and step through painstakingly. Sometimes you have to. But I'm looking for a faster, better way to ensure all the code has been tested.
Imagine setting a breakpoint and stepping through the code, but every line that was run gets highlighted and stays highlighted after it was run. So when the program finishes, you can easily look at the code and identify lines which were never run during your test.
Is there some kind of tool, extension, add-in, or something out there, that will let you run a program or test, and identify which lines were executed and which were not? This is useful for improving the quality of tests.
I am using Visual Studio and Xamarin Studio, but even if a tool like this exists for different languages and different IDE's, having that information will be helpful to find analogous answers on the IDE's and languages that I personally care about.
As paddy responded in a comment (not sure why it wasn't an answer), the term you're looking for is coverage. It's a pretty critical tool to pair with unit testing obviously because if some code isn't covered by a test and never runs, you'd want to know so you can more completely test. Of course, the pitfall is that knowing THAT a line of code was touched doesn't automatically tell you HOW the line was touched - you can have 100% test coverage without covering 100% of use cases of a particular LOC - maybe you have an inline conditional where one bit never gets hit, just as an example.
Coverage is also useful to know even outside of testing. By performing a coverage analysis on code in production, you can get a feel for what aspects of your codebase are most critical in real use and where you need to focus on bugfixing, testing, and robustness. Similarly, if there are areas of your codebase that rarely or never get hit in a statistically significant period of production uptime, maybe that piece of the codebase can or should get trimmed out.
dotCover and dotTrace seem like they'll probably put you on the right path. Coming from use in Python, I know I used to use Django-Nose, which comes with coverage testing and reporting integrated. There's coverage.py for standalone analysis, and of course these aren't the only tools in the ecosystem, just the ones I used.
According to this article, one can make a parameterized test in the GoogleTest framework with some code like this:
INSTANTIATE_TEST_CASE_P(InstantiationName,
MyStringTest,
::testing::Values("meek", "geek", "freek"));
TEST_P(MyStringTest, acceptsEekyWords)
{
ASSERT_TRUE(acceptName(GetParam()));
}
plus some scaffolding.
After going through the CxxTest User Guide, I couldn't help but notice the lack of any mention of parameterized tests. Are parameterized tests even possible with CxxTest?
This question seems to address something similar, but the answer is by no means trivial.
I'm new to C++ unit testing. Maybe parameterized tests aren't a big deal? Almost all of my tests were parameterized in my last C# NUnit project.
As I wrote in my answer to the other question you cited, there's no facility for generating tests at run time, which is what any parameter-based tester I've ever seen does. In CxxTest, the test list is determined at compile time by the Python-based C++ parser, cxxtestgen. You can generate suites dynamically, but you only have the option of generating zero or one copy of any suite.
I've getting some information about this, but I still can find a proper answer, I was asked recently in my company for this : "run a fxcop analysis on that code and tell me the results".
Ok, I have VS2010 Ultimate which has code analysis, but before making any comment, I browse it on the internet cause I want to implement the best choice...
So, let's say I'm gonna use the same rules on both analyzers:
Should I recommend using one above the other?
Should I say "hey, thats kinda old, let's use code analysis!"
Should I get the same results on different computers? (for what I undersand, fxcop gives you some "points" and for what I've read, sometimes it gives you diff points on diff computers, I don't know about this with code analysis
Thanks, any help would be appreciated
FxCop and Code Analysis are essentially the same thing, with the following differences:
Code Analysis includes a VS IDE extension. FxCop can be executed from and show results within the IDE, but the result is not as full-featured. On the other hand, FxCop includes a stand-alone UI that is more full-features with respect to certain types of results exploration.
Code Analysis includes more rules than FxCop. Part of this is because it includes an additional rules engine, but part is just extra rules that Microsoft decided we should pay for. (The extra rules can be run from FxCop if you want to see the results in the stand-alone FxCop UI.)
For any given rule, you should see exactly the same results on any given machine, regardless of which of the two tools you are using. The only cases in which you should see differences is when you do not specify culture settings for the FxCop analysis, and the system culture differs between machines.
MSTest + Visual Studio generates accessors which appear on the NCover coverage report. They usually have a low coverage in my case. I wonder if I use them in my tests if the method I'm testing will appear as covered in the Accessor but not in the original source code.
If this is the case, it is a problem and I need a solution.
The dev team was interested in this issue and created an MSTest private accessor project in VS 2010.
I ran coverage on it in NCover 3 with MSTest, and it appears that these accessors are reflecting the source code, not a generated version of it, at least in our test.
If you'd like to get the test project and try it out, just contact us at support#ncover.com.
Thanks!
If I understand your scenario, then any method called by your test will be shown as covered in the source code.
If the test adds accessors, those will also be shown as covered where they appear, but you can filter them out by excluding the get and set methods that are generated.
Here's a link to the syntax for the method exclude:
http://docs.ncover.com/ref/3-0/ncover-console/command-line/profiling-options#em
We've been successful here with the regex .*.get_.*, just as an example.
NCover Support
I have CLI/MFC application and I would like to begin to learn how to unit test with it. I have VS2008 Pro. If possible, I would like to use the built in unit testing. Thanks for your input!
I've had success with both CPPUnit and Google Test. For either you have to do a bit of work to get the test results to integrate back into Studio. The granularity of the results you want directly affects how much work. Do you want a pass/fail for the whole test set, or individual results? The former is a simple msbuild task, the latter requires outputting the result set to XML, massaging that with a transform, then pulling it back in.
We use Gallio & MbUnit to test our MFC & C++/CLI application. Simply write the tests in C++/CLI then you can test both managed and unmanged code in a single framework. We also use NMock2 for mocking the managed code.
If you have Team Test edition, you can use it to test C++/CLI applications and libraries. See here.