I'm beginning to get the grips of BDD and MSpec, but I'm still really bugged by the fact that I'm unable to debug my tests/specs, and that I have to leave the IDE to go to a html report file to see the results.
Currently, I have a post build event configured to run mspec.exe $(TargetFileName) --html “$(ProjectDir)Report.html”, but there must be some better way to do this.
Does anyone know any good add-ins, test runners or whatever that will let me
debug tests, instead of just running them
show the test results in a nice way inside Visual Studio
(Footnote: I'm running VS2010 Professional on Windows 7, if that matters.)
There are basically three options you have:
You can set up a custom tool in
Tools | External Tools to start
mspec.exe with the current project's
assembly to run the contexts and
generate the HTML report.
Install TestDriven.Net and
ensure that
Machine.Specifications.dll.tdnet
and
Machine.Specifications.TDNetRunner.dll
are in your project's copy of MSpec.
You can then run and debug your
contexts from the context menu: "Run
Tests", "Run With | Debugger" without further installation.
There's an example of what the MSpec folder
looks like for all of my projects.
If you use ReSharper 4.1, 4.5, 5.0
or the latest 5.1 EAP (== beta)
there are runners for each of these
versions.
The ZIP download
contains batch files that install
the runner for each respective
version of ReSharper. ReSharper's
unit test support is pretty
extensive in terms of UI
widgets/shortcut support, the reporting
tree view and debugging.
If you're
a dotTrace user you can also profile
right from within Visual Studio.
dotCover (another JetBrains product)
allows you to calcualate code
coverage results from your MSpec
runs.
On top of that, you get
all the nice coding and navigation features that ReSharper provides.
Be aware that only the first option will generate the HTML report as both the TestDriven.Net and ReSharper runners do not support HTML report generation. From my point of view this isn't an issue since the TD.Net and ReSharper runners offer fairly complete reporting mechanisms through the Visual Studio UI.
Another option that might work (I haven't used it myself) is to leverage the Gallio support that MSpec has. Gallio is a runner/framework for several testing frameworks; it might as well support debug runs with MSpec. Contact #smaclell if you have questions about Gallio support.
Related
I am trying to execute NUnit tests on a legacy project in Visual Studio 2022 professional. Migrating from NUnit 2 to NUnit 3 is not yet an option, and part of the team works with Visual Studio 2019. The problem I am facing is that the tests are not executed by VS 2022, without any error message on the attempt.
The test explorer displays the following after executing the test (and the console output finishes with a successful build):
The NuGet Package manager had NUnit 2.6.1 already installed. I additionally installed the packets NUnit.Extension.NUnitV2Driver and NUnit.Extension.NUnitV2ResultWriter. A test adapter seems only available for NUnit3 (NUnit3TestAdapter, which I also installed). I enabled all the packages for the individual project as was commented in this question. Is there anything else to install/configure that I could be missing to make this work?
Just a few suggested approaches...
The NUnit3TestAdapter, as the name indicates, is designed for NUnit 3. While it is able to run NUnit3 tests by use of the NUnitV2Driver extension, I don't think that will work under Visual Studio just by installing the package. You could experiment by modifying the .addins file installed along with the the adapter, but I can't give you precise instructions as I haven't tried it.
BTW, the V2 driver extension produces output in V3 format, which is probably what you want for use under Visual Studio. The V2 result writer is only needed if you have a reason to want an output file in V2 format.
I think your best bet may be to try using the last release of the NUnitTestAdapter (i.e. without '3' in the name), which only works with NUnit V2. If it will load under VS2022, it should do exactly what you want. In that case, you should stop loading the two extensions, which the V2 runner doesn't support.
Posts on the web suggest that you can profile tests in Visual Studio by right-clicking and selecting "Profile Test": http://adamprescott.net/2012/12/12/performance-profiling-for-unit-tests/ But I only see "Run Test" and "Debug Test" for my NUnit tests, and the same for xUnit.NET. What am I missing in order to profile tests? Is this just not supported in Community edition, or I am missing some configuration or component?
(It would seem odd if it's not supported in Community, given I can profile executables in Community, and thus could painfully work around this issue by creating an executable that runs the test, and profile that. Why support profiling executables but not profiling tests?)
Steps to reproduce for NUnit: created new C# library project in Visual Studio Community 2015, pasted content of http://nunit.org/index.php?p=quickStartSource&r=2.6.4 into new file, installed NuGet packages as follows:
<packages>
<package id="NUnit" version="2.6.4" targetFramework="net452" />
<package id="NUnit.Runners" version="2.6.4" targetFramework="net452" />
<package id="NUnitTestAdapter" version="2.0.0" targetFramework="net452" />
</packages>
Even restarted Visual Studio. Tests show up in Test Explorer and can be run, but no "Profile Test" option available on right-click menu. Also tried equivalent steps for xUnit.net, but no joy.
This ishow I was able to profile an NUnit test in VS Community 2015 earlier today.
Profiling the NUnit Test Runner as an executable
Be sure you're running VS2015 as Administrator.
Click Analyze > Performance Profiler... from the Toolbar.
Choose Performance Wizard and click Start.
Page 1: In my case, I wanted to see allocations so I clicked .NET memory allocation.
Page 2: Leave the option An executable (.EXE file) checked and continue.
Page 3: On this page you have to define the executable to run.
This will be the test runner nunit3-console.exe for NUnit, or whatever the equivalent is for your test framework.
What is the full path to the executable? C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe
Command-line arguments: bin\Debug\Test.dll --inprocess --test TestNamespace.TestClassName.Test_Method_Name
Working directory: \\MAC\Home\Documents\GitHub\ApplicationName\Test
You will need to substitute these paths with ones that make sense for your system. The --inprocess switch causes the tests to be run inline from the NUnit process. Without that switch, a child process is spawned and even though the profiler will appear to work, you'll just be profiling nunit3-console.exe, not your own code.
Page 4: Click Finish.
Keep in mind that the profiler will generate report files and save them to your working directory. In my case, since my working directory was a UNC share, it required me to pick a local folder path to save the reports to before the profiler would start.
A terminal window should appear briefly with the NUnit runner output in it. The window auto-closes, so if you see a flash of red text, you won't have time to read the error before it is gone. You can copy the command from Page 3 into a command prompt for more leisurely reading.
After the command runs (whether it succeeded or not), you should get a report where you can track how many allocations your test caused.
Unfortunately, allocations in a small test will probably get overshadowed by the allocations caused by the NUnit.Framework itself. I clicked around to see if there was a way to exclude them from the results, but didn't find a way to do it, so I just ignored them.
If you want to profile a different test, you can open the Performance Explorer and right-click nunit3-console.exe > Properties to change the command line arguments and then click Actions > Start Profiling to refresh the report.
Conclusion
This solution succeeds at profiling the results of a single NUnit test, but that statement comes with some caveats.
It was only marginally less obnoxious than creating a separate executable to profile, and that fact that the NUnit allocations appear in the report could make it a non-starter if you need to do some really sensitive profiling.
Maybe someone with more VS 2015 experience can help me improve this answer with some tips on how to exclude the NUnit.Framework dll from the report?
It's possible to profile MSTest tests (can also profile nunit/xunit with test adapters) using vstest.console.exe
Select executable program as the target from Analyze -> Performance Profiler
Provide the path of the vstest.console.exe (usually lives in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow)
Provide parameters as <full path to test dll file>.dll /TestCaseFilter:"FullyQualifiedName~<namespace>.<test class>" as the parameters
If you are using nUnit or xUnit, then provide nunit test adapter,or xunit test adapter path using /TestAdapterPath argument to vstest.console.exe
More info on how to provide parameters to vstest.exe
The answer appears to be in a comment on the Adam Prescott page referenced in the question:
August 16, 2013 at 4:57 pm
Unfortunately, per MSDN, this feature is
only available in the Premium and Ultimate editions.
http://msdn.microsoft.com/en-us/library/ms182372.aspx
The link in the comment currently refers to the 2015 edition of Visual Studio. The 2010 edition of the page clearly shows which versions of Visual Studio 2010 support profiling. The pages for the 2012 and later version omit a clear statement of which versions support profiling.
I recently installed Visual Studio 2013 and CodeLens is amazing! The problem that I find is that whenever I open a class file that has methods in it, it doesn't seem to find the Unit Tests associated to the method if it is not written in MSTest. Is there anything that I have to do so that it can find other Unit tests like MSpec?
Is it because Machine Specifications has a different approach when creating unit tests vs MSTest or other testing framework out there?
The tested by and test status indicators are powered by the test explorer.
So if your MSpec tests show up in the test explorer and are written in C# or VB, they should also show up in CodeLens.
(same answer as Visual Studio 2013 feature Code Lens with NUnit)
I just did it with NUnit on a clean machine by installing Nunit and the Nunit extension in extension manager. Here Main shows one reference and one passing test, and the test itself shows the checkmark after successfully running.
I did the same thing that John Gardner showed in his answer, except I split it into 2 projects (to mimic the set up that I was working with in my real solution). At first it didn't work even though I could see the tests in Test Explorer. After a bit of searching, I stumbled onto the right answer for my situation.
It turns out that you need to:
(1) Create a new Unit Test Project (not a class library)
(2) Add NUnit reference to the Unit Test project (I used NuGet) and have the NUnit Test Adapter installed in VS2013
(3) Move your tests to this new project
(4) Once you save and build, now you can go back to your production code and see the "x/y passing" message and see the unit tests in the Test Explorer window.
Most of us that have been using NUnit for a long time are used to creating class libraries for our code instead of Unit Test Projects. It would be nice if the CodeLens documentation actually directly covered this (the documentation states "Test status indicators appear automatically in test projects" which was my clue).
Trying to debug an NUnit test in VS2010 and is not stopping on the breakpoint. It was working the last time I tried need to debug the test (maybe 1 yr ago) but now it doesn't work.
I've tried the following:
How to debug with vs.net 2008 and nunit?
http://erraticdev.blogspot.com/2012/01/running-or-debugging-nunit-tests-from.html
I've tried attaching, rechecking my installation, updated the config file so it's using the 4.0 runtime, still doesn't hit the breakpoint.
What am I missing?
You can debug nunit tests in VS studio community version only. For this you need to install Nunit3testAdpter. You can add this in Tools -> Extension and updates. or from Nuget packages.
This is a long overdue question but I haven't seen the solution to debug NUnit tests directly within Visual Studio. This is quite easily possible and without any third party tools. All you have to do is configure your test project (which is just a plain old Class library project) so it will start NUnit test runner whenever you hit F5 or Ctrl-F5 (just run, no debugging).
Particularly you will have to configure your project properties to start an external program:
Step by step configuration is very well described in this blog post. It advises you to use NuGet to obtain NUnit test runner and configure project to execute the runner when you try running your library project. As simple as that. Using no specific tools which are usually not free.
There are multiple reasons why this may happen. If your application's target framework is different from NUnit's, then you won't be able to debug, because the tests are actually being run by nunit-agent.exe.
For me, my application was using 4.5, but NUnit's was using 3.5. (You can find this from one of the NUnit GUI applications -> Help -> About)
To fix this, change the nunit.exe.config file to include the following, inside the configuration section:
<startup>
<requiredRuntime version="4.0.30319" />
</startup>
This can also happen when your PDB generation is set to embedded. Switching it to full will allow the nunit test adapter to start with the debugger attached. At that point it can be switched back to embedded and it will debug again.
The setting is on the project properties pages, in the Build tab, then click the Advanced button. I always choose "All Configurations" at the top, so I make sure I'm emitting a useful PDB (or embedding the info) when building Release configuration assemblies too.
The other day I failed to hit a breakpoint in my NUnit debugging. It took me a while to realize that someone had copy/pasted the test into a different file and I was running a different test than the one that the breakpoint was in.
I recently had a similar problem. Changing the project's platform to AnyCPU solved it for me. Also worth checking:
1) Check NUnit's latest version (I was using 2.5.7, updated to 2.6.2)
2) The Target Framework for your test project. I was using .NET 4.0 (you can find it under Properties -> Application, on VS2010)
At least up to vs2010 the debugger can only debug 32 bit apps but not 64 bit apps.
With vs2010 I am using a 32bit nunit.exe version and use "attach to debugger". This workes well for me.
Please upgrade NUnit Test Extension and close the Visual studio so Visx (probably spell) will run and update NUnit tool.
After this you can open your project and run NUnit test project.
It will run.
I tested accurately.
Try changing the test to x64. From the menu do
Test -> Test Settings -> Default Processor Architecture -> x64.
i think u cant debug with Visual studio. better u install TestDriven.NET.
then
Put the BreakPoint on the test method
Right click on the test
method.
select TestWith->Debugger
I'm trying to find a way to migrate from mstest to xunit and still benefit from the IDE integration which made me choose mstest in the first place. Gallio seems to accomplish this noble goal, plus it's free (don't want Reshaper or TestDriven.net). But I can't get code coverage to work.
My solution contains of two projects: project SUT (the assembly I need to test) and project SUT.Tests which is a VS test project (this allows Gallio to display xunit tests in VS's Test View). I have code coverage enabled in Local.testSettings for SUT.dll and instrumentation is in place. After the test run completes there is no code coverage. Code Coverage Results window reports: Empty results generated: none of the instrumented binary was used. Look at test run details for any instrumentation problems. Unfortunately the test run details do not contain any "instrumentation problems". I tried unchecking the instrument assemblies in place checkbox and re-running the unit test; same result.
Any idea what's wrong?
My setup:
- Windows 7 x64
- VS 2010 Premium (SP1)
- xUnit 1.8
- Gallio 3.3.1 x64 (installer, not zip)
Apologies for answering my own question:
Turns out Gallio loads SUT.dll from SUT.Tests\Bin\Debug rather than SUT\Bin\Debug. I added this path to code coverage details, selected it instead of the default SUT.dll path and now it works! This is better described at here, towards the end of the post.