How can I debug step-by-step a project part of Visual Studio 2010 solution created by CMake?
The CMakeLists.txt has created the three projects ALL_BUILD, RUN_TEST and INSTALL, and other separate projects for each of the targets of the CMakeLists.txt
Tests are therefore created by CMake and so are listed in Visual Studio as separate projects to compile, in order to generate this way an executable for each test. I am not using the inner test framework offered by Visual Studio, because it seems that CMake are not taking it into consideration!!
Given that building RUN_TEST some tests fail, I would like to do a step-by-step debug of the tests, but it is not possible to do this by setting a test as start project. Moreover, the test has no command line arguments set if executed outside the RUN_TEST script.
Is there someone who knows CMake with Visual Studio 2010 ?
Does anyone know how I can debug my tests?
Tnx in advance
In case you are using the CppUnitTestFramework:
Open the Test View (Test / Windows / Test View). You should see a list of all tests there. Right click a test you want to debug and select "Debug Selection". That's it.
You may also select more than one test and debug them in this way.
Related
Posts on the web suggest that you can profile tests in Visual Studio by right-clicking and selecting "Profile Test": http://adamprescott.net/2012/12/12/performance-profiling-for-unit-tests/ But I only see "Run Test" and "Debug Test" for my NUnit tests, and the same for xUnit.NET. What am I missing in order to profile tests? Is this just not supported in Community edition, or I am missing some configuration or component?
(It would seem odd if it's not supported in Community, given I can profile executables in Community, and thus could painfully work around this issue by creating an executable that runs the test, and profile that. Why support profiling executables but not profiling tests?)
Steps to reproduce for NUnit: created new C# library project in Visual Studio Community 2015, pasted content of http://nunit.org/index.php?p=quickStartSource&r=2.6.4 into new file, installed NuGet packages as follows:
<packages>
<package id="NUnit" version="2.6.4" targetFramework="net452" />
<package id="NUnit.Runners" version="2.6.4" targetFramework="net452" />
<package id="NUnitTestAdapter" version="2.0.0" targetFramework="net452" />
</packages>
Even restarted Visual Studio. Tests show up in Test Explorer and can be run, but no "Profile Test" option available on right-click menu. Also tried equivalent steps for xUnit.net, but no joy.
This ishow I was able to profile an NUnit test in VS Community 2015 earlier today.
Profiling the NUnit Test Runner as an executable
Be sure you're running VS2015 as Administrator.
Click Analyze > Performance Profiler... from the Toolbar.
Choose Performance Wizard and click Start.
Page 1: In my case, I wanted to see allocations so I clicked .NET memory allocation.
Page 2: Leave the option An executable (.EXE file) checked and continue.
Page 3: On this page you have to define the executable to run.
This will be the test runner nunit3-console.exe for NUnit, or whatever the equivalent is for your test framework.
What is the full path to the executable? C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe
Command-line arguments: bin\Debug\Test.dll --inprocess --test TestNamespace.TestClassName.Test_Method_Name
Working directory: \\MAC\Home\Documents\GitHub\ApplicationName\Test
You will need to substitute these paths with ones that make sense for your system. The --inprocess switch causes the tests to be run inline from the NUnit process. Without that switch, a child process is spawned and even though the profiler will appear to work, you'll just be profiling nunit3-console.exe, not your own code.
Page 4: Click Finish.
Keep in mind that the profiler will generate report files and save them to your working directory. In my case, since my working directory was a UNC share, it required me to pick a local folder path to save the reports to before the profiler would start.
A terminal window should appear briefly with the NUnit runner output in it. The window auto-closes, so if you see a flash of red text, you won't have time to read the error before it is gone. You can copy the command from Page 3 into a command prompt for more leisurely reading.
After the command runs (whether it succeeded or not), you should get a report where you can track how many allocations your test caused.
Unfortunately, allocations in a small test will probably get overshadowed by the allocations caused by the NUnit.Framework itself. I clicked around to see if there was a way to exclude them from the results, but didn't find a way to do it, so I just ignored them.
If you want to profile a different test, you can open the Performance Explorer and right-click nunit3-console.exe > Properties to change the command line arguments and then click Actions > Start Profiling to refresh the report.
Conclusion
This solution succeeds at profiling the results of a single NUnit test, but that statement comes with some caveats.
It was only marginally less obnoxious than creating a separate executable to profile, and that fact that the NUnit allocations appear in the report could make it a non-starter if you need to do some really sensitive profiling.
Maybe someone with more VS 2015 experience can help me improve this answer with some tips on how to exclude the NUnit.Framework dll from the report?
It's possible to profile MSTest tests (can also profile nunit/xunit with test adapters) using vstest.console.exe
Select executable program as the target from Analyze -> Performance Profiler
Provide the path of the vstest.console.exe (usually lives in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow)
Provide parameters as <full path to test dll file>.dll /TestCaseFilter:"FullyQualifiedName~<namespace>.<test class>" as the parameters
If you are using nUnit or xUnit, then provide nunit test adapter,or xunit test adapter path using /TestAdapterPath argument to vstest.console.exe
More info on how to provide parameters to vstest.exe
The answer appears to be in a comment on the Adam Prescott page referenced in the question:
August 16, 2013 at 4:57 pm
Unfortunately, per MSDN, this feature is
only available in the Premium and Ultimate editions.
http://msdn.microsoft.com/en-us/library/ms182372.aspx
The link in the comment currently refers to the 2015 edition of Visual Studio. The 2010 edition of the page clearly shows which versions of Visual Studio 2010 support profiling. The pages for the 2012 and later version omit a clear statement of which versions support profiling.
I've been searching all morning for an answer online to this and I have tried a lot of the suggestions though all solutions seem to be for visual studio 2010 and I am running 2013 premium edition, I can't see anyone with the same problem.
I have premium VS 2013, I installed SpecFlow v1.9 through the extension manager and added the SpecFlow nuget package. I have created a CodedUI test project and added a feature file, some recorded codedui steps and a stepdef file. Nothing is complaining and it all looks like it should work. I have added :
<unitTestProvider name="MsTest"/>
to my App.config so I can run it from the test explorer in visual studio. I eventually want to run the tests via MTM but I will deal with that when I get this to work!
I have built and it is all happy so I go to test explorer and I can see my test so I right click, run the test. It fails with the following error, it does get to my Given step when I debug and falls over trying to open my application.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestTools.UITest.WindowsStoreUtility, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
I can not find the reference above when I have looked to add it to my project references, there is a similar one but it breaks some of my other references when I add it. I also had a look at my registry files and the path it is referring to does not exist for me to edit it. From what I have read codedui doesn't work nicely with specflow without a dll file. All references to this fix seem to be for vs 2010 and require that I copy a dll to my specflow source folder. I installed specflow from visual studio so I don't have a program files folder for it so I came to a bit of a sticking point with that solution.
Does anyone have specflow working with visual studio 2013 and a codedui test project? Am I missing a set up step? Or is this genuinely to do with my registry files / references. The error is confusing as I don't see why it is trying to find that reference especially when I can't find that reference anywhere when I try to find it manually. I'd be interested to know if you have had this error and managed to resolve it or if you could explain your set up steps so I can check I did not miss anything.
Thanks!
I believe you might be missing references to some of the required dlls. I just finished testing a test created using a Specflow, codedUI test project on my MS Visual Studio 2013 ultimate and it worked just fine.
Here are the steps which I followed:
Created a class library project.
Added reference to Specflow 1.9 from Extensions and Updates.
Added reference to Specflow CodedUI Attribute Generator (Install-Package SpecFlow.CodedUI -Version 1.0.0.23).
Added references to required libraries required for CodedUI and Test Tools to the project. See below the references I added.
Microsoft.VisualStudio.QualityTools.CodedUITestFramework
Microsoft.VisualStudio.QualityTools.UnitTestFramework
Microsoft.VisualStudio.TestTools.UITest.Common
Microsoft.VisualStudio.TestTools.UITest.Extension
Microsoft.VisualStudio.TestTools.UITest.ExtensionUtilities
Microsoft.VisualStudio.TestTools.UITesting
Added a separate CodedUI project to solution and added references of this project to the class library project.
Create a new Specflow feature file with some test and create step for it in step definition and reference to the function you created or recorded in the CodedUI project in the step definition.
Run the Specflow test from Test Explorer. It should work fine.
Note: The above steps should work just fine if you will create CodedUI project instead of a Class library project.
Mark the test class with the CodedUITest attribute.
I had this exact error message even though I had all the .dll's referenced correctly.
The solution for me was to regenerate the feature files.
(right click project and select "regenerate feature files")
Leaving this answer in case it helps someone in future.
I came across the same issue today, not with specflow but 2013 build controller and agent and codedui tests.
The solution was to install VS2013 Update 5 https://www.visualstudio.com/downloads/download-visual-studio-vs
I had same problem, found the following solution works well:
http://blog.majcica.com/2015/05/07/getting-started-with-specflow-and-codedui/
I am using Visual Studio 2010. Let`s say I have three projects in a solution:
MainProject
DependentA
DependentB
MainProject is my start up project and DependentA and DependentB are referenced to it. The build mode is set to Debug. Then I create a custom solution configuration Test and select it.
Now Visual studio sets the build option for DependentA and DependentB to Release automatically.
What I want to accomplish is to set some condidions. For example if Test configuration is selected build the DependentA and DependentB in Debug mode. I am quite new in this area but I suppose I can set some condition in the csproj file as it is done in this question:
Visual Studio 2010 Compiling with the Debug or Release version of third party library depending on if my project is being compiled Build or Release?
Any help with some example will be greatly appreciated. Thanks!
Have you used the Build Configuration Manager to select the desired project build configurations for the Test solution configuration? It's not clear from your question whether you're looking for anything more complicated than that.
Trying to debug an NUnit test in VS2010 and is not stopping on the breakpoint. It was working the last time I tried need to debug the test (maybe 1 yr ago) but now it doesn't work.
I've tried the following:
How to debug with vs.net 2008 and nunit?
http://erraticdev.blogspot.com/2012/01/running-or-debugging-nunit-tests-from.html
I've tried attaching, rechecking my installation, updated the config file so it's using the 4.0 runtime, still doesn't hit the breakpoint.
What am I missing?
You can debug nunit tests in VS studio community version only. For this you need to install Nunit3testAdpter. You can add this in Tools -> Extension and updates. or from Nuget packages.
This is a long overdue question but I haven't seen the solution to debug NUnit tests directly within Visual Studio. This is quite easily possible and without any third party tools. All you have to do is configure your test project (which is just a plain old Class library project) so it will start NUnit test runner whenever you hit F5 or Ctrl-F5 (just run, no debugging).
Particularly you will have to configure your project properties to start an external program:
Step by step configuration is very well described in this blog post. It advises you to use NuGet to obtain NUnit test runner and configure project to execute the runner when you try running your library project. As simple as that. Using no specific tools which are usually not free.
There are multiple reasons why this may happen. If your application's target framework is different from NUnit's, then you won't be able to debug, because the tests are actually being run by nunit-agent.exe.
For me, my application was using 4.5, but NUnit's was using 3.5. (You can find this from one of the NUnit GUI applications -> Help -> About)
To fix this, change the nunit.exe.config file to include the following, inside the configuration section:
<startup>
<requiredRuntime version="4.0.30319" />
</startup>
This can also happen when your PDB generation is set to embedded. Switching it to full will allow the nunit test adapter to start with the debugger attached. At that point it can be switched back to embedded and it will debug again.
The setting is on the project properties pages, in the Build tab, then click the Advanced button. I always choose "All Configurations" at the top, so I make sure I'm emitting a useful PDB (or embedding the info) when building Release configuration assemblies too.
The other day I failed to hit a breakpoint in my NUnit debugging. It took me a while to realize that someone had copy/pasted the test into a different file and I was running a different test than the one that the breakpoint was in.
I recently had a similar problem. Changing the project's platform to AnyCPU solved it for me. Also worth checking:
1) Check NUnit's latest version (I was using 2.5.7, updated to 2.6.2)
2) The Target Framework for your test project. I was using .NET 4.0 (you can find it under Properties -> Application, on VS2010)
At least up to vs2010 the debugger can only debug 32 bit apps but not 64 bit apps.
With vs2010 I am using a 32bit nunit.exe version and use "attach to debugger". This workes well for me.
Please upgrade NUnit Test Extension and close the Visual studio so Visx (probably spell) will run and update NUnit tool.
After this you can open your project and run NUnit test project.
It will run.
I tested accurately.
Try changing the test to x64. From the menu do
Test -> Test Settings -> Default Processor Architecture -> x64.
i think u cant debug with Visual studio. better u install TestDriven.NET.
then
Put the BreakPoint on the test method
Right click on the test
method.
select TestWith->Debugger
If I install Gallio 3.x will it also install a test runner plugin for Visual Studio?
Or must I use an additional plug-in like TestDriven.NET or Visual Nunit to run MbUnit test classes from within VS?
Install Gallio 3.1 on the dev machine. Then in VS2008, you'll have the option to create a "MbUnit v3 Test Project". This doesn't just include all of the Gallio dlls for you, it has a magic line in the project which identifies it to VS as a Test project.
You can now just use the in-built VS2008 Test runner.
If you have any existing projects with unit tests in, rather than making new projets, edit your existing project file and add the following line on line 9 (underneath the <ProjectGuid> on line 8):
<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
(If you have a VB project, it has a different second GUID: <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids> You can find the correct values by creating a new MbUnit test project from the templates installed with Gallio and then looking at the project file (.csproj or .vbproj) in a text editor.)
Now when you reload the project, VS2008 will recognise it as a test project.
A distinct advantage that I found over using Icarus was that debugging is now far more straight forward with break points being hit as expected.
Good Luck, Lee
TestDriven.Net works really well. Gallio also supports the ReSharper unit test runner and Visual Studio test tools. We will be shipping a new release of Gallio this week with support for R# 5.0 and VS 2010.
These are instruction for running MBUnit tests in Visual Studio 2012 and above using a neat NUnit trick.
Firstly, install the NUnit Test Adapter extension (yes, NUnit)
Tools > Extension and Updates > Online > search for NUnit > install
NUnit Test Adapter.
You may need to restart the Visual Studio IDE.
Then, you simply need to add a new NUnit test attribute to your test methods. See example code here (notice the using statements at the top) ...
//C# example
using MbUnit.Framework;
using NuTest = NUnit.Framework.TestAttribute;
namespace MyTests
{
[TestFixture]
public class UnitTest1
{
[Test, NuTest]
public void myTest()
{
//this will pass
}
}
}
You can run and debug the test in visual studio as NUnit and Gallio Icarus GUI Test Runner will run them as MBUnit (enabling parallel runs for example). You will need to stop Gallio from running the NUnit tests by deleting the NUnit folder in the gallio install location i.e. C:\Program Files\Gallio\bin\NUnit
Hope this helps, this is a simple working method so please vote up, many thanks.