I run my 700 unit tests in visual studio and they take less than 2 mins.
I also have Team Build set up to do a build and run my unit tests, which takes over 12 mins to run the same tests!
During my exploring of logs etc.. I've found this confusing info:
-If I open the MSTest .trx file (Visual Studio test results file) that was actually generated by Team Build on the server, it says the duration is about 2 mins.
-But when I view the Log from the build explorer, the log entry for unit tests (Run MSTest for Test Assemblies) shows the duration to be about 12 mins.
Anybody got any idea where the extra 10 mins have gone and perhaps more importantly is there something I can turn off to get them back?
TFS 2010, VS 2010.
Thanks,
Mike G
It's possible that this includes the publishing time (although it's very unlikely that publishing takes 10 mins). It could if you have a remote TFS server and you upload lots of big assemblies when publishing the test results. If that is indeed the case, I can look further to see if there is an option in the 2010 release that turns off publishing assemblies.
Related
Similar issue as this question (which does not have an answer), except we are using TFS 2013 Update 4. Our tests are written using NUnit and are configured to run through the build definition, which uses the default (not upgrade) template.
We just installed a full version of VS 2013 Premium on the build server (because prior tests were not being run at all). With that installed, the tests are run (if I did down into the log I can see them and the results, and if we force a failure the build appropriately fails) but the build summary says that 0/0 tests were run.
I followed the steps in the selected answer to this question, and it now shows the code coverage results, but still no dice on the actual tests or their results.
I also found this blog article that appears to be the same issue from TFS 2010. I have verified that all users have the View Test Runs permission set to allow, and I tried using a TFS administrator account just to be sure. Still no dice.
How do I make TFS show me what tests were run in the summary, and also in the Test Results window that the build summary links to?
Each individual test needs to be associated with a test case work item in Team Foundation Server. See this link for information on how to associate your tests. In my work, we're using TFS 2015, but I know this works in TFS 2013 as I have done this work in the past. Once the test case work item is associated with your automation (using Visual Studio to link the test automation to the test case) then there must be a build definition for the product under test that runs your test automation as part of the build. When a build is queued, your tests are run as part of the new build. Once the build is done, you can then view the detailed test results.
Plus the other StackOverflow post here describes how to view the detailed results.
I´m new to Visual Studio and this might be very easy to do but i've been researching for almost a week now and haven't got any results.
The thing is i'm trying to do some QA on a web project in visual studio 2010 that is under a TFS. I've recorded and coded some of my tests but i need this tests to run when there is a new check in by any of the developers.
I've read some of the How To's from MSDN but this step by step walkthroughs are not specific enough for the web performance test.
Thank you in advance
In you case you need to edit the build definition to include Build Verification Tests (BVTs) : theses tests can do check post build and also post deploy.
Web Performance Test, Load Test, Coded UI Test ... it's quite the same for TFS : some code to run to check something.
The good starting point is here.
I have Nunit unit test which i need to run as the part of my MS build.. I know that running all the test will slow up the build So, i need to run only the impacted test is there any way to find that out..
AFAIK running only impacted tests is not possible yet for NUnit tests. But this is possible for MSTests in Visual Studio 2010 Ultimate/Test Professional:
Recommending Tests to Run That are Affected by Code Changes
You can use Visual Studio Ultimate or Visual Studio Test Professional
2010 to help you determine which tests might have to be run, based on
coding changes that were made to the application you are testing. To
be able to use this functionality, you have to use Team Foundation
Build to build your application and use Microsoft Visual Studio 2010
for version control for your source code
Anyway you can use MSBuild Community NUnit Task to run tests from a set of the assemblies. You can do this as dependency target of standard AfterBuild target by specifying DependsOnTargets attribute.
<NUnit Assemblies="..."
IncludeCategory="..."
ExcludeCategory="..."
ToolPath="$(NUnitDllsPath)"
ProjectConfiguration="$(Configuration)"
OutputXmlFile="$(NUnitOutputPath)\UnitTests.xml"
ContinueOnError="true">
To know the impacted tests you need to track the test case code coverage. Only this way you can examine what test is impacted by the changes you are checking in. I don't know of any tool that does what you want besides Microsofts Team Foundation Server.
Running your tests as a part of the build can be done trough the Build Events properties of your project. You can execute the command line tool for NUnit.
But as PVitt already pointed out, I don't know if NUnit can work with Test Impact analysis.
We would like to automatically log the time we spend building our project every day, while devving. Builds can be initiated by pressing f5, or runnning our n-unit tests with TestDriven.Net. We want the entire combined time of building, not just for one or 2 projects. We want to do this to compare the impact on productivity of a slow pc versus a fast pc.
Any ideas?
Update: I've investigated MSBuild loggers. This seems like it could work, but I can't figure out how to send parameters to MSBuild when we do an f5 build from VS.
VSCommands for Visual Studio 2010 and 2012 is available from Visual Studio Gallery can produce detailed build statistics in output window after the build is finished:
Tools + Options, Projects and Solutions, Build and Run. Change the MSBuild project build output verbosity setting to Normal. You'll get a Time Elapsed measurement at the end of the build. You can turn this setting to 11 to get a detailed breakdown. Build log output is automatically saved to the buildlog.htm file in the project's build directory.
The equivalent command line option is /verbosity:normal (/v:n). You can get output logged to a file with redirection or the /fileLogger option (/fl). Type msbuild /? to get a summary of these options, there are a bunch more obscure ways to log.
In the project output directory there is a project_name.log file with the line
"Time Elapsed 00:00:00.00"
You can parse this
You can use Tools | Options | Projects and Solutions | VC++ Project Settings | Build Timing to control whether that copy of Visual Studio includes timing in VC++ build output.
But that time and the Time Elapsed mentioned by #Martin appear in the Output window of Visual Studio when you do F5 builds and on the console when you do command line builds.
It sounds like you really want to collect the build times for several builds in one place. One place to start looking might be MSBuild Loggers.
I realise that there are many older questions addressing the general question of NUnit v MSTest for versions of Visual Studio up to 2008 (such as this one).
Microsoft have a history of getting things right in their 3rd version. For MSTest, that is VS2010.
Have they done so with MSTest? Would you use it in a new project in preference to NUnit?
My specific concerns:
speed
running tests within CruiseControl.NET (either commandline or MSBuild task)
code coverage reports from CC.NET
can you run MSTest tests in debug mode
(We use ReSharper, so test-runners are not an issue for us. We have used NUnit for the last few years. We do not have TFS.)
List item speed is same, but MsTest may be a bit slower because it creates folder for test run every time
MSBuid and CC.Net is big pain. You can't run MSTest on computer without VS on it (not 100 sure about 2010, but with 2008 it is so)
not sure, sorry
yes you can, from visual studio
My recommendation is following: if NUnit satisfies you - use it, forget about MSTest
To correct some old information on the thread;
It IS possible to run 64 bit tests in 2010
From VS2008 forward it is not neccesary to have MSTEST create directories anc opy the binaries in, just disable deployment, in 2010 thats the default but you have to set it in 2008
2010 MSTEST is faster but as its a generalised test framework that also runs load/web/UI tests there are compromises in the design that will lead to it being slower. Jamie Cansdale appears to have managed to get perf increases with the lastest releases of TestDriven.net's support for MSTEST
I've mainly used NUnit, some xUnit and some MSTest. They seem functionality equivalent, but I don't like the MSTest test runner. It runs in visual studio so it either crowds the screen or is on another monitor getting in the way everytime I tab to visual studio. (I run NUnit on another monitor, but it doesn't cover everything on that monitor everytime I focus visual studio). It takes too many clicks to find out what test failed and why.
NUnit can run in the background until a test fails, at which point it shows you information about the breaking test. This seems like the ideal for keeping red/green/refactor going smoothly.
Nope. Same issues regarding appdomains and assembly resolving still exist. I would avoid unless you want the new goodness for other functional testing or integration with Team System.
I don't know much about CruseControl.net, but you can debug tests. We currently don't use TFS either, and the MSTest is working for us.
If you think you'll ever run your tests in 64 bit mode, use NUnit. MsTest is only x86.
One major difference between the two is that MSTest makes a copy of the current DLLs every time it runs a test. If you're doing TDD and running your tests frequently, this can eat up a lot of hard drive space.
If you're using MSTest, you can change this setting in Tools > Options > Test Tools > Test Execution. "Limit number of old Test Results to" is set to 25 by default in Visual Studio 2010. I usually change it to 1.
MSUnit runs your test cases under conditions that are different from the actual execution environment. Specifically, the deployed files differ from those that are deployed when you run your actual project. Nethertheless, there is the [DeploymentItem]-Attribute to specify which files shall be deployed by MSUnit. So if your application depends on any external
files, such as
database files
database configuration file
application configuration file
...
then MSUnit is not the right choice, because the MSUnit tests never cover what your file system is going to look like in execution environment. The Visual Studio Project File settings for deploying files (Copy always, Content, etc.) are ignored by the MSUnit runner. So those settings cannot be tested.