I created a Load Test that I would like to run every night so that I can compare the results over time.
I'm trying to trigger the load test to run from a TFS build. Tests are run by Visual Studio Online agents.
Here is the MSTest command that is being executed, however I'm getting an error and the load tests never run.
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe" /nologo /usestderr /testSettings:"F:\Builds\8269\Tests\LoadTests\src\Tests\xxxxx\Remote.testsettings" /searchpathroot:"F:\Builds\8269\Tests\LoadTests\bin" /resultsfileroot:"F:\Builds\8269\Tests\LoadTests\tst" /testcontainer:"F:\Builds\8269\Tests\LoadTests\bin\LoadTest2.loadtest" /publish:"https://xxxxx.visualstudio.com/defaultcollection" /noprompt /publishbuild:"vstfs:///Build/Build/13578" /teamproject:"Tests" /platform:"Any CPU" /flavor:"Release" /runtitle:"MySweetLoadTests"
Error...
The current SynchronizationContext may not be used as a TaskScheduler.
From here (The current SynchronizationContext may not be used as a TaskScheduler)
It says I need to setup a SynchronizationContext, however I'm not sure how to do that within a load test.
Thanks!
This now supported. Please refer to the blog.
http://blogs.msdn.com/b/visualstudioalm/archive/2015/08/24/cloud-load-test-support-in-mstest-exe-command-line-and-xaml-builds.aspx
Related
Previously, we were using Visual Studio 2017.
We've recently migrate to use Visual Studio 2019.
Every unit test is correctly discovered by VS2019 but, when we cannot run them. It sais that tests takes some time to execute (0.6s or 1.8s for example) but they don't produce any result and it don't reach any break points.
We tried to add MsTest.Adapter and NUnit3TestAdapter without any change.
Any idea ?
Found out what to do (or at least what can make it works) :
We removed every test package (MSTest.Adapter and NUnit3ATestAdapter) and add a direct reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework (C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll)
Now, every test works fine (even if the ouput is awful)
I recently installed XUnit and the XUnit VS runner (i.e. xunit.runner.visualstudio) on Visual Studio following the documentation. The installation process uses NuGet and on completion I can successfully see all tests in the MS Test Explorer.
However, if I start a new solution and reference all the required XUnit dll's, i.e., I want to use XUnit without using NuGet to install it, the tests never show up in MS Test Explorer (no problems seeing them in the ReSharper test explorer).
I think this has something to do with the XUnit test adapter that Visual Studio uses to find XUnit tests. It appears that NuGet somehow instructs Visual Studio to create the VisualStudioTestExplorerExtensions folder on each build putting the necessary XUnit test adapter dll's (e.g.xunit.runner.visualstudio.testadapter.dll) in that folder. The MS Test Explorer then uses this test adapter to find XUnit test (see SO Question).
When I attempt to use XUnit without NuGet, the VisualStudioTestExplorerExtensions folder never gets created on build. Adding the folder manually with all XUnit dll's in it also doesn't work. It's almost as if i need to instruct Visual Studio to look in that folder, but I can't see any obvious way of doing that.
How do I use/install XUnit without using NuGet? Or, how do I get VS to look in the `VisualStudioTestExplorerExtensions' when looking for tests?
If you want to install a test runner manually, copy all the necessary dlls to:
C:\Program Files (x86)\Microsoft Visual Studio {version}\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions
Replacing {version} with the appropriate number.
The drawback here is that this is machine wide (maybe desirable in your case), and that you can't quite as easily replicate this setup on a build server. You also will probably have to restart VS whenever you make changes to these files.
We are in the process of upgrading our TFS 2013 server to TFS 2015. In that regard I have run into an issue when trying to perform an application build on a build agent with the new version, using our XAML template build script (which was created in TFS 2013).
Most of our solutions use the above-mentioned template when we build them. The solutions (sadly, still) use setup projects (.vdproj) to generate an MSI package for the application. To build the setup project, and produce an MSI, for a given application; we call the devenv.exe (in our case, it points to Visual Studio 2013) in our template build script:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\devenv.exe /Build "Release|Any CPU" "d:\<path_to_solution>\solution.sln" /Out c:\temp\out.log
However, on our new build agent no MSI is produced, and calling devenv.exe yields neither any output, nor any error messages. If I try to call devenv.com, however, it starts building, but still no MSI is being produced.
When things start to build using devenv.com It seems like it is completely disregarding the setup project (I have checked the build configurations, and the setup project is checked for build on the "Release|Any CPU" configuration).
I have looked at build logs, build process activity logs, and I have also tried to perform the steps manually on the build agent, but I haven't been able to find anything that could point me in the direction as to what might be causing the issue.
Does anyone have an idea what could be the reason for this? Could it be an issue with TFS Build 2015? The only difference between the old setup, and the new - as far as I can tell - is the following:
VS 2013 was installed on C: rather than D: on the new build agent, i.e. the path to devenv would be different, but that should not matter as long as the build script is looking in the right place, and finds it...
We use a different drop location than what we used for the old setup
Appreciate any help and suggestions I can get.
Problem solved, and everything is now building successfully. I did not do a good enough job of looking at the diff between the setup we had on the build agent we used for TFS Build 2013 vs. the one we set up for TFS Build 2015.
Firstly, to be able to build setup projects in VS 2013, one needs to install the Visual Studio 2013 Installer Projects extension on the build agent.
However, there are some bugs with this extension that often causes building setup projects to yield the following error message: "An error occurred while validating. HRESULT = '8000000A'". This error can be fixed by modifying the registry, as described in this SO post.
Building the setup projects using devenv.exe, however, still doesn't work. I have not been able to figure out why, yet.
Use the Vnext build in TFS 2015 to build .vdproj. You just need to add one additional build step i.e "command line" then call devenv.com to build.
Such that,
tool :
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.com
arguments :
HelloWorldTestInstaller\HelloWorldTestInstaller.vdproj /build release
This will help you.
I migrated my web performance tests project's build to TFS 2013 build process template for auto nuget restore by following this document.
My solution has .webtest files which are the containers for web performance tests. But VsTest.console.exe doesn't recognize the .webtest tests. I looked at the discoverers and found that none of the ones installed support the .webtest extension.
I tried to find test adapters for running .webtests using vstest.console.exe but found nothing.
This document says the following for vstest.console.exe:
You can run automated unit and coded UI tests from the command line.
And this for mstest.exe:
You can run automated Web performance and load tests from the command line either locally or by using a test controller or test agents.
Why doesn't vstest.console.exe support .webtests ? Wasn't it designed as a replacement to mstest.exe for running any kind of tests using adapters ?
Also, if it means that I need to write a custom adapter that can read and run tests with .webtest extension, what does it take for me to do that ?
vstest.console can run webtests using the following:
"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" SmokeTest.orderedtest /Settings:..\..\..\local.testsettings"
You need to create an orderedtest and put your webtest in it.
vstest.console can run .webtest Webtests. Enter the path to the exe in quotation marks and then the path to the webtest:
"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" WebTest1.webtest
Looks like vstest.console.exe cannot run load tests and web tests. My only option is to use MsTest.exe through TFS 2013 by customizing the build process template to suit my needs.
Edit: I'm wrong. You can run web tests using vstest.console.exe.
I have Nunit unit test which i need to run as the part of my MS build.. I know that running all the test will slow up the build So, i need to run only the impacted test is there any way to find that out..
AFAIK running only impacted tests is not possible yet for NUnit tests. But this is possible for MSTests in Visual Studio 2010 Ultimate/Test Professional:
Recommending Tests to Run That are Affected by Code Changes
You can use Visual Studio Ultimate or Visual Studio Test Professional
2010 to help you determine which tests might have to be run, based on
coding changes that were made to the application you are testing. To
be able to use this functionality, you have to use Team Foundation
Build to build your application and use Microsoft Visual Studio 2010
for version control for your source code
Anyway you can use MSBuild Community NUnit Task to run tests from a set of the assemblies. You can do this as dependency target of standard AfterBuild target by specifying DependsOnTargets attribute.
<NUnit Assemblies="..."
IncludeCategory="..."
ExcludeCategory="..."
ToolPath="$(NUnitDllsPath)"
ProjectConfiguration="$(Configuration)"
OutputXmlFile="$(NUnitOutputPath)\UnitTests.xml"
ContinueOnError="true">
To know the impacted tests you need to track the test case code coverage. Only this way you can examine what test is impacted by the changes you are checking in. I don't know of any tool that does what you want besides Microsofts Team Foundation Server.
Running your tests as a part of the build can be done trough the Build Events properties of your project. You can execute the command line tool for NUnit.
But as PVitt already pointed out, I don't know if NUnit can work with Test Impact analysis.