How do you replace the 'old' test lists (as in a VS2008 vsmdi file) on a VS2013/TFS2013 build agent? We have VS2013 Professional and Team Foundation Server, and it seems that test lists are no longer supported, and the 'new' test playlists are not supported either for the TFS build agent. MSBUILD no longer seems to support the concept of lists.
Can someone clarify how you should execute selected lists of tests using MSBUILD on a TFS2013 build agent?
Instead of using MSBuild command, you need to choose to use MSTest command. However, by default the TfvcTemplate.12.xaml build template does not support the MSTest.exe option when you select the Test Runner in the build definition. Good news is there is a simple way for you to make it work:
Instead of using TFS2013 build process template, you can choose to use TFS2012 build process template which enables MSTest. You can get it here. Then select to use the TFS2012 build template, click to Automated Tests part, select MSTest Test Metadata File as test runner.
An alternative is to use Test Filters. Tag your unit tests in source control using attributes:
[Category("ListX")]
[TestMethod]
public void Test(){ /* ... */ }
Then you can add a Test Filter to your build definition (sorry for the typo in the screenshot, should read TestCategory=Critical, or ListX in the example above:
See also:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
Related
I have a suite of XUnit Tests in a .NET solution that is built in a TeamCity CI environment. The tests in question generate html templates. In our development environment, they write the results to the Visual Studio console using ITestOutputHelper. In Team City, the output is making the build artifacts way too big, and I want to find a way of suppressing the output as it is not needed in the build.
I imagine this will be achieved using an XUnit console runner command line option (but open to suggestions). I have found this list of command line options fsharp/FAKE#759, but it doesn't include one to suppress output. Is there any other (configuration-based) way of achieving what I want? (There are a lot of test classes and I don't want to have to refactor them all).
We have a TFS build definition that kicks off NUnit tests tagged with the 'Regression' test category. This uses the NUnit console runners annotation of
where cat = 'Regression'
However, we have multiple different environments where some tests will fail in one environment, they will pass in the other. We have not made much use of the Playlist feature, because I can not find a way to target a playlist when running remotely on TFS. Does anyone know how this can be done? Thanks!
Unfortunately, there is no way of specifying this/playlist in the TFS Build Definition for now. A related uservoice:
Allow test playlists to be used by TFS build servers
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3853614-allow-test-playlists-to-be-used-by-tfs-build-serve
As a workaround, you could use .orderedtest instead of .playlist.
Ordered tests can be created and edited in VS2013 and later. The format is otherwise similar to .playlist but it contains links to test GUIDs so its more complicated to modify programmatically.
TFS is able to run orderedtest in build pipeline, how to achieve this you could refer below links:
TFS - order of automata tests to execute
How to use Vnext build: ordered tests, distribute test, collect results
I'm new to Coded UI testing and TFS. I've written a coded UI test for my web application and looking for different options to automatically run it. Apparently the first option is to run it from visual studio. I have also found an article explaining how to associate a test method with a test case however I don't know whether I can run my test method (s) from TFS or not? How about running them with my build? I know the unit tests can be run with the build however I'm not sure about coded UI tests.
Yes, it is possible for you to run Coded UI test in the TFS build process.
Firstly, you need to configure the Team Build Service Host run as an interactive process. See this link.
Secondly, add the Coded UI test project into TFS Version Control.
Thirdly, create a build definition to build the Coded UI test project, and also specify tests to run.
Please have a check of this blog for the detailed steps: http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
I was wondering if someone would be able to explain what Build runners do and also what i would need to use for just a static HTML / CSS / JS site, or even an already compiled .NET site.
I will be hooking up each project to its equiv SVN and grabbing updates from there, but not 100% sure what the build runners do or which i should use as technically i dont need to build the site.
Sorry it may be too much to answer but i am just struggling to get my head round Team City
Thank you
Build runners are just a process for a specific task, for example the MSBuild runner is set up by putting information into specific fields which it then uses to call MSBuild on the target build agent. You could just as easily use the Command Line runner and build up the MSBuild run command manually.
Build runner is a part of TeamCity that allows integration with a
specific build tool (Ant, MSBuild, Command line, etc.). In a build
configuration, the build runner defines how to run a build and report
its results. Each build runner has two parts:
server-side settings that are configured through the web UI
agent-side part that executes a build on agent
You need to choose your runner depending on the task that you want to do and the technology that you have wrapped your project in. If there is no runner for your specific task then you can use the lowest common denominator which would be the Command Line runner.
The way I approach this would be to see how I can achieve what I want to from my own environment be that calling a rake, MSBuild or batch file. I then see how I can then apply that to a tool. Do not create a process around a tool but choose a tool that fits to your process.
I'm new to load testing in Visual Studio/MSTest, and I created a new Load Test recently to validate some high-traffic scenarios for a WCF service. I want to add this to the tests project for the service, but I don't want the test to be executed whenever I "Run All Tests in Solution" nor as part of our Continuous Integration build-verification process because a) it takes 5 minutes to run, and b) the service call that it is testing generates many thousands of email messages. Basically, I'd like to do the equivalent of adding the [Ignore] attribute to a unit test so that the load test is only executed when I explicitly choose to run it.
This MSDN Article ("How to: Disable and Enable Tests") suggests that the only to disable the test is to use Test Lists (.vsmdi files), but I don't have much experience with them, they seem like a hassle to manage, I don't want to have to modify our CI Build Definition, and this blog post says that Test Lists are deprecated in VS2012. Any other ideas?
Edit: I accepted Mauricio's answer, which was to put the load tests into a separate project and maintain separate solutions, one with the load tests and one without. This enables you to run the (faster-running) unit tests during development and also include the (slower-running) load tests during build verification without using test lists.
This should not be an issue for your CI Build Definition. Why?
To run unit tests as part of your build process you need to configure the build definition to point to a test container (usually a .dll file containint your test classes and methods). Load tests do not work this way, they are defined within .loadtest files (which are just xml files) that are consumed by the MSTest engine.
If you do not make any further changes to your CI Build definition the load test will be ignored.
If you want to run the test as part of a build, then you need to configure the build definition to use the .loadtest file.
Stay away from testlists. Like you said, they are being deprecated in VS11.
Edit: The simplest way to avoid running the load test as part of Visual Studio "Run All" tests is to create a different solution for your load tests.
Why don't you want to use Test Lists. I think is the best way to do that. Create different Test Lists for each test type (unit test, load test...) and then in your MSTest command run the Test List(s) you want:
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests (only UnitTests)
MSTest \testmetadata:testlists.vsmdi \testlist:LoadTests (only LoadTests)
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests \testlist:LoadTests (UnitTests & LoadTests)