I have a suite of XUnit Tests in a .NET solution that is built in a TeamCity CI environment. The tests in question generate html templates. In our development environment, they write the results to the Visual Studio console using ITestOutputHelper. In Team City, the output is making the build artifacts way too big, and I want to find a way of suppressing the output as it is not needed in the build.
I imagine this will be achieved using an XUnit console runner command line option (but open to suggestions). I have found this list of command line options fsharp/FAKE#759, but it doesn't include one to suppress output. Is there any other (configuration-based) way of achieving what I want? (There are a lot of test classes and I don't want to have to refactor them all).
Related
We have a TFS build definition that kicks off NUnit tests tagged with the 'Regression' test category. This uses the NUnit console runners annotation of
where cat = 'Regression'
However, we have multiple different environments where some tests will fail in one environment, they will pass in the other. We have not made much use of the Playlist feature, because I can not find a way to target a playlist when running remotely on TFS. Does anyone know how this can be done? Thanks!
Unfortunately, there is no way of specifying this/playlist in the TFS Build Definition for now. A related uservoice:
Allow test playlists to be used by TFS build servers
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3853614-allow-test-playlists-to-be-used-by-tfs-build-serve
As a workaround, you could use .orderedtest instead of .playlist.
Ordered tests can be created and edited in VS2013 and later. The format is otherwise similar to .playlist but it contains links to test GUIDs so its more complicated to modify programmatically.
TFS is able to run orderedtest in build pipeline, how to achieve this you could refer below links:
TFS - order of automata tests to execute
How to use Vnext build: ordered tests, distribute test, collect results
How do you replace the 'old' test lists (as in a VS2008 vsmdi file) on a VS2013/TFS2013 build agent? We have VS2013 Professional and Team Foundation Server, and it seems that test lists are no longer supported, and the 'new' test playlists are not supported either for the TFS build agent. MSBUILD no longer seems to support the concept of lists.
Can someone clarify how you should execute selected lists of tests using MSBUILD on a TFS2013 build agent?
Instead of using MSBuild command, you need to choose to use MSTest command. However, by default the TfvcTemplate.12.xaml build template does not support the MSTest.exe option when you select the Test Runner in the build definition. Good news is there is a simple way for you to make it work:
Instead of using TFS2013 build process template, you can choose to use TFS2012 build process template which enables MSTest. You can get it here. Then select to use the TFS2012 build template, click to Automated Tests part, select MSTest Test Metadata File as test runner.
An alternative is to use Test Filters. Tag your unit tests in source control using attributes:
[Category("ListX")]
[TestMethod]
public void Test(){ /* ... */ }
Then you can add a Test Filter to your build definition (sorry for the typo in the screenshot, should read TestCategory=Critical, or ListX in the example above:
See also:
https://jessehouwing.net/xaml-build-staged-execution-of-unit-tests/
I was wondering if someone would be able to explain what Build runners do and also what i would need to use for just a static HTML / CSS / JS site, or even an already compiled .NET site.
I will be hooking up each project to its equiv SVN and grabbing updates from there, but not 100% sure what the build runners do or which i should use as technically i dont need to build the site.
Sorry it may be too much to answer but i am just struggling to get my head round Team City
Thank you
Build runners are just a process for a specific task, for example the MSBuild runner is set up by putting information into specific fields which it then uses to call MSBuild on the target build agent. You could just as easily use the Command Line runner and build up the MSBuild run command manually.
Build runner is a part of TeamCity that allows integration with a
specific build tool (Ant, MSBuild, Command line, etc.). In a build
configuration, the build runner defines how to run a build and report
its results. Each build runner has two parts:
server-side settings that are configured through the web UI
agent-side part that executes a build on agent
You need to choose your runner depending on the task that you want to do and the technology that you have wrapped your project in. If there is no runner for your specific task then you can use the lowest common denominator which would be the Command Line runner.
The way I approach this would be to see how I can achieve what I want to from my own environment be that calling a rake, MSBuild or batch file. I then see how I can then apply that to a tool. Do not create a process around a tool but choose a tool that fits to your process.
Is it possible to enable Resharper to automatically run all unit tests within a solution automatically when performing a local build using Visual Studio? I believe NUnit has an option "re-run last tests run" but I don't see such an option through Resharper or VS.
Thanks in advance
JP
[Edit] This may or may not matter, but I should point out that I am a one-man-team writing a medium sized application.
Posting this one for anyone that comes across it in the future:
If you're running Visual Studio 2012, someone wrote a plugin that will do exactly what you are looking for:
http://visualstudiogallery.msdn.microsoft.com/5dca9c5c-29cf-4fd7-b3ff-573e5776f0bd
I don't think ReSharper has that option, but you could always turn to something like NCrunch or Continuous Testing one of the other continuous testing frameworks for .NET. Just be aware that they are still fairly new and a little rough around the edges.
I suggest creating a build script (MSBuild, nAnt, rake, whatever) - use this script to build your solution then run tests on it.
Separating your build process from your IDE will allow you to add more things to the build (other types of tests, static checks, style checks etc).
See this blog entry on the subject.
You need dotCover, which is technically part of ReSharper Ultimate. dotCover is JetBrains' testing coverage extension. Once you have this extension there is a "Continuous Testing" feature that you can turn on and configure it to run tests on build, save, etc in addition to code coverage.
I think you're on the right track though, test early and as often as possible while you are writing the code.
I am tasked to improve quality and implement TeamCity for continuous integration. My experience with TeamCity is very limited - I use mostly TFS myself and have some experience with CC.NET.
A lot should happen within a build process... actually the build is already pushed into three different configurations that will run one after the next.
My main problem is that in each of those I actually would need to start multiple runners. For example, the first build step shall consist of:
The generation of new AssemblyInfo.cs files for consistent in assembly numbering
The actual compilation
A partial unit test run (all tests that run fast and check core functionality)
An FxCop run
A StyleCop run
The current version of TeamCity only allows to configure one runner ... which leaves me stuck with a lot of things.
How you would approach this? My current idea is going towards using the MsBuild runner for everything and basically start my own MsBuild based script which then does all the things, pretty much the way that TFS handles it (and the same way i did things back in the cc.net way with my own Nant build script).
On a further problem the question is how to present statistical information, for example from unit tests running in different stages (build configurations). We have some further down that take some time to run and want that to run in a 2nd or 3rd step (the latest for example testing database generation code which, including loading base data, takes about 15+ minutes to run). OTOH we would really like test results to be somehow consolidated.
Anyone any ideas?
Thanks.
TeamCity 6.0 allows multiple build steps for a single build configuration. Isn't it what you're looking for?
You'll need to script this out, at least parts of it. TeamCity provides some nice UI based config for some of your needs, but not all. Here's my suggestion:
Create an msbuild script to handle your first two bullet points, AssemblyInfo generation and compilation. Configure the msbuild runner to run your script, and to run your tests. Collect your assemblies as artifacts.
Create a second build configuration for FxCop. Trigger it from the first build. Give it an 'artifact dependency' on the first build, which is how it gets a hold of your dlls.
For StyleCop, TC doesn't support it out of the box like it does FxCop. Add it to your msbuild script manually, and have it produce an html report (which TeamCity can then display).
You need to take a look at the Dependencies functionality in the TeamCity. This feature allows you to create a sequence of build configurations. In other words, you need to create a build configuration for each step and then link all them as dependencies.
For consolidating test results please take a loot at the Artifact Dependencies. It might help.