I have a few test sets in a single TFS build definition. I'm looking for a way to run all my tests sets in parallel to cut down on the time that the build takes to run.
Below is a screen shot of my build definition and my automated test configurations.
Does any one know how to accomplish this? I'm not seeing a setting in the build definition and I've seen test setting files being used to set this, but I'm not sure where to set the test setting file in the build definition .
The new vNext build system can run different build configurations in parallel (on different build agents) so you could, but I'm not sure how practical it would be.
Your tests sets would have to be in 4 different projects. Then you'd need to have different configurations:
- anycpu / release-with-tests1
- anycpu / release-with-tests2
- anycpu / release-with-tests3
- anycpu / release-with-tests4
With that you can enable (build) only 1 test project per configuration and set your build to run all tests (*test.dll), each configuration would only run one test project because the other ones were not built.
You enable the parallel build and it runs on up to 4 agents, depending on how many you have. Each agent would have to download the sources and build them before running the tests, so it could be interesting or not depending on how long your tests actually are.
There is a filter for the binaries you want to copy to your drop folder, so you can copy them for only 1 configuration and not waste disk space.
You can use the test frameworks that support multiple threaded execution to run the testing in parallel with multiple cores machines or distribute your test to multiple machines. Refer to this blog from MSDN for details: http://blogs.msdn.com/b/visualstudioalm/archive/2015/07/30/speeding-up-test-execution-in-tfs.aspx
Related
We use TeamCity with MsTest to manage and run an automation test suite for front-end testing of a WPF application.
Currently the test suite is running on one dedicated test agent (where TC is installed) and I'm now at a stage where I need to dramatically reduce the overall time it takes to run. I want to do this by adding another test agent to run the tests in parallel.
My test scenarios are large so I have them separated into approx 4 Specflow feature files that run in sequence. All these test scenarios are also categorised by their functional areas.
Firstly:
Is it possible to configure TeamCity to have one test agent managing the sending of tests to be run on each test agent? And then collating all results at the end!
Secondly:
And also keeping the categorised tests together that need to run in sequence?
I decided to use 2 separate project configurations in my TC setup. Each project is pointing at a different test agent using the Agent Requirements step. And I have simply divided up the test categories (that I have setup in my test scenarios anyhow) for each project (half and half).
Pro:
Simple solution and easy to maintain
Con:
Results for each build are separated in TC
I have a TeamCity (8.0.5) setup running with multiple agents and multiple projects with multiple configurations.
We have some build configurations for CI (eg compile and unit tests) and some of the configurations used for automated deployment.
The deployment configurations run scripts on shared services (eg IIS / sql etc) and so cannot be run in parallel with other deployment builds in any project.
Is there any way to limit this in team city?
I've have seen setting to "limit concurrent builds" but this only applies to a single configuration, I've also tried to limit the deployment builds to single common agent but I haven't been able to do this so far.
TeamCity 8 shipped with the Shared Resources plugin built-in and supported. It allows you to create a Shared Resource on any project in the hierarchy and under Build Features of a Build Configuration you can define what Shared Resources need to be available for the build to run, otherwise it'll be queued until the resource is available and a lock can be taken.
For more information, take at look at the TeamCity documentation: http://confluence.jetbrains.com/display/TCD8/Shared+Resources
I've been wondering how you go about doing CI-style testing when you're dealing with physical devices.
I imagine you have a suite of tests, and a pool of devices against which they can be run.
Additionally:
Some tests may require specific device models.
Some tests may require the use of more than one device.
What CI servers have support for this?
I'm still interested in those which have partial support, either natively or through plugins, as I'm interested in how it's done.
Continuous Integration enables a team integrate & test their work frequently. Automated builds are meant to compile, link and run unit-tests. You want your CI to run fast, especially if you run it at every check-in. That is why you would want to restrict CI activities to simple confirmation of the build and unit tests alone. What you're asking seems more-along the lines of quality assurance (QA) testing...and having QA failures mixed into your CI efforts would detract development efforts from progressing.
As such, I'm more under the impression activities associated with CI are not dependent upon the final physical machine said work may eventually be migrated to.
Now...this doesn't mean you CAN'T take the CI-compiled package and run it against some final target-machine....but again...that is really considered a seperate activity.
This seems to be re-enforced in the following article by Martin Fowler.
Notice he doesn't talk about the final targeted devices...only the build machine.
I can suggest Test Manager that is part of the TFS suite of Microsoft. I have not tried it with many different environments apart from windows based though I know there are many connectors. For windows based environments I believe it will satisfy most needs.
I use it for nightly builds to perform smoke tests (Turn it on, see if any smoke comes out) but you have to be careful to keep tests small in order to have them finished in a matter of hours and not days, if you want it to be part of your CI.
Then when you have a good enough quality you can proceed onto regression tests and integration tests if needed.
I wouldn't get too caught up in what a CI system is supposed to do or not do. Instead I would focus on the problem you are trying to solve. It sounds like that problem is to facilitate development on multiple platforms. You can use the concept of Continuous Integration and add to it successfully address the issue. I know, because I've done it in the past.
I implemented a build system for code that needed to compile and test successfully on 4 different platforms (nt, wince, linux-arm, linux-x86). The CI server would:
Used a linux and winnt build server compilation (and cross compilation)
The compiled tests and supporting libs would then be copied to the appropriate devices and an automated test run executed.
After the test suite was completed the log would be copied back, (or it was written to a network mounted fs)
If the test suite was successful we would tag the source, and package the libs and executables.
This same platform was reused for developer verification before commits. Developers would run a partial build and test (only updated source would be recompiled and those tests rerun). The CI would execute a full build (from scratch).
Our build were pretty fast because we had a proper DAG for build dependencies. This allowed for concurrent compilation within a platform build. Each platform build was also concurrent. As a result partial builds took a few seconds, full builds took ~30 minutes. Our build servers were quite beefy (optimized for fast compiles) and the codebase was of moderate size (I don't remember the stats).
I have many build configurations in TeamCity, each servicing a large project. In the past if a build is kicked off the Build Agent could be busy for up to 20min!
In order to improve throughput I installed a second Build Agent on the same machine such that if a build run is kicked off by say Build Agent 1 and it is busy for 20min and someone from another project makes a change then Build Agent 2 can do the build for the other project without needing to wait on the current build run to finish.
All was well until two successive check-ins resulted in both Build Agents running a build for a single build configuration in parallel. Since some resources are shared, IIS directories & databases, I don't want a single build configuration to run on both Build Agents in parallel.
How can I ensure a build isn't triggered if a build is currently running for that build configuration on a different build agent?
One way seems to involve environmental variables and ensuring a 50/50 split by Build Agent in terms of build configuration compatibility, but that seems a little clunky.
You can "Limit the number of simultaneously running builds" for a build configuration (general settings page).
Set it to 1, to fulfil your task.
I am tasked to improve quality and implement TeamCity for continuous integration. My experience with TeamCity is very limited - I use mostly TFS myself and have some experience with CC.NET.
A lot should happen within a build process... actually the build is already pushed into three different configurations that will run one after the next.
My main problem is that in each of those I actually would need to start multiple runners. For example, the first build step shall consist of:
The generation of new AssemblyInfo.cs files for consistent in assembly numbering
The actual compilation
A partial unit test run (all tests that run fast and check core functionality)
An FxCop run
A StyleCop run
The current version of TeamCity only allows to configure one runner ... which leaves me stuck with a lot of things.
How you would approach this? My current idea is going towards using the MsBuild runner for everything and basically start my own MsBuild based script which then does all the things, pretty much the way that TFS handles it (and the same way i did things back in the cc.net way with my own Nant build script).
On a further problem the question is how to present statistical information, for example from unit tests running in different stages (build configurations). We have some further down that take some time to run and want that to run in a 2nd or 3rd step (the latest for example testing database generation code which, including loading base data, takes about 15+ minutes to run). OTOH we would really like test results to be somehow consolidated.
Anyone any ideas?
Thanks.
TeamCity 6.0 allows multiple build steps for a single build configuration. Isn't it what you're looking for?
You'll need to script this out, at least parts of it. TeamCity provides some nice UI based config for some of your needs, but not all. Here's my suggestion:
Create an msbuild script to handle your first two bullet points, AssemblyInfo generation and compilation. Configure the msbuild runner to run your script, and to run your tests. Collect your assemblies as artifacts.
Create a second build configuration for FxCop. Trigger it from the first build. Give it an 'artifact dependency' on the first build, which is how it gets a hold of your dlls.
For StyleCop, TC doesn't support it out of the box like it does FxCop. Add it to your msbuild script manually, and have it produce an html report (which TeamCity can then display).
You need to take a look at the Dependencies functionality in the TeamCity. This feature allows you to create a sequence of build configurations. In other words, you need to create a build configuration for each step and then link all them as dependencies.
For consolidating test results please take a loot at the Artifact Dependencies. It might help.