I am using TeamCity to run around 15 different test suites consisting of hundreds of test cases whenever any new changes come in the existing project. It is configured in a way that it checks for any new changes in Git every half an hour and if it finds any changes, it starts building the project even in the middle of running the test suites. As a result, the database is getting outdated and the test cases are failing for the wrong reasons. How can I configure it so that if TeamCity is in the middle of running the test suites and if any changes come before it can finish running all the available test suites it will not start building the project. It will wait until all the test suites are run and then start building the project with the latest changes. Please help me out.
You may use a Resource Locker with a quota of 1.
Create the Locker at project level (a project both your builds are a part of). For the example I named it ServiceLocker.
Then, in both builds (Tests and Build project), add the Resource Locker you created.
As a result, when coming to the top of the build queue, both the builds will :
check that the Locker is used by less than 1 build (meaning 0 build) ;
if another build is using this Locker, then the build does not start. It will stay in the queue until the Locker is free ;
if/when this Locker is free, the build takes the Locker and starts.
In other words, it prevents the builds having this Locker to run simultaneously.
Related
When I run tests in Xcode, as long as I run all the tests collectively they execute normally. However, if I run only one individual test class or one individual test function within a test class, it doesn't execute any tests at all and prints:
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Upon debugging I've found that it does execute individual tests if I choose a particular other scheme I have, but my other two schemes don't execute the individual tests. I also noticed that if I edit one of the two problematic schemes to use the same Build Configuration as the working scheme, the individual tests do work (each of my schemes uses a different build configuration).
How do I make the problematic build configurations work to let me execute individual tests?
I was able to figure it out!
If I click on my project in the Project navigator, then click on my project in the left sidebar of the page that appears, and then go to the Info tab, I can see at the bottom of the Configurations section it says:
"Use ConfigurationName for command-line builds"
If I click the dropdown and change it to a different build configuration, it makes the scheme that uses that build configuration able to execute individual tests! And, as one might expect, it also makes the scheme previously able to execute individual tests no longer able to.
It would be nice if I could dynamically change the build configuration for command-line builds based on which scheme is active in Xcode, but as of yet I have not figured out how to do so.
So for now, just know that you can manually change this build configuration value whenever you want to run individual tests for a scheme.
I'm looking for a tool of some kind that i can integrate into our CI process to keep track of the manual steps we have.
As an example, we want to run through some manual test scripts on the integration server before pushing the version to the test server. Currently QA gets a notification when the build is done, executes the manual testing and then tells someone to push the version to test if it's okay.
What i would love to find is something that will keep track of when the manual tests have been successfully completed and automatically push the version to test.
It should will be possible to notify/trigger the tool from Visual Studio Online and have it trigger the next step in VSO as well.
I've been googling various different things, but can't seem to find anything close to what i'm looking for. Todo list tools like Asana doesn't seem to have the integration point we need, but maybe i'm just missing something?
You can use the new Release Management tools in conjunction with test cases to get what you want.
In VSTS you can create test cases to reflect the steps of the tests that you want and then create a Test Plan or Suit to reflect the list of tests that you need to run manually.
Then as part of your release process you could create a custom task that waits for a Test Run to be completed against you list. If that test run has all pass then move to the next step, if any tests fail then fail the release.
This should be fairly easy to setup and you need to call the API to check the Test Run. If your Testers use Microsoft Test Manager you can also have the results associated with the Build that you are deploying and get full traceability.
You can try with the Release Management tool in VSTS. It can achieve partial features you want.
I assume you have three steps in your build definition:
Build solution
Publish to integration server
Publish to test server
You can keep the first two steps in build definition. And then create a release definition in Release Management and add the third steps in it. Configure the definition to "Continuous deployment" and link it to your build definition. Assign your QA to be the approver for this release task. Now, a release task will be created as soon as a build task completes. But it will be pending for the approver (Your QA) to approve. Approve the release task after the test is passed, the build will be published to the test server. Otherwise, reject it.
I'm fairly new to TeamCity (but not to CI systems) and I've been trying to figure out how to work this configuration out:
I have latest TeamCity Professional Version 9.1.3 (3 build agents, 20 configs) installed
Here's my TeamCity Project layout:
Project A
-- Build Product X (WIN)
-- Build Product Y (WIN)
-- Build Product Z (Linux)
I have dedicated 3 agents to build the above build configurations accordingly - 2 on Windows and 1 agent on Linux
WIN products are built using a mix of batch, powershell and msbuild scripts
Linux product is built using shell script
Triggering these 3 builds (under Project A), manually, works just fine. However, this is not feasible as we have many feature branches and all of them would have similar build configurations - 3 clicks for each build + setting Build Parameters on each Build configuration is expensive.
So, Here are my questions:
Is there a way to trigger the entire Project to build with a Single click? doing so, should run these builds in parallel
If 1 is possible, then how do I set the same build number (build params) across these 3 build configurations?
Is it possible to set up a VCS trigger that would poll for changes on any of the repos that build these and trigger the entire project (provided 1 is possible)
Please note I tried configuring both snapshot and artifact dependencies to make this work, but creating dependencies only pauses the other build configurations to wait until the dependent project is complete but that's not feasible for us - they need to run in parallel. (our builds take approx 45mins to complete) - Yes, we have a huge product to package.
I'll be grateful for any pointers in the right direction
Thanks
Yes, one-click triggering of parallel runs is achievable: create a 4th (or 0th) build configuration that does nothing (no build steps). Let's call this config "Zero". Your three build configs will each have a "Finished build" trigger on Zero (trigger when Zero finishes) as well as a snapshot dependency on Zero.
The best bit: you only need to define the common/shared parameters in Zero and the other three configs can reuse these. For example, if you define %foo% in Zero, the other three can all use %dep.MyProject_Zero.foo%. This also means you can get at Zero's build number: %dep.MyProject_Zero.build.number%. In each of your three build configs, switch to the "General Settings" and set your "Build number format" to the above.
For VCS triggering, just set Zero to span all the three VCS areas. You're suggesting they are each in a different repo... I have no experience with that but assume that Zero can span all three repos.
Last, if you use feature branches, and your VCS is git, mercurial, or Perforce Streams, make sure you've read about TeamCity feature branches support; it can save you a lot of time!
We've recently found that an acceptance test project fails occasionally on our build server- due to our web drivers Type is not resolved error. I'm trying an experiment to see if its a question of timing of build steps. To this end I've tried to create a separate build step which launches the webdriver executable separately and then proceed onto the unit tests - the issue I have is when I launch the process it blocks the next step after it has successfully started.
eg. Type is not resolved for member 'OpenQA.Selenium.WebDriverException,WebDriver, Version=2.41.0.0, Culture=neutral, PublicKeyToken=null'..
Is there a way I can progress to the next build step after I call an exe?
Thanks
Do you have multiple build agents? Have you considered splitting this into separate build configurations and having triggers on previous build configurations?
You could have configurations:
Compile,
Webdriver, triggered on successful compile completion
Unit Tests, triggered on successful compile completion.
It's not nice, but it should give you the non-blocking behavior you're after. I can't see any other way of making it happen, build configurations are supposed to run as a linear sequence.
I have the following set-up in TeamCity 7.1.3:
Project A
Build & Deploy A
Test A (Quick)
Test A (Slow)
Test A (Very Slow)
Project B
Build & Deploy B
Test B (Quick)
Test B (Slow)
All of the tests for A depend on 'Build & Deploy A' and all of the tests for B depend on 'Build & Deploy B'. The dependencies are all snapshot dependencies with the following settings:
Do not run new build if there is a suitable one
Only use successful builds from suitable ones
Run build on the same agent
All of the tests have VCS triggers. The slow tests only run when particular folders in source control are changed. When a commit is made to project A, the following projects are added to the build queue:
Build & Deploy A
Test A (Quick)
Test A (Slow)
Test A (Very Slow)
Pre-commit builds run with higher priority in the build queue. As such, if while 'Build & Deploy A' is running a pre-commit check is made to project B, the build queue then looks like:
Build & Deploy B
Test B (Quick)
Test B (Slow)
Test A (Quick)
Test A (Slow)
Test A (Very Slow)
The problem with this set-up is that it will mean that the tests for A are run after project B has been deployed. This is an issue as the deployment of project B overwrites some of the files required by project A. The files which are being overwritten by the personal build are deployed files which are not in the checkout folder which is why they are not rolled back. Is there any way to enforce that the build chain for the tests isn't interrupted?
If I remove the 'Do not run new build if there is a suitable one' setting from the dependency, each of the tests will require a full build and deploy. The slower tests can't depend on other tests to make a linear chain as sometimes 'Test A (Very Slow)' is run, but 'Test A (Slow)' isn't.
Unfortunately it isn't currently possible to split the build and deploy to create artefacts.
Is there any way to efficiently set this up in TeamCity?
Here's what I'm gathering:
A and B should be considered mutually exclusive.
A should always have a lower build priority than B, but
Once started, A should not be interrupted.
There may be a way to fight TeamCity on this; I don't know. If you're open to changing your process a bit, one of these might work for you :
Clone your environment such that A and B can be deployed to different targets. This would allow them run completely independently of one another. Use multiple build agents (at least one each for A and B) to allow the faster, higher priority B to execute at the same time as the slower, lower priority A. (This, I think, would be a significant perk.)
Make your process more atomic. Merge the various build/deploy/test pieces into a single configuration, thus ensuring that pending portions of the A build chain can't be displaced by B.
I solved the problem by making both 'build & deploy' build configurations write a text file to the deployment folder containing the build type and build number. All tests compare the values in the text file to the current builds snapshot dependencies. If they values are different, re-trigger the build (forcing dependencies to be rebuilt) and then cancel the current build.
Cancel build by calling:
http://teamcity.jetbrains.com/guestAuth/ajax.html?comment=<CommentMessage>&submit=Stop&buildId=<BuildId>&kill
How to trigger build with dependencies set to rebuild:
Is it possible to force a TeamCity build to rebuild dependencies when triggered over HTTP?
This means that when a test is running out of step, the deployment will be re-triggered and then the tests will run.