Is it possible to stop a TeamCity build (the entire build, i.e. it won't execute subsequent steps) when a unit test fails? Ideally I'd also like it to terminate the currently executing step which in my case would be the Nunit Test Runner. In my circumstance there is no point in continuing the build if a single unit test fails. I've looked at Failure Conditions but I don't think they are applicable as the build continues to run.
Feature requested: http://youtrack.jetbrains.com/issue/YTF-3275
As you noted, TeamCity can run no further build steps on test failures in NUnit tests with "Only if build status is successful" step execution condition. However, that does not make the tests run stop until the step finish.
A related feature request is TW-23766.
The only workaround I can consider currently is not to use NUnit test runner and implement the logic inside the build script. For example, with nunit-console.exe like Manuel noted. If you choose to follow this route, consider using TeamCity Addin for NUNit.
You can do it using nunit-console.exe.
accordingly to the official documentation (http://nunit.org/index.php?p=consoleCommandLine&r=2.6.2) it provides a /stoponerror switch that does exactly what you need.
it can also generate an XML output that can be parsed by teamcity (there is a build feature for that) in order to populate the "test" tab.
on the build step after the unit tests, change the setting to Only if Build status is successful instead of if all previous steps finished successfully
see screenshot
also make sure that under failure conditions menu item the at least one test failed option is also ticked.
Related
We are running selenium tests using TestNG in TeamCity.
Is there a way to know the progress of the current run beyond the progress bar and build log?
Information of interest is:
Which tests were run and passed
Which tests were run and failed
...
Thx
As I know testng plugin for teamcity does not have such option. You can try real-time reporter e.g. extentreports. It could be easily connected via testng listener and pass results to report server.
Here is a link to another question which I answered recently which addresses something similar to what you're looking for with ExtentReports.
ITestListener - ExtentReport
TeamCity can report the tests in a build and you should really get that in order to use TeamCity at full power. It can list the test failures in a build with details, display test history, you can assign investigations of a test to your team members or mute the test failures...
If you use TeamCity's Ant or Maven build steps, TestNG tasks should be recognized automatically and build status should turn into something like "Tests failed: 1 (1 new), passed: 301".
Otherwise try to generate XML report in one of supported format and use XML Report build feature in TeamCity.
Yet another alternative is to report the tests in a completely custom way through TeamCity service messages.
All these approaches should update TeamCity build status and details as soon as the test is reported by the build tool. For service messages this is as soon as a test finish is reported, for Ant this is after each test run, for Maven this is after finishing tests of a module and for the XML test report build feature, this is while the XML is being saved on disk, even in incomplete form.
If you are rather looking for a specific test progress, that can be seen in the build log or you can notify TeamCity about stages in the build from within the test/script.
I'm looking for a tool of some kind that i can integrate into our CI process to keep track of the manual steps we have.
As an example, we want to run through some manual test scripts on the integration server before pushing the version to the test server. Currently QA gets a notification when the build is done, executes the manual testing and then tells someone to push the version to test if it's okay.
What i would love to find is something that will keep track of when the manual tests have been successfully completed and automatically push the version to test.
It should will be possible to notify/trigger the tool from Visual Studio Online and have it trigger the next step in VSO as well.
I've been googling various different things, but can't seem to find anything close to what i'm looking for. Todo list tools like Asana doesn't seem to have the integration point we need, but maybe i'm just missing something?
You can use the new Release Management tools in conjunction with test cases to get what you want.
In VSTS you can create test cases to reflect the steps of the tests that you want and then create a Test Plan or Suit to reflect the list of tests that you need to run manually.
Then as part of your release process you could create a custom task that waits for a Test Run to be completed against you list. If that test run has all pass then move to the next step, if any tests fail then fail the release.
This should be fairly easy to setup and you need to call the API to check the Test Run. If your Testers use Microsoft Test Manager you can also have the results associated with the Build that you are deploying and get full traceability.
You can try with the Release Management tool in VSTS. It can achieve partial features you want.
I assume you have three steps in your build definition:
Build solution
Publish to integration server
Publish to test server
You can keep the first two steps in build definition. And then create a release definition in Release Management and add the third steps in it. Configure the definition to "Continuous deployment" and link it to your build definition. Assign your QA to be the approver for this release task. Now, a release task will be created as soon as a build task completes. But it will be pending for the approver (Your QA) to approve. Approve the release task after the test is passed, the build will be published to the test server. Otherwise, reject it.
We've recently found that an acceptance test project fails occasionally on our build server- due to our web drivers Type is not resolved error. I'm trying an experiment to see if its a question of timing of build steps. To this end I've tried to create a separate build step which launches the webdriver executable separately and then proceed onto the unit tests - the issue I have is when I launch the process it blocks the next step after it has successfully started.
eg. Type is not resolved for member 'OpenQA.Selenium.WebDriverException,WebDriver, Version=2.41.0.0, Culture=neutral, PublicKeyToken=null'..
Is there a way I can progress to the next build step after I call an exe?
Thanks
Do you have multiple build agents? Have you considered splitting this into separate build configurations and having triggers on previous build configurations?
You could have configurations:
Compile,
Webdriver, triggered on successful compile completion
Unit Tests, triggered on successful compile completion.
It's not nice, but it should give you the non-blocking behavior you're after. I can't see any other way of making it happen, build configurations are supposed to run as a linear sequence.
I have been using the Build-Deploy-Test build workflow for TFS 2010 (see here http://msdn.microsoft.com/en-us/vstudio/gg131922.aspx) and have come up against a very annoying limitation.
Here is what I have done:
Setup a build to be queued
Configured it to restore to a test-ready snapshot
Added some deployment steps; in this case starting mongo db
I then came to the step where you configure which tests to run and hit an issue. Firstly you need configure a test plan with some test suites; a test suite consisting of test cases. The problem is that each test case can only be associated with a single unit test.
With having to create a test case per unit test, it means that every time a unit test is added a new test case must be created. Is there any way you can associate many unit tests with a test suite that can be ran in the Build-Deploy-Test build workflow.
One way around this problem is to update your test suite using the tcm.exe testcase /import /syncsuite: command. This command has to be run before test execution begins and a logical place to add it would be into the build template.
I suggest you to use Test List Editor in Test Tab on Visual Studio, create your differents lists of tests, organize your tests based on functionnal after that you can execute on your build list of tests by entering name of list.
link : http://msdn.microsoft.com/en-us/library/ms182463(v=vs.100).aspx
For configuring Build Definition, i suggets you to edit & select Automated Test Tab and insert list.
link : http://support.smartbear.com/articles/testcomplete/testcomplete-and-team-build/
I have a TFS build in VS2010. Following the build unit tests are executed. In the Build summary it tells me that "1 test run(s) completed - 100% average pass rate" but below this it states "No tests were impacted". I guess Impacted Tests relate to functionality providing the ability to only run tests that were impacted by code checked in? Is this correct and since I'm happy to run all tests following each build, how can I turn off the "No tests were impacted" message?
Thanks,
Rob
You're right about test impact analysis - it looks at the changesets since the last build and which tests intersect with that code. To disable test impact analysis for a build:
Right-click your build definition in Team Explorer and choose Edit...
Go to the Process category and expand the Advanced section
Set "Analyze Test Impact" to False
You can also set this value on a per-build basis - queue a manual build and the setting is available via the Parameters tab.