We've recently found that an acceptance test project fails occasionally on our build server- due to our web drivers Type is not resolved error. I'm trying an experiment to see if its a question of timing of build steps. To this end I've tried to create a separate build step which launches the webdriver executable separately and then proceed onto the unit tests - the issue I have is when I launch the process it blocks the next step after it has successfully started.
eg. Type is not resolved for member 'OpenQA.Selenium.WebDriverException,WebDriver, Version=2.41.0.0, Culture=neutral, PublicKeyToken=null'..
Is there a way I can progress to the next build step after I call an exe?
Thanks
Do you have multiple build agents? Have you considered splitting this into separate build configurations and having triggers on previous build configurations?
You could have configurations:
Compile,
Webdriver, triggered on successful compile completion
Unit Tests, triggered on successful compile completion.
It's not nice, but it should give you the non-blocking behavior you're after. I can't see any other way of making it happen, build configurations are supposed to run as a linear sequence.
Related
Is it possible to stop a TeamCity build (the entire build, i.e. it won't execute subsequent steps) when a unit test fails? Ideally I'd also like it to terminate the currently executing step which in my case would be the Nunit Test Runner. In my circumstance there is no point in continuing the build if a single unit test fails. I've looked at Failure Conditions but I don't think they are applicable as the build continues to run.
Feature requested: http://youtrack.jetbrains.com/issue/YTF-3275
As you noted, TeamCity can run no further build steps on test failures in NUnit tests with "Only if build status is successful" step execution condition. However, that does not make the tests run stop until the step finish.
A related feature request is TW-23766.
The only workaround I can consider currently is not to use NUnit test runner and implement the logic inside the build script. For example, with nunit-console.exe like Manuel noted. If you choose to follow this route, consider using TeamCity Addin for NUNit.
You can do it using nunit-console.exe.
accordingly to the official documentation (http://nunit.org/index.php?p=consoleCommandLine&r=2.6.2) it provides a /stoponerror switch that does exactly what you need.
it can also generate an XML output that can be parsed by teamcity (there is a build feature for that) in order to populate the "test" tab.
on the build step after the unit tests, change the setting to Only if Build status is successful instead of if all previous steps finished successfully
see screenshot
also make sure that under failure conditions menu item the at least one test failed option is also ticked.
We are currently running a Jenkins (Hudson) CI server to build and package our .net web projects and database projects. Everything is working great but I want to start writing unit tests and then only passing the build if the unit tests pass. We are using the built in msbuild task to build the web project. With the following arguments ...
MsBuild Version .NET 4.0
MsBuild Build File ./WebProjectFolder/WebProject.csproj
Command Line Arguments ./target:Rebuild /p:Configuration=Release;DeployOnBuild=True;PackageLocation=".\obj\Release\WebProject.zip";PackageAsSingleFile=True
We need to run automated tests over our code that run automatically when we build on our machines (post build event possibly) but also run when Jenkins does a build for that project.
If you run it like this it doesn't build the unit tests project because the web project doesn't reference the test project. The test project would reference the web project but I'm pretty sure that would be butchering our automated builds as they exist primarily to build and package our deployments. Running these tests should be a step in that automated build and package process.
Options ...
Create two Jenkins jobs. one to run the tests ... if the tests pass another build is triggered which builds and packages the web project. Put the post build event on the test project.
Build the solution instead of the project (make sure the solution contains the required tests) and put post build events on any test projects that would run the nunit console to run the tests. Then use the command line to copy all the required files from each of the bin and content directories into a package.
Just build the test project in jenkins instead of the web project in jenkins. The test project would reference the web project (depending on what you're testing) and build it.
Problems ...
There's two jobs and not one. Two things to debug not one. One to see if the tests passed and one to build and compile the web project. The tests could pass but the build could fail if its something that isn't used by what you're testing ...
This requires us to know exactly what goes into the build. Right now msbuild does it all for us. If you have multiple teams working on a project everytime an extra folder is created you have to worry about the possibly brittle command line statements.
This seems like a corruption of our main purpose here. The tests should be a step in this process not the overriding most important thing in this process. I'm also not 100% sure that a triggered build is the same as a normal build does it do all the same things as a normal build. Move all the correct files in the same way move them all into the same directories etc.
Initial problem.
We want to run our tests whenever our main project is built. But adding a post build event to the web project that runs against the test project doesn't work because the web project doesn't reference the test project and won't trigger a build of this project. I could go on ... but that's enough ...
We've spent about a week trying to make this work nicely but haven't succeeded. Feel free to edit this if you feel you can get a better response ...
In Jenkins/Hudson, it's quite ok to have many jobs. some for doing compilation triggered version control changes, some for running (unit) tests triggered by successful builds, some for doing more tests (integeration) trigered by successful earlier tests, some for deploying, triggered by successfully passing all tests.
Look at plugins like join, build pipeline, parametrized trigger and more to help out with this.
This will also allow things to happen in parallel, by using multiple nodes. Trying to cram everything in one job is not the way to go.
My build project have 3 steps:
-file preparation
-deployment
-functional tests
I have set all the dependencies between them, but I really wish also to hook the deployment step to the functional test, so that if the functional test are running and new code is committed the deployment wait till the functional tests finish.
I know there are the build triggering, the dependencies and the artifact dependencies but each of them doesn't seem good for my case.
The first run a deployment every time the functional test step finish and obviously is not what I want.
The second force the deployment to use the same code as in the functional test instead it should use new freshly committed code and for the third is more or less the same situation.
Where I'm thinking wrong? I'm missing something or there's a shortcut to use to make this working?
You can create 2 build configurations:
"Functional tests" configuration with VCS trigger.
"Deployment" configuration with snapshot dependency on first configuration and also VCS trigger (or other trigger, e.g. Schedule trigger with 'Trigger build only if there are pending changes' option selected).
Files will be deployed only if functional tests not failed on same code base.
It is what you need?
I am using TeamCity as my CI server.
I have defined three Build Steps. One builds my project from the .sln file, one runs tests using MSTest the final one packages up some of the dlls and exes the project produces into a deployment package.
Currently even if some tests fail the deployment package is still created. How can I stop Team City from processing further Build Steps if one has a problem?
Seems I am not the only one with this problem.
http://youtrack.jetbrains.net/issue/TW-12194
I am not sure how to adjust my scripts to know if things upstream have failed.
In the build conficguration genneral settings page, there is "Fail build if" section.
There is a checkbox in it which says "at least one test fails".
Is it checked?
I am tasked to improve quality and implement TeamCity for continuous integration. My experience with TeamCity is very limited - I use mostly TFS myself and have some experience with CC.NET.
A lot should happen within a build process... actually the build is already pushed into three different configurations that will run one after the next.
My main problem is that in each of those I actually would need to start multiple runners. For example, the first build step shall consist of:
The generation of new AssemblyInfo.cs files for consistent in assembly numbering
The actual compilation
A partial unit test run (all tests that run fast and check core functionality)
An FxCop run
A StyleCop run
The current version of TeamCity only allows to configure one runner ... which leaves me stuck with a lot of things.
How you would approach this? My current idea is going towards using the MsBuild runner for everything and basically start my own MsBuild based script which then does all the things, pretty much the way that TFS handles it (and the same way i did things back in the cc.net way with my own Nant build script).
On a further problem the question is how to present statistical information, for example from unit tests running in different stages (build configurations). We have some further down that take some time to run and want that to run in a 2nd or 3rd step (the latest for example testing database generation code which, including loading base data, takes about 15+ minutes to run). OTOH we would really like test results to be somehow consolidated.
Anyone any ideas?
Thanks.
TeamCity 6.0 allows multiple build steps for a single build configuration. Isn't it what you're looking for?
You'll need to script this out, at least parts of it. TeamCity provides some nice UI based config for some of your needs, but not all. Here's my suggestion:
Create an msbuild script to handle your first two bullet points, AssemblyInfo generation and compilation. Configure the msbuild runner to run your script, and to run your tests. Collect your assemblies as artifacts.
Create a second build configuration for FxCop. Trigger it from the first build. Give it an 'artifact dependency' on the first build, which is how it gets a hold of your dlls.
For StyleCop, TC doesn't support it out of the box like it does FxCop. Add it to your msbuild script manually, and have it produce an html report (which TeamCity can then display).
You need to take a look at the Dependencies functionality in the TeamCity. This feature allows you to create a sequence of build configurations. In other words, you need to create a build configuration for each step and then link all them as dependencies.
For consolidating test results please take a loot at the Artifact Dependencies. It might help.