My Hudson projects don't seem to properly aggregate downstream test results and I'm wondering if I've missed a configuration step somewhere. I have two projects, Foo and Foo-Tests, both of which are freestyle jobs.
On project Foo I have the following configuration:
Checked "Aggregate downstream test results".
Checked "Automatically aggregate all downstream tests" under the previous option.
Checked "Build other projects" and specified "Foo-Tests" to build.
On project Foo-Tests I have the following configuration:
Checked "Publish JUnit test result report" and specified my JUnit report XML files.
When Foo builds, it builds successfully and correctly triggers a Foo-Tests build. The Foo-Tests build is successful and publishes the JUnit reports correctly. However, when I look for the aggregated test results in Foo, there's a "Latest Test Result (no tests)" link for the build which sends me to a 404.
Here's what I've tried that doesn't solve the problem:
Tell Foo to "Publish JUnit test result report" with no parameters (there are no tests in project Foo, just Foo-Tests). This caused an error since there were no test files for it to process within the project.
Tell Foo-Tests to "Fingerprint all published artifacts" with no parameters (I was wondering if Hudson treated JUnit reports as artifacts behind the scenes). This caused an error since I hadn't explicitly defined any artifacts to publish.
I'm using Hudson 1.266.
Edit:
I should note that I've found two questions on the Hudson Users mailing list that have no answers and would possibly help solve this:
Question 1 (Nabble)
Question 2 (Nabble)
We're using Hudson ver. 1.324 and had similar trouble. Although you mention that you tried enabling fingerprinting and it didn't work for you, it did for us. We followed the instructions found here:
http://shotgunsandpenguins.blogspot.com/2009/07/how-to-aggregate-downstream-test.html
I was able to replicate your problem with Hudson 1.266. This a Hudson error, which was fixed in a later build (prior to 1.287), so either upgrade Hudson or use this two-click workaround: from the Project page, go first to Latest Build and then Aggregated Test Results.
The problem is that the Project page for Foo is only showing the Latest Test Results link, which has a URL like http://localhost:8080/hudson-1.266/job/Foo/lastBuild/testReport/. Since Foo has no tests of its own, this link has no JUnit XML file to reference and returns the error you mentioned. This was fixed somewhere between 1.266 and 1.287 by redirecting from latestBuild/testReport/ back up to latestBuild/ when there are no tests. The alternative for you in 1.266 is to, instead of clicking on Latest Test Results in the Project page, scroll down a bit and click on Latest Build under Permalinks. This will take you to the latestBuild/ URL, and from there you can click on Aggregated Test Result, which has a URL like http://localhost:8080/hudson-1.266/job/Foo/lastBuild/aggregatedTestReport/. On this page, all your test results from downstream projects will be available under the Drill Down section.
Unfortunately, there is still a problem with the Drill Down links, even in 1.287. From Foo, when you drill down to Foo-Tests as outlined above, you will be taken to a malformed URL, which looks like http://localhost:8080/hudson-1.287job/Foo-Tests/. You'll have to manually modify that URL to insert a / between the hudson context and the job path following it so that it looks like http://localhost:8080/hudson-1.287/job/Foo-Tests/. Then you will be able to actually see the downstream test results.
I haven't had a chance to look through the Hudson source to find the error, but there's already an issue open for this. Its issue 1574, and its been open for almost a year.
On a side note, I really love Hudson for CI, but their interface isn't as smooth as it could be. I look forward to their rework of the UI in ExtJS. Maybe thats what they're spending all their time on.
I fixed the missing '/' issue in Hudson 1.288
The key to using the aggregate downstream tests results is to run fingerprinting on both jobs. In this case, that would be 'Foo' and 'Foo-tests'
Hudson matches up the build with the downstream tests by finding files with identical fingerprints. So this means your fingerprints have to match. Kind of like a crime scene.
the two projects, Foo and Foo-Test, have to know they're on the same stream, so it takes fingerprints (and therefore archiving) of a common file.
i had to pick a file that didn't change between the running of Foo and Foo-Test, and still changed between times they are both run. for me that was an unrelated, temporary .jar generated by Foo in the custom/common workspace for my versions of Foo and Foo-Test.
i.e., i had to let both Foo and Foo-Test fingerprint the same file.
after that, at least with hudson 1.330, things work - aggregated links, drilldown, etc.
Related
so I set up a none-XAML-build on TFS 2015 to build and test my Maven project. Which works as expected. Then I added a build step to consolidate test results via **/TEST-*.xml and publish them as a test run to tfs, which also works, but...
on the testrun summary page it shows the correct number of passed tests and it also attached the JUnit XML-files, though on the test results page it does not show anything :( Those .xml-files are JUnit <testsuite> results with multiple <testcase> entries and I expected for TFS to display the result for each individual test including it's duration, but it does only show a blank page.
Anybody has the same problem or even a solution (except publishing by tfs api :p) Cheers!
There are no additional settings to show the test result info. It will automatically show the every detail information about each test.
Since the test run summary show the right info. The test must run successfully. Please check your build log whether there are some valuable info. And also double check your build template configuration. Here is a blog telling you how to use vnext build with Marven and run the tests for your reference:
Building Java code on Linux using VSO Build.vNext
We use the build in coverage application in TeamCity 6 (about to upgrade to 7.1)
If we wish to see the code coverage (or other metrics) of a particular build it is fine as we can navigate to that build, but it would be great if we could pluck out a few interesting metrics from all/some of the current projects/build configurations and display them all together.
For convenience I would expect the new display to be accessible from within TeamCity itself, however if there are solutions that require a separate solution we could look at them.
If you want to compare a set of common metrics (e.g. code coverage) across different projects and over time then SonarQube is probably what you want.
You can integrate it with TeamCity by adding a sonar-project.properties file to each project and calling sonar-runner from a command line build step.
I´ve created a Artifact path in TeamCity like this:
src\MyBuild\bin\Release\* => MyBuild.zip
Now I want to publish this created artifact through a download link:
http://localhost:8080/repository/downloadAll/BUILD_TYPE_ID/BUILD_ID:id?showAll=true
...But where I can get the BUILD_TYPE_ID and the BUILD_ID?
The documentation shows several different ways of getting to artifacts that might be easier.
To help you along though, I believe BUILD_ID is the internal ID of a particular build but unfortunately I don't know how to get it. I do know though that you can substitute ".lastSuccessful" to get the last successful build. BUILD_TYPE_ID is the ID of the project you want artifacts for. You can find it by going to your team city page and clicking on that project. You should see a URL that looks like
http://yourteamcity:8080/viewType.html?buildTypeId=bt1&tab=buildTypeStatusDiv
See the "buildTypeId=bt1"? That bt1 is the BUILD_TYPE_ID (it is likely to be something other than bt1 on your machine).
An example
We use the following URL pattern to get at artifacts.
http://yourteamcity:8080/repository/downloadAll/BUILD_TYPE_ID/BUILD_NUMBER
It still uses the BUILD_TYPE_ID, but it uses BUILD_NUMBER instead of BUILD_ID. BUILD_NUMBER is visible on the TeamCity project page.
So for us an example link to get build number 312.4 would look as follows -
http://yourteamcity:8080/repository/downloadAll/bt1/312.4
Your numbering scheme may be very different.
I am tasked to improve quality and implement TeamCity for continuous integration. My experience with TeamCity is very limited - I use mostly TFS myself and have some experience with CC.NET.
A lot should happen within a build process... actually the build is already pushed into three different configurations that will run one after the next.
My main problem is that in each of those I actually would need to start multiple runners. For example, the first build step shall consist of:
The generation of new AssemblyInfo.cs files for consistent in assembly numbering
The actual compilation
A partial unit test run (all tests that run fast and check core functionality)
An FxCop run
A StyleCop run
The current version of TeamCity only allows to configure one runner ... which leaves me stuck with a lot of things.
How you would approach this? My current idea is going towards using the MsBuild runner for everything and basically start my own MsBuild based script which then does all the things, pretty much the way that TFS handles it (and the same way i did things back in the cc.net way with my own Nant build script).
On a further problem the question is how to present statistical information, for example from unit tests running in different stages (build configurations). We have some further down that take some time to run and want that to run in a 2nd or 3rd step (the latest for example testing database generation code which, including loading base data, takes about 15+ minutes to run). OTOH we would really like test results to be somehow consolidated.
Anyone any ideas?
Thanks.
TeamCity 6.0 allows multiple build steps for a single build configuration. Isn't it what you're looking for?
You'll need to script this out, at least parts of it. TeamCity provides some nice UI based config for some of your needs, but not all. Here's my suggestion:
Create an msbuild script to handle your first two bullet points, AssemblyInfo generation and compilation. Configure the msbuild runner to run your script, and to run your tests. Collect your assemblies as artifacts.
Create a second build configuration for FxCop. Trigger it from the first build. Give it an 'artifact dependency' on the first build, which is how it gets a hold of your dlls.
For StyleCop, TC doesn't support it out of the box like it does FxCop. Add it to your msbuild script manually, and have it produce an html report (which TeamCity can then display).
You need to take a look at the Dependencies functionality in the TeamCity. This feature allows you to create a sequence of build configurations. In other words, you need to create a build configuration for each step and then link all them as dependencies.
For consolidating test results please take a loot at the Artifact Dependencies. It might help.
We have a lot of tests. I can break these up so that they run on seperate agents after an initial compile build happens, but is there a way I can recombine these results? Having 8 build configurations that all need to be green makes it hard to see if you've got one ubergreen build.
Is there a way in TeamCity to recombine / join builds once we've split them out? TW-9990 might help - allowing ANDs in the dependencies.
We found the answer which certainly works from TeamCity 5:
One compile build,
N test only builds that take compile.zip!** and copy to where the compile output would normally be. (via a template)
Consolidated finish:
Finish Build Trigger: Wait for a successful build in: ...
Snapshot Dependencies: Do not run new build if there is a suitable one
Only use successful builds from suitable ones
This all seems to work nicely and the whole shbang is easily copied for branches etc. Am very happy - this has worked well for us for many months now.
No idea how to do that natively. Here's my first thoughts on how I would try and tackle such a thing though:
Saving test results to files
Publishing the test result files as build artifacts
Creating a 'Merge build'
Adding artifact dependency onto the individual test projects
Writing a custom 'build' script using something like (N)Ant. This would parse the individual test results and publish the results as per the TC KB
Good luck!
Thinking outside the box you could have an overall build which doesn't really do anything (or use one of your test build configs as your 'master'), with snapshot dependencies on each of your split test builds. That way if any of them fail, the 'master' will fail because one the dependent build failed.
TW-9990 looks to be concerned with build triggering rather than dependencies.