so I set up a none-XAML-build on TFS 2015 to build and test my Maven project. Which works as expected. Then I added a build step to consolidate test results via **/TEST-*.xml and publish them as a test run to tfs, which also works, but...
on the testrun summary page it shows the correct number of passed tests and it also attached the JUnit XML-files, though on the test results page it does not show anything :( Those .xml-files are JUnit <testsuite> results with multiple <testcase> entries and I expected for TFS to display the result for each individual test including it's duration, but it does only show a blank page.
Anybody has the same problem or even a solution (except publishing by tfs api :p) Cheers!
There are no additional settings to show the test result info. It will automatically show the every detail information about each test.
Since the test run summary show the right info. The test must run successfully. Please check your build log whether there are some valuable info. And also double check your build template configuration. Here is a blog telling you how to use vnext build with Marven and run the tests for your reference:
Building Java code on Linux using VSO Build.vNext
Related
We are running selenium tests using TestNG in TeamCity.
Is there a way to know the progress of the current run beyond the progress bar and build log?
Information of interest is:
Which tests were run and passed
Which tests were run and failed
...
Thx
As I know testng plugin for teamcity does not have such option. You can try real-time reporter e.g. extentreports. It could be easily connected via testng listener and pass results to report server.
Here is a link to another question which I answered recently which addresses something similar to what you're looking for with ExtentReports.
ITestListener - ExtentReport
TeamCity can report the tests in a build and you should really get that in order to use TeamCity at full power. It can list the test failures in a build with details, display test history, you can assign investigations of a test to your team members or mute the test failures...
If you use TeamCity's Ant or Maven build steps, TestNG tasks should be recognized automatically and build status should turn into something like "Tests failed: 1 (1 new), passed: 301".
Otherwise try to generate XML report in one of supported format and use XML Report build feature in TeamCity.
Yet another alternative is to report the tests in a completely custom way through TeamCity service messages.
All these approaches should update TeamCity build status and details as soon as the test is reported by the build tool. For service messages this is as soon as a test finish is reported, for Ant this is after each test run, for Maven this is after finishing tests of a module and for the XML test report build feature, this is while the XML is being saved on disk, even in incomplete form.
If you are rather looking for a specific test progress, that can be seen in the build log or you can notify TeamCity about stages in the build from within the test/script.
I'm new to Coded UI testing and TFS. I've written a coded UI test for my web application and looking for different options to automatically run it. Apparently the first option is to run it from visual studio. I have also found an article explaining how to associate a test method with a test case however I don't know whether I can run my test method (s) from TFS or not? How about running them with my build? I know the unit tests can be run with the build however I'm not sure about coded UI tests.
Yes, it is possible for you to run Coded UI test in the TFS build process.
Firstly, you need to configure the Team Build Service Host run as an interactive process. See this link.
Secondly, add the Coded UI test project into TFS Version Control.
Thirdly, create a build definition to build the Coded UI test project, and also specify tests to run.
Please have a check of this blog for the detailed steps: http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
TFS2010 database is growing fast (50GB per month) because of our TFS builds.
We noticed that tbl_attachmentcontent table contains some binary (dll) files and some files related to instrumentation and TQAgents... All content seems to be unit test related.
I tried to find something how to stop growing of TFS DB because of unit tests.
I found this suggestion to disable publishing of unit test results to the TFS.
How can I disable publishing of binary files to the TFS DB?
Is it possible to disable publishing of specific content?
What will be missing in the TFS when I disable publising unit tets results to the TFS?
At about 400+ unit tests are running every build including code coverage results and huge build log (500+ C# compiled projects with a lot of dependencies).
I'd like to have build retention policy set to "Kepp all", because of build statistics.
I did some testing and here are the results:
1) You can disable unit test publishing. Your unit tests results file (trx) is not uploaded to the TFS DB. This is not problem for me, because I'm using Trx2Html tool to convert results to the HTML format. If test failed, converted HTML formatted report is sent by E-Mail to notify developer team.
2) You can enable compression on the dbo.tbl_AttachmentContent table. In our case compressed content is 5% from original.
3) Use this tool to clean up dbo.tbl_AttachmentContent table
All I know that test result is not a binary file, but it's XML file with (.trx) extension, for your question about what happen if you stop publish test result, the build server just attache the test result with the build so you can view the result by click on the test result on the build result, so you just download the file and start viewing the result by the visual studio, otherwise you will not be able to see it.
This is what I think.
Update
You can see this link:
http://blogs.msdn.com/b/anutthara/archive/2011/10/30/gsjgd.aspx
My Hudson projects don't seem to properly aggregate downstream test results and I'm wondering if I've missed a configuration step somewhere. I have two projects, Foo and Foo-Tests, both of which are freestyle jobs.
On project Foo I have the following configuration:
Checked "Aggregate downstream test results".
Checked "Automatically aggregate all downstream tests" under the previous option.
Checked "Build other projects" and specified "Foo-Tests" to build.
On project Foo-Tests I have the following configuration:
Checked "Publish JUnit test result report" and specified my JUnit report XML files.
When Foo builds, it builds successfully and correctly triggers a Foo-Tests build. The Foo-Tests build is successful and publishes the JUnit reports correctly. However, when I look for the aggregated test results in Foo, there's a "Latest Test Result (no tests)" link for the build which sends me to a 404.
Here's what I've tried that doesn't solve the problem:
Tell Foo to "Publish JUnit test result report" with no parameters (there are no tests in project Foo, just Foo-Tests). This caused an error since there were no test files for it to process within the project.
Tell Foo-Tests to "Fingerprint all published artifacts" with no parameters (I was wondering if Hudson treated JUnit reports as artifacts behind the scenes). This caused an error since I hadn't explicitly defined any artifacts to publish.
I'm using Hudson 1.266.
Edit:
I should note that I've found two questions on the Hudson Users mailing list that have no answers and would possibly help solve this:
Question 1 (Nabble)
Question 2 (Nabble)
We're using Hudson ver. 1.324 and had similar trouble. Although you mention that you tried enabling fingerprinting and it didn't work for you, it did for us. We followed the instructions found here:
http://shotgunsandpenguins.blogspot.com/2009/07/how-to-aggregate-downstream-test.html
I was able to replicate your problem with Hudson 1.266. This a Hudson error, which was fixed in a later build (prior to 1.287), so either upgrade Hudson or use this two-click workaround: from the Project page, go first to Latest Build and then Aggregated Test Results.
The problem is that the Project page for Foo is only showing the Latest Test Results link, which has a URL like http://localhost:8080/hudson-1.266/job/Foo/lastBuild/testReport/. Since Foo has no tests of its own, this link has no JUnit XML file to reference and returns the error you mentioned. This was fixed somewhere between 1.266 and 1.287 by redirecting from latestBuild/testReport/ back up to latestBuild/ when there are no tests. The alternative for you in 1.266 is to, instead of clicking on Latest Test Results in the Project page, scroll down a bit and click on Latest Build under Permalinks. This will take you to the latestBuild/ URL, and from there you can click on Aggregated Test Result, which has a URL like http://localhost:8080/hudson-1.266/job/Foo/lastBuild/aggregatedTestReport/. On this page, all your test results from downstream projects will be available under the Drill Down section.
Unfortunately, there is still a problem with the Drill Down links, even in 1.287. From Foo, when you drill down to Foo-Tests as outlined above, you will be taken to a malformed URL, which looks like http://localhost:8080/hudson-1.287job/Foo-Tests/. You'll have to manually modify that URL to insert a / between the hudson context and the job path following it so that it looks like http://localhost:8080/hudson-1.287/job/Foo-Tests/. Then you will be able to actually see the downstream test results.
I haven't had a chance to look through the Hudson source to find the error, but there's already an issue open for this. Its issue 1574, and its been open for almost a year.
On a side note, I really love Hudson for CI, but their interface isn't as smooth as it could be. I look forward to their rework of the UI in ExtJS. Maybe thats what they're spending all their time on.
I fixed the missing '/' issue in Hudson 1.288
The key to using the aggregate downstream tests results is to run fingerprinting on both jobs. In this case, that would be 'Foo' and 'Foo-tests'
Hudson matches up the build with the downstream tests by finding files with identical fingerprints. So this means your fingerprints have to match. Kind of like a crime scene.
the two projects, Foo and Foo-Test, have to know they're on the same stream, so it takes fingerprints (and therefore archiving) of a common file.
i had to pick a file that didn't change between the running of Foo and Foo-Test, and still changed between times they are both run. for me that was an unrelated, temporary .jar generated by Foo in the custom/common workspace for my versions of Foo and Foo-Test.
i.e., i had to let both Foo and Foo-Test fingerprint the same file.
after that, at least with hudson 1.330, things work - aggregated links, drilldown, etc.
I am working with an n-tiered architecture in Visual Studio 2008 (Developer Edition), and I have run into an issue.
We are running unit tests on every method of our services layer, and I am attempting to see the code coverage results, to ensure I'm hitting all the main paths through my methods.
When I attempt to view the results, I am informed that the test run did not have code coverage enabled. I tried editing the test run configuration through test -> edit test run configurations, only to discover that there were none found.
I have yet to discover how to create a test run configuration so that I can enable code coverage results.
How do I create the configuration?
Right-click the solution, choose Add, New item, choose Test run configuration in the resulting dialog. The only tricky part is that you don't get this choice unless you right-click the solution.