Does anyone know of a link describing how to create a TFS 2010 query or report showing number of unit tests run / passed / failed for a particular automated build over a period of time?
Related
Is there a possibility to add a custom file (e.g. custom Selenium report for all tests) into the body of Run summary itself (Runs -> Run Summary -> Attachments).
Tests are executed using the vNext "Run Functional Tests" task, the unit framework used in tests is a TestTools.UnitTesting.
I know that there is a TestContext.AddTestResult but it attaches file to a specific test, while I want to use some sort of a summary file for the whole run.
Test Run Summary in question
If you want to display your custom result in the test run summary or build result summary, seems you will have to write your own extensions. Help link: Overview of extensions for Visual Studio Team Services
Base for this were a lot of examples provided by MS on github:
Visual Studio Team Services Tasks
Visual Studio Team Services Sample Extensions
To get a first feeling what places on your TFS Web Portal can be extended/customized you can download and install this extension Contributions Guide from the Visual Studio Marketplace.
Similar issue as this question (which does not have an answer), except we are using TFS 2013 Update 4. Our tests are written using NUnit and are configured to run through the build definition, which uses the default (not upgrade) template.
We just installed a full version of VS 2013 Premium on the build server (because prior tests were not being run at all). With that installed, the tests are run (if I did down into the log I can see them and the results, and if we force a failure the build appropriately fails) but the build summary says that 0/0 tests were run.
I followed the steps in the selected answer to this question, and it now shows the code coverage results, but still no dice on the actual tests or their results.
I also found this blog article that appears to be the same issue from TFS 2010. I have verified that all users have the View Test Runs permission set to allow, and I tried using a TFS administrator account just to be sure. Still no dice.
How do I make TFS show me what tests were run in the summary, and also in the Test Results window that the build summary links to?
Each individual test needs to be associated with a test case work item in Team Foundation Server. See this link for information on how to associate your tests. In my work, we're using TFS 2015, but I know this works in TFS 2013 as I have done this work in the past. Once the test case work item is associated with your automation (using Visual Studio to link the test automation to the test case) then there must be a build definition for the product under test that runs your test automation as part of the build. When a build is queued, your tests are run as part of the new build. Once the build is done, you can then view the detailed test results.
Plus the other StackOverflow post here describes how to view the detailed results.
We use TFS 2012
Through a build definition we execute automated tests from MTM test plans.
How can we ensure TFS uses vstest.console.exe to run each test?
I've read online from other posts that TFS 2012 let's you choose either MSTest or VSTest as the test runner.
I've also read that if you use MTM you effectively are locked into using a test settings file(.testsettings) which means you are locked into using MSTest. This seems to go against TFS 2012 giving you a choice.
Test Manager currently is only supporting MsTest. This is a limitation in Test Manager. It's the same issue that prevents you from associating a xUnit or NUnit test automation on a test case.
So while TFS 2012 gives you the choice, each choice comes with its own set of limitations.
To be clear:
New test runner: Work in Continuous Integration, you specify which tests to run by selecting the new agile test runner (in TFS 2013 this is a fixed choice in the default templates, template customization is required to run mstest). It can run tests during build and you can directly invoke it from the commandline if you want to run tests after deployment. You can use a .runsettings file to specify certain options it should pick up. This option is required to execute 3rd party test frameworks like NUnit, XUnit.NET, Chutzpah etc.
The MTM test runner: Execute automated tests that are associated to a Test Case work item or execute tests using the Test Agent to run tests from a remote system using the Team Test infrastructure. You can use a .testsettings file to specify certain options it should pick up. These tests will be executed using MsTest and cannot contain 3rd party test frameworks.
In your Continuous Integration build you can configure multiple Test Runs, each can be configured to a specific test framework. So you can have both options in your build.
More on the differences and how you can use the different settings files.
Over time all test options will be moved to the new test runner. In TFS 2013 the option to configure a MsTest based test run in your Build Definition has been removed by default. You will need to customize the build process to select a different test runner.
I'm exploring using Microsoft Test Manager as an option for for my company to switch over to. We already use Visual Studio/TFS for our development, source control, builds, scrum process, etc.
I've been able to use Test Manager to run manual tests, but I can't find anyway to publish these results to a build or any easy way to associate these test runs with work items.
The build summary in Visual Studio does show that a test run as having completed after I run a test in MTM against that build, but when I try to view the results I get "Downloading the selected test run failed. Test run XX does not have a run summary attachment. The .trx file is either not published or upgraded."
Is it possible to publish results to TFS after running a manual test in MTM?
I run my 700 unit tests in visual studio and they take less than 2 mins.
I also have Team Build set up to do a build and run my unit tests, which takes over 12 mins to run the same tests!
During my exploring of logs etc.. I've found this confusing info:
-If I open the MSTest .trx file (Visual Studio test results file) that was actually generated by Team Build on the server, it says the duration is about 2 mins.
-But when I view the Log from the build explorer, the log entry for unit tests (Run MSTest for Test Assemblies) shows the duration to be about 12 mins.
Anybody got any idea where the extra 10 mins have gone and perhaps more importantly is there something I can turn off to get them back?
TFS 2010, VS 2010.
Thanks,
Mike G
It's possible that this includes the publishing time (although it's very unlikely that publishing takes 10 mins). It could if you have a remote TFS server and you upload lots of big assemblies when publishing the test results. If that is indeed the case, I can look further to see if there is an option in the 2010 release that turns off publishing assemblies.