I am setting up Test agents as part of build definition and now I want to run automation test case using TCM.exe, How can I pass the Test Environment name as an argument?
In TFS 2013 I did something like this-
start-Process -FilePath "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\tcm.exe" -ArgumentList " run /create /title:Nightly Run /planid:1334 /suiteid:123 /configid:123456 /settingsname:Test_xyz_TestSettings /testenvironment:xyzFuntional /collection:http://xyz:8080/tfs/ppp /teamproject:aa /include"
because in 2013 we were setting test controller and test agent as part of lab management and we knew the name of the test environment.
My requirement is to publish automation result in MTM ,any other way is really appreciated .Please note i can not use Test Hub for Test result publishing.
Thanks
Based on your screenshot, you are using the Run functional Test but not TCM.exe.
VStest, MStest and TCM.exe are different tools (Refer to this thread for more info.).
The tests in your scenario supposed to be used to associate test runs with builds and actually using vstest. MTM is displaying test results that are associated with test plans/suites which are somewhat independent of builds.
You can try to use tcm.exe to get what you're after. It's a command line interface to running tests, similar to mstest, but it runs the tests through the infrastructure you set up in MTM. Here's the documentation page: http://msdn.microsoft.com/en-us/library/dd465192.aspx
However, you can also try to follow the steps mentioned in below link to publish the VStest or MStest test results to MTM :
Uploading automated test results to MTM made "easy"
Use TCM to upload the resulting TRX file
Note There is problem with VSTS (formerly VSO) that causes these automated tests to stay in a state of “In Progress”*
Sample TCM command line:
TCM run /publish /suiteid:ID# /configid:CONFIG# /resultowner:"USER" /resultsfile:"C:\DLL\testResults.trx" /collection:"TFS_COLLECTION_URL" /teamproject:"PROJECT_NAME"
UPDATE:
The infrastructure (e.g. Test Environment you mentioned here ) is set in MTM, If you run the test with tcm.exe, generally it will pick up the parameters automatically. But if run test with VSTest or MStest the parameters may need to be defined with Set up environments to run continuous test tasks with your build tasks ...
Whatever, based on your description I think below articles should helps:
Continuous Delivery with VSO: Executing Automated Web Tests with
Microsoft Test Manager
Continuous Delivery with TFS: Running Automated Web Tests with
MTM
Related
Is there a way to automatically create Azure Dev Ops test cases from a test assembly or from a solution with test scripts? With TFS 2015, I know that you can automatically create cases using the tcm.exe command line tool. But we are migrating to Azure Dev Ops, and we need to migrate our test plans and test cases as well. As of now, we are told that we'll need to manually associate the test scripts with test cases one-by-one. Is there a way to replicate tcm.exe's /import functionality with Azure Dev Ops?
I agree with Matt. TCM.exe primary use case was to associate test automation to Test case work items so that they could be executed as part of the automated workflow.
However, with the new Pipelines capabilities, you no longer require the test automation to be associated with Test Case workitems. You can run the test automation directly within Pipelines.
See for more info:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops#arguments
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops
We have a TFS build definition that kicks off NUnit tests tagged with the 'Regression' test category. This uses the NUnit console runners annotation of
where cat = 'Regression'
However, we have multiple different environments where some tests will fail in one environment, they will pass in the other. We have not made much use of the Playlist feature, because I can not find a way to target a playlist when running remotely on TFS. Does anyone know how this can be done? Thanks!
Unfortunately, there is no way of specifying this/playlist in the TFS Build Definition for now. A related uservoice:
Allow test playlists to be used by TFS build servers
https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/3853614-allow-test-playlists-to-be-used-by-tfs-build-serve
As a workaround, you could use .orderedtest instead of .playlist.
Ordered tests can be created and edited in VS2013 and later. The format is otherwise similar to .playlist but it contains links to test GUIDs so its more complicated to modify programmatically.
TFS is able to run orderedtest in build pipeline, how to achieve this you could refer below links:
TFS - order of automata tests to execute
How to use Vnext build: ordered tests, distribute test, collect results
I want to make sure that I can run my unit test developed using specflow-c# from MTM on my local machine.
Background info:
- Developed the unit test methods using SpecFlow-C#
- Checked in my project to TFS
Problem:
- Now can anyone guide me step by step, how to achieve the integration of this unit methods.
I have gone through various articles of the MSDN, but somehow i get lost in there.
I have learnt that i need to create a build defination (But how to?), setup test controller and test agent (Again how?).
Please guide me in this.
Thanks in advance.
MTM needs three things to execute tests:
A build with which to associate the test run. It can only read from TFS. If you're not using TFS for build/deployment you can run the TFSBuild.exe tool to create a dummy build in TFS to point MTM to.
A test lab in which to run the test. Install both the controller and agent on your local machine. Open the agent configuration tool and register it to the controller. Open the controller configuration tool and register it with TFS Team Project Collection. Once this is done, you should be able to see the controller when setting up the lab environment in MTM.
A test case in TFS which is associated with a unit test. As far as I know, this association must be made in Visual Studio. In Team Explorer, open up the work items. Find your test case and click on the Associated Automation tab. In the Test Name field you select a unit test method to tie to the testcase. This is the test that gets run when the testcase is executed in MTM.
I have been using the Build-Deploy-Test build workflow for TFS 2010 (see here http://msdn.microsoft.com/en-us/vstudio/gg131922.aspx) and have come up against a very annoying limitation.
Here is what I have done:
Setup a build to be queued
Configured it to restore to a test-ready snapshot
Added some deployment steps; in this case starting mongo db
I then came to the step where you configure which tests to run and hit an issue. Firstly you need configure a test plan with some test suites; a test suite consisting of test cases. The problem is that each test case can only be associated with a single unit test.
With having to create a test case per unit test, it means that every time a unit test is added a new test case must be created. Is there any way you can associate many unit tests with a test suite that can be ran in the Build-Deploy-Test build workflow.
One way around this problem is to update your test suite using the tcm.exe testcase /import /syncsuite: command. This command has to be run before test execution begins and a logical place to add it would be into the build template.
I suggest you to use Test List Editor in Test Tab on Visual Studio, create your differents lists of tests, organize your tests based on functionnal after that you can execute on your build list of tests by entering name of list.
link : http://msdn.microsoft.com/en-us/library/ms182463(v=vs.100).aspx
For configuring Build Definition, i suggets you to edit & select Automated Test Tab and insert list.
link : http://support.smartbear.com/articles/testcomplete/testcomplete-and-team-build/
I'm new to load testing in Visual Studio/MSTest, and I created a new Load Test recently to validate some high-traffic scenarios for a WCF service. I want to add this to the tests project for the service, but I don't want the test to be executed whenever I "Run All Tests in Solution" nor as part of our Continuous Integration build-verification process because a) it takes 5 minutes to run, and b) the service call that it is testing generates many thousands of email messages. Basically, I'd like to do the equivalent of adding the [Ignore] attribute to a unit test so that the load test is only executed when I explicitly choose to run it.
This MSDN Article ("How to: Disable and Enable Tests") suggests that the only to disable the test is to use Test Lists (.vsmdi files), but I don't have much experience with them, they seem like a hassle to manage, I don't want to have to modify our CI Build Definition, and this blog post says that Test Lists are deprecated in VS2012. Any other ideas?
Edit: I accepted Mauricio's answer, which was to put the load tests into a separate project and maintain separate solutions, one with the load tests and one without. This enables you to run the (faster-running) unit tests during development and also include the (slower-running) load tests during build verification without using test lists.
This should not be an issue for your CI Build Definition. Why?
To run unit tests as part of your build process you need to configure the build definition to point to a test container (usually a .dll file containint your test classes and methods). Load tests do not work this way, they are defined within .loadtest files (which are just xml files) that are consumed by the MSTest engine.
If you do not make any further changes to your CI Build definition the load test will be ignored.
If you want to run the test as part of a build, then you need to configure the build definition to use the .loadtest file.
Stay away from testlists. Like you said, they are being deprecated in VS11.
Edit: The simplest way to avoid running the load test as part of Visual Studio "Run All" tests is to create a different solution for your load tests.
Why don't you want to use Test Lists. I think is the best way to do that. Create different Test Lists for each test type (unit test, load test...) and then in your MSTest command run the Test List(s) you want:
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests (only UnitTests)
MSTest \testmetadata:testlists.vsmdi \testlist:LoadTests (only LoadTests)
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests \testlist:LoadTests (UnitTests & LoadTests)