I am using Specflow (not specflow+runner) and Xunit for writing API Test. A bamboo build pipeline is created to execute all our api TC's. When I am running our TC's via Visual Studio it's taking 15 min to execute(~200 TC's) but when I am running via Bamboo pipeline it's taking 30 min.
All the TC's run inside bamboo pipeline are executed sequentially or we can run them in parallel so that we can reduce the execution time.
Related
I'm tracking the files of my golang app in a local TFS server integrated with git. I want to define build, test, and deploy steps but I can't find any tool for that.
How can I define a build and test step for the project that shows the test results in the Team's web site? thank you.
You can use the vNext build system in TFS2017 and add Command Line task in the build definition to run the build and test just like what you do on local machine:
And then add a "Publish Test Result" task to publish the test result to TFS so that you can check the test result from the TFS Website. A limitation is that "Publish Test Result" task only support four Test Result Format for now: JUnit, NUnit, VSTest and XUnit.
I'm trying to set up a build server script to run VS Load Tests. My preferred build server is Team City, but I would accept VSO's build server as well.
Thanks!
You should not use build server to run load test.
Visual Studio Online provides cloud-based load test, you can use it for your testing.
Instruction about VSO Load Test: https://www.visualstudio.com/en-us/get-started/test/load-test-your-app-vs
We use TFS 2012
Through a build definition we execute automated tests from MTM test plans.
How can we ensure TFS uses vstest.console.exe to run each test?
I've read online from other posts that TFS 2012 let's you choose either MSTest or VSTest as the test runner.
I've also read that if you use MTM you effectively are locked into using a test settings file(.testsettings) which means you are locked into using MSTest. This seems to go against TFS 2012 giving you a choice.
Test Manager currently is only supporting MsTest. This is a limitation in Test Manager. It's the same issue that prevents you from associating a xUnit or NUnit test automation on a test case.
So while TFS 2012 gives you the choice, each choice comes with its own set of limitations.
To be clear:
New test runner: Work in Continuous Integration, you specify which tests to run by selecting the new agile test runner (in TFS 2013 this is a fixed choice in the default templates, template customization is required to run mstest). It can run tests during build and you can directly invoke it from the commandline if you want to run tests after deployment. You can use a .runsettings file to specify certain options it should pick up. This option is required to execute 3rd party test frameworks like NUnit, XUnit.NET, Chutzpah etc.
The MTM test runner: Execute automated tests that are associated to a Test Case work item or execute tests using the Test Agent to run tests from a remote system using the Team Test infrastructure. You can use a .testsettings file to specify certain options it should pick up. These tests will be executed using MsTest and cannot contain 3rd party test frameworks.
In your Continuous Integration build you can configure multiple Test Runs, each can be configured to a specific test framework. So you can have both options in your build.
More on the differences and how you can use the different settings files.
Over time all test options will be moved to the new test runner. In TFS 2013 the option to configure a MsTest based test run in your Build Definition has been removed by default. You will need to customize the build process to select a different test runner.
We have been writing specifications for our JavaScript business logic using Jasmine. We're able to run our test suite within a browser, but how would we integrate this within TeamCity? Preferrably we do not want to use NodeJS, rather something as simple as possible.
I have created a modified version of run-jasmine.js that is found in the PhantomJS sources (original version is here. This version can be used within TeamCity (it will automatically detect that it is running in TeamCity). This updated version is using TeamCity service messages which allows for a nice integration.
You will need PhantomJS. You'll also need one of the following:
run-jasmine.js (for Jasmine 1.x).
run-jasmine.js (for Jasmine 2.x).
Add a build step in your TeamCity build configuration that can run this step:
phantomjs.exe run-jasmine.js index.html
index.html is your Jasmine runner page. If the build agents do not include PhantomJS, you can commit it to your repository along with your sources (this is what we do).
The result will look like this:
Test details:
The above is from a Tasks sample ASP.NET MVC project with this setup. It can be run in TeamCity using a Visual Studio (sln) build step. It will also run the tests within Visual Studio, as a pre-build step.
I'm exploring using Microsoft Test Manager as an option for for my company to switch over to. We already use Visual Studio/TFS for our development, source control, builds, scrum process, etc.
I've been able to use Test Manager to run manual tests, but I can't find anyway to publish these results to a build or any easy way to associate these test runs with work items.
The build summary in Visual Studio does show that a test run as having completed after I run a test in MTM against that build, but when I try to view the results I get "Downloading the selected test run failed. Test run XX does not have a run summary attachment. The .trx file is either not published or upgraded."
Is it possible to publish results to TFS after running a manual test in MTM?