We need to create load and stress tests for a cloud application. Most tests will have to be coded instead of being recorded. Thus we need to integrate unit tests in the text mix.
Currently, we are creating a spike with Visual Studio 2015 Load Test Framework. We want to run the load tests from our local infrastructure as well as use Azure's cloud testing.
We would rather use XUnit instead of MS Test to write the unit tests. Is that possible? Do we have to write an plugin to achieve that?
Please check this question for the answer. Both NUnit and XUnit tests can't be added to a load test.
Related
I need to somehow associate the automated tests with manual test plan in Azure. I already installed Visual Studio and open my automated test cases in VS. But when I am trying to click "Associate to Test Case" I got the error that functionality is applicable only with MSTest, XUNIT etc. And my framework Pytest is out of this list.
Does that mean I have no possibility to associate my automated test cases with manual one using this functionality? But maybe I can do it in other way? Maybe something can be done with yaml file? Or I can simply tagged automated test cases directly in Test Plan? Or maybe I can install third party plugin for pytest? Or maybe there is a way to build a matrix with manual/automated test cases?
I have no idea, but I need to have the visual picture of test cases matrix.
Does that mean I have no possibility to associate my automated test
cases with manual one using this functionality
Yes, we cannot associate framework Pytest to test case in Azure DevOps. It's not supported.
See What types of tests are supported? for details.
A: These are the limitations for each type of test:
Coded UI test, Selenium tests, and unit tests written using Version 1
of the MSTest framework can be associated with a test case.
Tests that use MSTest v2, NUnit, and xUnit frameworks can be associated with a test case workitem when using Visual Studio 15.9 Preview 2 or later. However, these tests cannot be run using Microsoft Test Manager and XAML builds.
Tests that use the .NET core framework can be associated with a test case workitem when using Visual Studio 15.9 Preview 2 or later. To run the .NET core tests the appropriate target framework must be specified in a runsettings file. However, these tests cannot be run using Microsoft Test Manager and XAML builds.
Tests that use other test frameworks such as Chutzpah (for JavaScript tests such as Mocha or QUnit), or Jest cannot be associated with a test case.
Associating generic tests may work, but running these tests is not supported.
However, you can reference the following blogs to run Python test with Azure DevOps pipeline:
Run Python test with Azure DevOps pipeline
Selenium Python Tests & Azure DevOps Integration – Made for each other!
I have a generic Selenium Test which needs to run over all projects how would i do that with Visual Studio online? I can get it to work on one project at a test but not sure how to make it so this test can run on all projects?
Generally you need to write the specific Selenium test for the specific project. However if all the projects you mentioned have the same structure/components/features, then you can reuse the Selenium test project for them.
I don't think it's this is applicable if you mean create tests together which want to apply for all the different projects.
You can reference below articles about the Selenium Test in TFS:
Get started with Selenium testing in a CD pipeline
Parallel Testing in a Selenium Grid with VSTS
I have a pretty simple question: What kind of project should I create for Integration Tests in Visual Studio? The only proposed project type under Test tab in Visual C# is Unit Test Project, but it's not integration test, so I'm not sure
This is what I read, but couldn't find an answer :
Integration Testing In Visual Studio With Different Project Types
How do you separate unit tests from integration tests in Visual Studio?
Thanks!
Use the unit test project template. You will create your integration tests in there.
The unit test project is just a template that provisions the necessary dependencies. You could just as easily create an empty project and manually add what it needed.
We use TFS 2012
Through a build definition we execute automated tests from MTM test plans.
How can we ensure TFS uses vstest.console.exe to run each test?
I've read online from other posts that TFS 2012 let's you choose either MSTest or VSTest as the test runner.
I've also read that if you use MTM you effectively are locked into using a test settings file(.testsettings) which means you are locked into using MSTest. This seems to go against TFS 2012 giving you a choice.
Test Manager currently is only supporting MsTest. This is a limitation in Test Manager. It's the same issue that prevents you from associating a xUnit or NUnit test automation on a test case.
So while TFS 2012 gives you the choice, each choice comes with its own set of limitations.
To be clear:
New test runner: Work in Continuous Integration, you specify which tests to run by selecting the new agile test runner (in TFS 2013 this is a fixed choice in the default templates, template customization is required to run mstest). It can run tests during build and you can directly invoke it from the commandline if you want to run tests after deployment. You can use a .runsettings file to specify certain options it should pick up. This option is required to execute 3rd party test frameworks like NUnit, XUnit.NET, Chutzpah etc.
The MTM test runner: Execute automated tests that are associated to a Test Case work item or execute tests using the Test Agent to run tests from a remote system using the Team Test infrastructure. You can use a .testsettings file to specify certain options it should pick up. These tests will be executed using MsTest and cannot contain 3rd party test frameworks.
In your Continuous Integration build you can configure multiple Test Runs, each can be configured to a specific test framework. So you can have both options in your build.
More on the differences and how you can use the different settings files.
Over time all test options will be moved to the new test runner. In TFS 2013 the option to configure a MsTest based test run in your Build Definition has been removed by default. You will need to customize the build process to select a different test runner.
I often have to restart my IDE when trying to run a test set from within Visual Studio 2010 since it keeps being hung/not-responding very often.
Do you have the same problem? If you do, do you have a work-around for that? Please share.
I'll share my setup with you. It won't solve your problem but might give you some ideas to improve your unit tests.
First, my unit tests do not connect to the database or reference cookies / session. In those scenarios, I use Dependency Injection (with Castle Windsor) to pass fake repositories to the unit tests. If your unit tests are hitting anything external (Web services, SQL, IIS) that might be a good place to look.
I use NUnit / Moq for the unit tests themselves. I had been using the NUnit client to run the unit tests but then I stumbled on a tool called Gallio that allows me to run (and debug) my unit tests from within Visual Studio.
I'm not sure if Gallio is compatible with your unit tests or not, but it might be beneficial to debug the unit tests to see exactly where they are hanging up.
EDIT: Thought I'd add that I'm using Visual Studio 2010 Premium and I have about 110 tests so far that complete in a few seconds. The tests hit ASP.NET MVC controller actions.