I'm implementing tests using SpecFlow that have nothing to do with each other. Is there config option for SpecFlow that enables parallel test execution? I'm using VS10 and MSTest runner that supports running "up to 5 parallel unit tests" as they claim in documentation.
Thanks,
max.yz
I moved away from MSTest to MbUnit to achieve this. You can achieve parallelism at test fixture level with MbUnit using the ParallelizableAttribute. However as the test fixtures are generated from the .feature Gherkin files I had to grab the SpecFlow source code and modify the MbUnitTestGeneratorProvider class in the TechTalk.SpecFlow.Generator project to output the ParallelizableAttribute. So you end up with something like this:
public class MbUnitTestGeneratorProvider : IUnitTestGeneratorProvider
{
private const string TESTFIXTURE_ATTR = "MbUnit.Framework.TestFixtureAttribute";
private const string PARALLELIZABLE_ATTR = "MbUnit.Framework.ParallelizableAttribute";
private const string TEST_ATTR = "MbUnit.Framework.TestAttribute";
private const string ROWTEST_ATTR = "MbUnit.Framework.RowTestAttribute";
private const string ROW_ATTR = "MbUnit.Framework.RowAttribute";
private const string CATEGORY_ATTR = "MbUnit.Framework.CategoryAttribute";
private const string TESTSETUP_ATTR = "MbUnit.Framework.SetUpAttribute";
private const string TESTFIXTURESETUP_ATTR = "MbUnit.Framework.FixtureSetUpAttribute";
private const string TESTFIXTURETEARDOWN_ATTR = "MbUnit.Framework.FixtureTearDownAttribute";
private const string TESTTEARDOWN_ATTR = "MbUnit.Framework.TearDownAttribute";
private const string IGNORE_ATTR = "MbUnit.Framework.IgnoreAttribute";
private const string DESCRIPTION_ATTR = "MbUnit.Framework.DescriptionAttribute";
public bool SupportsRowTests { get { return true; } }
public void SetTestFixture(CodeTypeDeclaration typeDeclaration, string title, string description)
{
typeDeclaration.CustomAttributes.Add(
new CodeAttributeDeclaration(
new CodeTypeReference(TESTFIXTURE_ATTR)));
typeDeclaration.CustomAttributes.Add(
new CodeAttributeDeclaration(
new CodeTypeReference(PARALLELIZABLE_ATTR)));
SetDescription(typeDeclaration.CustomAttributes, title);
}
If you compile this up and use it you'll end up with parallelizable test fixtures:
[System.CodeDom.Compiler.GeneratedCodeAttribute("TechTalk.SpecFlow", "1.6.1.0")]
[System.Runtime.CompilerServices.CompilerGeneratedAttribute()]
[MbUnit.Framework.TestFixtureAttribute()]
[MbUnit.Framework.ParallelizableAttribute()]
[MbUnit.Framework.DescriptionAttribute("Test")]
public partial class TestFeature
{
The only problem with this as it stands is that you will need to make sure that test fixtures do not conflict with one another. That is to say, a test from one fixture adds or modifies a database row that breaks a test that is running at the same time as it. There are ways around this but that is probably out of scope of your original question.
Alex.
There is a new tool called SpecRun that was recently released by the makers of SpecFlow. SpecRun will allow you to run these tests in parallel. If you use the SpecRun.Nunit package with it, you can run NUnit tests in parallel. We use SpecRun on our CI server to run the tests in parallel, but the developers use whatever their test runner of choice.
Changing your test framework can be disruptive. Because all our tests were in NUnit to start, we simply added the new SpecRun runner and nothing else changed. Very simple and transparent to the developers. And since it's available on NuGet, it was very easy to install.
I created a solution that generates a nant build file, which uses nunit in a custom parallel nant task:
https://github.com/MartyIce/SpecflowParallelizer
Due to how my legacy tests were written, I get backend concurrency problems, so it hasn't been succesful for me (yet) but hopefully this will work for someone else.
There is an option in the MSTest .testsettings file for the test project.
By default the test runner will only run 1 test at a time, by changing the parallelTestCount attribute of the Execute node to 0 it will run on as many threads as are avalable (for some reason limited to a max of 5)
Just right click the .teststtings file and pick open with; choose an XML editor and off you go.
You must not run any Coded UI tests or configure any datacollectors for this to work.
For a more detailed explanation see this article
Related
I have a test with multiple DataRow(s). When executing test, it only runs first test
[TestClass]
public class UnitTest1
{
[DataTestMethod]
[DataRow(1, 2, 3)]
[DataRow(2, 3, 5)]
[DataRow(3, 5, 8)]
public void AdditionTest(int a, int b, int result)
{
Assert.AreEqual(result, a + b);
}
}
Any idea if I am missing anything?
I was having the same problem, but only in the ReSharper Test Sessions window. If I ran the same tests from the Test Explorer window, all of the DataRow tests would run. It had previously worked in ReSharper as well. Before this started I had recently updated to MSTest 2.2.7 and changed the test project to use NuGet package references.
The problem was resolved by updating ReSharper to 2021.2.2.
If you're like me and were attempting to run a DataTestMethod on arrays, I think the issue is that visual studio will not run multiple DataRows with identical values, and "new float[]" is treated as the same value regardless of what its initialised to.
See more here: https://github.com/microsoft/testfx/issues/1016
My Hooks class like below :
#Before("#Firefox")
public void setUpFirefox() {}
#Before("#Chrome")
public void setUpChrome() {}
#After
public void tearDown(){}
When I run the following command mvn test -Dcucumber.options="--tags #Chrome" both #Before functions are calling.
How can I run specific #Before method depending on maven command?
My Runner class (I already tried with tags option, it is also not working for me) :
#RunWith(Cucumber.class)
#CucumberOptions(
plugin = {"pretty", "json:target/cucumber-reports/Cucumber.json",
"junit:target/cucumber-reports/Cucumber.xml",
"html:target/cucumber-reports"},
features = {"src/test/features/"},
glue = {"steps"})
public class RunCukesTest {
}
My feature file :
Feature: Storybook
#Test #Widgets #Smoke #Chrome #Firefox
Scenario: Storybook example
Given The user clicks on "storybook" index on the homepage
And Storybook HomePage should be displayed
It looks like it's because you've got both tags for that scenario, the before hooks seem to execute based on the scenario that's running. e.g.
-The tag command line --tags #Chrome etc. specifies which scenario to run
-Now based on that scenario, execute the before functions with the tags attached to that scenario (Test, Widgets, Smoke, Chrome, Firefox)
If you were to have a Before hook for the tag Smoke, I would imagine that would also run.
For example:
(It's in Scala)
Before("#test1") { _ =>
println("test1 before actioned")
}
Before("#test2") { _ =>
println("test2 before actioned")
}
With the feature file:
[...]
#test1
Scenario: Test scenario 1
Given Some precondition occurs
#test2
Scenario: Test scenario 2
Given Some precondition occurs
When I run either of those tags, I get the output
test1 before actioned or test2 before actioned
However, have both tags on one scenario then both lines are printed.
What's being actioned in those setupChome, setupFirefox functions, just setting the driver up? You could create new system property such as browser for example, match on the value and execute some setup then you could type in:
-Dbrowser=chrome and it would do the setup that way.
Recently moved from utilising AWS to Azure for the location of our load test agents, thus making the transition to making full use of VSTS.
It was described that, for the moment, to get a load test file working with VSTS to using our own VMs for testing, we need to provide two context parameters, UseStaticLoadAgents and StaticAgentsGroupName in each loadtest file.
Our load test solution is getting very large, and we have multiple loadtest files where we have to set these two values each time. This leads us into the situation where, if we were to change our agents group name for example, we would have to update each individual load test file with the new information.
Im looking at a way to centralise this until a nicer way is implemented by Microsoft. The idea was to use a load test plugin, to add these context parameters with the plugin drawing the needed values from a centralised config file.
However, it seems that none of the hooks in the load test plugin or simply using the initialise method to manually set these values is working. Likely because they are set after full initialisation.
Has anyone got a nice, code focused solution to manage this and stop us depending on adding brittle values in the editor? Or even gotten the above approach to work?
The loadtest file is the XML file, so you can update it programmatically, for example:
string filePath = #"XXX\LoadTest1.loadtest";
XmlDocument doc = new XmlDocument();
doc.Load(filePath);
XmlNamespaceManager nsmgr = new XmlNamespaceManager(doc.NameTable);
nsmgr.AddNamespace("ns", "http://microsoft.com/schemas/VisualStudio/TeamTest/2010");
XmlNode root = doc.DocumentElement;
XmlNode nodeParameters = root.SelectSingleNode("//ns:RunConfigurations/ns:RunConfiguration[#Name='Run Settings1']/ns:ContextParameters", nsmgr);
if(nodeParameters!=null)
{
//nodeParameters.SelectSingleNode("//ns:ContextParameter[#Name='UseStaticLoadAgents']").Value = "agent1";
foreach (XmlNode n in nodeParameters.ChildNodes)
{
switch (n.Attributes["Name"].Value)
{
case "Parameter1":
n.Attributes["Value"].Value = "testUpdate";
break;
case "UseStaticLoadAgents":
n.Attributes["Value"].Value = "agent1";
break;
case "StaticAgentsGroupName":
n.Attributes["Value"].Value = "group1";
break;
}
}
}
doc.Save(filePath);
Need your help in the following scenario:
I have a solution with 2 projects with different unit tests
Those projects generate 2 different dll: *deployment.dll and *database.dll
I have a build on TFS that I want to use to run those tests, I'm using "Test Case Filter" to filter the categories of my tests
(TestCategory=TEST1|TestCategory=TEST2|TestCategory=TEST3|TestCategory=TEST4)
and in "Test Sources Spec" I'm filtering both dll (*deployment.dll;*database.dll)
*.deployment.dll has TEST2, TEST3, TEST4
*.database.dll has TEST1
This doesn't work, tests of *database.dll does not run. Test selected in Visual Studio Test Runner
Could you please help on that? If I make the build with only 1 dll, for example, *.database.dll, TEST1 runs well.
(UPDATE) SCENARIO 1
Test Case Filter: TestCategory=TEST1|TestCategory=TEST1|TestCategory=TEST2|TestCategory=TEST3|TestCategory=TEST4
Test Sources Spec: *database.dll;*deployment.dll
only runs TEST1
(UPDATE) SCENARIO 2
Test Case Filter: TestCategory=TEST1|TestCategory=TEST1|TestCategory=TEST2|TestCategory=TEST3|TestCategory=TEST4
Test Sources Spec: **\*deployment.dll;*database.dll
only runs TEST2,TEST3,TEST4
(UPDATE) Does not find tests in Database.dll
I've tested in TFS 2015.3, XAML build, but couldn't reproduce your issue. I'd like to share my steps here for your reference:
I have a solution with some projects, 2 of them are UnitTest projects (UnitTestProject1, UnitTestProject2).
In UnitTest1 project, I added TestCategory for two test case, check screenshot below:
[TestMethod()]
[TestCategory("Pro")]
public void M1Test()
{
//
}
[TestMethod()]
[TestCategory("Dev")]
public void M2Test()
{
//
}
Similar to Step2, in UnitTest2 project, I added TestCategory for two test case, check screenshot below:
[TestMethod()]
[TestCategory("Pro1")]
public void M3Test()
{
//
}
[TestMethod()]
[TestCategory("Dev1")]
public void M4Test()
{
//
}
Edit "Test Case Filter" and "Test Sources Spec" in build definition like below screenshot and queue a build:
The test result is as expected. Only M1Test and M2Test in UnitTestProject1, M3Test and M4Test in UnitTestProject2 are tested.
Finally, it's solved :)
So, what I have done to solve this problem, was, change Build Process Template.
There are one step in this process, calling: "FindMatchingFiles"
I change this value as the below image shows. (however, from now on, I must use "**\*" in all my filters that use this process template). This operation make sure that I have the assemblies with fullpath destination complete.
If you have different solutions, please post here :)
Special thank you to #Cece - MSFT for all support
I have code such that:
[CodedUITest]
public class CodedUITest1
{
[TestMethod]
public void CodedUITestMethod1( )
{
using(var dlg = new MyWinForm( ))
{
dlg.Show();
System.Threading.Thread.Sleep(2000);
this.UIMap.AssertMethod1( );
this.UIMap.RecordedMethod1( );
this.UIMap.AssertMethod2( );
}
}
}
The code was running fine when I manually launched the app(before invoking the test) without the using clause to directly create the control.
I'd like to just use a reference to create an instance of the control and go from there instead of relying on trying to determine a path to the executable and opening it. The app just gets stuck with a ContextSwitchDeadlock.
Is there a way to do coded-Ui tests without doing a process launch? (using the reference and creating the control in the test code) or is there something wrong with the way I'm trying to do it?
It might be possible if you invoke the coded ui test portions (this.UIMap...) on a separate thread. But the way you have it now, they are both on the same thread, so you are going to get deadlocked.