Forcing MSTest to use a single thread - visual-studio

Given this test fixture:
[TestClass]
public class MSTestThreads
{
[TestMethod]
public void Test1()
{
Trace.WriteLine(Thread.CurrentThread.ManagedThreadId);
}
[TestMethod]
public void Test2()
{
Trace.WriteLine(Thread.CurrentThread.ManagedThreadId);
}
}
Running the test with MSTest through Visual Studio or command line prints two different thread numbers (yet they are run sequentially anyway).
Is there a way to force MSTest to run them using a single thread?

I solved this problem with locking:
public static class IntegrationTestsSynchronization
{
public static readonly object LockObject = new object();
}
[TestClass]
public class ATestCaseClass
{
[TestInitialize]
public void TestInitialize()
{
Monitor.Enter(IntegrationTestsSynchronization.LockObject);
}
[TestCleanup]
public void TestCleanup()
{
Monitor.Exit(IntegrationTestsSynchronization.LockObject);
}
//test methods
}
// possibly other test cases
This can of course be extracted to a base test class and reused.

I've fought for endless hours to make MSTest run in a single threaded mode on a large project that made heavy use of nHibernate and it's not-thread-safe (not a problem, it's just not) ISession.
We ended up more time writing code to support the multi-threaded nature of MSTest because - to the best of my and my teams knowledge - it is not possible to run MSTest in a single threaded mode.

You can derive your test class from
public class LinearTest
{
private static readonly object SyncRoot = new object();
[TestInitialize]
public void Initialize()
{
Monitor.Enter(SyncRoot);
}
[TestCleanup]
public void Cleanup()
{
Monitor.Exit(SyncRoot);
}
}

We try hard to make out tests isolated from each other. Many of them achieve this by setting up the state of a database, then restoring it afterwards. Although mostly tests set up different data, with some 10,000 in a run there is a fair chance of a collision unless the code author of a test takes care to ensure its initial data is unique (ie doesn't use the same primary keys as another test doing something similar). This is, frankly, unmanageable, and we do get occasional test failures that pass second time around. I am fairly sure this is caused by collisions that would be avoided running tests strictly sequentially.

The way to make an MSTest method run in single-threaded mode:
Nuget:
install-package MSTest.TestAdapter
install-package MSTest.TestFramework
In your test source on those methods that need to run while no other tests are running:
[TestMethod]
[DoNotParallelize]
public void myTest(){
//
}

Whilst it is a cop out answer, I would actually encourage you to make your code thread-safe. The behaviour of MSTest is to ensure isolation as Richard has pointed out. By encountering problems with your unit tests you are proving that there could be some problems in the future.
You could ignore them, use NUnit, or deal with them and continue to use MSTest.

I tried a bit of a different approach, because the underlying problem is that the names of the pipes are the problem. So I made a fakePipe, derived it from the one I use in the program. And named the pipe with the tests name.
[TestClass]
public class PipeCommunicationContractTests {
private PipeDummy pipe;
/// <summary>
///Gets or sets the test context which provides
///information about and functionality for the current test run.
///</summary>
public TestContext TestContext { get; set; }
[TestInitialize]
public void TestInitialize() {
pipe = new PipeDummy(TestContext.TestName);
pipe.Start();
}
[TestCleanup]
public void TestCleanup() {
{
pipe.Stop();
pipe = null;
}
...
[TestMethod]
public void CallXxOnPipeExpectResult(){
var result = pipe.Xx();
Assert.AreEqual("Result",result);
}
}
It appears to be a bit faster, since we can run on multiple cores and threads...

Related

Xunit Test for ASP.NET Boilerplate

For ASP.NET Boilerplate, I am trying to implement Xunit test for the project. Is there any method to provide the parameters to test methods and configure the test methods process order.
In the normal test project, I could use [InlineData("Test", 3)] to provide the value to test methods public async void AllInOne_Test(string userName, int count), but for ASP.NET Boilerplate, it did not pass the value to test method.
I also want to process TestB after TestB, how could I achieve this?
Update1:
For InlineData issue, I could inherit TheoryAttribute like below:
public sealed class MultiTenantTheoryAttribute : TheoryAttribute
{
public MultiTenantTheoryAttribute()
{
if (!CloudConsts.MultiTenancyEnabled)
{
Skip = "MultiTenancy is disabled.";
}
}
}
And then, use it like below:
[MultiTenantTheory]
[InlineData("Test_CreateCoupon")]
public async Task CreateCoupon_Test(string title)
{
//....
}
But, I still have an issue with test method processing order.
[MultiTenantTheory]
[InlineData("Test_CreateCoupon")]
public async Task CreateCoupon_Test(string title)
{
//...
}
[MultiTenantTheory]
[InlineData("Test_CreateCoupon", "Test_UpdateCoupon")]
public async Task UpdateCoupon_Test(string oldTitle, string updateTitle)
{
//。。。
}
In my tests, I have CreateCoupon_Test and UpdateCoupon_Test, I want to exec CreateCoupon_Test, then UpdateCoupon_Test when I run all tests. But, currently, its order is random, and I got UpdateCoupon_Test run before CreateCoupon_Test, then my tests failed, since I update the record which has not been created.

Mvvmcross Testing different view models fails when running together

I've come across an interesting error. I have two test files for my xamarin mobile application, both testing view models:
public class TestFirstViewModel : MvxIoCSupportingTest
{
public void AdditionalSetup() {
//Register services and dependencies here.
}
[Fact]
public TestMethod1() {
// Successful test code here.
}
}
That's in one file. In another file, I have:
public class TestSecondViewModel : MvxIoCSupportingTest
{
public void AdditionalSetup() {
//Register services and dependencies here, slightly different from first
}
[Fact]
public TestMethod2() {
// Successful test code here.
}
}
When I run these files individually (I'm using xunit), they work just fine. However, when I run them together, I get the following error on one of the test cases:
Result Message: Cirrious.CrossCore.Exceptions.MvxException : You cannot create more than one instance of MvxSingleton
Result StackTrace:
at Cirrious.CrossCore.Core.MvxSingleton`1..ctor()
at Cirrious.CrossCore.IoC.MvxSimpleIoCContainer..ctor(IMvxIocOptions options)
at Cirrious.CrossCore.IoC.MvxSimpleIoCContainer.Initialize(IMvxIocOptions options)
at Cirrious.MvvmCross.Test.Core.MvxIoCSupportingTest.ClearAll()
at Cirrious.MvvmCross.Test.Core.MvxIoCSupportingTest.Setup()
at Project.Test.TestFirstViewModel.TestMethod1() in ...
Can anyone tell me what's going on here?
The issue stems from the parallelization of XUnit without the option to do proper tear-down. You could diable parallelization in the AssemblyIndo.cs file in you test project by adding:
[assembly: CollectionBehavior(DisableTestParallelization = true)]
I ended up solving this question by changing testing frameworks. I had different ioc singleton initializations, because, well, they're different test cases and needed different inputs/mocks. Instead of using Xunit, I resorted to Nunit where their cache clearing was much more defined: Xunit doesn't exactly believe in setup and tear-down, so it made a test environment like this more difficult.
I fixed the issue by using the collection attribute.
[Collection("ViewModels")]
class ViewModelATest : BaseViewModelTest {
...
}
[Collection("ViewModels")]
class ViewModelBTest : BaseViewModelTest {
...
}
The base view model test class has the mock dispatcher and performs the singleton registrations in the additional setup method.
Each of my tests calls ClearAll() at the beginning.
I hade some success with setup things in a constructor and add this check:
public PaymentRepositoryTests()
{
if (MvxSingletonCache.Instance == null)
{
Setup();
}
//other registerings.
}`
Also I did implement the IDisposable Interface
public void Dispose()
{
ClearAll();
}
But tbh not sure how much impact that had..
It works ok with xunit
Copy MvxIocSupportingTest and Mvxtest in your xunit PCL project.
Modify MvxTest to remove the attributes and use a simple contructor:
public class MvxTest : MvxIoCSupportingTest
{
protected MockMvxViewDispatcher MockDispatcher { get; private set; }
public MvxTest()
{
Setup();
}
...
And in each of you test, derive from IClassFixture
public class TestRadiosApi : IClassFixture<MvxTest>
{
[Fact]
public async Task TestToken()
{
...
xunit will create the MvxTest class only once for all tests.

TDD : tests too close of method implementations

We are doing TDD for quite a while and we are facing some concerns when we refactor. As we are trying to respect as much as we can the SRP (Single responsibility principle), we created a lot of composition that our classes use to deal with common responsibilities (such as validation, logging, etc..).
Let's take a very simple example :
public class Executioner
{
public ILogger Logger { get; set; }
public void DoSomething()
{
Logger.DoLog("Starting doing something");
Thread.Sleep(1000);
Logger.DoLog("Something was done!");
}
}
public interface ILogger
{
void DoLog(string message);
}
As we use a mocking framework, the kind of test that we would do for this situation would be somthing like
[TestClass]
public class ExecutionerTests
{
[TestMethod]
public void Test_DoSomething()
{
var objectUnderTests = new Executioner();
#region Mock setup
var loggerMock = new Mock<ILogger>(MockBehavior.Strict);
loggerMock.Setup(l => l.DoLog("Starting doing something"));
loggerMock.Setup(l => l.DoLog("Something was done!"));
objectUnderTests.Logger = loggerMock.Object;
#endregion
objectUnderTests.DoSomething();
loggerMock.VerifyAll();
}
}
As you can see, the test is clearly aware of the method implementation that we are testing. I have to admit that this example is too simple, but we sometimes have compositions that cover responsibilities that don't add any value to a test.
Let's add some complexity to this example
public interface ILogger
{
void DoLog(LoggingMessage message);
}
public interface IMapper
{
TTarget DoMap<TSource, TTarget>(TSource source);
}
public class LoggingMessage
{
public string Message { get; set; }
}
public class Executioner
{
public ILogger Logger { get; set; }
public IMapper Mapper { get; set; }
public void DoSomething()
{
DoLog("Starting doing something");
Thread.Sleep(1000);
DoLog("Something was done!");
}
private void DoLog(string message)
{
var startMessage = Mapper.DoMap<string, LoggingMessage>(message);
Logger.DoLog(startMessage);
}
}
Ok, this is an example. I would include the Mapper stuff within the implementation of my Logger and keep a DoLog(string message) method in my interface, but it's an example to demonstrate my concerns
The corresponding test leads us to
[TestClass]
public class ExecutionerTests
{
[TestMethod]
public void Test_DoSomething()
{
var objectUnderTests = new Executioner();
#region Mock setup
var loggerMock = new Mock<ILogger>(MockBehavior.Strict);
var mapperMock = new Mock<IMapper>(MockBehavior.Strict);
var mockedMessage = new LoggingMessage();
mapperMock.Setup(m => m.DoMap<string, LoggingMessage>("Starting doing something")).Returns(mockedMessage);
mapperMock.Setup(m => m.DoMap<string, LoggingMessage>("Something was done!")).Returns(mockedMessage);
loggerMock.Setup(l => l.DoLog(mockedMessage));
objectUnderTests.Logger = loggerMock.Object;
objectUnderTests.Mapper = mapperMock.Object;
#endregion
objectUnderTests.DoSomething();
mapperMock.VerifyAll();
loggerMock.Verify(l => l.DoLog(mockedMessage), Times.Exactly(2));
loggerMock.VerifyAll();
}
}
Wow... imagine that we would use another way to translate our entities, I would have to change every tests that has some method that uses the mapper service.
Anyways, we really feel some pain when we do major refactoring as we need to change a bunch of tests.
I'd love to discuss about this kind of problem. Am I missing something? Are we testing too much stuff?
Tips:
Specify exactly what should happen and no more.
In your fabricated example,
Test E.DoSomething asks Mapper to map string1 and string2 (Stub out Logger - irrelevant)
Test E.DoSomething tells Logger to log mapped strings (Stub/Fake out Mapper to return message1 and message2)
Tell don't ask
Like you've yourself hinted, if this was a real example. I'd expect Logger to handle the translation internally via a hashtable or using a Mapper. So then I'd have a simple test for E.DoSomething
Test E.DoSomething tells Logger to log string1 and string2
The tests for Logger would ensure L.Log asks mapper to translate s1 and log the result
Ask methods complicate tests (ask Mapper to translate s1 and s2. Then pass the return values m1 and m2 to Logger) by coupling the collaborators.
Ignore irrelevant objects
The tradeoff for isolation via testing interactions is that the tests are aware of implementation.
The trick is to minimize this (via not creating interfaces/specifying expectations willy-nilly). DRY applies to expectations as well. Minimize the amount of places that an expectation is specified... ideally Once.
Minimize coupling
If there are lots of collaborators, coupling is high which is a bad thing. So you may need to rework your design to see which collaborators don't belong at the same level of abstraction
Your difficulties come from testing behavior rather than state. If you would rewrite the tests so that you look at what's in the log rather than verifying that the call to the log is made, your tests wouldn't break due to changes in the implementation.

Launching application from one class and using it in another class(Help me in Learning Selenium)

Launching Application from One Class and using it in another class.
Hi All,
I am new to selenium and Java. I just trying to work on selenium. I am
facing some problem. I wanted to create a class in which I just want
to launch application(say google.com), and the next other tasks like
executing test cases has to be done by other class. Can this be done,
Here I am pasting my code. Help me please in understanding this.
Here are my two classes
This class Launches Google on browser.
public class GoogleTest extends SeleneseTestCase
{
private static Selenium selenium;
#BeforeClass
public static void setUpBeforeClass() throws Exception
{
selenium = new DefaultSelenium("localhost", 4444, "*firefox", "http://www.google.com");
selenium.start();
selenium.windowMaximize();
}
#Test
public void testSearch() throws InterruptedException
{
selenium.open("http://www.google.com");
selenium.waitForPageToLoad("50000");
}
#AfterClass
public static void tearDownAfterClass() throws Exception {
selenium.stop();
}
}
This class should type in search field
public class FirstSeleniumTest extends GoogleTest
{ private static Selenium selenium;
#Test
public void testSearch()
{
selenium.type("q", "Selenium OpenQA");
selenium.waitForPageToLoad("50000");
}
}
When I run "GoogleTest" first class it runs perfectly, when I try to
run "FirstSeleniumTest" second class it gives error like this. FAILED:
testSearch on null(com.dev.rao.FirstSeleniumTest)
java.lang.NullPointerException at
com.dev.rao.FirstSeleniumTest.testSearch(FirstSeleniumTest.java:18)
Its normally difficult to do what you are trying to achieve. That's why new Selenium instance is better per test. and in the longer run easy to debug. But as per what you are trying I see two issues:
1.) selenium.stop() should be in the second script not the first. If you kill the instance how can it be extended in 2nd test.
2.) Declaring Selenium twice. (I think probably Selenium instance in 2nd test should also extend from first test.)

Allow own TestContext class to inject different connections Strings inVisual Studio Unit Test Project

In MBUnit I can annotate my Test class with a Factory and multiple getter returning an oracle/sql/mysql connection string which can be obtained by every test method in the test class.
How can I do this with the Unit Test Project for Visual Studio 2010?
e.g. I am not allowed to inherit from TestContext and pass that object to my ClassInit method?
[ClassInitialize()]
public static void MyClassInitialize(MyContextDerivedFromTextContext testContext)
{
}
I do not want to hardcode-annotate my TestMethods with such an attribute:
[DataSource("System.Data.SqlClient", "Data Source=.\\SQLEXPRESS;Initial Catalog=STM;Integrated Security=True;Pooling=False", "CustomerTable", DataAccessMethod.Sequential), TestMethod]
public void TestMethod1()
{
//
// TODO: Add test logic here
//
}
I want this:
public void TestMethod1(String testconnectionStringOracleORMySQLORMSSQLetc...)
{
//
// TODO: Add test logic here
//
}
I think you're out for some disappointment. MSTest supports data driven tests as you've already shown but it sucks compared to other test frameworks.
The only approach I can think of that might do what you want is to store your connection strings in an XML file and that as a data source for your MSTest tests to set up the connection and then do whatever else it is you need to do.

Resources