I am writing a custom language colorizer as a Visual Studio extension using MEF. Most of my code comes from the Ook Language Integration sample that is available here. My colorizer is pretty much ready, but now I need to provide an appropriate unit test suite for my implementation.
Given the MEF nature of this stuff, I wonder which is the appropriate approach to unit testing. Should I just directly reference my Tagger from my unit test and test the GetTags method? Should I involve MEF in my unit tests? Are there any examples out there about testing MEF based colorizers and custom Intellisense implementations?
OK, so I think I got an approach to do this. Don't know if it is the ideal approach, but allows me to test my core logic. The credits go to the guys behind the HQL Language Service for Visual Studio who showed me how to create stubs for ITextSnapshots and to Noah Richards whose Spell Checker extension has as set of unit tests that set the scenario for my ones.
Here the code for my TestFixture:
[TestFixture]
public class TokenTaggerTests
{
[Test]
public void CanGetTagsForSimpleSelect()
{
TestQuery("SELECT * FROM Books", new List<string>() { "SELECT", "*", "FROM", "Books"});
}
private void TestQuery(string query, List<string> expected)
{
ITextSnapshot snapshot = SnapshotUtil.CreateSnapshot(query);
NormalizedSnapshotSpanCollection spans = new NormalizedSnapshotSpanCollection(snapshot.CreateSpanFromLineNumber(0));
MyTokenTagProvider provider = new MyTokenTagProvider();
ITagger<ITag> tagger = provider.CreateTagger<ITag>(new Mock<ITextBuffer>().Object);
List<string> words = tagger.GetTags(spans).Select(s => query.Substring(s.Span.Start, s.Span.Length)).ToList();
string errorMessage = string.Format("Got list: [{0}]. Expected: [{1}]", string.Join(", ", words), string.Join(", ", expected));
CollectionAssert.AreEqual(expected, words, errorMessage);
}
}
In the TestQuery method I start by calling CreateSnapshot on SnapshotUtil. That is a useful class, found in the HQL Language Service test project that would create ITextSnapshot objects based on some input text, which simulates what VS would provide to my extension. Then I go and use MyTokenTagProvider to create a tagger for me. Here I use this powerful thing called Moq to create a Mock of an ITextBuffer object, which is the expected input parameter for CreateTagger.
Now that we have a tagger, the rest is just playing around with it to check if it is doing the job. GetTags is the core method I want to test here, as it is the one that will take my input query and output the series of tags it was able to identify. To get the actual words, we examine each tag's span and use it to extract the word from the original query. The CollectionAssert method of NUnit will make sure the output list of words matches the expected list.
Hopefully this is useful for anybody else out there creating and testing VS editor extensions.
Related
I've been working through Julie Lerman's books on Entity Framework, and I've run into somewhat of a snag...
In "Programming Entity Framework DbContext" on page 66 Julie suggests writing a Unit Test to make sure that dynamic proxies are being created, since the requirements for getting Entity Framework to create change tracking proxies are fairly simple, but also very easy to miss.
With that being said, I followed the general structure of Julie's example to write the following test for my code:
[Test]
public void IsDynamicProxy()
{
using (var scienceFairToGoContext = new ScienceFairToGoContext())
{
var scienceFair = scienceFairToGoContext.ScienceFairs.Create();
Assert.IsTrue(scienceFair is IEntityWithChangeTracker);
}
}
When I hover over the scienceFair object its pretty obvious its a change tracking proxy
System.Data.Entity.DynamicProxies.ScienceFair_D3C57A2F699E75F716E63553D950EF7EC75F0C603F69093FCD78122CC0D6452C
...but whenever I run the unit test it always fails because "scienceFair is IEntityWithChangeTracker" always evaluates to false.
It appears as though someone else ran into this issue as well and posted it to the O'Reilly forums, but there doesn't seem to be a solution posted, nor do my Google searches return any type of answer.
I'm currently using Visual Studio 2010, EF4, NUnit, and running my tests through ReSharper. It's also worth mentioning that if run the code in a simple console application and debug it I get the same results.
Actually, it looks like I figured out the issue. While working through Julie's book, I thought it would be a good idea to have all of the entities inherit from a base class, ScienceFairToGoEntity.
It looks like I forgot to mark the 4 properties I had on the base class (InsertBy, InsertDate, UpdateBy, UpdateDate) as virtual, so the dynamic proxies were for Lazy Loading/ Relationship Fix-up and not for Change Tracking.
It's weird to find this non-working piece of code in such a good book. You can however execute you test by:
using (var scienceFairToGoContext = new ScienceFairToGoContext())
{
var scienceFair = scienceFairToGoContext.ScienceFairs.Create();
Assert.IsTrue(scienceFair.GetType().IsSubclassOf(typeof(ScienceFair)));
}
although it is less generally applicable when there are entities in inheritance hierarchies. With derived entities you'd have to use
scienceFairToGoContext.BaseEntities.Create<TDerivedEntity>()
I've been looking at the way unit testing is done in the NuGetGallery. I observed that when controllers are tested, service classes are mocked. This makes sense to me because while testing the controller logic, I didn't want to be worried about the architectural layers below. After using this approach for a while, I noticed how often I was running around fixing my mocks all over my controller tests when my service classes changed. To solve this problem, without consulting people that are smarter than me, I started writing tests like this (don't worry, I haven't gotten that far):
public class PersonController : Controller
{
private readonly LESRepository _repository;
public PersonController(LESRepository repository)
{
_repository = repository;
}
public ActionResult Index(int id)
{
var model = _repository.GetAll<Person>()
.FirstOrDefault(x => x.Id == id);
var viewModel = new VMPerson(model);
return View(viewModel);
}
}
public class PersonControllerTests
{
public void can_get_person()
{
var person = _helper.CreatePerson(username: "John");
var controller = new PersonController(_repository);
controller.FakeOutContext();
var result = (ViewResult)controller.Index(person.Id);
var model = (VMPerson)result.Model;
Assert.IsTrue(model.Person.Username == "John");
}
}
I guess this would be integration testing because I am using a real database (I'd prefer an inmemory one). I begin my test by putting data in my database (each test runs in a transaction and is rolled back when the test completes). Then I call my controller and I really don't care how it retrieves the data from the database (via a repository or service class) just that the Model to be sent to the view must have the record I put into the database aka my assertion. The cool thing about this approach is that a lot of times I can continue to add more layers of complexity without having to change my controller tests:
public class PersonController : Controller
{
private readonly LESRepository _repository;
private readonly PersonService _personService;
public PersonController(LESRepository repository)
{
_repository = repository;
_personService = new PersonService(_repository);
}
public ActionResult Index(int id)
{
var model = _personService.GetActivePerson(id);
if(model == null)
return PersonNotFoundResult();
var viewModel = new VMPerson(model);
return View(viewModel);
}
}
Now I realize I didn't create an interface for my PersonService and pass it into the constructor of my controller. The reason is 1) I don't plan to mock my PersonService and 2) I didn't feel I needed to inject my dependency since my PersonController for now only needs to depend on one type of PersonService.
I'm new at unit testing and I'm always happy to be shown that I'm wrong. Please point out why the way I'm testng my controllers could be a really bad idea (besides the obvious increase in the time my tests will take to run).
Hmm. a few things here mate.
First, it looks like you're trying to test the a controller method. Great :)
So this means, that anything the controller needs, should be mocked. This is because
You don't want to worry about what happens inside that dependency.
You can verify that the dependency was called/executed.
Ok, so lets look at what you did and I'll see if i can refactor it to make it a bit more testable.
-REMEMBER- i'm testing the CONTROLLER METHOD, not the stuff the controller method calls/depends upon.
So this means I don't care about the service instance or the repository instance (which ever architectural way you decide to follow).
NOTE: I've kept things simple, so i've stripped lots of crap out, etc.
Interface
First, we need an interface for the repository. This can be implemented as a in-memory repo, an entity framework repo, etc.. You'll see why, soon.
public interface ILESRepository
{
IQueryable<Person> GetAll();
}
Controller
Here, we use the interface. This means it's really easy and awesome to use a mock IRepository or a real instance.
public class PersonController : Controller
{
private readonly ILESRepository _repository;
public PersonController(ILESRepository repository)
{
if (repository == null)
{
throw new ArgumentNullException("repository");
}
_repository = repository;
}
public ActionResult Index(int id)
{
var model = _repository.GetAll<Person>()
.FirstOrDefault(x => x.Id == id);
var viewModel = new VMPerson(model);
return View(viewModel);
}
}
Unit Test
Ok - here's the magic money shot stuff.
First, we create some Fake People. Just work with me here... I'll show you where we use this in a tick. It's just a boring, simple list of your POCO's.
public static class FakePeople()
{
public static IList<Person> GetSomeFakePeople()
{
return new List<Person>
{
new Person { Id = 1, Name = "John" },
new Person { Id = 2, Name = "Fred" },
new Person { Id = 3, Name = "Sally" },
}
}
}
Now we have the test itself. I'm using xUnit for my testing framework and moq for my mocking. Any framework is fine, here.
public class PersonControllerTests
{
[Fact]
public void GivenAListOfPeople_Index_Returns1Person()
{
// Arrange.
var mockRepository = new Mock<ILESRepository>();
mockRepository.Setup(x => x.GetAll<Person>())
.Returns(
FakePeople.GetSomeFakePeople()
.AsQueryable);
var controller = new PersonController(mockRepository);
controller.FakeOutContext();
// Act.
var result = controller.Index(person.Id) as ViewResult;
// Assert.
Assert.NotNull(result);
var model = result.Model as VMPerson;
Assert.NotNull(model);
Assert.Equal(1, model.Person.Id);
Assert.Equal("John", model.Person.Username);
// Make sure we actually called the GetAll<Person>() method on our mock.
mockRepository.Verify(x => x.GetAll<Person>(), Times.Once());
}
}
Ok, lets look at what I did.
First, I arrange my crap. I first create a mock of the ILESRepository.
Then i say: If anyone ever calls the GetAll<Person>() method, well .. don't -really- hit a database or a file or whatever .. just return a list of people, which created in FakePeople.GetSomeFakePeople().
So this is what would happen in the controller ...
var model = _repository.GetAll<Person>()
.FirstOrDefault(x => x.Id == id);
First, we ask our mock to hit the GetAll<Person>() method. I just 'set it up' to return a list of people .. so then we have a list of 3 Person objects. Next, we then call a FirstOrDefault(...) on this list of 3 Person objects .. which returns the single object or null, depending on what the value of id is.
Tada! That's the money shot :)
Now back to the rest of the unit test.
We Act and then we Assert. Nothing hard there.
For bonus points, I verify that we've actually called the GetAll<Person>() method, on the mock .. inside the Controller's Index method. This is a safety call to make sure our controller logic (we're testing for) was done right.
Sometimes, you might want to check for bad scenario's, like a person passed in bad data. This means you might never ever get to the mock methods (which is correct) so you verify that they were never called.
Ok - questions, class?
Even when you do not plan to mock an interface, I strongly suggest you to do not hide the real dependencies of an object by creating the objects inside the constructor, you are breaking the Single Responsibility principle and you are writing un-testable code.
The most important thing to consider when writing tests is: "There is no magic key to write tests". There are a lot of tools out there to help you write tests but the real effort should be put in writing testable code rather than trying to hack our existing code to write a test which usually ends up being an integration test instead of a unit-test.
Creating a new object inside a constructor is one of the first big signals that your code is not testable.
These links helped me a lot when I was making the transition to start writing tests and let me tell you that after you start, that will become a natural part of your daily work and you will love the benefits of writing tests I can not picture myself writing code without tests anymore
Clean code guide (used in Google): http://misko.hevery.com/code-reviewers-guide/
To get more information read the following:
http://misko.hevery.com/2008/09/30/to-new-or-not-to-new/
and watch this video cast from Misko Hevery
http://www.youtube.com/watch?v=wEhu57pih5w&feature=player_embedded
Edited:
This article from Martin Fowler explain the difference between a Classical and a Mockist TDD approach
http://martinfowler.com/articles/mocksArentStubs.html
As a summary:
Classic TDD approach: This implies to test everything you can without creating substitutes or doubles (mocks, stubs, dummies) with the exception of external services like web services or databases. The Classical testers use doubles for the external services only
Benefits: When you test you are actually testing the wiring logic of your application and the logic itself (not in isolation)
Cons: If an error occurs you will see potentially hundreds of tests failing and it will be hard to find the code responsible
Mockist TDD approach: People following the Mockist approach will test in isolation all the code because they will create doubles for every dependency
Benefits: You are testing in isolation each part of your application. If an error occurs, you know exactly where it occurred because just a few tests will fail, ideally only one
Cons: Well you have to double all your dependencies which makes tests a little bit harder but you can use tools like AutoFixture to create doubles for the dependencies automatically
This is another great article about writing testable code
http://www.loosecouplings.com/2011/01/how-to-write-testable-code-overview.html
There are some downsides.
First, when you have a test that depends on an external component (like a live database), that test is no longer really predictable. It can fail for any number of reasons - a network outage, a changed password on the database account, missing some DLLs, etc. So when your test suddenly fails, you cannot be immediately sure where the flaw is. Is it a database problem? Some tricky bug in your class?
When you can immediately answer that question just by knowing which test failed, you have the enviable quality of defect localization.
Secondly, if there is a database problem, all your tests that depend on it will fail at once. This might not be so severe, since you can probably realize what the cause is, but I guarantee it will slow you down to examine each one. Widespread failures can mask real problems, because you don't want to look at the exception on each of 50 tests.
And I know you want to hear about factors besides the execution time, but that really does matter. You want to run your tests as frequently as possible, and a longer runtime discourages that.
I have two projects: one with 600+ tests that run in 10 seconds, one with 40+ tests that runs in 50 seconds (this project does actually talk to a database, on purpose). I run the faster test suite much more frequently while developing. Guess which one I find easier to work with?
All of that said, there is value in testing external components. Just not when you're unit-testing. Integration tests are more brittle, and slower. That makes them more expensive.
Accessing the database in unit tests has the following consequences:
Performance. Populating the database and accessing it is slow. The more tests you have, the longer the wait. If you used mocking your controller tests may run in a couple of milliseconds each, compared to seconds if it was using the database directly.
Complexity. For shared databases, you'll have to deal with concurrency issues where multiple agents are running tests against the same database. The database needs to be provisioned, structure needs to be created, data populated etc. It becomes rather complex.
Coverage. You mind find that some conditions are nearly impossible to test without mocking. Examples may include verifying what to do when the database times out. Or what to do if sending an email fails.
Maintenance. Changes to your database schema, especially if its frequent, will impact almost every test that uses the database. In the beginning when you have 10 tests it may not seem like much, but consider when you have 2000 tests. You may also find that changing business rules and adapting the tests to be more complex, as you'll have to modify the data populated in the database to verify the business rule.
You have to ask whether it is worth it for testing business rules. In most cases, the answer may be "no".
The approach I follow is:
Unit classes (controllers, service layers etc) by mocking out dependencies and simulating conditions that may occur (like database errors etc). These tests verify business logic and one aims to gain as much coverage of decision paths as possible. Use a tool like PEX to highlight any issues you never thought of. You'll be surprised how much robust (and resilient) your application would be after fixing some of the issues PEX highlights.
Write database tests to verify that the ORM I'm using works with the underlying database. You'll be surprised the issues EF and other ORM have with certain database engines (and versions). These tests are also useful to for tuning performance and reducing the amount of queries and data being sent to and from the database.
Write coded UI tests that automates the browser and verifies the system actually works. In this case I would populate the database before hand with some data. These tests simply automate the tests I would have done manually. The aim is to verify that critical pieces still work.
With MSpec is it possible to create data driven tests?
For example, NUnit has the TestCase attribute that allows for multiple data driven cases.
[TestFixture]
public class ExampleOfTestCases
{
[TestCase(1,2,3)]
[TestCase(3,3,6)]
[TestCase(2,2,4)]
public void when_adding_two_numbers(int number1, int number2, int expected)
{
Assert.That(number1 + number2, Is.EqualTo(expected);
}
}
That's not possible. I would advise against driving MSpec with data, use NUnit or MbUnit if you need row tests or combinatorial tests (and MSpec when you describe behavior).
Follow-up: Aeden, TestCases/RowTests are not possible with MSpec and likely will never be. Please use NUnit for such cases, as it is the best tool for that job. MSpec excels when you want to specify system behavior (When an order is submitted => should notify the fulfilment service). For TestCases with MSpec you would need to create a context for every combination of inputs which might lead to class explosion.
MSpec is also great when you want to have a sane test structure that is easy to learn. Instead of starting with a blank sheet of paper (think NUnit's [Test] methods) MSpec gives you a template (Establish, Because, It) that you can build your specifications around. Contrast this to the example you give where Arrange, Act and Assert are combined into one line of code.
I have been working with Coded UI Test(CUIT) feature of VS2010 .
When recording the CodedUI framework generates a lots of hierarchical classes.
I was wondering whether coding(by hand) a CUIT would reduce the code created and would it be as optimized(in searching elements) as generated code??
Also what are the scenarios where a CUIT could be coded by hand?
CUITe (Coded UI Test enhanced) Framework is for people who prefer hand coding.
http://cuite.codeplex.com/
CUITe is a thin layer developed on top of Microsoft Visual Studio Team Test's Coded UI Test engine which helps reduce code, increases readability and maintainability, while also providing a bunch of cool features for the automation engineer.
CUITe allows you to define a much simpler Object Repository (== UIMap). Each page/window will be defined in a separate class file, and each UI control definition will be just a one liner. You can move common controls to a parent class which increases maintainability. You can also categorize the page/window definition classes into different folders as you deem fit.
I have been working on Coded UI, from my understanding recorded/generated code is too complex and difficult to maintain.
I always use hand coding, which is simple and easy to maintain.
Here is full sample hand coded UI script for Silver-light application
[TestMethod]
public void SilverlightHANDCODINGTest()
{
BrowserWindow br = BrowserWindow.Launch(#"http://localhost:1377/SilverlightApplication1TestPage.html");
UITestControl sCustom = new UITestControl(br);
sCustom.TechnologyName = "Web";
sCustom.SearchProperties.Add("ControlType", "Custom");
sCustom.SearchProperties.Add("TagName", "OBJECT");
sCustom.SearchProperties.Add("Type", "application/x-silverlight-2");
sCustom.SearchProperties.Add("TagName", "OBJECT");
// sCustom.DrawHighlight();
SilverlightControl sframe = new SilverlightControl(sCustom);
sframe.TechnologyName = "Silverlight";
sframe.SearchProperties.Add(SilverlightControl.PropertyNames.MaxDepth, "-1");
sframe.DrawHighlight();
SilverlightEdit sTextBox = new SilverlightEdit(sCustom);
sTextBox.TechnologyName = "Silverlight";
sTextBox.DrawHighlight();
Playback.Wait(2000);
sTextBox.SetProperty(SilverlightEdit.PropertyNames.Text, "Thank god");
SilverlightButton sButton = new SilverlightButton(sCustom);
sButton.TechnologyName = "Silverlight";
sButton.SearchProperties.Add(SilverlightButton.PropertyNames.DisplayText, "Button");
sButton.DrawHighlight();
Playback.Wait(2000);
Mouse.Click(sButton);
SilverlightComboBox sComboBox= new SilverlightComboBox(sCustom);
sComboBox.TechnologyName = "Silverlight";
sComboBox.DrawHighlight();
Playback.Wait(2000);
sComboBox.SetProperty(SilverlightComboBox.PropertyNames.SelectedItem,"Kishore");
}
Thanks,
You may hand write less code but its going to likely be less maintainable and more prone to breaking. You can use the partial class to effectively override the search clauses after the code has been generated.
I'm making my first steps in Test Driven Development with Visual Studio. I have some questions regarding how to implement generic classes with VS 2010.
First, let's say I want to implement my own version of an ArrayList.
I start by creating the following test (I'm using in this case MSTest):
[TestMethod]
public void Add_10_Items_Remove_10_Items_Check_Size_Is_Zero() {
var myArrayList = new MyArrayList<int>();
for (int i = 0; i < 10; ++i) {
myArrayList.Add(i);
}
for (int i = 0; i < 10; ++i) {
myArrayList.RemoveAt(0); //should this mean RemoveAt(int) or RemoveAt(T)?
//VS doesn't know. Any work arounds?
}
int expected = 0;
int actual = myArrayList.Size;
Assert.AreEqual(expected, actual);
}
I'm using VS 2010 ability to hit
ctrl + .
and have it implement classes/methods on the go.
I have been getting some trouble when implementing generic classes. For example, when I define an .Add(10) method, VS doesn't know if I intend a generic method(as the class is generic) or an Add(int number) method. Is there any way to differentiate this?
The same can happen with return types. Let's assume I'm implementing a MyStack stack and I want to test if after I push and element and pop it, the stack is still empty. We all know pop should return something, but usually, the code of this test shouldn't care for it. Visual Studio would then think that pop is a void method, which in fact is not what one would want. How to deal with this? For each method, should I start by making tests that are "very specific" such as is obvious the method should return something so I don't get this kind of ambiguity? Even if not using the result, should I have something like int popValue = myStack.Pop() ?
How should I do tests to generic classes? Only test with one generic kind of type? I have been using ints, as they are easy to use, but should I also test with different kinds of objects? How do you usually approach this?
I see there is a popular tool called TestDriven for .NET. With VS 2010 release, is it still useful, or a lot of its features are now part of VS 2010, rendering it kinda useless?
Whenever I define a new property in my test code, and ask VS to generate that method stub for me, it generates both a getter and a setter. If I have something like int val = MyClass.MyProperty i'd like to to understand that (at least yet) I only want to define a getter.
Thanks
I see there is a popular tool called TestDriven for .NET. With VS 2010 release, is it still useful, or a lot of its features are now part of VS 2010, rendering it kinda useless?
It's still useful in case you use one of a number of different unit testing frameworks (nunit, mbunit, xunit, csunit, etc).
There are also other tools (like Visual Nunit) that provide visual studio integration for running unit tests.
To your code sample, why would you have a method RemoveAt(T obj)?
You can do RemoveAt(int index) and Remove(T obj) instead. Take a look at Microsoft's APIs (for example, for List<T>) that see how they set up the Remove methods for a generic collection.
And now for your points:
1: What would Add(int number) do? If I understand your intentions correctly, Add(10) can only be intepreted as "Add value 10 at the end of my collection". If you wanted to add a value at a particular index, you can (and probably should) name that method Insert: Insert(int index, T value).
2: sure, Visual Studio will interpret the method as void at first, but you can edit it to be something like
public class MyStack<T>
{
public T Pop()
{
}
}
The stubs built by pressing Ctrl+. are a convenience, but not gospel. You don't HAVE to always assign a return value to a variable. If you don't need it in a test, don't do it. If you want VS to pick up on a return type other than void, you can write a different unit test (e.g. that Pop() returns the last pushed value).
3: I'd test with the types that I see most frequently used in my code. If you're writing a public API, then test with as many types as possible. If you're using NUnit, look into using the [TestCase] attribute to help you avoid writing some duplicate code.
4: I still use TestDriven, but I haven't tried going without it, so I can't really make a useful comparison.
5: Just delete the setter if you don't need it. Some addon frameworks like ReSharper support more advanced code generation, including read-only properties.