I am wonder if there is an easy way to finish a test after an Assert has failed. We used to use Galileo for all of our automated tested, but we have moved the Visual Studio Test framework. We had a method that would allow a test to fail, but continue on.
public static bool DoAssertAndContinue(Action assert)
{
try
{
assert();
return true;
}
catch (AssertionException ae)
{
ConfigContext.WriteLine(ae.Message);
return false;
}
}
This is what we used before...and it would be called like this:
assertionResults.Add(Automation.Utils.CommonMethods.DoAssertAndContinue(() => Assert.IsTrue(detail.ValidateName(boo, urns))));
I am just trying to figure out the best way to emulate what we had before without having to refactor all of our tests.
Instead of AssertionException, you should now catch UnitTestAssertException which is the base exception for all mstest assert failures.
You can use Try/Catch in MSTest as well. In Catch block you can catch the specific error and print it using Console.Write to know about the error. I would recommend you look into this thread for more details.
EDIT 1: Personally I don't use try/catch to sake of passing my test method. I write a test method to find defect in actual product. So if you are expecting that your calling method will give you some specific exception then I would suggest to use ExpectedException attribute. This is applicable if you are running your test method for single test data.
Now if you want to pass multiple test data into your test method. Then I would suggest go for Data driven test cases. Here you can keep all your test data inside a XML or XLS or in a DB. Then using that input file you can feed multiple test data into your test method. Try not to use try/catch here and for any test data if your calling method send you some exception then see if MSTest will handle it and move to the next test data or not. If it moves then in Test result window you will be able to see why for that specific test data your method get failed. For data driven concept see this link
The MsTest framework throws an exception on a failed assertion. It's propagation out of the unit test method is what causes the test to fail. If you want to continue on a failed assertion you just need to handle that exception and prevent it from escaping the method.
Related
I have some tests in ruby which call some framework methods and classes. The problem I'm facing, is that some methods can throw exceptions, since they contact services not in my control. I want in the teardown method of the test to actually have the result of the test (success, failed with XXX), so I can do some stuff based on that. Is there a way I can do that (different from wrapping the whole test in a begin/rescue block) ?
A code example of what you're trying to test might help. But depending on the framework API you're working with you can stub out the method that tries to contact the service, that would raise the exception. If you are calling one method that you are trying to test, and it also raises an exception (not very DRY/SRP code) then it makes it tricky. If that's the case, can either do a begin/rescue around that method call, or stub out the call to that larger method. The problem with doing the latter is it can obscure the value of your test more.
I have an odd looking test spec below.
expect(function(){
expect(null).not.toEqualMoment(testContext.moment1);
}).toThrow();
The inner expectation fails because of the check below inside my custom matcher, which throws an exception.
if(!moment.isMoment(actual)) {
throw new Error(_.string.sprintf('Actual: %s , is not a Moment object.', jasmine.pp(actual)));
}
I think an exception has to be thrown here and not just return a failing result because if not,
expect(null).not.toEqualMoment(null) will return true. ( If that makes sense to you ) .
So,
If I fail custom matchers with exceptions, how can I test this?
If you want to test your custom matcher, several problems occur when you do this by just apply your matchers.
For most simple custom matchers, you can just do expect(actual).toBeCustom(expected). If the test passes, the matcher is ok.
For the inverted case, just add .not.
However, in this way you are not testing the internal workings of the matcher, like what message is returned when a matcher fails, or - in your case - that the matcher throws and exception. You tried to test this by nesting expect calls, which is not possible.
Therefore, you need to add the matcher itself to your test and test the compare method.
expect(toEqualMomentMatcher.compare({}, momentjs())).toThrow('Actual: Object({ }) is not a Moment object');
Apart from your question, I advice you to let the matcher fail if actual is not a Moment. Just return a different message when actual is not a Moment.
First of all, I have to say, I'm new to mocking. So maybe I'm missing a point.
I'm also just starting to get used to the TDD approach.
So, in my actual project I'm working on a class in the business layer, while the data layer has yet to be deployed. I thought, this would be a good time to get started with mocking. I'm using Rhino Mocks, but I've come to the problem of needing to know the implementation details of a class before writing the class itself.
Rhino Mocks checks if alle the methods expected to be called are actually called. So I often need to know which mocked method is being called by the tested method first, even though they could be called in any order. Because of that I'm often writing complicated methods before I test them, because then I know already in which order the methods are being called.
simple example:
public void CreateAandB(bool arg1, bool arg2) {
if(arg1)
daoA.Create();
else throw new exception;
if(arg2)
daoB.Create();
else throw new exception;
}
if I want to test the error handling of this method, I'd have to know which method is being called first. But I don't want to be bugged about implementation details when writing the test first.
Am I missing something?
You have 2 choices. If the method should result in some change in your class the you can test the results of your method instead. So can you call CreateAandB(true,false) then then call some other method to see if the correct thing was created. In this situation your mock objects will probably be stubs which just provide some data.
If the doaA and doaB are objects which are injected into your class that actually create data in the DB or similar, which you can't verify the results of in the test, then you want to test the interaction with them, in which case you create the mocks and set the expectations, then call the method and verify that the expectations are met. In this situation your mock objects will be mocks and will verify the expected behaviour.
Yes you are testing implementation details, but your are testing the details of if your method is using its dependencies correctly, which is what you want to test, not how it is using them, which are the details you are not really interested in.
EDIT
IDao daoA = MockRepository.GenerateMock<IDao>(); //create mock
daoA.Expect(dao=>dao.Create); //set expectation
...
daoA.VerifyExpectations(); //check that the Create method was called
you can ensure that the expectations happen in a certain order, but not using the AAA syntax I believe (source from 2009, might have changed since,EDIT see here for an option which might work), but it seems someone has developed an approach which might allow this here. I've never used that and can't verify it.
As for needing to know which method was called first so you can verify the exception you have a couple of choices:
Have a different message in your exception and check that to determine which exception was raised.
Expect a call to daoA in addition to expecting the exception. If you don't get the call to daoA then the test fails as the exception must have been the first one.
Often times you just need fake objects, not mocks. Mock objects are meant to test component interaction, and often you can avoid this by querying the state of SUT directly. Most practical uses of mocks are to test interaction with some external system (DB, file system, webservice, etc.), and for other things you should be able to query system state directly.
I have got method which is using Assembly.LoadFrom(...) statement and returns the supported cultures from that satellite assembly, so how could I write unit tests for that type of methods.
What I did was, wrapped that static method/logic to return cultures in anther class and used it's instance method. Is this the right approach?
Is this the situation?
aMethod(whatToLoad) {
// other stuff
x = Assembly.LoadFrom( whatToLoad );
// code using x
}
First principle: We are focusing on testing aMethod(), the testing of Assembly.LoadFrom() is a separate problem. While we are building tests for aMethod() we don't try to test its dependencies.
So here what kind of tests might we need?
That we pass the right value for whatToLoad
That we correctly store/use the value returned
That we correctly handle errors or exceptions thrown from Assembly.LoadFrom()
It is easiest to do this if the test can supply a Mock implementation. Then we can test 1 by checking that the Mock received the expected value. Test 2 by returning a well defined value (or for mulitiple tests different interesting values) Test 3 by generating chosen error conditions.
So you have changed your code to something like this:
aMethod(loader, whatToLoad) {
// other code
x = loader.Load( whatToLoad );
// code using x
}
Maybe the loaded is injected in some other way, but the point is that you can now specify different tests my setting up a suitable loaoder. For example, for the first test.
testLoader = new MockLoaderThatRembers();
tested.aMethod(testLoader, loadThis);
assertEquals(testLoader.getLoader(), loadThis);
So if that's the kind of thing you are doing then yes, I'd say you're enabling TDD.
Visual Studio Test can check for expected exceptions using the ExpectedException attribute. You can pass in an exception like this:
[TestMethod]
[ExpectedException(typeof(CriticalException))]
public void GetOrganisation_MultipleOrganisations_ThrowsException()
You can also check for the message contained within the ExpectedException like this:
[TestMethod]
[ExpectedException(typeof(CriticalException), "An error occured")]
public void GetOrganisation_MultipleOrganisations_ThrowsException()
But when testing I18N applications I would use a resource file to get that error message (any may even decide to test the different localizations of the error message if I want to, but Visual Studio will not let me do this:
[TestMethod]
[ExpectedException(typeof(CriticalException), MyRes.MultipleOrganisationsNotAllowed)]
public void GetOrganisation_MultipleOrganisations_ThrowsException()
The compiler will give the following error:
An attribute argument must be a
constant expression, typeof expression
or array creation expression of an
attribute
Does anybody know how to test for an exception that has a message from a resource file?
One option I have considered is using custom exception classes, but based on often heard advice such as:
"Do create and throw custom exceptions
if you have an error condition that
can be programmatically handled in a
different way than any other existing
exception. Otherwise, throw one of the
existing exceptions." Source
I'm not expecting to handle the exceptions differently in normal flow (it's a critical exception, so I'm going into panic mode anyway) and I don't think creating an exception for each test case is the right thing to do. Any opinions?
I would recommend using a helper method instead of an attribute. Something like this:
public static class ExceptionAssert
{
public static T Throws<T>(Action action) where T : Exception
{
try
{
action();
}
catch (T ex)
{
return ex;
}
Assert.Fail("Exception of type {0} should be thrown.", typeof(T));
// The compiler doesn't know that Assert.Fail
// will always throw an exception
return null;
}
}
Then you can write your test something like this:
[TestMethod]
public void GetOrganisation_MultipleOrganisations_ThrowsException()
{
OrganizationList organizations = new Organizations();
organizations.Add(new Organization());
organizations.Add(new Organization());
var ex = ExceptionAssert.Throws<CriticalException>(
() => organizations.GetOrganization());
Assert.AreEqual(MyRes.MultipleOrganisationsNotAllowed, ex.Message);
}
This also has the benefit that it verifies that the exception is thrown on the line you were expecting it to be thrown instead of anywhere in your test method.
The ExpectedException Message argument does not match against the message of the exception. Rather this is the message that is printed in the test results if the expected exception did not in fact occur.
Just an opinion, but I would say the error text:
is part of the test, in which case getting it from the resource would be 'wrong' (otherwise you could end up with a consistantly mangled resource), so just update the test when you change the resource (or the test fails)
is not part of the test, and you should only care that it throws the exception.
Note that the first option should let you test multiple languages, given the ability to run with a locale.
As for multiple exceptions, I'm from C++ land, where creating loads and loads of exceptions (to the point of one per 'throw' statement!) in big heirachies is acceptable (if not common), but .Net's metadata system probably doesn't like that, hence that advice.
I think you can just do an explicit try-catch in your test code instead of relying on the ExpectedException attribute to do it for you. Then you can come up with some helper method that will read the resource file and compare the error message to the one that comes with the exception that was caught. (of course if there wasn't an exception then the test case should be considered a fail)
If you switch over to using the very nice xUnit.Net testing library, you can replace [ExpectedException] with something like this:
[Fact]
public void TestException()
{
Exception ex = Record.Exception(() => myClass.DoSomethingExceptional());
// Assert whatever you like about the exception here.
}
I wonder if NUnit is moving down the path away from simplicity... but here you go.
New enhancements (2.4.3 and up?) to the ExpectedException attribute allow you more control on the checks to be performed on the expected Exception via a Handler method. More Details on the official NUnit doc page.. towards the end of the page.
[ExpectedException( Handler="HandlerMethod" )]
public void TestMethod()
{
...
}
public void HandlerMethod( System.Exception ex )
{
...
}
Note: Something doesn't feel right here.. Why are your exceptions messages internationalized.. Are you using exceptions for things that need to be handled or notified to the user. Unless you have a bunch of culturally diverse developers fixing bugs.. you shouldn't be needing this. Exceptions in English or a common accepted language would suffice. But in case you have to have this.. its possible :)
I came across this question while trying to resolve a similar issue on my own. (I'll detail the solution that I settled on below.)
I have to agree with Gishu's comments about internationalizing the exception messages being a code smell.
I had done this initially in my own project so that I could have consistency between the error messages throw by my application and in my unit tests. ie, to only have to define my exception messages in one place and at the time, the Resource file seemed like a sensible place to do this since I was already using it for various labels and strings (and since it made sense to add a reference to it in my test code to verify that those same labels showed in the appropriate places).
At one point I had considered (and tested) using try/catch blocks to avoid the requirement of a constant by the ExpectedException attribute, but this seemed like it would lead to quite a lot of extra code if applied on a large scale.
In the end, the solution that I settled on was to create a static class in my Resource library and store my exception messages in that. This way there's no need to internationalize them (which I'll agree doesn't make sense) and they're made accessible anytime that a resource string would be accessible since they're in the same namespace. (This fits with my desire not to make verifying the exception text a complex process.)
My test code then simply boils down to (pardon the mangling...):
[Test,
ExpectedException(typeof(System.ArgumentException),
ExpectedException=ProductExceptionMessages.DuplicateProductName)]
public void TestCreateDuplicateProduct()
{
_repository.CreateProduct("TestCreateDuplicateProduct");
_repository.CreateProduct("TestCreateDuplicateProduct");
}