Let us say I have a login method that's working swimmingly. It's used in quite a number of places and saves plenty of lines of retyped code. For all intents and purposes, it does not make sense to make this method any shorter.
Is it proper to write a negative test by using exception handling? For example, if I wanted to ensure that a disabled user is not able to log into the system, is it too awkward to write something like:
begin
login(username, password)
fail
rescue Exception
page.should have_text('Sorry, your account is disabled!')
end
If login actually succeeds, the test should fail. And if we reach the exception, we find our error message and the test passes.
I'm wondering if this is too "clever" and may cause confusion. What's the best way of handling a negative test case here?
You would be testing the expected behavior so there would be no rescuing within your specs. In this case, you would want to disable the account in a before caluse and then ensure the exception is raised when you attempt to perform an operation (login, etc.).
Testing anything for the generic Exception is also a bad idea - you want to be very specific about the expected exception as a different issue could cause your specs to pass.
The code would look something like
expect { login(username, password) }.to raise_error LoginException
Related
I am writing a custom matcher for my logging output, mostly so that I can customize the error output to be more readable and helpful.
The thing being examined for the test is the array returned by LoggingSpecHelper.log_events, a module class method (i.e. not a module instance method). Therefore, it is available without the need for it to be passed as a parameter.
LoggingSpecHelper.log_events is kind of long to specify in each expectation, and anyway I'd prefer to hide that implementation detail from the caller, in case that implementation changes. That leads to my using expect(:logging), where :logging is a dummy value which has no meaning and is not examined. This, however, is awkward and confusing, leaving the reader scratching their head thinking "he's examining a symbol?"
Here is an example of how it is currently called; in this case I'm looking to see if a fatal error occurs in the log that contains 'something is misconfigured' (not a real production message):
expect(:logging).to have_log_output_match(:fatal, 'something is misconfigured')
The long form of this, and what is happening in the matcher, is below, but if I do this instead of calling the matcher, I don't get to see what is in the log if an error occurs (the matcher includes log content in the failure message):
expect(LoggingSpecHelper.log_events.count do |event|
event.level == :fatal && /something is misconfigured/.match?(event.message)
end).to be == 1
How would you suggest I handle this? I don't think shared examples are what I want here because this log expectation might not be the only expectation in the example.
As an aside, I should probably rename the matcher so that it is clear that I am testing for 1 and not >=1 occurrences.
I find mocking things with RSpec to be entirely problematic and I often don't know how much code to include, in terms of it being diagnostic. So I'll start with the situation I have and the code that I've isolated as causing the problem.
I have tests where I need to mock a browser. I have a mock driver I set up like this:
require "watir"
def mock_driver
browser = double("watir")
allow(browser).to receive(:is_a?).with(Watir::Browser).and_return(true)
allow(browser).to receive(:driver).and_return(true)
browser
end
The only problems I have in my test suite are these two tests:
context "an empiric driver is requested" do
it "a watir browser is provided" do
allow(Watir::Browser).to receive(:new).and_return(Empiric.browser)
Empiric.set_browser mock_driver
end
it "the requested watir browser can be shut down" do
#allow(Empiric.browser).to receive(:quit)
Empiric.quit_browser
#allow(mock_browser).to receive(:new).and_return(Empiric.browser)
#Empiric.set_browser mock_driver
end
end
(The commented out bits in the second test are on purpose to illustrate what's going on.)
With that one line in place in the second test, I get the following error on that test:
<Double "watir"> was originally created in one example but has leaked into another
example and can no longer be used. rspec-mocks' doubles are designed to only last for
one example, and you need to create a new one in each example you wish to use it for.
If I entirely comment out the first test above, that error doesn't happen so I know I've isolated the two tests that are interacting with each other.
Okay, now notice the final line of my second test that is commented out. That seems to be what the error is indicating to me. It's saying I need to create a new double in the other. Okay, so I'll change my last test:
it "the requested watir browser can be shut down" do
#allow(Empiric.browser).to receive(:quit)
Empiric.quit_browser
#allow(mock_browser).to receive(:new).and_return(Empiric.browser)
Empiric.set_browser mock_driver
end
So here I've uncommented the last line so I'm establishing the mock_driver in that test and not allowing the code to leak.
That, however, returns exactly the same error on exactly the same test.
I'm not sure if it would help to see the methods that are being called in that test, but here they are. First is set_browser:
def set_browser(app = :chrome, *args)
#browser = Watir::Browser.new(app, *args)
Empiric.browser = #browser
end
And here is quit_browser:
def quit_browser
#browser.quit
end
The fact that RSpec thought one test was "leaking" into the other made me think that perhaps my #browser instance was the problem, essentially being what's persisting between the two tests. But I don't see how to get around that. I thought that maybe if I quit the browser in the first test, that would help. So I changed the first test to this:
it "a watir browser is provided" do
Empiric.quit_browser
allow(Watir::Browser).to receive(:new).and_return(Empiric.browser)
Empiric.start_browser mock_driver
end
That, however, led to the above error being shown on both tests now.
My more likely accurate guess is that I simply don't know how to provide a mock in this context.
I think you have to use allow with the mock and not Watir::Browser.
For example, what happens if you allow the mock browser to receive whatever calls the browser would and have the it return the mock browser?
Right now you're allowing the "Watir::Browser" to receive those messages and that's returning an "Empiric.browser". Looking at your code, I understand why you put that in there but I think that might be what's screwing you up here.
Mocks in RSpec are horrible things that rarely if ever work correctly in situations like this. I would entirely recommend not using the mock_driver that you have set up. Rather, for each of your tests just do something similar to what you are doing in the mock_driver. My guess is you're including the mock driver as part of a shared context and that, too, is another thing that is very fragile in RSpec. Not recommended.
Instead you might want to use contexts to break up your tests. Then for each context block have a before block. I'm not sure if you should use before:all or before:each given that you're simulating a browser. But that way you can set up the browser in the before and tear it down in an after.
But I would recommend getting it working in each test individually first. Even if it's a lot of code duplication. Then once all tests are passing, refactor to put the browser stuff in those before/after blocks.
But, again, don't use mocks. Don't use shared contexts. It never ends well and honestly it makes your tests harder to reason about.
Given some advice from Micah, I wanted to provide an answer with a solution. I ended up doing this:
context "an empiric driver is requested" do
it "a watir browser is provided" do
allow(Watir::Browser).to receive(:new).and_return(Empiric.browser)
allow(Empiric.browser).to receive(:driver).and_return(true)
expect { Empiric.start_browser :some_browser }.not_to raise_error
end
it "the requested watir browser can be shut down" do
allow(Empiric.browser).to receive(:quit)
allow(Watir::Browser).to receive(:new).and_return(Empiric.browser)
allow(Empiric.browser).to receive(:driver).and_return(true)
expect { Empiric.quit_browser }.not_to raise_error
end
end
All of that was needed as it is or I would get some error or other. I removed my mock driver and, per Micah's suggestion, simply tried to incorporate what seemed to work. The above "contraption" is what I ended up with as the sweet spot.
This works in the sense of giving coverage of the methods in question. What was interesting was that I had to add this to my RSpec configuration:
RSpec.configure do |config|
config.mock_with :rspec do |mocks|
mocks.allow_message_expectations_on_nil = true
end
end
I needed to do this because RSpec was reporting that I was calling allowing something that was nil to receive a value.
This brought up some interesting things, if you think about it. I have a test that is clearly passing. And it adds to my code coverage. But is it actually testing the quit action on a browser? Well, not really since it was testing a quit action on something that it thought was nil.
But -- it does work. And it must be calling the lines of code in question because the code coverage, as reported my SimpleCov, indicates that the statements in question have been checked.
Is there any advantage to do:
ApiTokenExpired = Class.new(StandardError)
...
raise ApiTokenExpired if response.errorCode == 429
over this lazier alternative:
raise 'api token expired' if response.errorCode == 429
considering that this error detection happens only once in the code?
There is an important advantage of custom errors like ApiTokenExpired if you have to handle the error in an specific way somewhere in the call stack.
begin
# ...
rescue ApiTokenExpired => error
# handle the specific error
rescue => error
# default error handling
end
IMO the decision to create custom errors doesn't depend on the size of the project or future maintenance.
Since custom error classes cause an extra effort I don't use it by default. When a special error handling is needed then an error class should be introduced.
With the first one, you can selectively rescue that error by the class when that code is embedded in larger software. You can still do that with the second one using pattern match with the error message string, but in general, you might alter the message format over time due to refactoring (as opposed to unlikelihood of changing the exception class with your first form), and there can also be different exceptions that have a similar message string, both of which can break pattern matching. When you consider future maintenance, the first one is better.
It does if your app is designed to have error handling or plan to in the future(I mean, any successfull system will want that one day, right?), the only thing you can do with a string is read it in a log or display it in an error message, with types you can code all sorts of things in a decoupled fashion.
For example, ApiTokenExpired could maybe be handled with "try again" response since it's an external problem or be handled by the steps necessary to renew the API Token(and if you used a Typed Exception, you can plug it in later!), maybe you want all errors that are considered SEVERE or UNEXPECTED to send the system administrator an e-mail and also store statistics on the frequency of the error, typed exceptions allows you to code in anything you want in terms of what to do when your system is misbehaving.
Background:
In my Oracle database, I have plenty of database calls which can cause exceptions. I currently have exception handlers for all these, which call an error package. To cut a long story short, a raise_application_error is eventually raised, for expected errors, or a raise for unexpected errors, and this is sent back to the calling Java Groovy/Grails application layer.
So, for example, if a user enters an id and clicks search, I run a select query from the database. If the id doesn't exist, I have a NO_DATA_FOUND exception which performs a raise_application_error with a custom error message (i.e. "ID entered cannot be found.")
However, the application development team say they're struggling with this. They are trying to perform unit testing in Groovy and ideally want a variable returned. The SQL exceptions I am currently returning cause all tests to fail as it is an exception. Their code looks like this:
void nameOfProcedure() {
String result = storedProcedure.callDBProcedure(ConnectionType.MSSQL, val1, val2)
log.info "SQL Procedure query result value: "+ result
assertEquals("1", result)
}
They can add something like this above the test:
#Test (expected = SQLException.class)
But this means all returning SQLExceptions will pass, regardless of whether they are the right exceptions for the issue at hand.
Question:
What is the best solution to this issue? I'm being pressed to return variables from my exception blocks, rather than raise_application_errors - but I'm very reluctant to do this, as I've always been told this is simply terrible practice. Alternatively, they could make changes on their end, but are obviously reluctant to.
What's the next step? Should I be coding to return "expected" errors as variables, as opposed to exceptions? For example, if someone enters an ID that isn't found:
BEGIN
SELECT id
FROM table
WHERE id = entered_id
EXCEPTION
WHEN NO DATA FOUND THEN
RETURN 'ID cannot be found';
END
Or alternatively, should they be following a guide like this which advises using Hamcrest matchers to create their own custom exception property, which they can check against in their JUnit testing. What is best practice here?
You're right, it's terrible practice. It just 'wagging the dog'; they're being lazy to work good and wish you to spoil application design in order to please them.
Generally, unit test with exception returned should looks something like this:
try {
String result = callDBProcedure();
fail("Result instead of exception");}
catch (OracleSQLException e) {
assertEquals(e.errorCode, RAISE_APPLICATION_ERROR_CODE);}
catch (Throwable t) {
fail("Unexpected error");
}
They can upgrade this as they wish. For example, they can develop procedure 'call the SP and convert exception to anything they wish' and use it in their tests. But they should not affect application design outside testing. Never.
Assuming I have a WebCrawler class. There are several errors it can encounter. How should I propagate the errors upward?
Using exceptions:
class WebCrawler
class UrlBadFormatError < StandardError; end
class PageNotFoundError < StandardError; end
class UnauthorizedError < StandardError; end
def crawl(url)
if(! url =~ /some_format/)
raise UrlBadFormatError
response = get(url)
if(response.code == 404)
raise PageNotFoundError
if(response.code == 403)
raise UnauthorizedError
...
end
end
or constants:
class WebCrawler
URL_BAD_FORMAT = 1
PAGE_NOT_FOUND = 2
UNAUTHORZIED = 3
def crawl(url)
if(! url =~ /some_format/)
return URL_BAD_FORMAT
response = get(url)
if(response.code == 404)
return PAGE_NOT_FOUND
if(response.code == 403)
return UNAUTHORZIED
...
end
end
or symbols:
class WebCrawler
def crawl(url)
if(! url =~ /some_format/)
return :url_bad_format
response = get(url)
if(response.code == 404)
return :page_not_found
if(response.code == 403)
return :unauthorized
...
end
end
which is best? or it depends(on what?)
For something which indicates programmer error, such as the wrong type of argument passed to a method, definitely throw an exception. The exception will crash the program, drawing the programmer's attention to the fact that they are using your class incorrectly, so they can fix the problem. In this case, returning an error code wouldn't make sense, because the program will have to include code to check the return value, but after the program is debugged, such errors shouldn't ever happen.
In your WebCrawler class, is it expected that crawl will receive a bad URL as an argument sometimes? I think the answer is probably no. So raising an exception would be appropriate when a bad URL is passed.
When an exception is raised, the flow of execution suddenly "jumps" to the innermost handler. This can be a useful way to structure code when the exception is not expected to happen most of the time, because you can write the "main flow" of your method as simple, straight-line code without including a lot of details about what will happen when some rare error condition occurs. Those details can be separated from the "main flow" code, and put in an exception handler. When an error condition is expected to happen under normal conditions, though, it can be better to put the error handling code inline with the "main flow", to make it clearer what is going on. If the control flow of your program "jumps around" (as is the case when exceptions are used for normal flow control), that means the reader also has to "jump around" in the program text as they are figuring out how it works.
For the other two, I think it is expected that at least sometimes, the HTTP request will return an error code. To determine whether an exception or special return value is the best way to indicate such a condition, I would think about how often those conditions are going to happen under normal usage. Think also about how the client code will read either way. If you use exceptions, they will have to write something like:
urls.map do |url|
begin
crawl(url)
rescue PageNotFoundError
""
rescue UnauthorizedError
""
end
end
(By the way, I think this code example shows something: it might be a good idea if both of your custom exceptions inherit from a common superclass, so you can catch both of them with a single rescue clause if desired.) Or if you use error codes, it would look something like:
urls.map do |url|
response = crawl(url)
if [:page_not_found, :unauthorized].include? response
""
else
response
end
end
Which do you think reads better? It's really up to you. The one thing which you do not want to do is use integer constants for errors. Why use integers? When you print them in a debug trace, you'll have to go look at the list of constants to see what each one means. And using symbols is just as efficient computationally.
Why wouldn't you throw exceptions? They can encapsulate additional information besides just the type, are trivially rescued, and if you're using an IDE, are first-class citizens.
If it's an exception then by all means raises an exception! All three of those cases are, in my opinion, exceptions. While some may argue that 4xx status codes aren't exception-worthy since you may expect them to happen, they are still client errors.
You may also read about Ruby's throw/catch, which offer exception-like behavior for cases where "don't use exceptions for control flow" applies (though I don't think that's the case here).
You should raise errors. If you encounter a malformed URL, or if the page isn't found, or if you weren't authorized to access the page, it means you cannot continue crawling. Raising an error or exception returns from the method and lets the caller deal with the unusual situation.
It should also include information about the error, such as error codes, the URL which resulted in an error and any other relevant information. It can help in deciding how best to handle the error and can later be formatted into a helpful message for the user.
What you should not do, ever, is return numeric error codes. Ruby is not C. Just use symbols instead.
I am against the use of exceptions upon encountering 403s, 404s, malformed urls and similar common occurences on the web. Exceptions are meant for "internal" errors. In the World Wild Web, bad URLs are entirely unexceptional. There should be a method(s) for handling each different URL disease. I would personally return special values as symbols, or some "SpecialCase" objects recording what happened. There is also underused catch...throw statement.