So, here's the deal. I'm currently working in a Ruby on Rails environment and have been for ~1 year now. Before that I was in C++/Java land for almost a decade. I'm (still) trying to figure out what the Ruby way is when it comes to asserts.
I'm not worried about the technical detail. I know TestUnit has asserts which can be used in the testing environment and I know I can add my own assert methods to my Ruby project and use them in production Rails to lock down known conditions. The question is: What is the Ruby way for ensuring something in code that I know should/not happen?
For the record, I've been asserting in tests and raising in production. I still can't help but miss my production asserts...
Asserts really shouldn't be used in production code for two reasons.
assert x is very functional, and as such hard to read. Using a raise/if combo adds readability.
assert doesn't make it clear what error will be raised if the condition fails. While,
raise ObscureButInformitiveError if condition
lets the application layers further up do something relveant. Such as emailing an admin, or writing to a perticular log.
Let the error happen, then check the logs for what went wrong, then fix it.
Rails catches all uncaught exceptions automatically, it will only mess up the single request the error happened in.
There's no official non-test assertions in Ruby, but there are gems.
For instance Jim Weirich's Given looks promising. Although its main focus is testing environments (rspec / minitest), but it also:
... provides three assertions meant to be used in
non-test/non-spec code. For example, here is a square root function
decked out with pre and post-condition assertions.
require 'given/assertions'
require 'given/fuzzy_number'
include Given::Assertions
include Given::Fuzzy
def sqrt(n)
Precondition { n >= 0 }
result = Math.sqrt(n)
Postcondition { result ** 2 == about(n) }
result
end
To use the
non-testing assertions, you need to require the 'given/assertions'
file and then include the Given::Assertions module into what ever
class is using the Precondition / Postcondition / Assert methods. The code
block for these assertions should always be a regular Ruby true/false
value (the should and expect methods from RSpec are not available).
Note that this example also uses the fuzzy number matching, but that
is not required for the assertions themselves.
Related
Scenario
I'm writing a library (no Ruby on Rails) for which I'd like to have very detailed Cucumber features. This especially includes describing errors/exceptions that should be thrown in various cases.
Example
The most intuitive way to write the Cucumber steps would probably be something like
When I do something unwanted
Then an "ArgumentError" should be thrown
Problem
There are two issues I have to address:
The first step should not fail when an exception is thrown.
The exception that the first step throws should be accessible to the second step in order to do some assertion magic.
Unelegant And Cumbersome Solution
The best approach I've been able to come up with is caching the exception in the first step and putting it into an instance variable that the second step can access, like so:
When /^I do something unwanted$/ do
begin
throw_an_exception!
rescue => #error
end
end
Then /^an "(.*)" should be thrown$/ do |error|
#error.class.to_s.should == error
end
However, this makes the first step more or less useless in cases where I don't want it to fail, and it requires an instance variable, which is never a good thing.
So, can anyone help me out with an at least less cumbersome solution? Or should I write my features differently anyway? Any help would be much appreciated.
I thought about it once more, and maybe the answer is:
There is no elegant solution, because the Given-When-Then-Scheme is violated in your case.
You expect that "Then an exception should be thrown" is the outcome of "When I do something unwanted".
But when you think about it, this is not true! The exception is not the outcome of this action, in fact the exception just shows that the "When"-Statement failed.
My solution to this would be to test at a higher level:
When I do something unwanted
Then an error should be logged
or
When I do something unwanted
Then the user should get an error message
or
When I do something unwanted
Then the program should be locked in state "error"
or a combination of these.
Then you would "cache the exception" in your program - which makes perfect sense, as you most likely need to do that anyway.
The two problems you've stated would be solved, too.
In case you really must test for exceptions
Well, i guess then cucumber isn't the right test suite, hmm? ;-)
As the Given-When-Then-Scheme is violated anyway, I would simply write
When I do something unwanted it should fail with "ArgumentError"
and in the step definitions something like (untested, please correct me if you try it)
When /^I do something unwanted it should fail with "(.*)"$/ do |errorstring|
expect {
throw_an_exception!
}.to raise_error(errorstring)
end
As said above, that is horribly wrong as the scheme is broken, but it would serve the purpose, wouldn't it? ;-)
You'll find further documentation at testing errors at rspec expectations.
One option is to mark the scenario with #allow-rescue and check the page's output and status code. For example
In my_steps.rb
Then(/^the page (?:should have|has) content (.+)$/) do |content|
expect(page).to have_content(content)
end
Then(/^the page should have status code (\d+)$/) do |status_code|
expect(page.status_code.to_s).to eq(status_code)
end
Then /^I should see an error$/ do
expect(400..599).to include(page.status_code)
end
In my_feature.feature
#allow-rescue
Scenario: Make sure user can't do XYZ
Given some prerequisite
When I do something unwanted
Then the page should have content Routing Error
And the page should have status code 404
or alternatively:
#allow-rescue
Scenario: Make sure user can't do XYZ
Given some prerequisite
When I do something unwanted
Then I should see an error
This may not be exactly what you were hoping for, but it might be an acceptable workaround for some people who come across this page. I think it will depend on the type of exception, since if the exception is not rescued at any level then the scenario will still fail. I have used this approach mostly for routing errors so far, which has worked fine.
It is possible to raise an exception in a When block and then make assertions about it in the following Then blocks.
Using your example:
When /^I do something unwanted$/ do
#result = -> { throw_an_exception! }
end
Then /^an "(.*)" should be thrown$/ do |error|
expect{ #result.call }.to raise_error(error)
end
That example uses RSpec's matchers but the important part is the -> (Lambda); which allows the reference to the throw_an_exception! method to be passed around.
I hope that helps!
I'm answering from the perspective of someone who uses Cucumber features in a Behavior-Driven Development situation, so take it or leave it...
Scenarios should be written to test a 'feature' or functionality of the application, as opposed to being used to test the code itself. An example being:
When the service is invoked
Then a success code should be returned
It sounds like your test case (i.e. If I do this, then this exception should be thrown) is a candidate for unit or integration testing - in my case, we would use some Mocking or unit testing framework.
My suggestion would be to re-evaluate your feature scenarios to see if they are really testing what you intend them to test. From personal experience, I've found that if my test classes are becoming abnormally complex, then my features are 'wrong.'
When running RSpec tests in Ruby on Rails 2.3 with ActiveRecord, the database gets rolled back to the state after a before :all block after each example (it block).
However, I want to spec the lifecycle of an object, which means going through a number of examples one by one, changing the state and testing postconditions. This is impossible with the rollback behaviour.
So to clarify:
describe MyModel
before :all { #thing = MyModel.create }
it "should be settable" do
lambda { #thing.a_number = 42 }.should_not raise_exception
end
it "should remember things" do
#thing.a_number.should == 42
# this fails because the database was rolled back ☹
end
end
Is there some way to persist changes made in examples?
I agree with normalocity, in this case it looks like you would be better off with a single spec containing two assertions.
There are cases in which it is helpful to turn off rollbacks, e.g. for higher level tests with Capybara and Selenium, in which case you can use the use_transactional_fixtures configuration option. You can put thi
RSpec.configure do |config|
config.use_transactional_fixtures = false
end
Well, that depends on what you're trying to do. If you're testing the life cycle (a series of things that happen over time), that's more the realm of integration tests, which you can build more in tools such as Cucumber, etc. Spec is more designed to do small tests of small bits of code.
It's technically possible for you to simply write a long spec test, with multiple .should statements, and so long as all of them pass, then you've effectively got the kind of test you're describing. However, that's not really, in my experience, what spec is designed to give you.
I guess what I'm saying is, don't try to prevent the rollback - that's not what it's there to do. Either use a tool more designed to do the kinds of tests you're looking to build, or write a longer test that has multiple .should statements.
I tend to use before blocks to set instance variables. I then use those variables across my examples. I recently came upon let(). According to RSpec docs, it is used to
... to define a memoized helper method. The value will be cached across multiple calls in the same example but not across examples.
How is this different from using instance variables in before blocks? And also when should you use let() vs before()?
I always prefer let to an instance variable for a couple of reasons:
Instance variables spring into existence when referenced. This means that if you fat finger the spelling of the instance variable, a new one will be created and initialized to nil, which can lead to subtle bugs and false positives. Since let creates a method, you'll get a NameError when you misspell it, which I find preferable. It makes it easier to refactor specs, too.
A before(:each) hook will run before each example, even if the example doesn't use any of the instance variables defined in the hook. This isn't usually a big deal, but if the setup of the instance variable takes a long time, then you're wasting cycles. For the method defined by let, the initialization code only runs if the example calls it.
You can refactor from a local variable in an example directly into a let without changing the
referencing syntax in the example. If you refactor to an instance variable, you have to change
how you reference the object in the example (e.g. add an #).
This is a bit subjective, but as Mike Lewis pointed out, I think it makes the spec easier to read. I like the organization of defining all my dependent objects with let and keeping my it block nice and short.
A related link can be found here: http://www.betterspecs.org/#let
The difference between using instances variables and let() is that let() is lazy-evaluated. This means that let() is not evaluated until the method that it defines is run for the first time.
The difference between before and let is that let() gives you a nice way of defining a group of variables in a 'cascading' style. By doing this, the spec looks a little better by simplifying the code.
I have completely replaced all uses of instance variables in my rspec tests to use let(). I've written a quickie example for a friend who used it to teach a small Rspec class: http://ruby-lambda.blogspot.com/2011/02/agile-rspec-with-let.html
As some of the other answers here says, let() is lazy evaluated so it will only load the ones that require loading. It DRYs up the spec and make it more readable. I've in fact ported the Rspec let() code to use in my controllers, in the style of inherited_resource gem. http://ruby-lambda.blogspot.com/2010/06/stealing-let-from-rspec.html
Along with lazy evaluation, the other advantage is that, combined with ActiveSupport::Concern, and the load-everything-in spec/support/ behavior, you can create your very own spec mini-DSL specific to your application. I've written ones for testing against Rack and RESTful resources.
The strategy I use is Factory-everything (via Machinist+Forgery/Faker). However, it is possible to use it in combination with before(:each) blocks to preload factories for an entire set of example groups, allowing the specs to run faster: http://makandra.com/notes/770-taking-advantage-of-rspec-s-let-in-before-blocks
It is important to keep in mind that let is lazy evaluated and not putting side-effect methods in it otherwise you would not be able to change from let to before(:each) easily.
You can use let! instead of let so that it is evaluated before each scenario.
In general, let() is a nicer syntax, and it saves you typing #name symbols all over the place. But, caveat emptor! I have found let() also introduces subtle bugs (or at least head scratching) because the variable doesn't really exist until you try to use it... Tell tale sign: if adding a puts after the let() to see that the variable is correct allows a spec to pass, but without the puts the spec fails -- you have found this subtlety.
I have also found that let() doesn't seem to cache in all circumstances! I wrote it up in my blog: http://technicaldebt.com/?p=1242
Maybe it is just me?
Dissenting voice here: after 5 years of rspec I don't like let very much.
1. Lazy evaluation often makes test setup confusing
It becomes difficult to reason about setup when some things that have been declared in setup are not actually affecting state, while others are.
Eventually, out of frustration someone just changes let to let! (same thing without lazy evaluation) in order to get their spec working. If this works out for them, a new habit is born: when a new spec is added to an older suite and it doesn't work, the first thing the writer tries is to add bangs to random let calls.
Pretty soon all the performance benefits are gone.
2. Special syntax is unusual to non-rspec users
I would rather teach Ruby to my team than the tricks of rspec. Instance variables or method calls are useful everywhere in this project and others, let syntax will only be useful in rspec.
3. The "benefits" allow us to easily ignore good design changes
let() is good for expensive dependencies that we don't want to create over and over.
It also pairs well with subject, allowing you to dry up repeated calls to multi-argument methods
Expensive dependencies repeated in many times, and methods with big signatures are both points where we could make the code better:
maybe I can introduce a new abstraction that isolates a dependency from the rest of my code (which would mean fewer tests need it)
maybe the code under test is doing too much
maybe I need to inject smarter objects instead of a long list of primitives
maybe I have a violation of tell-don't-ask
maybe the expensive code can be made faster (rarer - beware of premature optimisation here)
In all these cases, I can address the symptom of difficult tests with a soothing balm of rspec magic, or I can try address the cause. I feel like I spent way too much of the last few years on the former and now I want some better code.
To answer the original question: I would prefer not to, but I do still use let. I mostly use it to fit in with the style of the rest of the team (it seems like most Rails programmers in the world are now deep into their rspec magic so that is very often). Sometimes I use it when I'm adding a test to some code that I don't have control of, or don't have time to refactor to a better abstraction: i.e. when the only option is the painkiller.
let is functional as its essentially a Proc. Also its cached.
One gotcha I found right away with let... In a Spec block that is evaluating a change.
let(:object) {FactoryGirl.create :object}
expect {
post :destroy, id: review.id
}.to change(Object, :count).by(-1)
You'll need to be sure to call let outside of your expect block. i.e. you're calling FactoryGirl.create in your let block. I usually do this by verifying the object is persisted.
object.persisted?.should eq true
Otherwise when the let block is called the first time a change in the database will actually happen due to the lazy instantiation.
Update
Just adding a note. Be careful playing code golf or in this case rspec golf with this answer.
In this case, I just have to call some method to which the object responds. So I invoke the _.persisted?_ method on the object as its truthy. All I'm trying to do is instantiate the object. You could call empty? or nil? too. The point isn't the test but bringing the object ot life by calling it.
So you can't refactor
object.persisted?.should eq true
to be
object.should be_persisted
as the object hasn't been instantiated... its lazy. :)
Update 2
leverage the let! syntax for instant object creation, which should avoid this issue altogether. Note though it will defeat a lot of the purpose of the laziness of the non banged let.
Also in some instances you might actually want to leverage the subject syntax instead of let as it may give you additional options.
subject(:object) {FactoryGirl.create :object}
"before" by default implies before(:each). Ref The Rspec Book, copyright 2010, page 228.
before(scope = :each, options={}, &block)
I use before(:each) to seed some data for each example group without having to call the let method to create the data in the "it" block. Less code in the "it" block in this case.
I use let if I want some data in some examples but not others.
Both before and let are great for DRYing up the "it" blocks.
To avoid any confusion, "let" is not the same as before(:all). "Let" re-evaluates its method and value for each example ("it"), but caches the value across multiple calls in the same example. You can read more about it here: https://www.relishapp.com/rspec/rspec-core/v/2-6/docs/helper-methods/let-and-let
Note to Joseph -- if you are creating database objects in a before(:all) they won't be captured in a transaction and you're much more likely to leave cruft in your test database. Use before(:each) instead.
The other reason to use let and its lazy evaluation is so you can take a complicated object and test individual pieces by overriding lets in contexts, as in this very contrived example:
context "foo" do
let(:params) do
{ :foo => foo, :bar => "bar" }
end
let(:foo) { "foo" }
it "is set to foo" do
params[:foo].should eq("foo")
end
context "when foo is bar" do
let(:foo) { "bar" }
# NOTE we didn't have to redefine params entirely!
it "is set to bar" do
params[:foo].should eq("bar")
end
end
end
I use let to test my HTTP 404 responses in my API specs using contexts.
To create the resource, I use let!. But to store the resource identifier, I use let. Take a look how it looks like:
let!(:country) { create(:country) }
let(:country_id) { country.id }
before { get "api/countries/#{country_id}" }
it 'responds with HTTP 200' { should respond_with(200) }
context 'when the country does not exist' do
let(:country_id) { -1 }
it 'responds with HTTP 404' { should respond_with(404) }
end
That keeps the specs clean and readable.
I really don't see a sane use for these. There is already rescue and raise, so why the need for throw and catch? It seems they are supposed to be used to jump out of deep nesting, but that just smells like a goto to me. Are there any examples of good, clean use for these?
Note: It looks like a few things have changed with catch/throw in 1.9. This answer applies to Ruby 1.9.
A big difference is that you can throw anything, not just things that are derived from StandardError, unlike raise. Something silly like this is legal, for example:
throw Customer.new
but it's not terribly meaningful. But you can't do:
irb(main):003:0> raise Customer.new
TypeError: exception class/object expected
from (irb):3:in `raise'
from (irb):3
from /usr/local/bin/irb:12:in `<main>'
They can be really useful in simplifying DSLs for end users by passing control out of the DSL without the need for complex case / if statements
I have a Ruby app which allows users to extend it via an internal DSL. Some of the functions in the DSL need to return control to specific parts of my application. Let's take a simple example. Suppose the user is developing a simple extension concerning dates
if today is a holiday then
do nothing
end
week_of_year = today.week.number
if week_of_year < 10 then
...
The do nothing bit triggers a throw which passes control out of the exec statement and back to me.
Rather than continuing to execute the DSL, on some condition, we want it to exit and hand control back to my application. Now you could get the user to use lots of embedded if statements and have the DSL end naturally but that just obscures what the logic is trying to say.
Throw really is a goto which is 'considered dangerous' but damn it sometimes they are the best solution.
It's basically a goto, and slightly more akin to a call/cc, except that the control flow is wired up implicitly by name instead of explicitly as a parameter. The difference between throw/catch and raise/rescue is that the former is intended to be used for control flow instead of only exceptional situations, and it doesn't waste time putting together a stack trace.
Sinatra uses throw/catch for HTTP error codes, where a handler can use throw to cede control to the Sinatra library in a structured way. Other sorts of HTTP frameworks use exceptions, or by returning a different class of response, but this lets Sinatra (for example) try another request handler after catching it.
The difference between the two is that you can only 'raise' exceptions but can 'throw' anything (1.9). Other than that, they should be interchangeable, that is, it should be possible to rewrite one with another, just like the example given by #john-feminella.
I'm expanding my Ruby understanding by coding an equivalent of Kent Beck's xUnit in Ruby. Python (which Kent writes in) has an assert() method in the language which is used extensively. Ruby does not. I think it should be easy to add this but is Kernel the right place to put it?
BTW, I know of the existence of the various Unit frameworks in Ruby - this is an exercise to learn the Ruby idioms, rather than to "get something done".
No it's not a best practice. The best analogy to assert() in Ruby is just raising
raise "This is wrong" unless expr
and you can implement your own exceptions if you want to provide for more specific exception handling
I think it is totally valid to use asserts in Ruby. But you are mentioning two different things:
xUnit frameworks use assert methods for checking your tests expectations. They are intended to be used in your test code, not in your application code.
Some languages like C, Java or Python, include an assert construction intended to be used inside the code of your programs, to check assumptions you make about their integrity. These checks are built inside the code itself. They are not a test-time utility, but a development-time one.
I recently wrote solid_assert: a little Ruby library implementing a Ruby assertion utility and also a post in my blog explaining its motivation. It lets you write expressions in the form:
assert some_string != "some value"
assert clients.empty?, "Isn't the clients list empty?"
invariant "Lists with different sizes?" do
one_variable = calculate_some_value
other_variable = calculate_some_other_value
one_variable > other_variable
end
And they can be deactivated, so assert and invariant get evaluated as empty statements. This let you avoid performance problems in production. But note that The Pragmatic Programmer: from journeyman to master recommends against deactivating them. You should only deactivate them if they really affect the performance.
Regarding the answer saying that the idiomatic Ruby way is using a normal raise statement, I think it lacks expressivity. One of the golden rules of assertive programming is not using assertions for normal exception handling. They are two completely different things. If you use the same syntax for the two of them, I think your code will be more obscure. And of course you lose the capability of deactivating them.
Some widely-regarded books that dedicate whole sections to assertions and recommend their use:
The Pragmatic Programmer: from Journeyman to Master by Andrew Hunt and David Thomas
Code Complete: A Practical Handbook of Software Construction by Steve McConnell
Writing Solid Code by Steve Maguire
Programming with
assertions
is an article that illustrates well what assertive programming is about and
when to use it (it is based in Java, but the concepts apply to any
language).
What's your reason for adding the assert method to the Kernel module? Why not just use another module called Assertions or something?
Like this:
module Assertions
def assert(param)
# do something with param
end
# define more assertions here
end
If you really need your assertions to be available everywhere do something like this:
class Object
include Assertions
end
Disclaimer: I didn't test the code but in principle I would do it like this.
It's not especially idiomatic, but I think it's a good idea. Especially if done like this:
def assert(msg=nil)
if DEBUG
raise msg || "Assertion failed!" unless yield
end
end
That way there's no impact if you decide not to run with DEBUG (or some other convenient switch, I've used Kernel.do_assert in the past) set.
My understanding is that you're writing your own testing suite as a way of becoming more familiar with Ruby. So while Test::Unit might be useful as a guide, it's probably not what you're looking for (because it's already done the job).
That said, python's assert is (to me, at least), more analogous to C's assert(3). It's not specifically designed for unit-tests, rather to catch cases where "this should never happen".
How Ruby's built-in unit tests tend to view the problem, then, is that each individual test case class is a subclass of TestCase, and that includes an "assert" statement which checks the validity of what was passed to it and records it for reporting.