There's some tests that sometimes pass, and sometimes fail. I'd like to fix them, but I'm not able to at the moment, for reasons beyond the scope of this question. Are there any alternatives to pending or skip for them in RSpec 3?
pending isn't suitable, because as of the current version of RSpec, when the tests pass, RSpec will tell me that they passed, and therefore shouldn't be marked as pending, and mark the build as broken.
skip isn't suitable. I only use skip to avoid specs that cause the suite to crash. If the tests consistently stop failing, I'd like to know that that's the case.
I'd like something that runs the tests, displays whether it passes or not, but doesn't cause the build to be broken whether they pass or fail.
An additional gem to add this behaviour is ok.
There's nothing built in, but it's trivial to do this yourself:
module AllowFailure
def allow_failure(reason)
yield
rescue Exception => e
pending(reason)
raise
end
end
RSpec.configure do |config|
config.include AllowFailure
end
Then just wrap the body of the flickering tests with allow_failure(reason) { ... }.
(Caveat: The code above is off-the-cuff and I haven't tried it so it may not be exactly correct -- but it should be close).
Related
I know that in RSpec you can easily mark an example as incomplete by omitting the block:
it 'is not a complete example'
But is it possible to get RSpec to fail any example which does not contain any expectations?
IE. RSpec will happily pass this example:
it 'does not have any expectations but still passes' do
end
I want to emulate the behavior of PHPUnit which fails any test not containing any assertions.
There isn't a built-in way, but it's a feature I'd like to see added at some point.
In the meantime, it's not too hard to add yourself:
http://blog.sorah.jp/2012/12/17/rspec-warn-for-no-expectations
I encountered an issue with some Web UI Automation using RSpec and Selenium/Capybara/SitePrism. It's related to the combination of before :all and before :each clauses that occur in the code, and perhaps especially in my spec_helper file.
Up until now, I've been running rspec against one spec file at a time. Each spec file required a spec_helper file, which included the following:
spec_helper.rb:
RSpec.configure do |config|
config.before(:all) do
# 1) Code to configure WebDriver and launch Browser is here
end
end
The spec files themselves contain their own before blocks for various reasons. Most of them contain something like the following:
test_a_spec.rb:
describe "Page A" do
before :all do
# 2) Log in to web site, maybe load the test page in question
end
it "does this thing" do
# 3) Test this thing
end
it "does that thing" do
# 4) Test that thing
end
end
This worked fine as long as I was running RSpec against individual spec files. When I tried tagging some of my examples and then running against the whole spec folder, I had a problem. The before :all block in spec_helper.rb didn't prepend itself to every file, as I thought it would, but instead ran once at the beginning. All the spec files after the first were expecting a clean browser to be launched by spec_helper and to do the log in part themselves, but the browser wasn't clean and was already logged in, so that wasn't good.
Changing the before :all in spec_helper.rb to a before :each seemed like the natural solution, so I did, and suddently my tests instantly failed with an error claiming that rack-test requires a rack application, but none was given. This happened to some of my tests but not all, and through process of elimination I realized it was only tests that had their own before :all blocks that were failing. It appears that the before :all in the spec file is superceding the before :each in the spec_helper file. So it was trying to log in before it had launched a browser.
This vexes me. I am terribly vexed. I had sort of assumed two things:
I assumed before :each was on equal footing with before :all, and in fact imagine it like a before :all plus more, in a sense. The idea that a before :all would supercede a before :each seems weird to me.
I assume all these before blocks were subject to nesting in a reasonable way. That is, a before :all block should fire once before all the things below it, which would mean firing within each iteration of a before :each block that might contain the before :all block. Also, if a before :each contains some code and then a describe statement below it has a before :all, the before :each code should still fire before the before :all code. Maybe it does do that, I'm just not sure at this point.
Questions:
1) What kind of behavior is my spec_helper file actually producing? If I were to take that same behavior and put it into one of the spec files itself, for instance, what would that look like? Would the before :each with the browser launch code wrap around all the code in the spec file in some sort of implicit describe block? Does the before :each get inserted into the outermost describe block, and therefore have to compete with the spec file's before :all block?
2) I could "solve" this by abandoning all before :all blocks in my tests, but I like the flexibility of the way things are, and since this is UI Automation and speed is a factor, I don't really want to be opening a new browser and logging in for each describe, even though I'm aware that this would be ideal for keeping every test separated from the others. Do I have to/want to do just that?
Thanks!
To address the part of your question not already covered by https://stackoverflow.com/a/22489263/1008891, the require method will not reload any file already loaded previously, as described in https://stackoverflow.com/a/22489263/1008891. This contrasts with the load method, which acts like a "copy/paste" of the file contents wherever it is used.
I'm not saying that if you changed all your requires to loads that you'd get the same result as running the individual files, as there may be other global state that affects the behavior you're seeing.
The RSpec documentation (http://rspec.info/documentation/3.3/rspec-core/RSpec/Core/Hooks.html#before-instance_method) provides some very useful information about how before blocks interact and the order in which they are run, but they do recommend avoiding before(:context) [aka before(:all)].
FYI, note that the after blocks are run in the reverse order fin which before blocks are.
Scenario
I'm writing a library (no Ruby on Rails) for which I'd like to have very detailed Cucumber features. This especially includes describing errors/exceptions that should be thrown in various cases.
Example
The most intuitive way to write the Cucumber steps would probably be something like
When I do something unwanted
Then an "ArgumentError" should be thrown
Problem
There are two issues I have to address:
The first step should not fail when an exception is thrown.
The exception that the first step throws should be accessible to the second step in order to do some assertion magic.
Unelegant And Cumbersome Solution
The best approach I've been able to come up with is caching the exception in the first step and putting it into an instance variable that the second step can access, like so:
When /^I do something unwanted$/ do
begin
throw_an_exception!
rescue => #error
end
end
Then /^an "(.*)" should be thrown$/ do |error|
#error.class.to_s.should == error
end
However, this makes the first step more or less useless in cases where I don't want it to fail, and it requires an instance variable, which is never a good thing.
So, can anyone help me out with an at least less cumbersome solution? Or should I write my features differently anyway? Any help would be much appreciated.
I thought about it once more, and maybe the answer is:
There is no elegant solution, because the Given-When-Then-Scheme is violated in your case.
You expect that "Then an exception should be thrown" is the outcome of "When I do something unwanted".
But when you think about it, this is not true! The exception is not the outcome of this action, in fact the exception just shows that the "When"-Statement failed.
My solution to this would be to test at a higher level:
When I do something unwanted
Then an error should be logged
or
When I do something unwanted
Then the user should get an error message
or
When I do something unwanted
Then the program should be locked in state "error"
or a combination of these.
Then you would "cache the exception" in your program - which makes perfect sense, as you most likely need to do that anyway.
The two problems you've stated would be solved, too.
In case you really must test for exceptions
Well, i guess then cucumber isn't the right test suite, hmm? ;-)
As the Given-When-Then-Scheme is violated anyway, I would simply write
When I do something unwanted it should fail with "ArgumentError"
and in the step definitions something like (untested, please correct me if you try it)
When /^I do something unwanted it should fail with "(.*)"$/ do |errorstring|
expect {
throw_an_exception!
}.to raise_error(errorstring)
end
As said above, that is horribly wrong as the scheme is broken, but it would serve the purpose, wouldn't it? ;-)
You'll find further documentation at testing errors at rspec expectations.
One option is to mark the scenario with #allow-rescue and check the page's output and status code. For example
In my_steps.rb
Then(/^the page (?:should have|has) content (.+)$/) do |content|
expect(page).to have_content(content)
end
Then(/^the page should have status code (\d+)$/) do |status_code|
expect(page.status_code.to_s).to eq(status_code)
end
Then /^I should see an error$/ do
expect(400..599).to include(page.status_code)
end
In my_feature.feature
#allow-rescue
Scenario: Make sure user can't do XYZ
Given some prerequisite
When I do something unwanted
Then the page should have content Routing Error
And the page should have status code 404
or alternatively:
#allow-rescue
Scenario: Make sure user can't do XYZ
Given some prerequisite
When I do something unwanted
Then I should see an error
This may not be exactly what you were hoping for, but it might be an acceptable workaround for some people who come across this page. I think it will depend on the type of exception, since if the exception is not rescued at any level then the scenario will still fail. I have used this approach mostly for routing errors so far, which has worked fine.
It is possible to raise an exception in a When block and then make assertions about it in the following Then blocks.
Using your example:
When /^I do something unwanted$/ do
#result = -> { throw_an_exception! }
end
Then /^an "(.*)" should be thrown$/ do |error|
expect{ #result.call }.to raise_error(error)
end
That example uses RSpec's matchers but the important part is the -> (Lambda); which allows the reference to the throw_an_exception! method to be passed around.
I hope that helps!
I'm answering from the perspective of someone who uses Cucumber features in a Behavior-Driven Development situation, so take it or leave it...
Scenarios should be written to test a 'feature' or functionality of the application, as opposed to being used to test the code itself. An example being:
When the service is invoked
Then a success code should be returned
It sounds like your test case (i.e. If I do this, then this exception should be thrown) is a candidate for unit or integration testing - in my case, we would use some Mocking or unit testing framework.
My suggestion would be to re-evaluate your feature scenarios to see if they are really testing what you intend them to test. From personal experience, I've found that if my test classes are becoming abnormally complex, then my features are 'wrong.'
When running RSpec tests in Ruby on Rails 2.3 with ActiveRecord, the database gets rolled back to the state after a before :all block after each example (it block).
However, I want to spec the lifecycle of an object, which means going through a number of examples one by one, changing the state and testing postconditions. This is impossible with the rollback behaviour.
So to clarify:
describe MyModel
before :all { #thing = MyModel.create }
it "should be settable" do
lambda { #thing.a_number = 42 }.should_not raise_exception
end
it "should remember things" do
#thing.a_number.should == 42
# this fails because the database was rolled back ☹
end
end
Is there some way to persist changes made in examples?
I agree with normalocity, in this case it looks like you would be better off with a single spec containing two assertions.
There are cases in which it is helpful to turn off rollbacks, e.g. for higher level tests with Capybara and Selenium, in which case you can use the use_transactional_fixtures configuration option. You can put thi
RSpec.configure do |config|
config.use_transactional_fixtures = false
end
Well, that depends on what you're trying to do. If you're testing the life cycle (a series of things that happen over time), that's more the realm of integration tests, which you can build more in tools such as Cucumber, etc. Spec is more designed to do small tests of small bits of code.
It's technically possible for you to simply write a long spec test, with multiple .should statements, and so long as all of them pass, then you've effectively got the kind of test you're describing. However, that's not really, in my experience, what spec is designed to give you.
I guess what I'm saying is, don't try to prevent the rollback - that's not what it's there to do. Either use a tool more designed to do the kinds of tests you're looking to build, or write a longer test that has multiple .should statements.
So, here's the deal. I'm currently working in a Ruby on Rails environment and have been for ~1 year now. Before that I was in C++/Java land for almost a decade. I'm (still) trying to figure out what the Ruby way is when it comes to asserts.
I'm not worried about the technical detail. I know TestUnit has asserts which can be used in the testing environment and I know I can add my own assert methods to my Ruby project and use them in production Rails to lock down known conditions. The question is: What is the Ruby way for ensuring something in code that I know should/not happen?
For the record, I've been asserting in tests and raising in production. I still can't help but miss my production asserts...
Asserts really shouldn't be used in production code for two reasons.
assert x is very functional, and as such hard to read. Using a raise/if combo adds readability.
assert doesn't make it clear what error will be raised if the condition fails. While,
raise ObscureButInformitiveError if condition
lets the application layers further up do something relveant. Such as emailing an admin, or writing to a perticular log.
Let the error happen, then check the logs for what went wrong, then fix it.
Rails catches all uncaught exceptions automatically, it will only mess up the single request the error happened in.
There's no official non-test assertions in Ruby, but there are gems.
For instance Jim Weirich's Given looks promising. Although its main focus is testing environments (rspec / minitest), but it also:
... provides three assertions meant to be used in
non-test/non-spec code. For example, here is a square root function
decked out with pre and post-condition assertions.
require 'given/assertions'
require 'given/fuzzy_number'
include Given::Assertions
include Given::Fuzzy
def sqrt(n)
Precondition { n >= 0 }
result = Math.sqrt(n)
Postcondition { result ** 2 == about(n) }
result
end
To use the
non-testing assertions, you need to require the 'given/assertions'
file and then include the Given::Assertions module into what ever
class is using the Precondition / Postcondition / Assert methods. The code
block for these assertions should always be a regular Ruby true/false
value (the should and expect methods from RSpec are not available).
Note that this example also uses the fuzzy number matching, but that
is not required for the assertions themselves.