I'm trying to run a scenario several (30) times in order to get a nice statistical sample. However the block is only executing once; each subsequent time results in the scenario being called and not executing (although it says that the scenario did successfully complete with a time of around 5 ms).
Around('#mass_benchmark') do |scenario, block|
$seconds_taken = "SECONDS TAKEN NOT SET"
#time_array = []
30.times do
before_hook(scenario)
block.call
after_hook(scenario)
#time_array << $seconds_taken
end
write_time_array_to_file(#time_array, scenario_name)
end
The tag #mass_benchmark executes this block, as opposed to ~#mass_benchmark, which just executes the scenario normally. The methods before_hook and after_hook replicate the Before ('~#mass_benchmark') and After ('~#mass_benchmark') hooks (which actually just call the same method).
The variable $seconds_taken is set around the specific area for which I am timing. I am not timing the whole test there, just a critical portion of it; the remainder of the test is getting to that point, etc, which is not to be part of the timed portion, so I cannot just move the timing portion outside of this.
The issue may be with something I'm doing in those methods, but as far as I can tell, everything works normally (as indicated by well-placed puts statements). Any ideas are appreciated!
Currently Cucumber does not seem to support calling the block twice in an around hook. This can be demonstrated by the following feature file:
Feature: This scenario will print a line
Scenario: Print a line
When I print a line
And step definitions:
Around do |scenario, block|
Kernel.puts "START AROUND, status=#{scenario.status}"
block.call
Kernel.puts "BETWEEN CALLS, status=#{scenario.status}"
block.call
Kernel.puts "END AROUND, status=#{scenario.status}"
end
When /^I print a line$/ do
Kernel.puts "IN THE STEP DEFINITION"
end
When this is executed, Cucumber will print:
Scenario: Print line # features/test1.feature:3
START AROUND, status=skipped
IN THE STEP DEFINITION
When I print a line # features/test.rb:9
BETWEEN CALLS, status=passed
When I print a line # features/test.rb:9
END AROUND, status=passed
Evidently since the status of the scenario is already "passed", Cucumber does not re-execute it, though the output formatter receives the steps. I have not found any way to "reset" the status in the scenario API to get them to be re-run.
There are other problems with around hooks as well, for example you cannot set variables to the World in around hooks (like you can in before hooks). See also Cucumber issues 52 and 116 for more gory details.
One possibility might be to keep the passed-in block as it is, and call the ".call" method on a duplicate?
Something like (untested):
Around do |scenario, block|
30.times do
duplicate = block.dup
before_hook(scenario)
duplicate.call
after_hook(scenario)
end
end
Just make sure not to use ".clone" on the block, since clone will create an object with the same Id, resulting in every change made to the duplicate also affecting the original.
Related
I would like to keep track at runtime of the RSpec test currently being executed, preferably including its file name and line number, so that if a test hangs I can know where to find it. I'm thinking of simply writing the file name and line number to a file in a before(:each).
I realize I can see in the output the most recently executed test, but I would like to know the currently executing test.
before(:each) yields an RSpec::Core::Example instance representing the current test/example, which should contain the information you're after.
before(:each) do |example|
puts example.description
puts example.location
puts example.metadata
end
I'm trying to get some easy-to-read time metrics for how long steps of each scenario take to run.
I have methods that insert rows into a logging table like this:
puts "#{Time.now} - Starting Given"
puts "#{Time.now} - Ending Given"
etc...
And I want them to be called automatically at the start and end of each step, but I can only find ways to run them before or after the entire scenario, rather than each step.
Is there a way to do this?
Ruby comes with a benchmarking module to handle this stuff pretty easily for you.
Edit to address the comments below
In Cucumber, there is a hook called Around that allows you to wrap your scenario however you wish. For example, straight from the docs:
Around('#fast') do |scenario, block|
Timeout.timeout(0.5) do
block.call #actually runs the scenario
end
end
I am working with a custom testing framework and we are trying to expand some of the assert functionality to include a custom error message if the assert fails. The current assert is called like this:
assert_compare(first_term, :neq, second_term) do
puts 'foobar'
end
and we want something with the functionality of:
assert_compare(first_term, :neq, second_term, error_message) do
puts 'foobar'
end
so that if the block fails the error message will describe the failure. I think this is ugly, however, as the framework we are moving away from did this and i have to go through a lot of statements that look like:
assert.compare(variable_foo['ARRAY1'][2], variable_bar['ARRAY2'][2], 'This assert failed because someone did something unintelligent when writing the test. Probably me, since in am the one writing this really really long error statement on the same line so that you have to spend a quarter of your day scrolling to the side just to read it')
This type of method call makes it difficult to read, even when using a variable for the error message. I feel like a better way should be possible.
assert_compare(first_term, :neq, second_term) do
puts 'foobar'
end on_fail: 'This is a nice error message'
This, to me, is the best way to do it but i don't know how or if it is even possible to accomplish this in ruby.
The goal here is to make it as aesthetic as possible. Any suggestions?
You could make on_fail a method of whatever assert_compare returns and write
assert_compare(first_term, :neq, second_term) do
puts 'foobar'
end.on_fail: 'This is a nice error message'
In short, no. Methods in ruby take a block as the final parameter only. As Chuck mentioned you could attempt to make the on_fail method a method of whatever assert_compare returns and that is a good solution. The solution I've come up with is not what you are looking for, but it works:
def test block, param
block.call
puts param
end
test proc { puts "hello"}, "hi"
will result in
"hello"
"hi"
What I've done here is create a Proc (which is essentially a block) and then passed it as a regular parameter.
I recently asked how to test in RSpec if a block was called and the answers to that question seem to work in a simple case. The problem is when the initialization with the block is more complex. Then it is done in before and reused by a number of different tests in the context, among them the one testing if the block was evaluated. See the example:
context "the node definition using block of code" do
before do
#n=node do
# this block should be called
end
# some more complex setup concerning #n
end
it "should call the block" do
# how to test it?
end
# here a bunch of other tests using #n
end
In this case the solution with side effect changing value of a local variable does not work. Raising an exception from the block is useless since the whole statement must be properly evaluated to be used by the other tests.
Obviously I could do the tests separately, but it seems to stink, since I must copy-paste the initialization part and since the was-the-block-called test inherently belongs to this very context.
How to test if the block was evaluated in such a case?
Explanation for question asked by #zetetic below.
The context is that I'm implementing a kind of DSL, with nodes defined by their parameters and blocks of code (that can define something else in the scope of node). Since the things defined by the node's block can be pretty generic, at least for the first attempt I just need to be sure the block is evaluated and that what a user provides there will be considered. For now does not matter what it is.
Probably I should refactor my tests now and using mocks make them test behaviors rather then implementation. However it will be a little bit tricky, for the sake of some mixins and dynamic handling of messages sent to objects. For now the cincept of such tests is a little bit fuzzy in my head ;-)
Anyway your answers and comments helped me to better understand how RSpec works and explained why what I'm trying to do looks as if it did not fit to the RSpec.
Try something like this (untested by me):
context "the node definition using block of code" do
let(:node){
node = Node.new "arg1", "arg2", node_block
# more complex stuff here
node
}
context "checking the block is called" do
let(:node_block) {
double = double("node_block")
double.should_receive("some kind of arg").and_return("something")
# this will now cause a fail if it isn't called
double
}
it "should call the block" do
node.blah()
end
end
let(:node_block) {
# some real code
}
subject { node.blah() }
it { should == 2 }
# ...
end
So that's a very shaky piece of code (you'll have to fill in the gaps as you didn't give very much to go on, and let is obviously a lambda too, which could mean you've got to play around with it a bit) that uses let and a double to check it's called, and avoids using before, which is really for side effects not setting up variables for use in the specs.
#zetetic makes a very insightful comment that you're not testing behaviour here. I'm not against using rspec for doing more unit test style stuff (guidelines are made to be broken), but you might ask how later tests will pass when using a real block of code if that block isn't being called? In a way, I'm not even sure you need to check the block is called, but only you know.
In Cucumber, in my env.rb file, I have a before & after hook set up (well, a few of them, some linked to specific tags) but have found the after hooks don't output anything when I put a puts in them.
For example, this works:
Before do
puts "before the scenario"
end
but this doesn't:
After do
puts "after the scenario"
end
It seems that the after hooks do get run (as there's a particular line I'm having a problem with in the after hook & in trying to debug that, I've found this problem) but they're just not outputting anything.
All my searching has proved fruitless, can't find anyone else with similar problems. Can anyone tell if I'm doing something wrong?
Cucumber overrides the puts message in the RbWorld class so that anything written with puts gets properly broadcasted to all formatters. In the case of the pretty formatter, these go into a delayed_messages collection, until it calls print_messages, which it appears to do after each step name has been printed, presumably so that messages appear to be nested under the step in which they were generated.
For some reason there is no final call to print_messages in the pretty formatter, I'm not sure if it's an omission or deliberate since it would look less 'pretty' to have spurious messages in the output.
Interestingly if you add a second scenario, you'll see 'after the scenario' printed as the first message when the second scenario gets run, that's the delayed_messages collection in action.
In summary, you're doing nothing wrong, it's just how Cucumber hijacks the puts method. If you're not too bothered about these messages being nicely formatted, then you can just replace puts with STDOUT.puts to bypass Cucumber's formatting.
Before do
p "Go!"
puts "Go!"
end
After do
p "Stop!"
puts "Stop!"
$stdout.puts "Stop!"
end
output of this snippet may help to understand why 'puts' not working in After hook.