I'm adding logging to a test suite. It uses Ruby, Cucumber and Rspec to run. I've already set up an env.rb to log each test and can capture Scenarios and Steps and drop them into the logger but I am at a loss how to grab the actual results.
Each test run will put something like
1 scenario (1 passed)
27 steps (27 passed)
0m0.176s
into the console.
I know I can grab feature name, scenario name, tags and steps using the Before do |scenario| below but I cannot find anyway of accessing the results.
$logger.level = :info
$logger.add_appenders \
Logging.appenders.stdout,
Logging.appenders.file("logging/#{Time.now.strftime("%m-%d-%Y--%H:%M")}.log")
user = `id -un` #Mac specific, sorry windows users
user.delete!("\n")
$logger.debug "Test run by #{user} on #{Time.now.strftime("%m/%d/%Y")}"
Before do |scenario|
#feature_name = scenario.feature.name
#scenario_name = scenario.name
#scenario_tags = scenario.source_tag_names
$logger.info "FEATURE: #{#feature_name}"
$logger.info "SCENARIO: #{#scenario_name}"
$step_index = 0
$stop_count = scenario.test_steps.count
#scenario = scenario
end
AfterStep do |step|
if $step_index < $stop_count
$logger.info "#{#scenario.test_steps[$step_index].text}\n"
end
$step_index += 2
end
It needs to output to a .log which is why I've so far avoided the html report builders.
Found that this does exactly what I needed
AfterConfiguration do |config|
config.on_event :test_case_finished do |event|
p event.result
end
end
Related
I am using rspec to do some data driven testing. My test reads from a csv file, grabs an entry which is inserted into the text box on the page and is then compared to expected text which is also read from the csv file. All this is working as expected, I am able to read and compare without any issues.
Below is my code:
Method for reading csv file:
def user_data
user_data = CSV.read Dir.pwd + '/user_data.csv'
descriptor = user_data.shift
descriptor = descriptor.map { |key| key.to_sym }
user_data.map { |user| Hash[ descriptor.zip(user) ] }
end
Test:
describe "Text box tests" do
before :all do
#homepage = Homepage.new
end
it "should display the correct name" do
visit('http://my test url')
sleep 2
user_data.each do |entry|
#homepage.enter_name(entry[:name])
#homepage.click_go()
sleep 2
begin
expect(page).to have_css("#firstname", text: entry[:expected_name])
end
end
end
end
The problem is with failures. If I have a failure with one of the tests (i.e the expected text is not displayed on the page) then the test stops and all subsequent entries in the csv are not tested. If I put in a rescue after the expect statement like this:
rescue Exception => error
puts error.message
Then the error is logged to the console, however at the end of my test run it says no failures.
So basically I am looking for is, in the event of a failure for my test to keep running(until all entries in the csv have been covered), but for the test run to be marked as failed. Does anyone know how I can achieve this?
Try something like this:
context "when the user is on some page" do
before(:context) { visit('http://example.org/') }
user_data.each do |entry|
it "should display the correct name: #{entry[:name]}" do
#homepage.enter_name(entry[:name])
#homepage.click_go
expect(page).to have_css("#firstname", text: entry[:expected_name])
end
end
end
You will also need to change def user_data to def self.user_data
I would advise mapping over the entries and calling the regular capybara method has_css? instead of the rspec helper method. It would look like this:
results = user_data.map do |entry|
#homepage.enter_name(entry[:name])
#homepage.click_go()
sleep 2
page.has_css?("#firstname", text: entry[:expected_name])
end
expect(results.all?) to be_truthy
if you want to keep track of which ones failed, you cann modify it a bit:
missing_entries = []
user_data.each do |entry|
#homepage.enter_name(entry[:name])
#homepage.click_go()
sleep 2
has_entry = page.has_css?("#firstname", text: entry[:expected_name])
unless has_entry
missing_entries.push entry[:expected_name]
end
end
expect(missing_entries).to be_empty
I'm using the AfterStep hooks inside calabash-ios/cucumber.
I want to know the last executed step inside my hook.
AfterStep do |scenario|
puts "Step: #{scenario.name} #{scenario.title} #{scenario.gherkin_statement}"
end
I can see that the scenario is passed in, but how do I access the currently running step? I don't see any information inside the scenario docs regarding this.
I would assume that the step would be passed into the AfterStep hook. Any clues?
You may refer to this example code, which works with step indexes within the AfterStep hook.
Example:
CALABASH_COUNT = {:step_index => 0, :step_line => nil}
#TODO change this approach as it breaks scenario outlines
Before do |scenario|
begin
CALABASH_COUNT[:step_index] = 0
CALABASH_COUNT[:step_line] = scenario.raw_steps[CALABASH_COUNT[:step_index]].line
rescue Exception => e
puts "#{Time.now} - Exception:#{e}"
end
end
AfterStep do |scenario|
CALABASH_COUNT[:step_index] = CALABASH_COUNT[:step_index] + 1
raw = scenario.raw_steps[CALABASH_COUNT[:step_index]]
CALABASH_COUNT[:step_line] = raw.line unless raw.nil?
end
The Behave BDD framework, in Python allows a more simple step.name type accessor, but others seem more difficult, requiring the above technique of counting the current step and then using the index to find the name from the raw steps text.
While running automated test scripts I will oftentimes get the following warning messages and others:
QFont::setPixelSize: Pixel size <= 0 (0)
QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.
I've searched around and these outputs have nothing to do with my test scripts. They don't impact the results in any way. Thus, I don't wish to see them.
While looking for a way to solve this, I found code which is supposed to redirect stderr to a class which will filter out specific messages. However, when I try to use this code, none of my scripts work.
The class that suppresses the warnings:
class WarningSuppressor
# array to hold warnings to suppress
SUPPRESS_THESE_WARNINGS = [
'QFont::setPixelSize: Pixel size <= 0 (0)',
'QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once.'
]
class << self
def write(message)
if suppress_warning? message
0
end
end
def suppress_warning? message
SUPPRESS_THESE_WARNINGS.any? { |suppressable_warning| message.chomp.include? suppressable_warning }
end
end
end
The configuration code that is supposed to redirect stderr:
Capybara.configure do |config|
config.default_driver = :webkit
config.javascript_driver = :webkit
config.run_server = false # prevents Capybara from booting up a rack application automatically
config.app_host = 'http://local.xxxxxxxx.com'
# Sends output to a custom warning supressor
config.register_driver :webkit do |app|
Capybara::Driver::Webkit.new(app, stderr: WarningSuppressor)
end
# 10 second wait for ajax to finish
config.default_wait_time = 10
end
If I insert a binding.pry in the block statement, app is nil but Capybara::Driver::Webkit exists.
Does anyone have a better way/way to fix this; a method of hiding certain warnings from being displayed while running my automated scripts?
This question sounds stupid, but I never found an answer online to do this.
Assume you have a test suite like this page:
http://en.wikibooks.org/wiki/Ruby_Programming/Unit_testing
or code:
require "simpleNumber"
require "test/unit"
class TestSimpleNumber < Test::Unit::TestCase
def test_simple
assert_equal(4, SimpleNumber.new(2).add(2) )
assert_equal(4, SimpleNumber.new(2).multiply(2) )
end
def test_typecheck
assert_raise( RuntimeError ) { SimpleNumber.new('a') }
end
def test_failure
assert_equal(3, SimpleNumber.new(2).add(2), "Adding doesn't work" )
end
end
Running the code:
>> ruby tc_simpleNumber2.rb
Loaded suite tc_simpleNumber2
Started
F..
Finished in 0.038617 seconds.
1) Failure:
test_failure(TestSimpleNumber) [tc_simpleNumber2.rb:16]:
Adding doesn't work.
<3> expected but was
<4>.
3 tests, 4 assertions, 1 failures, 0 errors
Now, how to use a variable (what kind?) to save the testing results?
e.g., an array like this:
[{:name => 'test_simple', :status => :pass},
{:name => 'test_typecheck', :status => :pass},
{:name => 'test_failure', :status => :fail},]
I am new to testing, but desperate to know the answer...
you need to execute your test script file, that's it, the result will display pass or fails.
Suppose you save file test_unit_to_rspec.rb, after that execute below command
ruby test_unit_to_rspec.rb
Solved the problem with setting a high verbose level, in a test runner call.
http://ruby-doc.org/stdlib-1.8.7/libdoc/test/unit/rdoc/Test/Unit/UI/Console/TestRunner.html
require 'test/unit'
require 'test/unit/ui/console/testrunner'
class MySuperSuite < Test::Unit::TestSuite
def self.suite
suites = self.new("My Super Test Suite")
suites << TestSimpleNumber1
suites << TestSimpleNumber2
return suites
end
end
#run the suite
# Pass an io object
#new(suite, output_level=NORMAL, io=STDOUT)
runner = Test::Unit::UI::Console::TestRunner.new(MySuperSuite, 3, io)
results will be saved in the io stream in a nice format fo each test case.
What about using '-v' (verbose):
ruby test_unit_to_rspec.rb -v
This should show you a lot more information
You can check out another of Nat's posts for a way to capture the results. The short answer to your question is that there is no variable for capturing the results. All you get is:
Loaded suite My Special Tests
Started
..
Finished in 1.000935 seconds.
2 tests, 2 assertions, 0 failures, 0 errors
Which is not very helpful if you want to report to someone else what happened. Nat's other post shows how to wrap the Test::Unit in rspec to get a better result and more flexibility.
class Test::Unit::TestCase
def setup
#id = self.class.to_s()
end
def teardown
#test_result = "pass"
if(#_result.failure_count > 0 || #_result.error_count > 0)
#test_result = "fail"
# making sure no errors/failures exist before the next test case runs.
i = 0
while(i < #_result.failures.length) do
#_result.failures.delete_at(i)
i = i + 1
end
while(i < #_result.errors.length) do
#_result.errors.delete_at(i)
i = i + 1
end
#test_result = "fail"
end # if block ended
puts"#{#id}: #{#test_result}"
end # teardown definition ended
end # class Test::Unit::TestCase ended
Example Output :
test1: Pass
test2: fail
so on....
I'm using a loop to load and execute Ruby scripts in a directory. At the moment the script will load the script, but how do I execute it when the only reference to it is the filename in the form of a string?
Dir.foreach('tests') do |item|
next if item == '.' or item == '..' #removes extra "." or ".."
load dirname + '/' + item #successfully loads the script
if item # the scripts return true/false
numberPassed+=1
else
numberFailed+=1
failed.push(item)
end
numberTested+=1
end
For some reason I'm getting 2 Passed, but it never actually runs the scripts "item" represents.
EDIT: here is an example of a script that would need to be loaded. They all follow this format:
require "watir-webdriver"
class TestScript
puts 'Testing etc etc"...'
browser = Watir::Browser.new :ie
browser.goto "webpage.htm"
browser.text_field(:name => "j_username").set "username"
browser.text_field(:name => "j_password").set "password"
browser.link(:id, "watSubmitLogin").click
browser.wait
browser.link(:id=> 'watCommDir').fire_event("onmouseover")
browser.link(:id=> 'watAddFi').click
browser.wait
...
browser.link(:href, "javascript: submitForm();").click
browser.wait
if browser.text.include?( 'The user already Exists')
puts 'Passed'
browser.close
return true
else
puts 'Failed'
browser.close
return false
end
end
I need to somehow tell the main script whether the sub-scripts pass or fail so I can keep track of how many pass/fail/error/total and create a report of all the tests that failed.
Looks like you are doing acceptance testing with Watir and try to do custom test results reporting.
I would recommend to use existing test runners to run all your tests and build custom output formatter for your needs. Existing test runners already solve a lot of issues you will encounter during creation of your own test runner (like how to run tests from specified folder, how to identify failing/successful test etc).
One of the commmon test runners for acceptance tests in Ruby community is Cucumber. Another good alternative is RSpec. Both these libraries support custom formatters:
In RSpec you would need to subclass RSpec::Core::Formatters::BaseFormatter.
In Cucumber you would need to implement class with methods specified in this documentation.
If you want to stay with the current simple implementation, here is one possible approach that is inpired by ruby Regexps: Inside the test set global variable, e.g. $test_succeeded (like $~, $& etc. global variables generated by ruby regular expressions) and then examine this value in your test runner.
In tests
if browser.text.include?( 'The user already Exists')
puts 'Passed'
browser.close
$test_succeeded = true
# ...
In tests runner
Dir.foreach('tests') do |item|
next if item == '.' or item == '..' #removes extra "." or ".."
load dirname + '/' + item #successfully loads the script
if $test_succeeded
# ...
If you have problems running the script then I can recommend to define special method to run tests (similar to RSpec approach):
def test
test_res = yield # call test
$test_results ||= {}
$test_results << test_res # and store its result in arra of test results
end
Then your tests will look like:
require 'file_with_test_method'
require 'watir-webdriver'
test do
# your test code
browser.text.include?( 'The user already Exists') # last expression in the block will be the test result
end