I'm maintaining a standalone test automation suite written in Rspec & Capybara and SitePrism (No Rails).
Recently I started integrating it with Testrail for reporting - I used rspec-testrail gem for that but I had to modify it a bit, because I also wanted to send Slack notifications with test progress and report.
Anyway, the integration works smoothly, except the fact that example processing is relying on example having an exception and lack of exception causes setting the test status as Passed on Testrail.
As it appears, after :each nor after :example in Rspec.configure doesn't guarantee that the example has finished running.
I also tried around :example and around :each as described here, but to no avail.
I inspected contents of example.metadata and it looks that example.metadata[:execution_result] has only started_at variable, but a finished example would have also finished_at and status variables, according to the docs
My suspicion (after reading the relish docs) is that :aggregated_failures is the cause of different metadata structure and multiple expects running in threads that are later merged into one backtrace.
Do you know how can I wait for the example to finish or how to hook into the state where it's finished?
Or maybe I should create a custom formatter where I would hook after example notifications printed to the console (I would like to keep the stacktrace there).
My code is as follows:
Test (both assertions are failing):
require 'spec_helper'
feature 'Sign in' do
let(:login_page) { LoginPage.new }
let(:user) { { email: ENV['ADMIN_EMAIL'], password: ENV['ADMIN_PASSWORD'] } }
scenario 'is successful and user is redirected to dashboard for user with correct credentials', testrail_id: 5694 do
login_page.load
login_page.form.email.set(user[:email])
login_page.form.password.set(user[:password])
login_page.form.submit_btn.click
expect(login_page.sidebar).to have_jobs(text: "some_nonexistenttext")
login_page.logout
expect(current_url).to have_content "google.com"
end
end
Console output from the above test:
Failures:
1) Sign in is successful and user is redirected to dashboard for user with correct credentials
Got 2 failures:
1.1) Failure/Error: expect(login_page.sidebar).to have_jobs(text: "blala")
expected #has_jobs?({:text=>"some_nonexistenttext"}) to return true, got false
# ./spec/auth/login_spec.rb:13:in `block (3 levels) in <top (required)>'
1.2) Failure/Error: expect(current_url).to have_content "google.com"
expected to find text "google.com" in "https://example.com/"
# ./spec/auth/login_spec.rb:15:in `block (3 levels) in <top (required)>'
Finished in 53.91 seconds (files took 1.45 seconds to load)
4 examples, 1 failure
Failed examples:
rspec ./spec/auth/login_spec.rb:8 # Sign in is successful and user is redirected to dashboard for user with correct credentials
Spec helper:
require 'rubygems'
require 'capybara/rspec'
require 'selenium-webdriver'
require 'site_prism'
require 'slack-notifier'
require_relative '../config'
require_relative '../lib/testrail'
...
RSpec.configure do |config|
config.define_derived_metadata do |meta|
meta[:aggregate_failures] = true
end
config.example_status_persistence_file_path = 'examples.txt'
config.before :all do
testrail_initialize_test_run!
end
config.after :example, testrail_id: proc { |value| !value.nil? } do |example|
RSpec::Testrail.process(example)
end
end
processing method (slightly modified from original)
def process(example)
if example.exception
status = 5
message = example.exception.message
slack_notifier.publish(message_text "Failed")
elsif example.skipped? || example.pending?
puts 'Example skipped or pending'
status = 10
message = 'Pending or not implemented'
else
status = 1
message = ''
end
client.send_post("add_result_for_case/#{testrun['id']}/#{example.metadata[:testrail_id]}",
status_id: status,
comment: message)
end
So basically all I had to was to use a reporter listener and process notifications inside it :)
config.reporter.register_listener RSpec::Testrail::Listener.new, :start, :example_failed, :example_passed, :example_pending, :stop
Related
This question probably has a simple answer but I can't find any examples for using Roda with RSpec3, so it is difficult to troubleshoot.
I am using Marston and Dees "Effective Testing w/ RSpec3" book which uses Sinatra instead of Roda. I am having difficulty passing an object to API.new, and, from the book, this is what works with Sinatra but fails with a "wrong number of arguments" error when I substitute Roda.
Depending on whether I pass arguments with super or no arguments with super(), the error switches to indicate that the failure occurs either at the initialize method or in the call to Rack::Test::Methods post in the spec.
I see that in Rack::Test, in the Github repo README, I may have to use Rack::Builder.parse_file("config.ru") but that didn't help.
Here are the two errors that rspec shows when using super without brackets:
Failures:
1) MbrTrak::API POST /users when the user is successfully recorded returns the user id
Failure/Error: post '/users', JSON.generate(user)
ArgumentError:
wrong number of arguments (given 1, expected 0)
# ./spec/unit/app/api_spec.rb:21:in `block (4 levels) in <module:MbrTrak>'
And when using super():
1) MbrTrak::API POST /users when the user is successfully recorded returns the user id
Failure/Error: super()
ArgumentError:
wrong number of arguments (given 0, expected 1)
# ./app/api.rb:8:in `initialize'
# ./spec/unit/app/api_spec.rb:10:in `new'
# ./spec/unit/app/api_spec.rb:10:in `app'
# ./spec/unit/app/api_spec.rb:21:in `block (4 levels) in <module:MbrTrak>'
This is my api_spec.rb:
require_relative '../../../app/api'
require 'rack/test'
module MbrTrak
RecordResult = Struct.new(:success?, :expense_id, :error_message)
RSpec.describe API do
include Rack::Test::Methods
def app
API.new(directory: directory)
end
let(:directory) { instance_double('MbrTrak::Directory')}
describe 'POST /users' do
context 'when the user is successfully recorded' do
it 'returns the user id' do
user = { 'some' => 'user' }
allow(directory).to receive(:record)
.with(user)
.and_return(RecordResult.new(true, 417, nil))
post '/users', JSON.generate(user)
parsed = JSON.parse(last_response.body)
expect(parsed).to include('user_id' => 417)
end
end
end
end
end
And here is my api.rb file:
require 'roda'
require 'json'
module MbrTrak
class API < Roda
def initialize(directory: Directory.new)
#directory = directory
super()
end
plugin :render, escape: true
plugin :json
route do |r|
r.on "users" do
r.is Integer do |id|
r.get do
JSON.generate([])
end
end
r.post do
user = JSON.parse(request.body.read)
result = #directory.record(user)
JSON.generate('user_id' => result.user_id)
end
end
end
end
end
My config.ru is:
require "./app/api"
run MbrTrak::API
Well roda has defined initialize method that receives env as an argument which is being called by the app method of the class. Looks atm like this
def self.app
...
lambda{|env| new(env)._roda_handle_main_route}
...
end
And the constructor of the app looks like this
def initialize(env)
When you run your config.ru with run MbrTrack::API you are actually invoking the call method of the roda class which looks like this
def self.call(env)
app.call(env)
end
Because you have redefined the constructor to accept hash positional argument this no longer works and it throws the error you are receiving
ArgumentError:
wrong number of arguments (given 0, expected 1)
Now what problem are you trying to solve, if you want to make your API class configurable one way to go is to try out dry-configurable which is part of the great dry-ruby gem collection.
If you want to do something else feel free to ask.
It has been a long time since you posted your question so hope you will still find this helpful.
The class being tested qa.rb contains the code:
class QA
def initialize(bugs: 0)
#bugs = bugs
end
def speak
"Hello!"
end
def happy?
#bugs > 0
end
def debug
#bugs = 0
end
end
The RSpec file qa_spec.rb contains the code:
require 'rspec'
require_relative 'qa'
RSpec.describe QA do
describe '#happy?' do
context 'when bugs are more than 0' do
it 'returns true' do
subject { described_class.new(bugs: 1) }
expect(subject).to be_happy
end
end
end
end
The test fails when I run it, and gives me this error:
PS C:\Users\Jobla\repos\TDD> rspec qa_spec.rb
F
Failures:
1) QA#happy? when bugs are more than 0 returns true
Failure/Error: expect(subject).to be_happy
expected `#<QA:0x2e0d640 #bugs=0>.happy?` to return true, got false
# ./qa_spec.rb:9:in `block (4 levels) in <top (required)>'
Finished in 0.02999 seconds (files took 0.16995 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./qa_spec.rb:7 # QA#happy? when bugs are more than 0 returns true
However, when I edit qa_spec.rb and I swap the it and subject lines, the test suddenly passes:
require 'rspec'
require_relative 'qa'
RSpec.describe QA do
describe '#happy?' do
context 'when bugs are more than 0' do
subject { described_class.new(bugs: 1) } #swapped with line below
it 'returns true' do #swapped with line above
expect(subject).to be_happy
end
end
end
end
Tests pass:
PS C:\Users\Jobla\repos\TDD> rspec qa_spec.rb
.
Finished in 0.01003 seconds (files took 0.17993 seconds to load)
1 example, 0 failures
Please could someone explain why does swapping the it and subject lines change the result of the test?
subject is designed to be set in context or describe block, but not in it.
If you do not set subject before it then subject would be set automatically by calling new without parameters on described_class. bugs will be set to default 0. After that, you call it with a block subject { described_class.new(bugs: 1) } inside it, it's the same as if you call described_class.new { described_class.new(bugs: 1) } because subject inside it is an instance of QA class.
I'm not able to understand why the following Rspec test does not pass -
require "rspec"
require_relative "file-to-be-tested"
describe Customer do
it "is valid with firstname" do
customer = Customer.new("handy")
expect(customer).to be_valid
end
end
for the corresponding Class definition -
class Customer
attr_reader :firstname
def initialize(firstname)
#firstname = firstname
end
end
these two code snippets are in separate files in the same folder, so when i run ~rspec <first-filename> in the terminal, I get the following error -
F
Failures:
1) Customer is valid with firstname
Failure/Error: expect(customer).to be_valid
expected #<Customer:0x007f90e50f3110> to respond to `valid?`
# ./poodr/poodr_rspec.rb:8:in `block (2 levels) in <top (required)>'
Finished in 0.00551 seconds (files took 0.52876 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./poodr/poodr_rspec.rb:6 # Customer is valid with firstname
be_valid is an rspec-rails method, but it looks like you're using just straight rspec. you could do something like:
require "rspec"
require_relative "file-to-be-tested"
describe Customer do
it "is valid with firstname" do
expect { Customer.new('handy') }.to_not raise_error
end
end
What are you expecting the to be_valid test to do? The issue is that the Customer class has no method called valid? which your test is trying to test.
A hack to move your test along if your doing test driven development:
class Customer
def valid?
true
end
end
You now have a method called valid and your test will pass. Obviously it shouldn't always be true so your next step would be to expand the definition of valid?. What check needs to be done to know if a customer is valid or not?
Something strange is going on with capybara and rspec, which I'm setting up on Ruby 1.9.3, Padrino 0.10.7, rspec 2.11.0, capybara 2.0.2.
A basic Padrino project set up haml and rspec (no custom code, yet!) other than just enough to load a "/" page (which I verified does render as expected by "puts page.content" within the specs below). Here's the simple spec. "Bogus" doesn't exist, but "Home" does...note that when I puts to console, the expected true/false are CORRECT, but for some reason, the matchers aren't seeing the true/false correctly.
The one clue I have so far lies in the 2nd spec using should have_content('Bogus') which is reporting that Proc is expected...
./spec/controllers/hello_world_spec.rb
require 'spec_helper'
require 'capybara'
require 'capybara/rspec'
describe 'The HelloWorld App', :type => :feature do
context "per documentation" do
it "has bogus content" do
visit '/'
page.has_content?('Bogus')
end
it "does not have bogus content" do
visit '/'
page.should have_content("Bogus")
end
end
context "should tests" do
it "has bogus content" do
visit '/'
page.has_content?('Bogus').should == true
end
it "does not have bogus content" do
visit '/'
page.has_content?('Bogus').should == false
end
end
context "variables" do
it "has bogus content" do
visit '/'
result = page.has_content?('Bogus')
puts result
result.should == true
end
it "has Home content (expect TRUE!)" do
visit '/'
result = page.has_content?('Home')
puts result
result.should == true
end
it "does not have bogus content" do
visit '/'
result = page.has_content?('Bogus')
puts result
result.should == false
end
end
end
spec_helper.rb
PADRINO_ENV = 'test' unless defined?(PADRINO_ENV)
require File.expand_path(File.dirname(__FILE__) + "/../config/boot")
def app
##
# You can handle all padrino applications using instead:
Padrino.application
# Askme.tap do |app|
# end
end
RSpec.configure do |conf|
conf.include Rack::Test::Methods
Capybara.app = app
end
Output:
11:40:57:website >> bundle exec rspec spec/app/controllers/hello_world_controller_spec.rb
WARNING: Nokogiri was built against LibXML version 2.8.0, but has dynamically loaded 2.7.8
The HelloWorld App
per documentation
has bogus content
does not have bogus content (FAILED - 1)
should tests
has bogus content
does not have bogus content
variables
false
has bogus content
true
has Home content (expect TRUE!)
false
does not have bogus content
Failures:
1) The HelloWorld App per documentation does not have bogus content
Failure/Error: page.should have_content("Bogus")
TypeError:
wrong argument type Capybara::RSpecMatchers::HaveText (expected Proc)
# ./spec/app/controllers/hello_world_controller_spec.rb:16:in `block (3 levels) in <top (required)>'
Finished in 1.66 seconds
7 examples, 1 failure
Failed examples:
rspec ./spec/app/controllers/hello_world_controller_spec.rb:14 # The HelloWorld App per documentation does not have bogus content
Turns out the culprit was having both "bacon" and "rspec" in the Gemfile. Capybara was being introduced to a project that utilized bacon for the test suite and examples being tried were rspec. Once bacon was removed from the bundled gems, the capybara specs ran per documentation.
Since the bacon scripts run more or less as-is under rspec, the project decision is to remove bacon and go with rspec for the test suite and make the minor tweaks to the bacon scripts to run all specs under rspec.
I'm having some problems using Sinatra with Capybara.
I want to test a pure javascript application. It's just a plain index.html that is being served by Sinatra.
require "sinatra"
get "/" do
File.read("public/index.html")
end
Let's say for example that I want to test this code.
$("a.link").click(function(){
$(this).replaceWith("New String");
});
Click me!
Then the test would look something like this.
describe "requests", js: true do
it "should display a message" do
visit "/"
click_link "Click me!"
page.should have_content("New String")
end
end
The problem is that nothing happens. According to Ryan Bates screencast Firefox should start and run the test if js: true is added to the describe block.
Here is my spec_helper file.
require "rspec"
require "capybara"
require "capybara/dsl"
Capybara.javascript_driver = :selenium
require_relative "./../server"
Capybara.app = Sinatra::Application
Capybara.javascript_driver = :selenium
Capybara.default_wait_time = 10
RSpec.configure do |config|
config.mock_with :rspec
config.include Capybara
end
Here is the output when running rspec rspec/request_spec.rb.
requests
should display a message (FAILED - 1)
Failures:
1) requests should display a message
Failure/Error: page.should have_content("New String")
expected #has_content?("New String") to return true, got false
# ./spec/request_spec.rb:5:in `block (2 levels) in <top (required)>'
Finished in 4.38 seconds
1 example, 1 failure
Failed examples:
rspec ./spec/request_spec.rb:2 # requests should display a message
I created an complete example project on Github that can be found here:
https://github.com/oleander/capybara-js-fails
Anyone knows why it fails?
Here is the original answer from Jonas Nicklas.
You need to require 'capybara/rspec' and set :type => :request.
See the Capybara README section on "Using Capybara with RSpec".
/Jonas
Here is a working example on Github.