This is the contents of my .rspec_parallel file. I am using parallel_tests gem to run tests in multiple browser instances. To my knowledge, the gem uses the same formatter options available in RSpec.
--format html --out results<%= ENV['TEST_ENV_NUMBER'] %>.html
This works fantastic and I'm able to get the HTML output I normally see from RSpec. However, all of the 'puts' messages and basic standard output is logged to my console window, and not to the HTML files.
How can I get this output into each individual HTML file that I have set up?
puts will output to $stdout where as output is actually an instance variable of the RSpec::Core::Formatters::BaseFormatter class. output is defaulted to $stdout but when you pass in a string it determines that it should create a new StringIO and then output this to the given file name. Thus puts will not append to the #output variable.
You could do something ugly like create a runner file like
File.open('some_file_name.html','w+') do |file|
file << `rspec spec --format html`
end
Then this file will have the output from $stdoutbut your puts code will not be html formatted in this case. Other than that you could try building your own Custom Formatter but it will probably take quite a bit of source searching to make sure you can capture everything appropriately.
That being said it does seem the reporter was exposed for adding custom messages but I am uncertain of how to use this appropriately See Pull Request 1866
Seems it would be something like
it "has a name" do |ex|
ex.reporter.message("Custom Message Here")
#actual test
end
but the html formatter seems to ignore this. I can see the output in $stdout but not in the html file itself.
Best of luck.
Related
I have a Ruby project with a UNIX executable file called parse located in a bin subfolder in my project root directory.
At the moment it's just this:
#!/usr/bin/env ruby
# frozen_string_literal: true
puts 'hello world'
The file can be executed on the command line when this command is run from the project root directory: bin/parse
It works fine, but I also want to write a passing Rspec test for it.
I have this spec file:
RSpec.describe "end-to-end application behaviour" do
subject { system('bin/parse') }
it 'prints the expected messsage to stdout' do
expect { subject }.to output(
'hello world'
).to_stdout
end
end
When I run it I get the test failure:
expected block to output "hello world" to stdout, but output nothing
This is the location of my spec file relative to my project root: spec/integration/parse_spec.rb
I tried placing require and require_relative statements in that spec file with the paths to the parse executable, in case that would help, but I just kept getting:
LoadError: cannot load such file
Does anyone know how I can write a test in that file that will pass and prove the parse executable behaviour works?
Don't Use the RSpec Output Matcher
RSpec has a built-in output matcher than can test both where output goes, as well as its contents. However, it's testing where your Ruby output goes, not whether some external application is using standard input or standard error. You're going to have to make some different assumptions about your code.
You can avoid driving yourself nuts by comparing strings rather than testing the underlying shell or your output streams. For example, consider:
RSpec.describe "parse utility output" do
it "prints the right string on standard output" do
expect(`echo hello world`).to start_with("hello world")
end
it "shows nothing on standard output when it prints to stderr" do
expect(`echo foo >&2 > /dev/null`).to be_empty
end
end
Just replace the echo statements with the correct invocation of parse for your system, perhaps by setting PATH directly in your shell, using a utility like direnv, or by modifying ENV["PATH"] in your spec or spec_helper.
As a rule of thumb, RSpec isn't really meant for testing command-line applications. If you want to do that, consider using the Aruba framework to exercise your command-line applications. It's best to use RSpec to test the results of methods or the output of commands, rather than trying to test basic functionality. Of course, your mileage may vary.
Use to_stdout_from_any_process instead of to_stdout:
expect { subject }.to output('hello world').to_stdout_from_any_process
According to this piece of documentation, it is possible to achieve the following output format by selecting the format documentation:
something
does something that passes
does something that fails (FAILED - 1)
does something that is pending (PENDING: Not Yet Implemented)
Is it possible to slightly edit this so it outputs as:
something
does something (SUCCESS)
does something (FAIL)
does something (PENDING)
Basically I would like the result of the test displayed no matter what - instead of just logging explicitly the failures and the pending ones.
I was able to do this by subclassing RSpec::Core::Formatters::DocumentationFormatter. Create the following file as spec/formatters/custom_formatter.rb:
class CustomFormatter < RSpec::Core::Formatters::DocumentationFormatter
RSpec::Core::Formatters.register self
private
def passed_output(example)
format_output(example, 'SUCCESS', :success)
end
def pending_output(example, _message)
format_output(example, 'PENDING', :pending)
end
def failure_output(example)
format_output(example, 'FAILED', :failure)
end
def format_output(example, status_text, code_or_symbol)
RSpec::Core::Formatters::ConsoleCodes.wrap(
"#{current_indentation}#{example.description.strip} (#{status_text})",
code_or_symbol
)
end
end
Then run the specs using this:
rspec --require formatters/custom_formatter --format CustomFormatter
Instead of --require formatters/custom_formatter, you can also require the formatter on spec/spec_helper.rb, e.g.
require_relative 'formatters/custom_formatter'
Then you only need to run this:
rspec --format CustomFormatter
If you want CustomFormatter to be the default formatter, you can add the command line options to .rspec configuration file at your project root. Here's how it should look like:
--require spec_helper
--require /path/to/custom_formatter.rb
--format CustomFormatter
With that, you no longer need to specify any command line arguments to use CustomFormatter.
Documentations and references:
https://relishapp.com/rspec/rspec-core/v/3-6/docs/formatters/custom-formatters
http://www.rubydoc.info/gems/rspec-core/RSpec/Core/Formatters
https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/formatters/documentation_formatter.rb
You can not change the existing RSpec formatters, but you can create your own
When RSpec's built-in output formatters don't, however, give you everything
you need, you can write your own custom formatter and tell RSpec to use that
one instead. The simplest way is to subclass RSpec's BaseTextFormatter, and
then override just the methods that you want to modify.
I am trying to execute rspec from ruby, and get the status or number of failures from a method or something like that. Actually I am running something like this:
system("rspec 'myfilepath'")
but I only can get the string returned by the function. Is there any way to do this directly using objects?
I think the best way would be using RSpec's configuration and Formatter. This would not involve parsing the IO stream, also gives much richer result customisation programmatically.
RSpec 2:
require 'rspec'
config = RSpec.configuration
# optionally set the console output to colourful
# equivalent to set --color in .rspec file
config.color = true
# using the output to create a formatter
# documentation formatter is one of the default rspec formatter options
json_formatter = RSpec::Core::Formatters::JsonFormatter.new(config.output)
# set up the reporter with this formatter
reporter = RSpec::Core::Reporter.new(json_formatter)
config.instance_variable_set(:#reporter, reporter)
# run the test with rspec runner
# 'my_spec.rb' is the location of the spec file
RSpec::Core::Runner.run(['my_spec.rb'])
Now you can use the json_formatter object to get result and summary of a spec test.
# gets an array of examples executed in this test run
json_formatter.output_hash
An example of output_hash value can be found here:
RSpec 3
require 'rspec'
require 'rspec/core/formatters/json_formatter'
config = RSpec.configuration
formatter = RSpec::Core::Formatters::JsonFormatter.new(config.output_stream)
# create reporter with json formatter
reporter = RSpec::Core::Reporter.new(config)
config.instance_variable_set(:#reporter, reporter)
# internal hack
# api may not be stable, make sure lock down Rspec version
loader = config.send(:formatter_loader)
notifications = loader.send(:notifications_for, RSpec::Core::Formatters::JsonFormatter)
reporter.register_listener(formatter, *notifications)
RSpec::Core::Runner.run(['spec.rb'])
# here's your json hash
p formatter.output_hash
Other Resources
Detailed work through
Gist example
I suggest you to take a look into rspec source code to find out the answer. I think you can start with example_group_runner
Edit: Ok here is the way:
RSpec::Core::Runner::run(options, err, out)
Options - array of directories, err & out - streams. For example
RSpec::Core::Runner.run(['spec', 'another_specs'], $stderr, $stdout)
Your problem is that you're using the Kernel#system method to execute your command, which only returns true or false based on whether or not it can find the command and run it successfully. Instead you want to capture the output of the rspec command. Essentially you want to capture everything that rspec outputs to STDOUT. You can then iterate through the output to find and parse the line which will tell you how many examples were run and how many failures there were.
Something along the following lines:
require 'open3'
stdin, stdout, stderr = Open3.popen3('rspec spec/models/my_crazy_spec.rb')
total_examples = 0
total_failures = 0
stdout.readlines.each do |line|
if line =~ /(\d*) examples, (\d*) failures/
total_examples = $1
total_failures = $2
end
end
puts total_examples
puts total_failures
This should output the number of total examples and number of failures - adapt as needed.
This one prints to console and at the same time captures the message. The formatter.stop is just a stub function, I don't know what it is for normally, I had to include it to use DocumentationFormatter. Also the formatter output contains console coloring codes.
formatter = RSpec::Core::Formatters::DocumentationFormatter.new(StringIO.new)
def formatter.stop(arg1)
end
RSpec.configuration.reporter.register_listener(formatter, :message, :dump_summary, :dump_profile, :stop, :seed, :close, :start, :example_group_started)
RSpec::Core::Runner.run(['test.rb','-fdocumentation'])
puts formatter.output.string
I'm running a bunch of rspec tests in parallel using the parallel_tests framework. Before I parallelized the tests, I was outputting the results from the tests into an html file like so:
rspec --format html --out tmp/index.html <pattern>
Now it looks more like this:
parallel:spec --format html --out tmp/index.html <pattern>
However, now that the tests are running in parallel, each test is generating its own html file, and since they all use the same path (tmp/index.html), the last test to finish overwrites the output html file and I'm left with a report of only that one test. How can I generate either a single html file which contains the aggregated results of all of my tests (this would be ideal)? And if that's not possible, how can I output each test to its own output html file so they don't all overwrite each other?
I tried using the built-in loggers in the parallel_test project (ParallelTests::RSpec::RuntimeLogger, ParallelTests::RSpec::SummaryLogger, and ParallelTests::RSpec::FailuresLogger) but those all just generate simple text files instead of the nice html files like rspec does. I also saw this question here but I'm not using cucumber, so this doesn't really apply to me. I tried putting --format html --out tmp/report<%= ENV['TEST_ENV_NUMBER'] %>.html in my .rspec_parallel file, but that didn't have any effect.
I had to write my own formatter, here's the code in case anyone else runs into this problem:
require 'fileutils'
RSpec::Support.require_rspec_core "formatters"
RSpec::Support.require_rspec_core "formatters/helpers"
RSpec::Support.require_rspec_core "formatters/base_text_formatter"
RSpec::Support.require_rspec_core "formatters/html_printer"
RSpec::Support.require_rspec_core "formatters/html_formatter"
# Overrides functionality from base class to generate separate html files for each test suite
# https://github.com/rspec/rspec-core/blob/master/lib/rspec/core/formatters/html_formatter.rb
class ParallelFormatter < RSpec::Core::Formatters::HtmlFormatter
RSpec::Core::Formatters.register self, :start, :example_group_started, :start_dump,
:example_started, :example_passed, :example_failed,
:example_pending, :dump_summary
# TEST_ENV_NUMBER will be empty for the first one, then start at 2 (continues up by 1 from there)
def initialize(param=nil)
output_dir = ENV['OUTPUT_DIR']
FileUtils.mkpath(output_dir) unless File.directory?(output_dir)
raise "Invalid output directory: #{output_dir}" unless File.directory?(output_dir)
id = (ENV['TEST_ENV_NUMBER'].empty?) ? 1 : ENV['TEST_ENV_NUMBER'] # defaults to 1
output_file = File.join(output_dir, "result#{id}.html")
opened_file = File.open(output_file, 'w+')
super(opened_file)
end
end
I am using rspec for my test in a ruby project, and I want to spec that my program should not output anything when the -q option is used. I tried:
Kernel.should_not_receive :puts
That did not result in a failed test when there was output to the console.
How do I verify the absents of text output?
puts uses $stdout internally. Due to the way it works, the easiest way to check is to simply use: $stdout.should_not_receive(:write)
Which checks nothing is written to stdout as expected.
Kernel.puts (as above) would only result in a failed test when it
is explictely called as such (e.g. Kernel.puts "Some text"), where
as most cases it's call in the scope of the current object.
The accepted answer above is incorrect. It "works" because it doesn't receive a :write message but it might have received a :puts message.
The correct line should read:
$stdout.should_not_receive(:puts)
Also you need to make sure you put the line before the code that will write to STDIO. For instance:
it "should print a copyright message" do
$stdout.should_receive(:puts).with(/copyright/i)
app = ApplicationController.new(%w[project_name])
end
it "should not print an error message" do
$stdout.should_not_receive(:puts).with(/error/i)
app = ApplicationController.new(%w[project_name])
end
That's an actual working RSpec from a project