I'm running a bunch of rspec tests in parallel using the parallel_tests framework. Before I parallelized the tests, I was outputting the results from the tests into an html file like so:
rspec --format html --out tmp/index.html <pattern>
Now it looks more like this:
parallel:spec --format html --out tmp/index.html <pattern>
However, now that the tests are running in parallel, each test is generating its own html file, and since they all use the same path (tmp/index.html), the last test to finish overwrites the output html file and I'm left with a report of only that one test. How can I generate either a single html file which contains the aggregated results of all of my tests (this would be ideal)? And if that's not possible, how can I output each test to its own output html file so they don't all overwrite each other?
I tried using the built-in loggers in the parallel_test project (ParallelTests::RSpec::RuntimeLogger, ParallelTests::RSpec::SummaryLogger, and ParallelTests::RSpec::FailuresLogger) but those all just generate simple text files instead of the nice html files like rspec does. I also saw this question here but I'm not using cucumber, so this doesn't really apply to me. I tried putting --format html --out tmp/report<%= ENV['TEST_ENV_NUMBER'] %>.html in my .rspec_parallel file, but that didn't have any effect.
I had to write my own formatter, here's the code in case anyone else runs into this problem:
require 'fileutils'
RSpec::Support.require_rspec_core "formatters"
RSpec::Support.require_rspec_core "formatters/helpers"
RSpec::Support.require_rspec_core "formatters/base_text_formatter"
RSpec::Support.require_rspec_core "formatters/html_printer"
RSpec::Support.require_rspec_core "formatters/html_formatter"
# Overrides functionality from base class to generate separate html files for each test suite
# https://github.com/rspec/rspec-core/blob/master/lib/rspec/core/formatters/html_formatter.rb
class ParallelFormatter < RSpec::Core::Formatters::HtmlFormatter
RSpec::Core::Formatters.register self, :start, :example_group_started, :start_dump,
:example_started, :example_passed, :example_failed,
:example_pending, :dump_summary
# TEST_ENV_NUMBER will be empty for the first one, then start at 2 (continues up by 1 from there)
def initialize(param=nil)
output_dir = ENV['OUTPUT_DIR']
FileUtils.mkpath(output_dir) unless File.directory?(output_dir)
raise "Invalid output directory: #{output_dir}" unless File.directory?(output_dir)
id = (ENV['TEST_ENV_NUMBER'].empty?) ? 1 : ENV['TEST_ENV_NUMBER'] # defaults to 1
output_file = File.join(output_dir, "result#{id}.html")
opened_file = File.open(output_file, 'w+')
super(opened_file)
end
end
Related
I have a Ruby project with a UNIX executable file called parse located in a bin subfolder in my project root directory.
At the moment it's just this:
#!/usr/bin/env ruby
# frozen_string_literal: true
puts 'hello world'
The file can be executed on the command line when this command is run from the project root directory: bin/parse
It works fine, but I also want to write a passing Rspec test for it.
I have this spec file:
RSpec.describe "end-to-end application behaviour" do
subject { system('bin/parse') }
it 'prints the expected messsage to stdout' do
expect { subject }.to output(
'hello world'
).to_stdout
end
end
When I run it I get the test failure:
expected block to output "hello world" to stdout, but output nothing
This is the location of my spec file relative to my project root: spec/integration/parse_spec.rb
I tried placing require and require_relative statements in that spec file with the paths to the parse executable, in case that would help, but I just kept getting:
LoadError: cannot load such file
Does anyone know how I can write a test in that file that will pass and prove the parse executable behaviour works?
Don't Use the RSpec Output Matcher
RSpec has a built-in output matcher than can test both where output goes, as well as its contents. However, it's testing where your Ruby output goes, not whether some external application is using standard input or standard error. You're going to have to make some different assumptions about your code.
You can avoid driving yourself nuts by comparing strings rather than testing the underlying shell or your output streams. For example, consider:
RSpec.describe "parse utility output" do
it "prints the right string on standard output" do
expect(`echo hello world`).to start_with("hello world")
end
it "shows nothing on standard output when it prints to stderr" do
expect(`echo foo >&2 > /dev/null`).to be_empty
end
end
Just replace the echo statements with the correct invocation of parse for your system, perhaps by setting PATH directly in your shell, using a utility like direnv, or by modifying ENV["PATH"] in your spec or spec_helper.
As a rule of thumb, RSpec isn't really meant for testing command-line applications. If you want to do that, consider using the Aruba framework to exercise your command-line applications. It's best to use RSpec to test the results of methods or the output of commands, rather than trying to test basic functionality. Of course, your mileage may vary.
Use to_stdout_from_any_process instead of to_stdout:
expect { subject }.to output('hello world').to_stdout_from_any_process
For integrating our Rubocop reports into Jenkins, I'd like to generate both Checkstyles -- XML output and HTML output -- in one go.
I can achieve this when running Rubocop directly:
$ rubocop -r rubocop/formatter/checkstyle_formatter \
--format html --out reports/html/index.html \
--format RuboCop::Formatter::CheckstyleFormatter --out reports/checkstyle-result.xml
However, I can't figure out how to achieve the same result with RuboCop::RakeTask and the formatters array.
Here's what I have tried:
Using multiple --out entries in options
RuboCop::RakeTask.new(:rubocop_ci) do |t|
t.formatters = ['html', 'RuboCop::Formatter::CheckstyleFormatter']
t.options = ['--out', 'reports/checkstyle-result.xml',
'--out', 'reports/html/index.html']
end
This generates reports/checkstyle-result.xml, but prints the HTML output to STDOUT (presumably, Rubocop ignores the second --out parameter).
Put everything into the options, and ignore formatters completely
RuboCop::RakeTask.new(:rubocop_ci2) do |t|
t.options = ['--format', 'RuboCop::Formatter::CheckstyleFormatter',
'--out', 'reports/checkstyle-result.xml',
'--format', 'html',
'--out', 'reports/html/index.html']
end
This works, but is ugly; is it possible to use multiple entries in formatters with different output files?
According to this piece of documentation, it is possible to achieve the following output format by selecting the format documentation:
something
does something that passes
does something that fails (FAILED - 1)
does something that is pending (PENDING: Not Yet Implemented)
Is it possible to slightly edit this so it outputs as:
something
does something (SUCCESS)
does something (FAIL)
does something (PENDING)
Basically I would like the result of the test displayed no matter what - instead of just logging explicitly the failures and the pending ones.
I was able to do this by subclassing RSpec::Core::Formatters::DocumentationFormatter. Create the following file as spec/formatters/custom_formatter.rb:
class CustomFormatter < RSpec::Core::Formatters::DocumentationFormatter
RSpec::Core::Formatters.register self
private
def passed_output(example)
format_output(example, 'SUCCESS', :success)
end
def pending_output(example, _message)
format_output(example, 'PENDING', :pending)
end
def failure_output(example)
format_output(example, 'FAILED', :failure)
end
def format_output(example, status_text, code_or_symbol)
RSpec::Core::Formatters::ConsoleCodes.wrap(
"#{current_indentation}#{example.description.strip} (#{status_text})",
code_or_symbol
)
end
end
Then run the specs using this:
rspec --require formatters/custom_formatter --format CustomFormatter
Instead of --require formatters/custom_formatter, you can also require the formatter on spec/spec_helper.rb, e.g.
require_relative 'formatters/custom_formatter'
Then you only need to run this:
rspec --format CustomFormatter
If you want CustomFormatter to be the default formatter, you can add the command line options to .rspec configuration file at your project root. Here's how it should look like:
--require spec_helper
--require /path/to/custom_formatter.rb
--format CustomFormatter
With that, you no longer need to specify any command line arguments to use CustomFormatter.
Documentations and references:
https://relishapp.com/rspec/rspec-core/v/3-6/docs/formatters/custom-formatters
http://www.rubydoc.info/gems/rspec-core/RSpec/Core/Formatters
https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/formatters/documentation_formatter.rb
You can not change the existing RSpec formatters, but you can create your own
When RSpec's built-in output formatters don't, however, give you everything
you need, you can write your own custom formatter and tell RSpec to use that
one instead. The simplest way is to subclass RSpec's BaseTextFormatter, and
then override just the methods that you want to modify.
This is the contents of my .rspec_parallel file. I am using parallel_tests gem to run tests in multiple browser instances. To my knowledge, the gem uses the same formatter options available in RSpec.
--format html --out results<%= ENV['TEST_ENV_NUMBER'] %>.html
This works fantastic and I'm able to get the HTML output I normally see from RSpec. However, all of the 'puts' messages and basic standard output is logged to my console window, and not to the HTML files.
How can I get this output into each individual HTML file that I have set up?
puts will output to $stdout where as output is actually an instance variable of the RSpec::Core::Formatters::BaseFormatter class. output is defaulted to $stdout but when you pass in a string it determines that it should create a new StringIO and then output this to the given file name. Thus puts will not append to the #output variable.
You could do something ugly like create a runner file like
File.open('some_file_name.html','w+') do |file|
file << `rspec spec --format html`
end
Then this file will have the output from $stdoutbut your puts code will not be html formatted in this case. Other than that you could try building your own Custom Formatter but it will probably take quite a bit of source searching to make sure you can capture everything appropriately.
That being said it does seem the reporter was exposed for adding custom messages but I am uncertain of how to use this appropriately See Pull Request 1866
Seems it would be something like
it "has a name" do |ex|
ex.reporter.message("Custom Message Here")
#actual test
end
but the html formatter seems to ignore this. I can see the output in $stdout but not in the html file itself.
Best of luck.
I have some test on Rspec, which looks like this:
describe "description" do
before :each do
do_before()
end
it "something_1" do
...
end
it "something_2" do
...
end
end
I know that I can get name of current test ("something_1") by using
example.description
Is there any way to get array of all descriptions in before :each area?
rspec -f d --color --dry-run filename
Works for me in rspec 3.5.2, lists all tests without running them
There used to be a way to do this using the tag --dry-run, but that has been removed and no longer works.
You can use the -fd which is for format = documentation. This will show you a long list of all the specs that you've done and what they look like. It does, however, still run the test and show you any errors you have in the process. That said it's still a great way to list of all of your tests.