I have a inspec test, this is great:
inspec exec scratchpad/profiles/forum_profile --reporter yaml
Trouble is I want to run this in a script and output this to an array
I cannot find the documentation that indicated what method i need to use to simulate the same
I do this
def my_func
http_checker = Inspec::Runner.new()
http_checker.add_target('scratchpad/profiles/forum_profile')
http_checker.run
puts http_checker.report
So the report method seems to give me load of the equivalent type and much more - does anyone have any documentation or advice on returning the same output as the --reporter yaml type response but in a script? I want to parse the response so I can share output with another function
I've never touched inspec, so take the following with a grain of salt, but according to https://github.com/inspec/inspec/blob/master/lib/inspec/runner.rb#L140, you can provide reporter option while instantiating the runner. Looking at https://github.com/inspec/inspec/blob/master/lib/inspec/reporters.rb#L11 I think it should be smth. like ["yaml", {}]. So, could you please try
# ...
http_checker = Inspec::Runner.new(reporter: ["yaml", {}])
# ...
(chances are it will give you the desired output)
Related
According to this piece of documentation, it is possible to achieve the following output format by selecting the format documentation:
something
does something that passes
does something that fails (FAILED - 1)
does something that is pending (PENDING: Not Yet Implemented)
Is it possible to slightly edit this so it outputs as:
something
does something (SUCCESS)
does something (FAIL)
does something (PENDING)
Basically I would like the result of the test displayed no matter what - instead of just logging explicitly the failures and the pending ones.
I was able to do this by subclassing RSpec::Core::Formatters::DocumentationFormatter. Create the following file as spec/formatters/custom_formatter.rb:
class CustomFormatter < RSpec::Core::Formatters::DocumentationFormatter
RSpec::Core::Formatters.register self
private
def passed_output(example)
format_output(example, 'SUCCESS', :success)
end
def pending_output(example, _message)
format_output(example, 'PENDING', :pending)
end
def failure_output(example)
format_output(example, 'FAILED', :failure)
end
def format_output(example, status_text, code_or_symbol)
RSpec::Core::Formatters::ConsoleCodes.wrap(
"#{current_indentation}#{example.description.strip} (#{status_text})",
code_or_symbol
)
end
end
Then run the specs using this:
rspec --require formatters/custom_formatter --format CustomFormatter
Instead of --require formatters/custom_formatter, you can also require the formatter on spec/spec_helper.rb, e.g.
require_relative 'formatters/custom_formatter'
Then you only need to run this:
rspec --format CustomFormatter
If you want CustomFormatter to be the default formatter, you can add the command line options to .rspec configuration file at your project root. Here's how it should look like:
--require spec_helper
--require /path/to/custom_formatter.rb
--format CustomFormatter
With that, you no longer need to specify any command line arguments to use CustomFormatter.
Documentations and references:
https://relishapp.com/rspec/rspec-core/v/3-6/docs/formatters/custom-formatters
http://www.rubydoc.info/gems/rspec-core/RSpec/Core/Formatters
https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/formatters/documentation_formatter.rb
You can not change the existing RSpec formatters, but you can create your own
When RSpec's built-in output formatters don't, however, give you everything
you need, you can write your own custom formatter and tell RSpec to use that
one instead. The simplest way is to subclass RSpec's BaseTextFormatter, and
then override just the methods that you want to modify.
At the moment, when using minitest, if you do:
bundle exec rake TESTOPTS='--verbose'
you get an output like this:
Text you typed in the describe#test_0001_test description = 0.11 s = .
Text you typed in the describe#test_0003_another test description = 0.10 s = .
...etc.
I want to have acces to this programmatically, so that I can select the slowest tests, sort them by the time they took to run, and print them out to stdout in any format I want. Ideally I would define a new Rake task for this and then run something like:
bundle exec rake mytask
or something.
However, I can't seem to find anything online on how to access this information programmatically. I searched about custom reporters, but apparently you have to monkey-patch Minitest for that, and I don't want to do that. The other option is to install the minitest-reporters gem, but I don't want nor need all that functionality, what I want to do is very simple. I've also read through the code in the Minitest repo, but couldn't wrap my head around what to inherit from if I wanted to create my own class, and what to access in order to get the time spent running and the name of the test.
Is there any way to have access to this information programmatically? For example accessing the reports produced by minitest once all tests have finished running? How do you do it, those of you who write custom reporters without requiring a gem or monkey-patching minitest? I feel like this should be an easy thing to do.
You can write your own reporter (no need to monkey-patch, or use minitest-reporters), but there is an easier way. The verbose output is formatted in such a way that you can parse it using sort:
bundle exec rake TESTOPTS='--verbose' | sort -t = -k 2 -g
This is the contents of my .rspec_parallel file. I am using parallel_tests gem to run tests in multiple browser instances. To my knowledge, the gem uses the same formatter options available in RSpec.
--format html --out results<%= ENV['TEST_ENV_NUMBER'] %>.html
This works fantastic and I'm able to get the HTML output I normally see from RSpec. However, all of the 'puts' messages and basic standard output is logged to my console window, and not to the HTML files.
How can I get this output into each individual HTML file that I have set up?
puts will output to $stdout where as output is actually an instance variable of the RSpec::Core::Formatters::BaseFormatter class. output is defaulted to $stdout but when you pass in a string it determines that it should create a new StringIO and then output this to the given file name. Thus puts will not append to the #output variable.
You could do something ugly like create a runner file like
File.open('some_file_name.html','w+') do |file|
file << `rspec spec --format html`
end
Then this file will have the output from $stdoutbut your puts code will not be html formatted in this case. Other than that you could try building your own Custom Formatter but it will probably take quite a bit of source searching to make sure you can capture everything appropriately.
That being said it does seem the reporter was exposed for adding custom messages but I am uncertain of how to use this appropriately See Pull Request 1866
Seems it would be something like
it "has a name" do |ex|
ex.reporter.message("Custom Message Here")
#actual test
end
but the html formatter seems to ignore this. I can see the output in $stdout but not in the html file itself.
Best of luck.
I am trying to execute rspec from ruby, and get the status or number of failures from a method or something like that. Actually I am running something like this:
system("rspec 'myfilepath'")
but I only can get the string returned by the function. Is there any way to do this directly using objects?
I think the best way would be using RSpec's configuration and Formatter. This would not involve parsing the IO stream, also gives much richer result customisation programmatically.
RSpec 2:
require 'rspec'
config = RSpec.configuration
# optionally set the console output to colourful
# equivalent to set --color in .rspec file
config.color = true
# using the output to create a formatter
# documentation formatter is one of the default rspec formatter options
json_formatter = RSpec::Core::Formatters::JsonFormatter.new(config.output)
# set up the reporter with this formatter
reporter = RSpec::Core::Reporter.new(json_formatter)
config.instance_variable_set(:#reporter, reporter)
# run the test with rspec runner
# 'my_spec.rb' is the location of the spec file
RSpec::Core::Runner.run(['my_spec.rb'])
Now you can use the json_formatter object to get result and summary of a spec test.
# gets an array of examples executed in this test run
json_formatter.output_hash
An example of output_hash value can be found here:
RSpec 3
require 'rspec'
require 'rspec/core/formatters/json_formatter'
config = RSpec.configuration
formatter = RSpec::Core::Formatters::JsonFormatter.new(config.output_stream)
# create reporter with json formatter
reporter = RSpec::Core::Reporter.new(config)
config.instance_variable_set(:#reporter, reporter)
# internal hack
# api may not be stable, make sure lock down Rspec version
loader = config.send(:formatter_loader)
notifications = loader.send(:notifications_for, RSpec::Core::Formatters::JsonFormatter)
reporter.register_listener(formatter, *notifications)
RSpec::Core::Runner.run(['spec.rb'])
# here's your json hash
p formatter.output_hash
Other Resources
Detailed work through
Gist example
I suggest you to take a look into rspec source code to find out the answer. I think you can start with example_group_runner
Edit: Ok here is the way:
RSpec::Core::Runner::run(options, err, out)
Options - array of directories, err & out - streams. For example
RSpec::Core::Runner.run(['spec', 'another_specs'], $stderr, $stdout)
Your problem is that you're using the Kernel#system method to execute your command, which only returns true or false based on whether or not it can find the command and run it successfully. Instead you want to capture the output of the rspec command. Essentially you want to capture everything that rspec outputs to STDOUT. You can then iterate through the output to find and parse the line which will tell you how many examples were run and how many failures there were.
Something along the following lines:
require 'open3'
stdin, stdout, stderr = Open3.popen3('rspec spec/models/my_crazy_spec.rb')
total_examples = 0
total_failures = 0
stdout.readlines.each do |line|
if line =~ /(\d*) examples, (\d*) failures/
total_examples = $1
total_failures = $2
end
end
puts total_examples
puts total_failures
This should output the number of total examples and number of failures - adapt as needed.
This one prints to console and at the same time captures the message. The formatter.stop is just a stub function, I don't know what it is for normally, I had to include it to use DocumentationFormatter. Also the formatter output contains console coloring codes.
formatter = RSpec::Core::Formatters::DocumentationFormatter.new(StringIO.new)
def formatter.stop(arg1)
end
RSpec.configuration.reporter.register_listener(formatter, :message, :dump_summary, :dump_profile, :stop, :seed, :close, :start, :example_group_started)
RSpec::Core::Runner.run(['test.rb','-fdocumentation'])
puts formatter.output.string
I have some test on Rspec, which looks like this:
describe "description" do
before :each do
do_before()
end
it "something_1" do
...
end
it "something_2" do
...
end
end
I know that I can get name of current test ("something_1") by using
example.description
Is there any way to get array of all descriptions in before :each area?
rspec -f d --color --dry-run filename
Works for me in rspec 3.5.2, lists all tests without running them
There used to be a way to do this using the tag --dry-run, but that has been removed and no longer works.
You can use the -fd which is for format = documentation. This will show you a long list of all the specs that you've done and what they look like. It does, however, still run the test and show you any errors you have in the process. That said it's still a great way to list of all of your tests.