Background: My project's continuous integration build runs RSpec in several parallel runs. Specs are partitioned across parallel runs by spec file. That means long spec files dominate test suite run time. So I want to know the time each spec file takes to run (not just the time each example takes to run).
How can I get RSpec to tell me the time each spec file takes to run? Several of RSpec's stock formatters tell me the time each example takes to run, but they don't sum the time for each spec.
I'm using RSpec 3.2.
I addressed this need by writing my own RSpec formatter. Put the following class in spec/support, make sure it's required, and run rspec like so:
rspec --format SpecTimeFormatter --out spec-times.txt
class SpecTimeFormatter < RSpec::Core::Formatters::BaseFormatter
RSpec::Core::Formatters.register self, :example_started, :stop
def initialize(output)
#output = output
#times = []
end
def example_started(notification)
current_spec = notification.example.file_path
if current_spec != #current_spec
if #current_spec_start_time
save_current_spec_time
end
#current_spec = current_spec
#current_spec_start_time = Time.now
end
end
def stop(_notification)
save_current_spec_time
#times.
sort_by { |_spec, time| -time }.
each { |spec, time| #output << "#{'%4d' % time} seconds #{spec}\n" }
end
private
def save_current_spec_time
#times << [#current_spec, (Time.now - #current_spec_start_time).to_i]
end
end
Related
I have a large amount of Minitest unit tests (methods), over 300. They all take some time, from a few milliseconds to a few seconds. Some of them hang up, sporadically. I can't understand which one and when.
I want to apply Timeout to each of them, to make sure anyone fails if it takes longer than, say, 5 seconds. Is it achievable?
For example:
class FooTest < Minitest::Test
def test_calculates_something
# Something potentially too slow
end
end
You can use the Minitest PLugin loader to load a plugin. This is, by far, the cleanest solution. The plugin system is not very well documented, though.
Luckily, Adam Sanderson wrote an article on the plugin system.
The best news is that this article explains the plugin system by writing a sample plugin that reports slow tests. Try out minitest-snail, it is probably almost what you want.
With a little modification we can use the Reporter to mark a test as failed if it is too slow, like so (untested):
File minitest/snail_reporter.rb:
module Minitest
class SnailReporter < Reporter
attr_reader :max_duration
def self.options
#default_options ||= {
:max_duration => 2
}
end
def self.enable!(options = {})
#enabled = true
self.options.merge!(options)
end
def self.enabled?
#enabled ||= false
end
def initialize(io = STDOUT, options = self.class.options)
super
#max_duration = options.fetch(:max_duration)
end
def record result
#passed = result.time < max_duration
slow_tests << result if !#passed
end
def passed?
#passed
end
def report
return if slow_tests.empty?
slow_tests.sort_by!{|r| -r.time}
io.puts
io.puts "#{slow_tests.length} slow tests."
slow_tests.each_with_index do |result, i|
io.puts "%3d) %s: %.2f s" % [i+1, result.location, result.time]
end
end
end
end
File minitest/snail_plugin.rb:
require_relative './snail_reporter'
module Minitest
def self.plugin_snail_options(opts, options)
opts.on "--max-duration TIME", "Report tests that take longer than TIME seconds." do |max_duration|
SnailReporter.enable! :max_duration => max_duration.to_f
end
end
def self.plugin_snail_init(options)
if SnailReporter.enabled?
io = options[:io]
Minitest.reporter.reporters << SnailReporter.new(io)
end
end
end
I am using a rake task to run tests written in Ruby.
The rake task:
desc "This Run Tests on my ruby app"
Rake::TestTask.new do |t|
t.libs << File.dirname(__FILE__)
t.test_files = FileList['test*.rb']
t.verbose = true
end
I would like to create a timeout so that if any test (or the entire suite) hangs a timeout exception will be thrown and the test will fail.
I tried to create a new task that would run the test task with a timeout:
desc "Run Tests with timeout"
task :run_tests do
Timeout::timeout(200) do
Rake::Task['test'].invoke
end
end
The result was that a timeout was thrown, but the test continued to run.
I've been looking for something similar, and ended up writing this:
require 'timeout'
# Provides an individual timeout for every test.
#
# In general tests should run for less then 1s so 5s is quite a generous timeout.
#
# Timeouts can be overridden per-class (in rare cases where tests should take more time)
# by setting for example `self.test_timeout = 10 #s`
module TestTimeoutHelper
def self.included(base)
class << base
attr_accessor :test_timeout
end
base.test_timeout = 5 # s
end
# This overrides the default minitest behaviour of measuring time, with extra timeout :)
# This helps out with: (a) keeping tests fast :) (b) detecting infinite loops
#
# In general however benchmarks test should be used instead.
# Timeout is very unreliable by the way but in general works.
def time_it
t0 = Minitest.clock_time
Timeout.timeout(self.class.test_timeout, Timeout::Error, 'Test took to long (infinite loop?)') do
yield
end
ensure
self.time = Minitest.clock_time - t0
end
end
This module should be included into either specific test cases, or a general test case.
It works with a MiniTest 5.x
This code adds a timeout for entire suite:
def self.suite
mysuite = super
def mysuite.run(*args)
Timeout::timeout(600) do
super
end
end
mysuite
end
I'm trying to convert a test file from Rspec to Minitest, which up to this point has been going well (very simple tests thus far). But I've come across some stubs and I can't get them to function correctly.
Here is the original Rspec test:
it "takes exactly 1 second to run a block that sleeps for 1 second (with stubs)" do
fake_time = #eleven_am
Time.stub(:now) { fake_time }
elapsed_time = measure do
fake_time += 60 # adds one minute to fake_time
end
elapsed_time.should == 60
end
My attempt to convert that to Minitest:
it "takes exactly 1 second to run a block that sleeps for 1 second (with stubs)" do
fake_time = #eleven_am
Time.stub(:now, fake_time) do
elapsed_time = measure { fake_time += 60 }
elapsed_time.must_equal 60
end
end
And the method these are testing:
def measure(rep=1, &block)
start_time = Time.now
rep.times { block.call }
Time.now - start_time
end
The problem I'm having is that with Rspec, the stubs update dynamically with the method execution. When fake_time gets changed in the block, Time.now is immediately updated to correspond with that, meaning the final Time.now call is updated in my method, and it returns the appropriate difference (60).
With Minitest, I seem to override the Time.now response successfully, but it doesn't update with execution, so when fake_time gets adjusted, Time.now does not. This causes it to always return 0, as start_time and Time.now remain identical.
This is probably correct behavior for what it is, I'm just not sure how to get what I want out of it.
How do I make the Minitest stub act like the Rspec stub?
I received an answer from Chris Kottom on Reddit that I will share here.
The solution is to use a lambda:
Time.stub(:now, -> { fake_time }) do
...
end
This creates a dynamic stub that updates with your code execution.
If fake_time is a variable instead of a method (e.g. static vs. dynamic), you can represent this via:
Time.stub(:now, fake_time) do
...
end
I recently decided to write a simple test runtime profiler for our Rails 3.0 app's test suite. It's a very simple (read: hacky) script that adds each test's time to a global, and then outputs the result at the end of the run:
require 'test/unit/ui/console/testrunner'
module ProfilingHelper
def self.included mod
$test_times ||= []
mod.class_eval do
setup :setup_profiling
def setup_profiling
#test_start_time = Time.now
end
teardown :teardown_profiling
def teardown_profiling
#test_took_time = Time.now - #test_start_time
$test_times << [name, #test_took_time]
end
end
end
end
class ProfilingRunner < Test::Unit::UI::Console::TestRunner
def finished(elapsed_time)
super
tests = $test_times.sort{|x,y| y[1] <=> x[1]}.first(100)
output("Top 100 slowest tests:")
tests.each do |t|
output("#{t[1].round(2)}s: \t #{t[0]}")
end
end
end
Test::Unit::AutoRunner::RUNNERS[:profiling] = proc do |r|
ProfilingRunner
end
This allows me to run the suites like so rake test:xxx TESTOPTS="--runner=profiling" and get a list of Top 100 tests appended to the end of the default runner's output. It works great for test:functionals and test:integration, and even for test:units TEST='test/unit/an_example_test.rb'. But if I do not specify a test for test:units, the TESTOPTS appears to be ignored.
In classic SO style, I found the answer after articulating clearly to myself, so here it is:
When run without TEST=/test/unit/blah_test.rb, test:units TESTOPTS= needs a -- before its contents. So the solution in its entirety is simply:
rake test:units TESTOPTS='-- --runner=profiling'
I've got an interesting conundrum. I'm in the midst of developing a library to parse PSDs in Ruby. Also, a buddy is simultaneously working on a library to parse PSDs in JavaScript. We would like to share the same unit tests via a git submodule.
We've decided to use a simple JSON DSL to define each test. A single test might look like:
{
"_name": "Layer should render out",
"_file": "test/fixtures/layer_out.psd",
"_exports_to": "test/controls/layer_out_control.png"
}
So, now it's up to us to build the appropriate test harnesses to translate the JSON into the appropriate native unit tests. I've been using MiniTest to get myself up to speed, but I'm running into a few walls.
Here's what I've got so far. The test harness is named TargetPractice for the time being:
# run_target_practice.rb
require 'target_practice'
TargetPractice.new(:test) do |test|
test.pattern = "test/**/*.json"
end
and
# psd_test.rb
class PSDTest < MiniTest::Unit::TestCase
attr_accessor :data
def tests_against_data
# do some assertions
end
end
and
# target_practice.rb
class TargetPractice
attr_accessor :libs, :pattern
def initialize(sym)
#libs = []
#pattern = ""
yield self
run_tests
end
def run_tests
FileList[#pattern].to_a.each do |file|
test_data = JSON.parse(File.open(file).read)
test = PSDTest.new(test_data["_name"]) do |t|
t.data = test_data
end
end
end
end
Unfortunately, I'm having trouble getting a yield in the initialize to stick in my PSDTest class. Also, it appears that a test will run immediately on initialization.
I would like to dynamically create a few MiniTest::Unit::TestCase objects, set their appropriate data properties and then run the tests. Any pointers are appreciated!
I think you are overcomplicating things a bit here. What you need is a parameterized test, which is pretty trivial to implement using mintest/spec:
describe "PSD converter" do
def self.tests(pattern = 'test/**/*.json')
FileList[pattern].map{|file| JSON.parse(File.read(file))}
end
tests.each do |test|
it "satisfies test: " + test["_name"] do
# some assertions using test["_file"] and test["_exports_to"]
end
end
end