We have some tests in our MiniTest::Spec test suite that pass when each test directory is run individually, but fail when the entire suite is run.
Instead of spending hours trying to track down the reason for the failures, I thought it would be easier to just run each test directory in its own TestTask instance, like so:
Dir.new('spec').each do |f|
next if f.start_with? '.'
if File.directory? File.join('spec', f)
Rake::TestTask.new('test') do |t|
t.libs << "app" << "spec"
t.test_files = FileList["spec/#{f}/*_spec.rb"]
t.verbose = true
end
end
end
This causes the tests in each directory to run in "parallel", however. TestTask.new seems to execute each test in a separate thread (though I haven't figured out why yet).
Is there any way to avoid this behavior? I want each directory to run sequentially, since without this I'm running into "stack too deep" errors.
Thanks!
Related
I'm writing some tests for a webpage that I'd like to run in several environments. The idea is that the test will run in one, then repeat in the next. The two environments are preview and uat.
I've written an Around hook to set the environment variables. Below:
Around do |scenario, block|
def test_envs
chosen_env = ENV['test_env'] || 'preview'
chosen_env.split(',').map(&:strip)
end
test_envs.each do |test_env|
$base_url = "https://#{test_env}.webpage.com"
end
block.call
end
I have then written a method to execute the navigation step:
def navigate_to(path)
visit $base_url + path
end
My Scenario step_definition is:
navigate_to '/login'
The tests will work in either environment, Preview by default or UAT if I set test_env=uat
However, I was aiming to set test_env=preview,uat and have them run consecutively in both environments.
Is there something obvious that I've missed here?
Thanks
If I'm understanding you correctly, it's the 'parallel' aspect that you're asking about.
Rspec can be used with parallel tests (the parallel_tests gem) but I wouldn't be so sure that calling something like 3.times { blk.call } in an around hook will run each block in parallel.
An alternative may be do so some metaprogramming with your example definitions, i.e.
test_envs.each do |env_name|
it "does something in #{env_name}" do
# do something with the specific environment
end
end
Now, I haven't actually used this gem and I don't know for sure it would work. I think the simplest solution may be to just write a wrapper script to call the tests
# run_tests.rb
environments = ENV["TEST_ENV"]&.split(",") || []\
filename = ENV["filename"]
environments.each do |env_name|
Thread.new do
system <<-SH
env TEST_ENV=#{env_name} bundle exec rspec #{filename}
SH
end
end
Running it like env TEST_ENV=foo,bar ruby run_tests.rb would call the following commands in their own threads:
env TEST_ENV=foo bundle exec rspec
env TEST_ENV=bar bundle exec rspec
I like this approach because it means you don't have to touch your existing test code.
I want to execute some code before an arbitrary RSpec test is run, but only in cases where the example groups to be tested are either in a specific directory or carry a specific tag.
For example, if I have the following groups:
## spec/file_one.rb
describe "Spec One - A group which needs the external app running", :external => true do
describe "Spec Two - A group which does not need the external app running" do
## spec/file_two.rb
describe "Spec Three - A group which does NOT need the external app running" do
## spec/always_run/file_three.rb
describe "Spec Four - A group which does need the external app running"
Then I want the code to be executed only when a test run contains Spec One or Spec Four.
This is relatively easy to do when I can rely on the filename, but harder when relying on the tag. How can I check what files examples will be run and then check their tags?
I'd just have a support setup like this:
PID_FILE = File.join(Rails.root, "tmp", "pids", "external.pid")
def read_pid
return nil unless File.exists? PID_FILE
File.open(PID_FILE).read.strip
end
def write_pid(pid)
File.open(PID_FILE, "w") {|f| f.print pid }
end
def external_running?
# Some test to see if the external app is running here
begin
!!Process.getpgid(read_pid)
rescue
false
end
end
def start_external
unless external_running?
write_pid spawn("./run_server")
# Maybe some wait loop here for the external service to boot up
end
end
def stop_external
Process.kill read_pid if external_running?
end
RSpec.configure do |c|
before(:each) do |example|
start_external if example.metadata[:external]
end
after(:suite) do
stop_external
end
end
Each test tagged with :external would attempt to start the external process if it's not already started. Thus, the first time you run a test that needs it, the process would be booted. If no tests with the tag are run, the process is never booted. The suite then cleans up after itself by terminating the process as a part of the shutdown process.
This way, you don't have to pre-process the test list, your tests aren't interdependent, and your external app is automatically cleaned up after. If the external app is running before the test suite gets a chance to invoke it, it will read the pid file and use the existing instance.
Rather than relying on metadata[:external] you could parse the full name of the example and determine if it needs the external app for a more "magical" setup, but that's kind of smelly to me; example descriptions are for humans, not for the spec suite to parse.
I have some unit tests written using Test::Unit::TestCase, with XML generated by ci_reporter. However, due to circumstances beyond my control, they may occasionally fluctuate, and randomly fail. I'd like to detect when a test fails, and attempt to re-run it.
I tried doing this by monkey-patching 'teardown' to check 'passed?', and re-running the tests on a failure. However, the XML output will still show the first failed case, and not the second (now passing) run.
This sounds a bit like the opposite of Multiple tests with minitest
Perhaps this is a possibility: Copy your test case in an own file. As an example, try the following test:
#store it as file 'testcase.rb'
gem 'test-unit'
require 'test/unit'
class X < Test::Unit::TestCase
def test_1
num = rand(10)
assert_true( num < 2, "Value is #{num}")
end
end
Then define your test call in a rake task:
require 'rake'
TEST_REPETION = 10
task :test do
success = false
TEST_REPETION.times{
stdout = `ruby testcase.rb`
if stdout =~ /Failure/
puts "Failure occured - redo the test"
else
puts 'Tests ok'
success = true
exit
end
}
puts "Stopped after #{TEST_REPETION} tries" unless success
end
Now the test is called, until the test succeed or TEST_REPETION are done.
Remarks:
Rake isn't needed, you may do the call without rake (My template was a rake task)
This works only, if your xml changes for each run (it must be regenerated before the test. else you test always the same).
You may store the test result (stdout) in a file and use it later to analyze, which tests failed and try to retest them.
I have three RSpec2 test files, each of which passes individually. But running the suite with rspec spec (or jruby -S rspec spec) fails.
The problem: ARGV is being set to ["spec"] and running my program with a spec argument changes its behavior. I try to handle this in my tests with:
before(:each) do
ARGV.clear # also tried: ARGV.delete_if { |val| true }
end
but the puts ARGV statement in my code indicates ARGV is still being set to ["spec"].
I've also created a spec/spec_helper.rb file with:
RSpec.configure do |config|
config.before(:suite) do
ARGV.clear
end
end
with the same result. When I run tests individually, ARGV is empty. But when I run rspec spec, ARGV is ["spec"].
Possibly relevant background: I'm running under rbenv.
Rewrite your code such that ARGV isn't mentioned in the methods you're testing.
For example, if you need to test that you can parse "play_jukebox", then do
def test_play_jukebox
parse_options(["play_jukebox"])
end
and in your bin file, have
if $0 == __FILE__
parse_options(ARGV)
end
I want to use MiniTest::Spec, I found a couple resources to get started, but none of them mentioned what the test files (or spec files) should be named, and where they should be placed:
test/test_*
spec/*_spec.rb
So which one should I use?
Assuming you are using rake you can specify your own path in the rake file e.g.
Rake::TestTask.new do |t|
t.libs << "test"
t.test_files = FileList['test/test_*.rb']
t.verbose = true
end
It's up to you obviously, but I'd recommend using spec/**/*_spec.rb -- the thinking here being, if MiniTest::Spec is modeled after RSpec in syntax, then you might as well put the tests in the same place that RSpec puts them so people won't be put off guard.