How can I get the number of RSpec examples tested at runtime? I am maintaining a large test suite that takes a long time to run and seems to have a memory leak, and want to periodically output various diagnostics during the test run. In my spec_helper, I start a thread as in the code below. I would like to include in those diagnostics the number of tests already run. (The total number of tests to test would be great too, if that is available.)
Thread.new do
start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
classes = [
SemanticLogger::Logger,
SemanticLogger::Appender::IO,
# ...
]
loop do
STDERR.puts
classes.each do |klass|
instance_count = ObjectSpace.each_object(klass).count
uptime = (Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time).round(0)
STDERR.puts format("%5d: %10d %s\n", uptime, instance_count, klass)
end
STDERR.puts
sleep 10
end
end
One way to do this is to count the examples explicitly in a before(:each) block that applies to all test examples (for example, in a universally included spec_helper.rb file). For example:
RSpec.configure do |config|
$rspec_example_count = 0
config.before(:each) do |example|
$rspec_example_count += 1
(If you can get by without using a global variable, even better, but the global variable is probably fine given that it is only used in an rspec test suite.)
Related
Let's say I've got two ruby scripts - a.rb and b.rb. Both are web-scrapers used for different pages. They can work for many, many hours and I would like to run them simultaneously. In order to do that I've tried to run them by third script using 'promise' gem with the following code:
def method_1
require 'path to my file\a'
end
def method_2
require 'path to my file\b'
end
require 'future'
x=future{method_1}
y=future{method_2}
x+y
However this solution throws an error(below) and only one script is executed.
An operation was attempted on something that is not a socket.
(Errno::ENOTSOCK)
I also tried playing with Thread class:
def method_one
require 'path to my file\a'
end
def method_two
require 'path to my file\b'
end
x = Thread.new{method_one}
y = Thread.new{method_two}
x.join
y.join
And it gives me the same error as for 'promise' gem.
I've also run those scripts in separate shells- then they work at the same time, but the performance is much worse (aprox. about 50% slower).
Is it any way to run them at the same time and keep high performance?
You can use concurrent-ruby for this, here is how you can run both your scripts in parallel:
require 'concurrent'
# Create future for running script a
future1 = Concurrent::Promises.future do
require 'path to file\a'
:result
end
# Create future for running script b
future2 = Concurrent::Promises.future do
require 'path to file\b'
:result
end
# Combine both futures to run them in parallel
future = Concurrent::Promises.zip(future1, future1)
# Wait until both scripts are completed
future.value!
I want to execute some code before an arbitrary RSpec test is run, but only in cases where the example groups to be tested are either in a specific directory or carry a specific tag.
For example, if I have the following groups:
## spec/file_one.rb
describe "Spec One - A group which needs the external app running", :external => true do
describe "Spec Two - A group which does not need the external app running" do
## spec/file_two.rb
describe "Spec Three - A group which does NOT need the external app running" do
## spec/always_run/file_three.rb
describe "Spec Four - A group which does need the external app running"
Then I want the code to be executed only when a test run contains Spec One or Spec Four.
This is relatively easy to do when I can rely on the filename, but harder when relying on the tag. How can I check what files examples will be run and then check their tags?
I'd just have a support setup like this:
PID_FILE = File.join(Rails.root, "tmp", "pids", "external.pid")
def read_pid
return nil unless File.exists? PID_FILE
File.open(PID_FILE).read.strip
end
def write_pid(pid)
File.open(PID_FILE, "w") {|f| f.print pid }
end
def external_running?
# Some test to see if the external app is running here
begin
!!Process.getpgid(read_pid)
rescue
false
end
end
def start_external
unless external_running?
write_pid spawn("./run_server")
# Maybe some wait loop here for the external service to boot up
end
end
def stop_external
Process.kill read_pid if external_running?
end
RSpec.configure do |c|
before(:each) do |example|
start_external if example.metadata[:external]
end
after(:suite) do
stop_external
end
end
Each test tagged with :external would attempt to start the external process if it's not already started. Thus, the first time you run a test that needs it, the process would be booted. If no tests with the tag are run, the process is never booted. The suite then cleans up after itself by terminating the process as a part of the shutdown process.
This way, you don't have to pre-process the test list, your tests aren't interdependent, and your external app is automatically cleaned up after. If the external app is running before the test suite gets a chance to invoke it, it will read the pid file and use the existing instance.
Rather than relying on metadata[:external] you could parse the full name of the example and determine if it needs the external app for a more "magical" setup, but that's kind of smelly to me; example descriptions are for humans, not for the spec suite to parse.
I'm trying to write a custom tool that runs ruby unit tests with my customizations.
What I need it to do is to load a certain TestCase from given file(through require or whatever), and then run it after doing some calculations and initializations.
Problem is, the moment I require "test/unit" and a test case, it runs immediately.
What can I do with this?
Thanks.
Since you're running 1.9 and test/unit in 1.9 is merely a wrapper for MiniTest, the following approach should work:
implement your own custom Runner
set MiniTest's runner to your custom runner
Something like (shameless plug from EndOfLine Custom Test Runner, adjusted to Ruby 1.9):
fastfailrunner.rb:
require 'test/unit'
class FastFailRunner19 < MiniTest::Unit
def _run args = []
puts "fast fail runner"
end
end
~
example_test.rb:
require 'test/unit'
class ExampleTest < Test::Unit::TestCase
def test_assert_equal
assert_equal 1, 1
end
def test_lies
assert false
end
def test_exceptions
raise Exception, 'Beware the Jubjub bird, and shun the frumious Bandersnatch!'
end
def test_truth
assert true
end
end
run.rb:
require_relative 'fast_fail_runner'
require_relative 'example_test'
MiniTest::Unit.runner= FastFailRunner19.new
If you run this with
ruby run.rb
the custom FastFailRunner19 will be used, which does nothing.
What about reading file content as a regular text file and doing eval on its content after you initialize/calculate things you say? It may not be sufficient for your needs and may require manual setup and execution of testing framework.
Like that (I put heredoc instead of reading file). Basically content is just a string containing your test case code.
content = <<TEST_CASE
class YourTestCase
def hello
puts 'Hello from eval'
end
end
YourTestCase.new.hello
TEST_CASE
eval content
Note: Altough I'd rather not use eval if there is another way. One should be extra careful when evaling code from string manually in any language.
You could collect the test cases you want to deferred its executions and store them in an array. Afterwards you would create a block execution code. For instance:
test_files = ['test/unit/first_test.rb'] #=> Testcases you want to run
test_block = Proc.new {spec_files.each {|f|load f} } #=> block storing the actual execution of those tests.
Once you're ready to call those testcases you just do test_block.call.
To generalize a bit, when thinking about deferring or delaying code executions, closures are a very elegant and flexible alternative.
I have some unit tests written using Test::Unit::TestCase, with XML generated by ci_reporter. However, due to circumstances beyond my control, they may occasionally fluctuate, and randomly fail. I'd like to detect when a test fails, and attempt to re-run it.
I tried doing this by monkey-patching 'teardown' to check 'passed?', and re-running the tests on a failure. However, the XML output will still show the first failed case, and not the second (now passing) run.
This sounds a bit like the opposite of Multiple tests with minitest
Perhaps this is a possibility: Copy your test case in an own file. As an example, try the following test:
#store it as file 'testcase.rb'
gem 'test-unit'
require 'test/unit'
class X < Test::Unit::TestCase
def test_1
num = rand(10)
assert_true( num < 2, "Value is #{num}")
end
end
Then define your test call in a rake task:
require 'rake'
TEST_REPETION = 10
task :test do
success = false
TEST_REPETION.times{
stdout = `ruby testcase.rb`
if stdout =~ /Failure/
puts "Failure occured - redo the test"
else
puts 'Tests ok'
success = true
exit
end
}
puts "Stopped after #{TEST_REPETION} tries" unless success
end
Now the test is called, until the test succeed or TEST_REPETION are done.
Remarks:
Rake isn't needed, you may do the call without rake (My template was a rake task)
This works only, if your xml changes for each run (it must be regenerated before the test. else you test always the same).
You may store the test result (stdout) in a file and use it later to analyze, which tests failed and try to retest them.
I have an app with some specs written into minitest. As usual, I start them using rake.
Because some times I got a random results, my specs can pass one time, and fail an other time.
In this case, I can keep the sequence number and replay it later, after fixing.
Because I have this kind of tests (with a random result), I generally run rake many time, just to be sure that the app is okay.
I would like to know if there is a nice way to perform multiple rake tests (100 times for example), and stop them if there any failure or any error?
I think you should think again about your test, not about the test call. A test with a random result looks wrong for me.
What's the random factor in your test? Can you write a mock-element for the random factor and repeat the test with different values for the mock-element. So you get a "complete" test.
I created a dummy test with random result to simulate your situation:
#store it as file 'testcase.rb'
gem 'test-unit'
require 'test/unit'
class X < Test::Unit::TestCase
def test_1
num = rand(10)
assert_true( num < 5, "Value is #{num}")
end
end
The following task calls the test 10 times and stops after the first failure:
TEST_REPETION = 10
task :test do
TEST_REPETION.times{
stdout = `ruby testcase.rb`
if stdout =~ /\d+\) Failure/
puts "Failure occured"
puts stdout
exit
else
puts 'Tests ok'
end
}
end
For real usage I would adapt some parts:
Instead puts 'Tests ok' define a counter to see how often the test was succussfull
Instead puts stdoutyou may store the result in a result file?