I have an app with some specs written into minitest. As usual, I start them using rake.
Because some times I got a random results, my specs can pass one time, and fail an other time.
In this case, I can keep the sequence number and replay it later, after fixing.
Because I have this kind of tests (with a random result), I generally run rake many time, just to be sure that the app is okay.
I would like to know if there is a nice way to perform multiple rake tests (100 times for example), and stop them if there any failure or any error?
I think you should think again about your test, not about the test call. A test with a random result looks wrong for me.
What's the random factor in your test? Can you write a mock-element for the random factor and repeat the test with different values for the mock-element. So you get a "complete" test.
I created a dummy test with random result to simulate your situation:
#store it as file 'testcase.rb'
gem 'test-unit'
require 'test/unit'
class X < Test::Unit::TestCase
def test_1
num = rand(10)
assert_true( num < 5, "Value is #{num}")
end
end
The following task calls the test 10 times and stops after the first failure:
TEST_REPETION = 10
task :test do
TEST_REPETION.times{
stdout = `ruby testcase.rb`
if stdout =~ /\d+\) Failure/
puts "Failure occured"
puts stdout
exit
else
puts 'Tests ok'
end
}
end
For real usage I would adapt some parts:
Instead puts 'Tests ok' define a counter to see how often the test was succussfull
Instead puts stdoutyou may store the result in a result file?
Related
How can I get the number of RSpec examples tested at runtime? I am maintaining a large test suite that takes a long time to run and seems to have a memory leak, and want to periodically output various diagnostics during the test run. In my spec_helper, I start a thread as in the code below. I would like to include in those diagnostics the number of tests already run. (The total number of tests to test would be great too, if that is available.)
Thread.new do
start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
classes = [
SemanticLogger::Logger,
SemanticLogger::Appender::IO,
# ...
]
loop do
STDERR.puts
classes.each do |klass|
instance_count = ObjectSpace.each_object(klass).count
uptime = (Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time).round(0)
STDERR.puts format("%5d: %10d %s\n", uptime, instance_count, klass)
end
STDERR.puts
sleep 10
end
end
One way to do this is to count the examples explicitly in a before(:each) block that applies to all test examples (for example, in a universally included spec_helper.rb file). For example:
RSpec.configure do |config|
$rspec_example_count = 0
config.before(:each) do |example|
$rspec_example_count += 1
(If you can get by without using a global variable, even better, but the global variable is probably fine given that it is only used in an rspec test suite.)
I assume this is very newbie stuff but I'm learning Ruby by doing, and I'm developing a small CLI tool that receives a couple of parameters in order to do its stuff properly. This is my current workflow:
I want to test (using Minitest) all the possible flows:
Exits with 0 and help message is shown if ARGV.count != 2
Exits with 1 if first param is not correct
Exits with 1 if second param is not correct
Exits with 1 if both params are not correct
Exits with 0 and does stuff if all params are correct
Now, if I run tests the only thing I see is the help output as there is no parameter being passed.
So, a couple of questions:
How can I pass arguments to the main program in tests?
How can I test the output? (I'm using puts)
Thanks!
nice diagram!
you can either use helpers like aruba https://github.com/cucumber/aruba
or dig into ruby internals in order to bend it to your will!
# test.rb
pseudoIO = StringIO.new
$stdout = pseudoIO
puts "hi #{ARGV.join(', ')}"
ARGV.replace ["file1"]
puts "now its #{ARGV.join(', ')}"
abort "captured: #{pseudoIO.string}"
output should be
ruby text.rb "whutup"
# => captured: hi whutup
# => now its file1
I'm trying to write a custom tool that runs ruby unit tests with my customizations.
What I need it to do is to load a certain TestCase from given file(through require or whatever), and then run it after doing some calculations and initializations.
Problem is, the moment I require "test/unit" and a test case, it runs immediately.
What can I do with this?
Thanks.
Since you're running 1.9 and test/unit in 1.9 is merely a wrapper for MiniTest, the following approach should work:
implement your own custom Runner
set MiniTest's runner to your custom runner
Something like (shameless plug from EndOfLine Custom Test Runner, adjusted to Ruby 1.9):
fastfailrunner.rb:
require 'test/unit'
class FastFailRunner19 < MiniTest::Unit
def _run args = []
puts "fast fail runner"
end
end
~
example_test.rb:
require 'test/unit'
class ExampleTest < Test::Unit::TestCase
def test_assert_equal
assert_equal 1, 1
end
def test_lies
assert false
end
def test_exceptions
raise Exception, 'Beware the Jubjub bird, and shun the frumious Bandersnatch!'
end
def test_truth
assert true
end
end
run.rb:
require_relative 'fast_fail_runner'
require_relative 'example_test'
MiniTest::Unit.runner= FastFailRunner19.new
If you run this with
ruby run.rb
the custom FastFailRunner19 will be used, which does nothing.
What about reading file content as a regular text file and doing eval on its content after you initialize/calculate things you say? It may not be sufficient for your needs and may require manual setup and execution of testing framework.
Like that (I put heredoc instead of reading file). Basically content is just a string containing your test case code.
content = <<TEST_CASE
class YourTestCase
def hello
puts 'Hello from eval'
end
end
YourTestCase.new.hello
TEST_CASE
eval content
Note: Altough I'd rather not use eval if there is another way. One should be extra careful when evaling code from string manually in any language.
You could collect the test cases you want to deferred its executions and store them in an array. Afterwards you would create a block execution code. For instance:
test_files = ['test/unit/first_test.rb'] #=> Testcases you want to run
test_block = Proc.new {spec_files.each {|f|load f} } #=> block storing the actual execution of those tests.
Once you're ready to call those testcases you just do test_block.call.
To generalize a bit, when thinking about deferring or delaying code executions, closures are a very elegant and flexible alternative.
I have a Ruby script that needs to run about one time a second. I am using a Ruby script to keep track of modifications of files in a directory and want the script to track updates in "live" time.
Basically, I want my script to do the same kind of thing as running "top" on a Unix shell, where the screen is updated every second or so. Is there an equivalent to setInterval in Ruby like there is in JavaScript?
There are a few ways to do this.
The quick-and-dirty versions:
shell (kornish):
while :; do
my_ruby_script.rb
sleep 1
done
watch(1):
shell$ watch -n 1 my_ruby_script.rb
This will run your script every second and keep the output of the most recent run displayed in your terminal.
in ruby:
while true
do_my_stuff
sleep 1
end
These all suffer from the same issue: if the actual script/function takes time to run, it makes the loop run less than every second.
Here is a ruby function that will make sure the function is called (almost) exactly every second, as long as the function doesn't take longer than a second:
def secondly_loop
last = Time.now
while true
yield
now = Time.now
_next = [last + 1,now].max
sleep (_next-now)
last = _next
end
end
Use it like this:
secondly_loop { my_function }
You may find interesting this gem whenever
You can code repeating tasks this way:
every 1.second do
#your task
end
As stated in another answer, rb-inotify is well suited to this sort of thing. If you don't want to use it, then a simple approach is to use threads:
a = Thread.new { loop { some_method; Thread.stop } }
b = Thread.new { loop { sleep 1; break unless a.alive?; a.run } }
To stop polling, use a.kill or make sure that some_method kills its own thread with Thread.kill when some condition is met.
Using two threads like this ensures that some_method runs at least every second, regardless of the length of the operation, without having to do any time checking yourself (within the granularity of the thread scheduling, of course).
You might want to consider using something like rb-inotify to get notifications of changes of files. This way you can avoid "sleep" and keep the "live" feeling.
There is some useful information at the "Efficient Filesystem Handling" section of the Guard Gem documentation: https://github.com/guard/guard#efficient-filesystem-handling
Or you could use TimerTask from the Concurrent Ruby gem
timer_task = Concurrent::TimerTask.new(execution_interval: 1) do |task|
task.execution_interval.times{ print 'Boom! ' }
print "\n"
task.execution_interval += 1
if task.execution_interval > 5
puts 'Stopping...'
task.shutdown
end
end
timer_task.execute
Code extracted from the same link
I have some unit tests written using Test::Unit::TestCase, with XML generated by ci_reporter. However, due to circumstances beyond my control, they may occasionally fluctuate, and randomly fail. I'd like to detect when a test fails, and attempt to re-run it.
I tried doing this by monkey-patching 'teardown' to check 'passed?', and re-running the tests on a failure. However, the XML output will still show the first failed case, and not the second (now passing) run.
This sounds a bit like the opposite of Multiple tests with minitest
Perhaps this is a possibility: Copy your test case in an own file. As an example, try the following test:
#store it as file 'testcase.rb'
gem 'test-unit'
require 'test/unit'
class X < Test::Unit::TestCase
def test_1
num = rand(10)
assert_true( num < 2, "Value is #{num}")
end
end
Then define your test call in a rake task:
require 'rake'
TEST_REPETION = 10
task :test do
success = false
TEST_REPETION.times{
stdout = `ruby testcase.rb`
if stdout =~ /Failure/
puts "Failure occured - redo the test"
else
puts 'Tests ok'
success = true
exit
end
}
puts "Stopped after #{TEST_REPETION} tries" unless success
end
Now the test is called, until the test succeed or TEST_REPETION are done.
Remarks:
Rake isn't needed, you may do the call without rake (My template was a rake task)
This works only, if your xml changes for each run (it must be regenerated before the test. else you test always the same).
You may store the test result (stdout) in a file and use it later to analyze, which tests failed and try to retest them.