Mutation Testing with Rspec - ruby

I'm attempting to perform a mutation-test on some ruby code using rspec. I am just learning ruby and I don't really know if I'm doing it right. Part of the code I'm trying to test is:
class Cipher
def initialize()
#a_offset = 65 #'A'.unpack('C').first
#z_offset = 90 #'Z'.unpack('C').first
end
def encode(key, plain_text)
key_offset = key.upcase.unpack('C').first
cipher_text = plain_text.upcase.split('').collect do |letter|
cipher_letter = (letter.unpack('C').first + key_offset - #a_offset)
if cipher_letter > #z_offset
cipher_letter -= ( #z_offset - #a_offset + 1 )
end
cipher_letter.chr
end
return cipher_text.join
end
So far my test suite looks like this:
require 'rspec'
require 'Cipher'
describe "#initialize" do
it "should have correct #a_offset" do
encoder = Cipher.new()
expect(encoder.instance_variable_get(:#a_offset)).to eq 65
end
it "should have correct #z_offset" do
encoder = Cipher.new()
expect(encoder.instance_variable_get(:#z_offset)).to eq 90
end
end
describe "#encode" do
it "It should correctly encode Caesar with key = A"do
encoder = Cipher.new()
expect(encoder.encode('R', 'CAESAR')).to eq ("TRVJRI")
end
end
When running rspec my 3 tests pass. However when I use mutation testing on this suite I only kill 3/343 which is not very good.

Since your goal is to develop a mutation-adequate test suite, the unkilled mutants essentially provide guidance for what to test for. Ideally, your question should have provided some examples for generated mutants that are not killed by your test suite so that others can comment on how to write tests that kill them.
Let me try to best answer your question with general recommendations and examples.
I am not familiar with the particular mutation testing tool you are using, but in general you should follow a process similar to the following*:
Select a mutant that is not killed by your test suite and determine whether it can be killed (a mutant might be semantically equivalent to your program -- if so, discard it from the pool of mutants).
Write a test that kills the selected mutant -- that is, write a test that passes on your original program but fails on the mutant.
Determine suitable inputs that trigger the faulty behavior, introduced by the mutant.
Determine a suitable test oracle that asserts on the expected behavior/output of your program (i.e., a test oracle that passes on your original program but fails on the mutant).
Please keep in mind that your goal should not be to write minimal tests that simply kill all mutants, but rather tests that make sense in the context of your project and as a byproduct kill the mutants. Again, if you could provide some concrete examples for mutants that your test suite does not kill, I and others can better comment on what tests are missing.
Example: Suppose you have a mutant that introduces the following fault (i.e., changes the relational operator in the if statement of your original program):
- if cipher_letter > #z_offset
+ if cipher_letter >= #z_offset
Killing this mutant requires a test that checks for the boundary condition -- that is, a test that encodes at least one character to a 'Z'. Your current test suite does not have such a test.
*This process describes traditional mutation testing. Recent research [1], [2] suggests that (1) not all killable mutants should be killed (some mutants elicit tests that simply don't make sense) and (2) equivalent mutants may indicate a problem in your program (e.g., unwanted redundancy or ambiguity) that you might want to fix instead of discarding the equivalent mutant.

Related

In RSpec, Is there a way to get information about the example to be tested for use in before(:each)?

I would like to keep track at runtime of the RSpec test currently being executed, preferably including its file name and line number, so that if a test hangs I can know where to find it. I'm thinking of simply writing the file name and line number to a file in a before(:each).
I realize I can see in the output the most recently executed test, but I would like to know the currently executing test.
before(:each) yields an RSpec::Core::Example instance representing the current test/example, which should contain the information you're after.
before(:each) do |example|
puts example.description
puts example.location
puts example.metadata
end

rspec context blocks not executing in the correct order

Using rspec and the selenium webdriver for chrome, ive written code that has this format:
describe x do
[some code]
context y do
[some code]
it a do
[more code]
end
end
context z do
[some other code]
it b do
[even more code]
end
end
end
It intuitively seems like the program flow should go straight down the program, going into context y and then executing the 'it a do' statement. However, after it finishes with the [some code] in context y, it immediately skips to context z without hitting the it block.
Is there a command line argument or some other piece of code that needs to be included in order for the program flow to work linearly down through the code?
Rspec randomizes the order of the tests.
Each test is supposed to be working and should not be dependent of its order compared to the other tests.
You can, however, bypass this behaviour if you run rspec with the --order default option

RSpec -- test if block called with block defined in before

I recently asked how to test in RSpec if a block was called and the answers to that question seem to work in a simple case. The problem is when the initialization with the block is more complex. Then it is done in before and reused by a number of different tests in the context, among them the one testing if the block was evaluated. See the example:
context "the node definition using block of code" do
before do
#n=node do
# this block should be called
end
# some more complex setup concerning #n
end
it "should call the block" do
# how to test it?
end
# here a bunch of other tests using #n
end
In this case the solution with side effect changing value of a local variable does not work. Raising an exception from the block is useless since the whole statement must be properly evaluated to be used by the other tests.
Obviously I could do the tests separately, but it seems to stink, since I must copy-paste the initialization part and since the was-the-block-called test inherently belongs to this very context.
How to test if the block was evaluated in such a case?
Explanation for question asked by #zetetic below.
The context is that I'm implementing a kind of DSL, with nodes defined by their parameters and blocks of code (that can define something else in the scope of node). Since the things defined by the node's block can be pretty generic, at least for the first attempt I just need to be sure the block is evaluated and that what a user provides there will be considered. For now does not matter what it is.
Probably I should refactor my tests now and using mocks make them test behaviors rather then implementation. However it will be a little bit tricky, for the sake of some mixins and dynamic handling of messages sent to objects. For now the cincept of such tests is a little bit fuzzy in my head ;-)
Anyway your answers and comments helped me to better understand how RSpec works and explained why what I'm trying to do looks as if it did not fit to the RSpec.
Try something like this (untested by me):
context "the node definition using block of code" do
let(:node){
node = Node.new "arg1", "arg2", node_block
# more complex stuff here
node
}
context "checking the block is called" do
let(:node_block) {
double = double("node_block")
double.should_receive("some kind of arg").and_return("something")
# this will now cause a fail if it isn't called
double
}
it "should call the block" do
node.blah()
end
end
let(:node_block) {
# some real code
}
subject { node.blah() }
it { should == 2 }
# ...
end
So that's a very shaky piece of code (you'll have to fill in the gaps as you didn't give very much to go on, and let is obviously a lambda too, which could mean you've got to play around with it a bit) that uses let and a double to check it's called, and avoids using before, which is really for side effects not setting up variables for use in the specs.
#zetetic makes a very insightful comment that you're not testing behaviour here. I'm not against using rspec for doing more unit test style stuff (guidelines are made to be broken), but you might ask how later tests will pass when using a real block of code if that block isn't being called? In a way, I'm not even sure you need to check the block is called, but only you know.

How do you unit test code that generates or uses shell commands?

I'm writing a Ruby library that will automate some shell commands. The commands will depend on quite a complex set of business logic.
I want to unit test the code that generates those shell commands. I'm not sure how would I do that; I don't care much about the exact text of the commands, but I want to make sure the commands are affected by logic - which is of course the purpose of the unit tests.
My current idea is to encapsulate the command generation into very thin template-like objects (or just templates), and have logic code call those objects when it needs to produce a shell command, and then I can check that logic code calls expected methods on the template code. This still leaves the template code untested, but that's not as important as verifying the logic. Basically
class DirectoryMaker
def initialize(generator)
#generator = generator
end
def make_directory(name)
if name=='baz'
raise 'YOURE NOT ALLOWED TO DO THAT'
else
#generator.mkdir(name)
end
end
end
describe DirectoryMaker do
it 'should produce a mkdir command' do
generator = double('Generator')
generator.should_receive('mkdir').with('foo').and_return('')
described_class.new(generator).make_directory('foo')
end
it 'should raise an exception when passed baz as a name' do
expect {
described_class.new(double('Generator')).make_directory('baz')
}.to raise_error
end
end
Ah, it seems I've answered my question, but if you have better suggestions I'd be happy to hear them.
i'm not sure i understand you correctly.
yes you can make the thin layer that is an abstraction of your console output (model). you can test if your logic (controller) produces correct objects from this abstraction layer. and you can test if your templates renders correct shell output from the model (view). but what for? do you really need the full MVC pattern here?
you have your logic that should produce an output - a string. why can't you just test if some input parameters produce desired strings?

How do I run a Test::Unit suite to completion with a tally of tests/failures?

I've inherited a large suite of Test::Unit tests, and one of my first tasks is to have the suite run to completion rather than exit after the first test failure.
I'm currently rescuing AssertionFailedError and ensuring an output string, but this just seems wrong. What's the better way? Seems like it would be a configuration option.
Some more background would be helpful. I can't say I understand the behavior you're seeing.
I'm using both the core test/unit lib that comes with ruby 1.8, as well as several versions of the gem with ruby 1.9. The normal behavior for both of these is to run the entire loaded suite to completion and summarize the results.
Running a script that says require 'test/unit' will add an on-exit hook to run Test::Unit::AutoRunner with a Test::Unit::Collector::ObjectSpace collector (i.e. runs every instance of Test::Unit::TestCase currently loaded in the global object space).
It's also fairly easy to write your own custom test runner which manually loads your test classes and packs them into a Test::Unit::TestSuite, and capture its results.
But with every version of test/unit I've used, I've always seen the entire suite finish and report both failures and errors. In lieu of further info, I'd suggest experimenting with a single dummy test so see how you should expect test/unit to behave.
For example
require 'test/unit'
class Foo < Test::Unit::TestCase
def testFoo
flunk 'bad foo'
end
end
class Bar < Test::Unit::TestCase
def testBar
raise 'bar bar'
end
end
gives
Loaded suite foo
Started EF Finished in 0.001000 seconds.
1) Error:
testBar(Bar)
RuntimeError: bad bar
foo.rb:9:in `testBar`
2) Failure:
testFoo(Foo) [foo.rb:4]:
bad foo
2 tests, 1 assertions, 1 failures, 1 errors, 0 skips
Lastly: where are you trying to rescue/ensure? In the test methods? Under normal circumstances, there's no reason to catch an AssertionFailedError. This is for Test::Unit::TestCase to do, as a way to count failures and give you the report you want. Catching this interferes with what test/unit is written to do.

Resources