I want to execute some code before an arbitrary RSpec test is run, but only in cases where the example groups to be tested are either in a specific directory or carry a specific tag.
For example, if I have the following groups:
## spec/file_one.rb
describe "Spec One - A group which needs the external app running", :external => true do
describe "Spec Two - A group which does not need the external app running" do
## spec/file_two.rb
describe "Spec Three - A group which does NOT need the external app running" do
## spec/always_run/file_three.rb
describe "Spec Four - A group which does need the external app running"
Then I want the code to be executed only when a test run contains Spec One or Spec Four.
This is relatively easy to do when I can rely on the filename, but harder when relying on the tag. How can I check what files examples will be run and then check their tags?
I'd just have a support setup like this:
PID_FILE = File.join(Rails.root, "tmp", "pids", "external.pid")
def read_pid
return nil unless File.exists? PID_FILE
File.open(PID_FILE).read.strip
end
def write_pid(pid)
File.open(PID_FILE, "w") {|f| f.print pid }
end
def external_running?
# Some test to see if the external app is running here
begin
!!Process.getpgid(read_pid)
rescue
false
end
end
def start_external
unless external_running?
write_pid spawn("./run_server")
# Maybe some wait loop here for the external service to boot up
end
end
def stop_external
Process.kill read_pid if external_running?
end
RSpec.configure do |c|
before(:each) do |example|
start_external if example.metadata[:external]
end
after(:suite) do
stop_external
end
end
Each test tagged with :external would attempt to start the external process if it's not already started. Thus, the first time you run a test that needs it, the process would be booted. If no tests with the tag are run, the process is never booted. The suite then cleans up after itself by terminating the process as a part of the shutdown process.
This way, you don't have to pre-process the test list, your tests aren't interdependent, and your external app is automatically cleaned up after. If the external app is running before the test suite gets a chance to invoke it, it will read the pid file and use the existing instance.
Rather than relying on metadata[:external] you could parse the full name of the example and determine if it needs the external app for a more "magical" setup, but that's kind of smelly to me; example descriptions are for humans, not for the spec suite to parse.
Related
I'm writing some tests for a webpage that I'd like to run in several environments. The idea is that the test will run in one, then repeat in the next. The two environments are preview and uat.
I've written an Around hook to set the environment variables. Below:
Around do |scenario, block|
def test_envs
chosen_env = ENV['test_env'] || 'preview'
chosen_env.split(',').map(&:strip)
end
test_envs.each do |test_env|
$base_url = "https://#{test_env}.webpage.com"
end
block.call
end
I have then written a method to execute the navigation step:
def navigate_to(path)
visit $base_url + path
end
My Scenario step_definition is:
navigate_to '/login'
The tests will work in either environment, Preview by default or UAT if I set test_env=uat
However, I was aiming to set test_env=preview,uat and have them run consecutively in both environments.
Is there something obvious that I've missed here?
Thanks
If I'm understanding you correctly, it's the 'parallel' aspect that you're asking about.
Rspec can be used with parallel tests (the parallel_tests gem) but I wouldn't be so sure that calling something like 3.times { blk.call } in an around hook will run each block in parallel.
An alternative may be do so some metaprogramming with your example definitions, i.e.
test_envs.each do |env_name|
it "does something in #{env_name}" do
# do something with the specific environment
end
end
Now, I haven't actually used this gem and I don't know for sure it would work. I think the simplest solution may be to just write a wrapper script to call the tests
# run_tests.rb
environments = ENV["TEST_ENV"]&.split(",") || []\
filename = ENV["filename"]
environments.each do |env_name|
Thread.new do
system <<-SH
env TEST_ENV=#{env_name} bundle exec rspec #{filename}
SH
end
end
Running it like env TEST_ENV=foo,bar ruby run_tests.rb would call the following commands in their own threads:
env TEST_ENV=foo bundle exec rspec
env TEST_ENV=bar bundle exec rspec
I like this approach because it means you don't have to touch your existing test code.
Yes, I know I can just use load instead of require. But that is not a good solution for my use case:
When the app boots, it requires a config file. Each environment has its own config. The config sets constants.
When the app boots, only one environment is required. However, during testing, it loads config files multiple times to make sure there are no syntax errors.
In the testing environment, the same config file may be loaded more than once. But I don't want to change the require to load because every time the a spec runs, it reloads the config. This should be done via require, because if the config has already been loaded, it raises already initialized constant warnings.
The cleanest solution I can see is to manually reset the require flag for the config file after any config spec.
Is there a way to do that in Ruby?
Edit: adding code.
When the app boots it calls the init file:
init.rb:
require "./config/environments/#{ ENV[ 'RACK_ENV' ]}.rb"
config/environments/test.rb:
APP_SETTING = :foo
config/environments/production.rb:
APP_SETTING = :bar
spec/models/config.rb: # It's not a model spec...
describe 'Config' do
specify do
load './config/environments/test.rb'
end
specify do
load './config/environments/production.rb'
end
Yes it can be done. You must know the path to the files that you want to reload. There is a special variable $LOADED_FEATURES which stores what has been loaded, and is used by require to decide whether to load a file when it is requested again.
Here I am assuming that the files you want to re-require all have the unique path /myapp/config/ in their name. But hopefully you can see that this would work for any rule about the path name you can code.
$LOADED_FEATURES.reject! { |path| path =~ /\/myapp\/config\// }
And that's it . . .
Some caveats:
require does not store or follow any kind of dependency tree, to know what it "should" have loaded. So you need to ensure the full chain of requires starting with the require command you run in the spec to re-load the config, and including everything you need to be loaded, is covered by the removed paths.
This will not unload class definitions or constants, but simply re-load the files. In fact that is literally what require does, it just calls load internally. So all the warning messages about re-defining constants will also need to be handled by un-defining the constants you expect to see defined in the files.
There is probably a design of your config and specs that avoids the need to do this.
if you really want to do this, here's one approach that doesn't leak into your test process. Fork a process for every config file you want to test, communicate the status back to the test process via IO.pipe and fail/succeed the test based on the result.
You can go as crazy as you want with the stuff you send down the pipe...
Here's some quick and dirty example to show you what I mean.
a config
# foo.rb
FOO = "from foo"
another config
# bar.rb
FOO = "from bar"
some faulty config
# witherror.rb
asdf
and your "test"
# yourtest.rb
def load_config(writer, config_file)
fork do
begin
require_relative config_file
writer.write "success: #{FOO}\n"
rescue
writer.write "fail: #{$!.message}\n"
end
writer.close
exit # maybe this is even enough to NOT make it run your other tests...
end
end
rd, writer = IO.pipe
load_config(writer, "foo.rb")
load_config(writer, "bar.rb")
load_config(writer, "witherror.rb")
writer.close
puts rd.read
puts rd.read
puts rd.read
puts FOO
The output is:
success: from foo
success: from bar
fail: undefined local variable or method `asdf' for main:Object
yourtest.rb:24:in `<main>': uninitialized constant FOO (NameError)
as you can see, the FOO constant doesn't leak into your test process etc.
Of course you're only through half way because there's more to it like, making sure only one process runs the test etc.
Frankly, I don't think this is a good idea, no matter what approach you chose because you'll open a can of worms and imho there's no really clean way to do this.
Both examples are going to STDOUT, but cucumber only sees the first one. The second scenario fails with:
Then the stdout should contain "test" # aruba-0.4.11/lib/aruba/cucumber.rb:82
expected "" to include "test" (RSpec::Expectations::ExpectationNotMetError)
features/test.feature:13:in `Then the output should contain "test"'
The features:
Scenario: echo test
Given a blank slate
When I run `echo "test"`
The stdout should contain "test"
Scenario: puts test
Given a blank slate
When I start the program
The stdout should contain "test"
The step definitions:
When /^I start the program$/ do
TestModule::Main.new.start
end
The code:
module TestModule
class Main
def initialize
end
def start
$stdout.puts "test"
end
end
end
I'm not that familiar with Aruba, but a quick peek into it's source code suggests that the assertions it makes against STDOUT (or any output) only apply to processes that it started itself, and not all content that's been written to STDOUT. The code that you invoke yourself, in the second scenario, is outside of the control of Aruba, so it's output won't be tracked.
If you think about it, it couldn't really work any other way - if Aruba captured all STDOUT for assertions, then it would contain Cucumber's own test output as well...
It looks like you're trying to test your program in-process without using Aruba to invoke a separate Ruby process. If that's the case I'd suggest modifying the program to make it possible to pass in a STDOUT replacement e.g.
def initialize(output=$stdout)
Then when you start the code:
When /^I start the program$/ do
TestModule::Main.new(#output).start
end
And you can change your assertion:
Then the stdout should contain "(.+)" do |string|
#output.should include string
end
Assume there are potentially expensive operations to be performed in setup or teardown that are same for all tests and whose results doesn't get messed with during test runs. It seems not right to me to have them run before/after every single test.
So is there a preferred way to run setup/teardown code only before the first test is executed and only after the last test ran?
Edit: The particular case I'm working on should test some extensions to Net::FTP and thus establishes a FTP connection and sets up some remote objects for testing:
class TestFTPExtensions < Test::Unit::TestCase
def setup
# Setup connection
#ftp = Net::FTP.new 'localhost', 'anonymous'
#ftp.passive = true
# Create remote test directory
#ftp.mkdir 'dir'
# Create remote test file
path = File.join Dir.tmpdir, 'file'
File.open path, 'w' do |f|
#ftp.put f
end
File.delete path
end
def teardown
#ftp.rmdir 'dir'
#ftp.delete 'file'
#ftp.close
end
# imagine some tests here that don't change/remove any remote objects
end
Thanks to Andrew I found an answer to this here on stackoverflow.
However in the course of trying to find an answer I also noticed that in the 1.9.x branch the standard testing framework was switched to MiniTest. So actually I'm using that for testing right now. This answer explains how to achieve the same with MiniTest.
I am creating a Rubygem that will let me generate jekyll post files. One of the reasons I am developing this project is to learn TDD. This gem is strictly functional on the command line, and it has to make a series of checks to make sure that it finds the _posts directory. This depends on two things:
Wether or not a location option was passed
Is that location option valid?
A location option was not passed
Is the posts dir in the current directory?
Is the posts dir the current working directory?
At that point, I am really having a hard time testing that part of the application. So I have two questions:
is it acceptable/okay to skip tests for small parts of the application like the one described above?
If not, how do you test file manipulation in ruby using minitest?
Some projects I've seen implement their command line tools as Command objects (for example: Rubygems and my linebreak gem). These objects are initialized with the ARGV simply have a call or execute method which then starts the whole process. This enables these projects to put their command line applications into a virtual environment. They could, for example hold the input and output stream objects in instance variables of the command object to make the application independant of using STDOUT/STDIN. And thus, making it possible to test the input/output of the command line application. In the same way I imagine, you could hold your current working directory in an instance variable to make your command line application independent of your real working directory. You could then create a temporary directory for each test and set this one as the working directory for your Command object.
And now some code:
require 'pathname'
class MyCommand
attr_accessor :input, :output, :error, :working_dir
def initialize(options = {})
#input = options[:input] ? options[:input] : STDIN
#output = options[:output] ? options[:output] : STDOUT
#error = options[:error] ? options[:error] : STDERR
#working_dir = options[:working_dir] ? Pathname.new(options[:working_dir]) : Pathname.pwd
end
# Override the puts method to use the specified output stream
def puts(output = nil)
#output.puts(output)
end
def execute(arguments = ARGV)
# Change to the given working directory
Dir.chdir(working_dir) do
# Analyze the arguments
if arguments[0] == '--readfile'
posts_dir = Pathname.new('posts')
my_file = posts_dir + 'myfile'
puts my_file.read
end
end
end
end
# Start the command without mockups if the ruby script is called directly
if __FILE__ == $PROGRAM_NAME
MyCommand.new.execute
end
Now in your test's setup and teardown methods you could do:
require 'pathname'
require 'tmpdir'
require 'stringio'
def setup
#working_dir = Pathname.new(Dir.mktmpdir('mycommand'))
#output = StringIO.new
#error = StringIO.new
#command = MyCommand.new(:working_dir => #working_dir, :output => #output, :error => #error)
end
def test_some_stuff
#command.execute(['--readfile'])
# ...
end
def teardown
#working_dir.rmtree
end
(In the example I'm using Pathname, which is a really nice object oriented file system API from Ruby's standard library and StringIO, which is useful for for mocking STDOUT as it's an IO object which streams into a simple String)
In the acutal test you could now use the #working_dir variable to test for existence or content of files:
path = #working_dir + 'posts' + 'myfile'
path.exist?
path.file?
path.directory?
path.read == "abc\n"
From my experience (and thus this is VERY subjective), I think it's ok sometimes to skip unit testing in some areas which are difficult to test. You need to find out what you get in return and the cost for testing or not. My rule of thumb is that the decision to not test a class should be very unusual (around less than 1 in 300 classes)
If what you're trying to test is very difficult, because of the dependencies with the file system, I think you could try to extract all the bits that interact with the file system.