I have a script to run all tests in a directory, using require; I do:
Dir.files.each
...
and then
require 'path'
...
I try to use another loop to run those tests a few times, passing different arguments. I have my arguments in an array, and I go through the array, then run the same code as above to run all tests in a directory. In this loop, I print a line:
puts executing tests for a[i]
and the next line is the require that runs the set of tests.
The problem is that the print executes e.g., ten times (ten lines printed together), but require only runs in the end, passing only the very last element of the array. I tried different statements, and they all run fine, so I don't believe it's a problem in the loop; I think it's the require. I tried load, but didn't see any difference. 'exec' only runs the first test in the set. Any ideas?
some more details:
thanks for the replies! the system command is much closer to what I wanted - it runs all tests for me.
I have below an example of what I'm trying to do.
When I run the script once passing specific arguments 'a', I get the following results:
#### Run all tests for 'a' ####
Loaded suite
............................
Finished in 220.123 seconds
If I put my arguments in an array eg. ar = ['a','b','c','d']
I get
#### Run all tests for 'a' ####
#### Run all tests for 'b' ####
#### Run all tests for 'c' ####
#### Run all tests for 'd' ####
Loaded suite
............................
Finished in 220.123 seconds
ie. the tests run for the last option only (d)
If I use 'system' every single file runs individually - which makes it hard to go through the results for say 100 tests for a few different runs.
The code snippet is :
for i in 0 .. #ar.length-1 do
puts'## Running : '+ #ar[i] + ' ##'
Dir.entries('./suite_dir').each do | file |
require './suite_dir/'+ file
end
end
Don't use require like that, it's not intended to be executable...it only works once.
From http://www.ruby-doc.org/core-1.9.3/Kernel.html#method-i-require
require(name) → true or false
Loads the given name, returning true if successful and false if the
feature is already loaded.
You can use something like this to run all files from the directory:
Dir.foreach(path) do |file|
puts "Testing #{file}"
system(path+file)
end
Related
I have an rspec test defined like below. I know that I can run rspec -e "guessing" to run the full block of tests, but I want to run only specific contexts at a time. This way I can code the "correctly" tests first, and then later the "incorrectly" portion. Is there a command line way run the specified tests without naming them individually?
describe 'guessing' do
context 'correctly' do
it 'changes correct guess list' do
end
it 'returns true' do
end
end
context 'incorrectly' do
it 'changes wrong guess list' do
end
it 'returns true' do
end
end
end
You can use -e to match any part of the description, so rspec -e incorrectly will run those two tests. (I don't know of a way to only match "correctly", as it's also a substring of "incorrectly".)
You can also use a line number reference to match a context block: rspec your_spec.rb:2 (given the above content exactly, with context 'correctly' do on line 2) will run that set of specs.
I have two load tests below with each one being in their separate test cases. This is using SOAP UI free:
Currently I have to manually select a load test, run it manually, wait until it finishes and then manually export the results before manually moving onto the next load test and performing the same actions.
Is there a way (and if so how) to be able to automatically run all the load tests (one by one) and extract each of it's own set of results in a file (test step, min, max avg, etc). This is to save the tester having to do manual intervention and can just let the test run whilst they do other stuff.
You can use the load tests command line, the doc is here.
Something like
loadtestrunner -ehttp://localhost:8080/services/MyService c:\projects\my-soapui-project.xml -r -f folder_name
Using these two options:
r : Turns on exporting of a LoadTest statistics summary report
f : Specifies the root folder to which test results should be exported
Then file like LoadTest_1-statistics.txt will be in your specified folder with csv statistics results.
inspired with answer of #aristotll )
loadtestrunner.bat runs the following class : com.eviware.soapui.tools.SoapUITestCaseRunner
from groovy you can call the same like this:
com.eviware.soapui.tools.SoapUITestCaseRunner.main([
"-ehttp://localhost:8080/services/MyService",
"c:\projects\my-soapui-project.xml",
"-r",
"-f",
"folder_name"
])
but the method main calls System.exit()...
and soapui will exit in this case.
so let's go deeper:
def res = new com.eviware.soapui.tools.SoapUITestCaseRunner().runFromCommandLine([
"-ehttp://localhost:8080/services/MyService",
"c:\projects\my-soapui-project.xml",
"-r",
"-f",
"folder_name"
])
assert res == 0 : "SoapUITestCaseRunner failed with code $res"
PS: did not tested - just an idea
Is there a Ruby Cucumber test hook for at_start? I tried at_start and it didn't work.
I have something like this in support/hooks.rb and I want to print a single global message before any of the tests start:
Before do
print '.'
end
at_exit do
puts ''
puts 'All Cucumber tests finished.'
end
It seems like if they have an at_exit hook, they should have a before-start hook as well right?
There is some documentation for "global hooks" at https://github.com/cucumber/cucumber/wiki/Hooks
You don't need to wrap it in any special method such as Before or at_exit. You just execute the code at the root level in any file contained in the features/support directory, such as env.rb. To copy and paste the example they've given:
# these following lines are executed at the root scope,
# accomplishing the same thing that an "at_start" block might.
my_heavy_object = HeavyObject.new
my_heavy_object.do_it
# other hooks can be defined in the same file
at_exit do
my_heavy_object.undo_it
end
They also give an example of how to write a Before block that gets executed only once. Basically you have this block exit if some global variable is defined. The first time the block is run, the global variable is defined which prevents it from being executed multiple times. See the "Running a Before hook only once" section on that page I linked.
Inside TestKitchen describe blocks I'm running a command, loading its output into a variable then running multiple expect statements over that output validating different parts of it. The end goal is using this as part of CI builds to do blackbox testing.
In this instance I'm calling Jmeter (using it to run a remote agent to perform off-DUT tests) then running through the results that it returns checking each test (yeah yeah... it's a little nasty but it works a treat):
describe "Test Transparent Proxy (JMeter)" do
$jmeter_run = command("/usr/local/apache-jmeter-2.13/bin/jmeter -n -t /root/jmx/mytest.jmx -r -Jremote_hosts=192.168.7.252 -Gdut_ip=#$internal_ip -X -l /dev/stdout 2>&1").stdout
it 'test1' do
expect($jmeter_run).to match /text_to_match/
end
it 'test2' do
expect($jmeter_run).to match /more_text to match/
end
end
The tests themselves run fine, but I'm finding that I'm getting multiple jmeter runs (different test sets) being run out-of-order as to how they're defined in the test spec. I have other blocks that are being executed around the Jmeter tests. Here is my flow:
block 1
block 2
block 3 (Jmeter1)
block 4
block 5 (Jmeter2)
What I'm getting though is this:
block5
block3
block1
block2
block4
None of the documentation I've found seems to give me any clues as to how to avoid this. I don't want to put the command execution inside a should/expect chunk of its own as I want/need to be able to tell if an individual test has failed. I would also like to avoid running 50-odd individual Jmeter tests (they're about 5 secs each even with an avg of 20 tests in each run).
Help? :D
Well I managed to resolve this issue myself.
After a lot of tinkering I ended up running the command inside a test:
it 'JMeter executed correctly' do
$jmeter_run1 = command("/usr/local/apache-jmeter-2.13/bin/jmeter -n -t /root/jmx/mytest.jmx -r -Jremote_hosts=192.168.7.252 -Gdut_ip=#$internal_ip -X -l /dev/stdout 2>&1").stdout
expect($jmeter_run1).not_to be_empty
end
Everything now runs nicely in order like it is supposed to and everything is happy.
Both examples are going to STDOUT, but cucumber only sees the first one. The second scenario fails with:
Then the stdout should contain "test" # aruba-0.4.11/lib/aruba/cucumber.rb:82
expected "" to include "test" (RSpec::Expectations::ExpectationNotMetError)
features/test.feature:13:in `Then the output should contain "test"'
The features:
Scenario: echo test
Given a blank slate
When I run `echo "test"`
The stdout should contain "test"
Scenario: puts test
Given a blank slate
When I start the program
The stdout should contain "test"
The step definitions:
When /^I start the program$/ do
TestModule::Main.new.start
end
The code:
module TestModule
class Main
def initialize
end
def start
$stdout.puts "test"
end
end
end
I'm not that familiar with Aruba, but a quick peek into it's source code suggests that the assertions it makes against STDOUT (or any output) only apply to processes that it started itself, and not all content that's been written to STDOUT. The code that you invoke yourself, in the second scenario, is outside of the control of Aruba, so it's output won't be tracked.
If you think about it, it couldn't really work any other way - if Aruba captured all STDOUT for assertions, then it would contain Cucumber's own test output as well...
It looks like you're trying to test your program in-process without using Aruba to invoke a separate Ruby process. If that's the case I'd suggest modifying the program to make it possible to pass in a STDOUT replacement e.g.
def initialize(output=$stdout)
Then when you start the code:
When /^I start the program$/ do
TestModule::Main.new(#output).start
end
And you can change your assertion:
Then the stdout should contain "(.+)" do |string|
#output.should include string
end