The project uses MongoDB, so this answers weren't helpfull:
1, 2, 3
I can successfully execute every test file, or even small groups of test files, but they fail when running all together. rspec spec/*
Finished in 23 minutes 45 seconds (files took 23.02 seconds to load)
2071 examples, 1357 failures, 28 pending
The project is a little bit old. MongoDB 3.6, Ruby 2.5.1 with Padrino as framework. Also, we use DatabaseCleaner and Fabrication. A lot of errors show the classic message of Sinatra doesn’t know this ditty.
61) BlogController POST :create Microsite request should return code 12 for invalid token
Failure/Error: #response_body = JSON.parse(last_response.body)
JSON::ParserError:
784: unexpected token at '<!DOCTYPE html>
<html>
<head>
<style type="text/css">
body { text-align:center;font-family:helvetica,arial;font-size:22px;
color:#888;margin:20px}
#c {margin:0 auto;width:500px;text-align:left}
</style>
</head>
<body>
<h2>Sinatra doesn’t know this ditty.</h2>
<img src='http://localhost.la/__sinatra__/404.png'>
<div id="c">
Try this:
<pre># in app.rb
class Microsite
post '/v1/microsite/blog' do
"Hello World"
end
end
</pre>
</div>
</body>
</html>
'
In addition, my boss uses Apple and he had a likewise problem, related with WiredTiger. However, I'm using Ubuntu and MongoDB log file doesn't show WiredTiger errors while executing tests.
It seems that he fixed it with a system configuration. Apparently Rspec was trying to open all the files in order to execute the tests and the process runned out of memory or something like that. Should I have to do some similar configuration in Ubuntu?
config.ru
spec/spec_helper.rb
When test A passes in isolation, but not within the whole test suite, this usually means, that a previous test B changes the state of the test environment (e.g. doesn't clean up the database or something similar) so that test B fails.
What you need to do is to find this test B, which changes the state of the test environment and then refactor this test, so that it leaves back a clean state.
I usually do this by running test A in isolation and then go the folder structure up until test A fails and then have a look, which tests ran before.
I had a look at your spec_helper.rb and have seen, that you run your specs "in order". Once you fixed this issue I would recommend you to run them in random order. This helps you to avoid dependencies between tests. Please have a look at the Rspec documentation for details.
Add this config to your spec_helper.rb
RSpec.configure do |config|
# other config
config.order = "random"
Kernel.srand config.seed
# more config
end
I also recommend you, to check the response.status and the response.header['content-type'] before you parse the response.body with JSON.parse(last_response.body).
Related
Normally Ruby test-unit will display a summary of tests run after they are finished, something like this:
Finished in 0.117158443 seconds.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10 tests, 10 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
298.74 tests/s, 0.00 assertions/s
This was working, but now something has changed and when the unit tests are run it shows the dots but then stops. I tried re-organizing some test file into different directories and made absolutely sure to change the filepaths in the testrunner. Also, the dots do not match the number of tests/assertions.
Loaded suite test
Started
.................$prompt> // <<-- does not put newline here.
I notice that if I run the testrunner from another directory, the summary will show, but it will cause errors with the test dependencies. I should be able to run the testrunner from the same directory. This is an example of the testrunner I am using: https://test-unit.github.io/test-unit/en/file.how-to.html. What are the reasons that this would not display at the end?
Seems like it could be an issue with not having the test-unit.yml file in the same directory as which you run the script.
See here in the code or in the same section in that document you posted.
See how it's configured here, for example:
runner: console
console_options:
color_scheme: inverted
color_schemes:
inverted:
success:
name: red
bold: true
failure:
name: green
bold: true
This part of the code documentation really stuck out:
# ## Test Runners
#
# So, now you have this great test class, but you still
# need a way to run it and view any failures that occur
# during the run. There are some test runner; console test
# runner, GTK+ test runner and so on. The console test
# runner is automatically invoked for you if you require
# 'test/unit' and simply run the file. To use another
# runner simply set default test runner ID to
# Test::Unit::AutoRunner:
Maybe you need to have that runner specified in your YML file?
Without seeing how you are calling your script and your directory organization, it will be hard to tell what is causing that issue but I think it begins with it not reading that yaml file for instance.
If all else fails, let me recommend two great unit testing libraries for Ruby if you feel compelled to switch to a more widely-used library:
minitest
RSpec
Edit: You could also revert your directories to be in the same order as before and hardcode in your Gemfile for that test-unit gem the last version that worked for you, like `"test-unit": "3.4.0".
So check this out: it appears as though, upon running bundle exec rspec, there's a race between jekyll serve and puma/rspec's boot up. Sometimes I run the command, and my tests run fine. Other times, I get the error for each of my spec files: cannot load such file -- /path/to/project/sitename_jekyll/_layouts/spec/form_spec.rb which is interesting cause that's not where my spec files are located. They're in /path/to/project/sitename_jekyll/spec/form_spec.rb.
What's crazy is that I can literally just re-run the command over and over and over again and sometimes it'll go through and run the spec tests in the correct location, and sometimes it'll look for them in _layouts and error out. It probably runs correctly maybe once out of ever three or five attempts. All the other times I get the following errors:
Here's what my spec_helper.rb looks like: https://gist.github.com/johnhutch/2cddfafcde0485ff021501d5696c0c2d
And here's an example test file:
https://gist.github.com/johnhutch/a35d15c170f5fd9ca07998bf035d111d
My .rspec only contains two lines:
--color
--require spec_helper
And here's the output, both successful and unsuccesful, back to back:
https://gist.github.com/johnhutch/7927d609170ef5c70a595735502b128d
HEEELLLLLP!
This sounds like jekyll is changing the current directory while building the site, which since it is being run in a thread also affects the tests RSpec is trying to run (See https://bugs.ruby-lang.org/issues/9785 for why Dir.chdir is not threadsafe) - leading to attempts to load things from incorrect locations.
A potential solution to this would be to wait for the Jekyll site to be built before actually running your tests. A comment in your spec_helper seems to state that someone thought passing force_build: true would do this but from a quick perusal of the jekyll-rack code I don't think that's true and you actually need to wait for compiling? to return false (v 0.5) (complete? to return true in the current master branch) to ensure building has finished (as well as passing force_build). This could either be done in a loop sleeping and checking (simpler)
sleep 0.1 while <jekyll app>.compiling?
or (if using the master branch) via the mutex/conditional Rack::Jekyll exposes like in its test suite - https://github.com/adaoraul/rack-jekyll/blob/master/test/helper.rb#L49
Note: Also check my comment about your tests that aren't actually testing anything.
As per Thomas Walpole's super helpful responses this ended up working:
sleep 0.1 while Capybara.app.compiling?
inserted right after:
51 Capybara.app = Rack::Jekyll.new(force_build: true)
in my spec_helper.rb
Thanks again, Thomas!
UPDATE: when I run rake test from root, I get:
C:\workspace\faker>rake test
DL is deprecated, please use Fiddle
C:/RailsInstaller/Ruby2.1.0/bin/ruby.exe -w -I"lib;test;." -I"C:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib" "C:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/rake_test_loader.r
b" "test/test_array_sample_method_compat.rb" "test/test_avatar.rb" "test/test_en_au_ocker_locale.rb" "test/test_en_ca_locale.rb" "test/test_en_locale.rb" "test/test_en_ug_locale.rb" "test/test_en_us_locale.rb" "test/test_es_l
ocale.rb" "test/test_faker.rb" "test/test_faker_app.rb" "test/test_faker_bitcoin.rb" "test/test_faker_book.rb" "test/test_faker_business.rb" "test/test_faker_city.rb" "test/test_faker_code.rb" "test/test_faker_color.rb" "test
/test_faker_commerce.rb" "test/test_faker_company.rb" "test/test_faker_date.rb" "test/test_faker_hacker_talk.rb" "test/test_faker_internet.rb" "test/test_faker_lorem.rb" "test/test_faker_name.rb" "test/test_faker_number.rb" "
test/test_faker_shakespear.rb" "test/test_faker_shakespeare.rb" "test/test_faker_slack_emoji.rb" "test/test_faker_street.rb" "test/test_faker_team.rb" "test/test_faker_time.rb" "test/test_faker_university.rb" "test/test_flexi
ble.rb" "test/test_helper.rb" "test/test_locale.rb" "test/test_pl_locale.rb" "test/test_uk_locale.rb"`
I don't fully understand what rake is doing at this point, but the system hangs infinitely. What am I missing?
So I have built some added functionality for the faker gem, and want to run the tests I wrote before submitting a pull request. However, all my attempts to run the existing tests, and my new tests, have failed. I am worried that it is a Windows machine related issue.
I have tried running rake test:units but it gives me an error and then hangs, while also ringing the Bell on my system 5 times. Strange right?
I have also tried running bundle exec rspec spec, but no tests are found becasue they are not RSpec unit tests, or at least not in the right location.
To see how the gem is formatted, check it out at the repo
All the test files, including my new one, live in the test directory. But for the life of me, I can't run them.
I tried this and works fine:
bundle install
bundle exec ruby test
I'm trying to use RSpec from within existing ruby runtime and run specs every time when file changes. This is because of JRuby and JVM startup time. To eliminate this on every run I'd like to start ruby once, then only reload changed files and run specs. I was using guard (with diffrent extensions) and watchr but all seem to suffer from an issue described below.
I nailed the issue down to RSpec itself. The problem is, when running RSpec via RSpec::Core::Runner.run several times it works fine until spec file is reloaded using load. Then RSpecs starts running specs twice.
I've created sample project showing this issue live: https://github.com/mostr/rspec_double_run_issue
Below is sample output:
ruby run_spec_in_loop.rb
Running spec from within ruby runtime
.
Finished in 0.00047 seconds
1 example, 0 failures
loading spec file via 'load' as if it was changed and we wanted changes to be picked up
Running spec from within ruby runtime
..
Finished in 0.001 seconds
2 examples, 0 failures
Is there any way to tell RSpec to clear its context between subsequent runs when run from within existing ruby runtime? I've also raised this as an issue #826 for RSpec Core project.
Summarizing the answer here in order to remove this question from the "Unanswered" filter...
Per RJHunter's observation, the explanation has been documented on the GitHub RSpec Core project here:
https://github.com/rspec/rspec-core/issues/826#issuecomment-15089030
For posterity (in case the above link dies), here are the details:
The RSpec runner is already calling load internally, your second load is what's causing the double run issue.
I quickly knocked up a script based off your example which reruns a single spec file, changes the specs to something else, then reruns them, work's correctly without the second load
See: https://gist.github.com/JonRowe/5192007
The aforementioned Gist contains:
require 'rspec'
spec_file = 'spec/sample_spec.rb'
File.open(spec_file, 'w') { |file| file.write 'describe { specify { expect(true).to eq false } }' }
1.upto(5) do |i|
puts "Running spec from within ruby runtime"
::RSpec::Core::Runner.run([spec_file], STDERR, STDOUT)
#rewriting the spec file
File.open(spec_file, 'w') { |file| file.write "describe { specify { expect(#{i}).to eq false } }" }
end
I've been playing around with autotest trying to make it work all day.. but am having some problems...
I've been following https://github.com/rspec/rspec/wiki/autotest, I'm running with:
Ruby 1.9.3-p194
rspec 2.10.0
ZenTest 4.8.1
I also created a .rspec file.
So with this setup, I run autotest, and it works - my test runs, it passes, hooray!. When I stick a failure into my test e.g. false.should == true, then the test starts looping, over and over again.
what happens is that it's an integration test, and I'm writing to an sqlite db. If I run find . -mmin -1 then I'm able to see that my db folder has changed - so I figured this is the problem.
So I edit .autotest and add the following:
Autotest.add_hook :initialize do |autotest|
%w{db}.each { |exception| autotest.add_exception(exception) }
false
end
But now when I run autotest, it just says the following:
loading autotest/rspec2
and that's it, it won't do anything anymore. Previously the output was:
loading autotest/rspec2
/home/me/.rbenv/versions/1.9.3-p194/bin/ruby -rrubygems -S '/home/me/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/rspec-core-2.10.1/exe/rspec' ``--tty '/home/me/Workspace/myproject/spec/integration/db/lead_spec.rb'
and then it'd run my test and show the result...
Anyone know what could be going on? it's very frustrating, and I feel like I've come to a road block....
Thanks for your help!
Autotest checks if defined exceptions match any part of the filename. Your spec has db in it's path so it is ignored by autotest.
If you want to ignore db folder, then do the following:
Autotest.add_hook :initialize do |a|
a.add_exception %r{^\./db}
end