Rspec suite fails running on AWS Ubuntu with ECONNREFUSED - but each test passes individually - ruby

Question

We are trying to migrate our rails build from a local network server to an AWS EC2 instance running Ubuntu. The test suite works fine on all the developers machines (combination of mac and ubuntu) and the old build server (linux) but refuses to pass on the amazon instance.
The symptom is:
An error occurred in an after hook
Errno::ECONNREFUSED: Connection refused - connect(2)
occurred at /home/ubuntu/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/net/http.rb:878:in 'initialize'
which is repeated a bunch of times. There is no error logged before this one.
However this issue only occurs when running multiple tests together (eg, bundle exec rspec) - when I run each test individually they all pass. Locally none of the dev team have seen this issue before.
Our tests use Selenium and Firefox which should run headless on ubuntu using Xvfb. We also use WebMock for some tests.
I have researched this issue for a day or two now and tried everything I can think of, to no avail. I have checked the versions of firefox and all the gems installed are the same on aws as locally.
My gemfile and spec_helper are below. I have investigated the firefox and xvfb installation, but because each test is able to pass individually I feel the system is generally ok ... but clearly something is not happy.
Version details
ruby 2.0.0p481
rails 3.2.16
rspec 2.14.8
firefox 29
ubuntu 14.04
Gemfile
source 'https://rubygems.org'
#Base gems
gem 'rails', '~> 3.2.16'
gem 'sass-rails'
gem 'bcrypt-ruby', '~> 3.0.0'
gem 'faker', '~> 1.2.0'
gem 'jquery-rails', '~> 3.0.0'
#gem 'jquery-rails', '~> 2.1.4'
gem 'jquery-migrate-rails'
# threading gem
gem 'sucker_punch', '~> 1.0'
gem 'mixpanel-ruby'
# Mongo gems
gem 'mongoid', '~> 3.1.6'
gem 'bson_ext', '~> 1.9.2'
gem 'origin', '~> 1.1.0'
gem 'mongoid-enum' , '~> 0.1.1'
#filemaker gem
#gem 'ginjo-rfm', '~> 2.1.7'
# data gird gem
gem 'datagrid', '1.0.5'
# New Relic and Engine Yard Gems
gem 'ey_config', '~> 0.0.6'
gem 'newrelic_rpm'
#Authorsiation
gem 'cancan', '~> 1.6.10'
#paging
gem 'kaminari', '~> 0.15'
#client side validation
gem 'jquery-validation-rails'
#file upload
gem 'carrierwave-mongoid', :require => 'carrierwave/mongoid'
gem 'rmagick', '~> 2.13.2'
gem 'fog' , '~> 1.19.0'
gem 'unf', '~> 0.1.3'
gem 'remotipart', '~> 1.2'
gem 'listen', '2.7.6'
# reports for jenkins - see if having it here allows engineyard to deploy
gem 'ci_reporter', '1.9.1'
#NOTE YOU MUST INSTALL IMG MAGIC IF bundle install fails
#http://www.imagemagick.org/script/binary-releases.php#unix
#sudo apt-get install libmagickwand-dev imagemagick
gem 'timecop', '~> 0.7.1'
#file upload
gem 'jquery-fileupload-rails', '~> 0.4.1'
gem 'jquery-cookie-rails', '~> 1.3.1'
#configuring Rails
gem 'figaro'
#History tracking
gem 'mongoid-history'
gem 'nav_lynx'
#modal window requirements
gem 'jquery-ui-rails'
gem 'jquery-modal-rails'
gem 'itree'
gem 'lumberjack'
gem 'newrelic_moped'
group :development, :test do
gem 'rspec-rails', '~> 2.14.0'
gem 'guard-rspec', '~> 4.2.0'
gem 'guard-spork', '~> 1.5.1'
gem 'childprocess', '0.3.9'
gem 'spork', '~> 1.0.0rc4'
gem 'spork-rails', '~> 4.0.0'
gem 'bullet'
end
group :development, :staging do
gem 'rack-mini-profiler'
end
#Used to annotate model files generated
group :development do
gem 'annotate', '~> 2.6.0'
gem 'letter_opener', '~> 1.1.2'
end
# Gems used only for assets and not required
# in production environments by default.
group :assets do
gem 'coffee-rails', '~> 3.2.2'
gem 'uglifier', '~> 2.3.2'
end
group :test do
gem 'capybara', '~> 2.2.1'
gem 'selenium-webdriver', '2.39.0'
gem 'database_cleaner', '~> 1.2.0'
#needs sudo apt-get install xvfb
gem 'headless'
#static security analysis tool
gem 'brakeman', :require => false
#generating mock objects in tests
gem 'factory_girl_rails', '~> 4.3.0'
gem 'webmock', '~> 1.16.0'
#for windows
gem 'rb-fchange' if /win32/ =~ RUBY_PLATFORM
gem 'rb-notifu' if /win32/ =~ RUBY_PLATFORM
gem 'win32console' if /win32/ =~ RUBY_PLATFORM
#for linux
gem 'rb-inotify', '~> 0.9.2' if /linux/ =~ RUBY_PLATFORM
gem 'libnotify', '~> 0.8.2' if /linux/ =~ RUBY_PLATFORM
#for macosx
gem 'growl', '~> 1.0.3' if /darwin/ =~ RUBY_PLATFORM
gem 'rb-fsevent', '~> 0.9.3' if /darwin/ =~ RUBY_PLATFORM
# for rcov in Jenkins
gem 'simplecov'
gem 'simplecov-rcov'
#additional matches for rspect
gem 'shoulda-matchers', '~> 2.5.0'
#open browser on demand to debug tests
gem 'launchy'
#http rest client
gem 'rest-client', '~> 1.6.7'
gem 'show_me_the_cookies'
end
spec_helper.rb
require 'rubygems'
require 'spork'
require 'webmock/rspec'
require 'sucker_punch'
require 'sucker_punch/testing/inline'
Spork.prefork do
if ENV["RUBYMINE_HOME"]
$:.unshift(File.expand_path("rb/testing/patch/common", ENV["RUBYMINE_HOME"]))
$:.unshift(File.expand_path("rb/testing/patch/bdd", ENV["RUBYMINE_HOME"]))
end
# setup simple cov for Jenkins
require 'simplecov'
require 'simplecov-rcov'
SimpleCov.formatter = SimpleCov::Formatter::RcovFormatter
SimpleCov.start 'rails'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
require 'rspec/rails'
require 'rspec/autorun'
# Requires supporting ruby files with custom matchers and macros, etc,
# in spec/support/ and its subdirectories.
Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
RSpec.configure do |config|
config.include ShowMeTheCookies
# Include Factory Girl syntax to simplify calls to factories
config.include FactoryGirl::Syntax::Methods
#Don't run any tests that contain :skip_test => true
config.filter_run_excluding :skip_test => true
# If true, the base class of anonymous controllers will be inferred
# automatically. This will be the default behavior in future versions of
# rspec-rails.
config.infer_base_class_for_anonymous_controllers = false
# Run specs in random order to surface order dependencies. If you find an
# order dependency and want to debug it, you can fix the order by providing
# the seed, which is printed after each run.
# --seed 1234
config.order = "random"
config.include Capybara::DSL
config.before(:suite) do
Headless.new.start
end
config.before(:all) do
#manually clear database
YearDates.destroy_all
CoursePlan.destroy_all
Topic.destroy_all
FactoryGirl.create(:stub_year_dates)
end
config.after(:all) do
Timecop.return
end
#Mongoid database cleaner
config.before(:suite) do
DatabaseCleaner[:mongoid, {:connection => :unit_test}].clean_with(:truncation)
DatabaseCleaner[:mongoid, {:connection => :unit_test}].strategy = :truncation
end
config.before(:each) do
new_time = Time.local(2014, 1, 20) #week two of the year
Timecop.travel(new_time)
Timecop.baseline = new_time
DatabaseCleaner[:mongoid, {:connection => :unit_test}].start
FactoryGirl.create(:stub_year_dates) #recreate this on each test as it is needed all the time, but edited sometimes
#need to ensure these exist before the other factories are called
FactoryGirl.create(:english_subject)
FactoryGirl.create(:chemistry_subject)
FactoryGirl.create(:maths_ext_2_subject)
FactoryGirl.create(:maths_ext_1_tutorials_only_subject)
end
config.after(:each) do
Timecop.return_to_baseline
DatabaseCleaner[:mongoid, {:connection => :unit_test}].clean
end
Capybara.default_wait_time = 5 #wait longer for ajax requests
end
end
Spork.each_run do
# This code will be run each time you run your specs.
FactoryGirl.reload
end
WebMock.allow_net_connect!
I would greatly appreciate any assistance!
update
I have determined that the results are inconsistent between test runs. At first I thought it was the same each time, but the number of tests failing varies each run. I thought I had fixed the problem when all the tests passed except one, but the next run it was back to over a hundred failures.
update 2
Upon closer inspection (oops) I have discovered there were some tests failing with a different error the the majority. Instead of ECONNREFUSED, they are reporting 'End Of File Reached'. When I disabled these tests (5 in total) the remaining tests pass reliably. I suspect that xvfb is encountering some javascript or ajax related error, which is then preventing any of the following tests from passing. We are now working on diagnosing what is happening in these tests that is causing the failure.

Answer 1

File this one under 'not sure'. We ended up running up a new AWS server and setting it up again from scratch, and this time it worked fine. Some where suspect an issue with the javascript driver, possibly permissions related, but it is not worth our time investigating.