undefined method `query_options' for nil:NilClass, resque, unicorn, Rails 4, mysql - ruby

I get this error when i try to send an update mail.
undefined method `query_options' for nil:NilClass
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/connection_adapters/mysql2_adapter.rb:218:in `execute'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:300:in `execute_and_free'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:461:in `columns'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/connection_adapters/schema_cache.rb:114:in `block in prepare_default_proc'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/connection_adapters/schema_cache.rb:56:in `yield'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/connection_adapters/schema_cache.rb:56:in `default'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/connection_adapters/schema_cache.rb:56:in `columns'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/model_schema.rb:208:in `columns'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/model_schema.rb:217:in `columns_hash'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/dynamic_matchers.rb:60:in `block in valid?'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/dynamic_matchers.rb:60:in `each'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/dynamic_matchers.rb:60:in `all?'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/dynamic_matchers.rb:60:in `valid?'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activerecord-4.0.0/lib/active_record/dynamic_matchers.rb:18:in `method_missing'
/home/user_name/work/app_name/app/mailers/user_mailer.rb:12:in `update_message'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/actionpack-4.0.0/lib/abstract_controller/base.rb:189:in `process_action'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/actionpack-4.0.0/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:373:in `_run__1485230712795781007__process_action__callbacks'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/activesupport-4.0.0/lib/active_support/callbacks.rb:80:in `run_callbacks'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/actionpack-4.0.0/lib/abstract_controller/callbacks.rb:17:in `process_action'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/actionpack-4.0.0/lib/abstract_controller/base.rb:136:in `process'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/actionpack-4.0.0/lib/abstract_controller/rendering.rb:44:in `process'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/actionmailer-4.0.0/lib/action_mailer/base.rb:503:in `process'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/actionmailer-4.0.0/lib/action_mailer/base.rb:497:in `initialize'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/resque_mailer-2.2.6/lib/resque_mailer.rb:48:in `new'
/home/user_name/.rvm/gems/ruby-1.9.3-p551#app_name/gems/resque_mailer-2.2.6/lib/resque_mailer.rb:48:in `perform'
We are using Rails 4, unicorn 4.7 and resque 1.25.
For sending emails on the application we use different mailer.
The recovery email that uses the object to send the emails, works fine, but the email that sends the profile update, which queries the user by id, does not work and throws the above error.
This is the unicorn config:
worker_processes 1
working_directory "correct/path" # available in 0.94.0+
listen "correct/path", :backlog => 64
timeout 30
pid "correct/path"
stderr_path "correct/path"
stdout_path "correct/path"
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
GC.copy_on_write_friendly = true
before_fork do |server, worker|
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
old_pid = "#{server.config[:pid]}.oldbin"
if old_pid != server.pid
begin
sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
Process.kill(sig, File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
end
end
end
after_fork do |server, worker|
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end
I have tried various answer from stack overflow, or Github issues with no result.

I finally fixed the problem, I had to restart the worker(s) after removing the octopus gem.
That error message is because octopus does not work very well out of the box with unicorn so I did configured it, but I forgot to close the worker, and it kept the octopus setting thus throwing this error.
Unfortunately I could not find a good way to use octopus with our applications because we got this error, that as far as I could see is related to Resque, more precisely when any of the workers forks a process, it fails to connect to an outside database, ex: x model connecting to y database., which works well for the application using octopus_connect, but not for the workers.
Our solution was to use instead taskrabbit/makara as the connection proxy.

Related

Occurrence of Sequel::DatabaseDisconnectError at puma worker, even though the db is disconnected in puma's before_fork-hook

I have a hanami 1.3 app, but the issue should be unrelated to hanami. I want to connect to a second db with plain Sequel-gem. Therefore I define the connection in hanami's config/environment.rb:
# config/environment.rb
# ...
DWH = Sequel.connect(ENV['DWH'], :loggers => [Logger.new($stdout)])
#...
In production I have a puma-config like that:
# config/puma.rb
require_relative './environment'
workers 5
threads_count = 1
threads threads_count, threads_count
daemonize true
preload_app!
rackup DefaultRackup
port 2300
environment 'production'
before_fork do
DWH.disconnect
end
on_worker_boot do
Hanami.boot
end
I used the before_fork hook to disconnect the db (http://sequel.jeremyevans.net/rdoc/files/doc/fork_safety_rdoc.html). But after some time I get errors like that:
Sequel::DatabaseDisconnectError: PG::UnableToSend: SSL SYSCALL error: EOF detected
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:166:in `async_exec'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:166:in `block in execute_query'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/database/logging.rb:49:in `log_connection_yield'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:166:in `execute_query'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:153:in `block in execute'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:129:in `check_disconnect_errors'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:153:in `execute'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:515:in `_execute'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:327:in `block (2 levels) in execute'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:537:in `check_database_errors'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:327:in `block in execute'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/database/connecting.rb:301:in `block in synchronize'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/connection_pool/threaded.rb:107:in `hold'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/database/connecting.rb:301:in `synchronize'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:327:in `execute'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/dataset/actions.rb:1135:in `execute'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/adapters/postgres.rb:680:in `fetch_rows'
/home/usr/vendor/bundle/ruby/2.6.0/gems/sequel-4.49.0/lib/sequel/dataset/actions.rb:155:in `each'
/home/usr/app/lib/repositories/dwh_repository.rb:39:in `to_a'
I had a conversation with the author of Sequel. It seems, that the puma config and the connection-approach is correct.
It seems, the DB-connections are dropped by another network-part (i.e. tcp-timeout, firewall, ...).
In such a case this is the expected behavior of Sequel:
The application lost connection to the database. When this happens, a DatabaseDisconnectError is raised and Sequel removes the connection from the connection pool. New connection will be created as needed up to the maximum pool size.
The best way to solve the issue, is to fix the reason for the connection-dropping (change setup [DB, server]). A pragmatic solution could be putting DB and application on same server.
If that is not possible, there's a Sequel-extension, which could be a workaround:
https://sequel.jeremyevans.net/rdoc-plugins/files/lib/sequel/extensions/connection_validator_rb.html
I had the same problem, to fix it just increase the number or threads in config, I go with preloaded 32 and 0 workers (actualy one but not in cluster mode, that would be enabled if I used workers 1)
In short, you need this
workers 0
threads 32, 32
My full puma config for custom rack app
# set app root
Dir.chdir File.expand_path("../..", __FILE__)
# load RACK_ENV
require 'dotenv'
Dotenv.load
require 'bundler/setup'
Bundler.require
plugin :tmp_restart
port 3000
log_requests false
nakayoshi_fork true
pidfile './tmp/puma.pid'
state_path './tmp/puma.state'
# activate_control_app('tcp://127.0.0.1:9000', no_token: true)
if ENV['RACK_ENV'] == 'production'
stdout_redirect './log/puma.log', './log/puma_errros.log'
environment 'production'
workers 0
threads 32, 32
# restart if there is no response on /ok, check every 10 seconds
Thread.new do
loop do
sleep 10
unless `curl -si -m 5 http://localhost:3000/ok`.include?('200 OK')
Logger.new('log/app_boot.log').error 'http://localhost:3000/ok fail, restarting'
`touch tmp/restart.txt`
end
end
end
end

Why is aws-sdk-ruby connecting to '169.254.169.254' on initialization?

I installed the gem webmock, which blocks external connections during the test suite. After I installed it, I got an unexpected result when running the test suite.
I'm somewhat confused why the aws-sdk needs to connect to 169.254.169.254 on initialization?
rspec spec/models/concerns/posconcern_spec.rb
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.
An error occurred while loading ./spec/models/concerns/posconcern_spec.rb.
Failure/Error: require File.expand_path('../../config/environment', __FILE__)
WebMock::NetConnectNotAllowedError:
Real HTTP connections are disabled. Unregistered request: GET http://169.254.169.254/latest/meta-data/iam/security-credentials/ with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'User-Agent'=>'Ruby'}
You can stub this request with the following snippet:
stub_request(:get, "http://169.254.169.254/latest/meta-data/iam/security-credentials/").
with(headers: {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'User-Agent'=>'Ruby'}).
to_return(status: 200, body: "", headers: {})
============================================================
# /home/andey/.rvm/gems/ruby-2.3.4/gems/webmock-3.1.0/lib/webmock/http_lib_adapters/net_http.rb:114:in `request'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:109:in `http_get'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:90:in `block (2 levels) in get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:105:in `open_connection'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:88:in `block in get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:121:in `retry_errors'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:87:in `get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:73:in `block in refresh'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:121:in `retry_errors'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:72:in `refresh'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/refreshing_credentials.rb:20:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:51:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:90:in `new'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:90:in `instance_profile_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:12:in `block in resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:11:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:11:in `resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/plugins/request_signer.rb:37:in `block in <class:RequestSigner>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:70:in `call'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:205:in `block in resolve_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:57:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:57:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:204:in `resolve_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:200:in `value_at'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:189:in `block in resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:189:in `resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:177:in `apply_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:150:in `build!'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:68:in `build_config'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:19:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:105:in `new'
# ./config/initializers/aws.rb:1:in `<top (required)>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `block in load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:240:in `load_dependency'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:652:in `block in load_config_initializer'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/notifications.rb:166:in `instrument'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:651:in `load_config_initializer'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:615:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:615:in `block in <class:Engine>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `instance_exec'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `run'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:55:in `block in run_initializers'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:44:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:44:in `tsort_each_child'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:54:in `run_initializers'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/application.rb:352:in `initialize!'
# ./config/environment.rb:5:in `<top (required)>'
# ./spec/rails_helper.rb:3:in `require'
# ./spec/rails_helper.rb:3:in `<top (required)>'
# ./spec/models/concerns/posconcern_spec.rb:1:in `require'
# ./spec/models/concerns/posconcern_spec.rb:1:in `<top (required)>'
Relevant link to repo:
https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/lib/aws-sdk-core/instance_profile_credentials.rb#L45
It is trying to connect to the AWS metadata server to get the AWS credentials. All AWS servers should be able to connect to the internal metadata server.
I installed the gem webmock, which blocks external connections during
the test suite
How are you blocking it? Firewall? Add a rule to allow traffic to 169.254.169.254 or stub it.
The script is trying to connect to AWS services. For that it needs credentials which can be provided in many ways. One way is to use AWS IAM role and fetch the credentials dynamically from the metadata server (169.254.169.254). Your script is connecting to 169.254.169.254 to fetch the credentials. It can be used to connect to AWS services later.
This is a follow up to #helloV 's answer.
Full documentation on
http://169.254.169.254/latest/meta-data/iam/security-credentials/
An application on the instance retrieves the security credentials
provided by the role from the instance metadata item
iam/security-credentials/role-name. The application is granted the
permissions for the actions and resources that you've defined for the
role through the security credentials associated with the role. These
security credentials are temporary and we rotate them automatically.
We make new credentials available at least five minutes prior to the
expiration of the old credentials.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials
Had a tough time researching on how to stub it. Will share my findings here hope it'll help other people.
Since AWS uses service name to fetch credentials for multiple services such as S3, any IAM Role, SQS etc.
But when dealing with IAM roles AWS hits http://169.254.169.254/latest/meta-data/iam/security-credentials/ which return the role name attached onto the machine and using the response of the same it hits the curl http://169.254.169.254/latest/meta-data/iam/security-credentials/#{role}.
So we need to stub both of the curls in order to successfully mock the IAM role. Below is the code for the same.
def self.auth(params = {})
status = params[:status].presence || 200
url = 'http://169.254.169.254/latest/meta-data/iam/security-credentials/'
WebMock.stub_request(:get, url)
.with({headers: request_headers}.compact)
.to_return(status: status, body: 'SampleRole', headers: {})
end
def self.role_auth(role = 'SampleRole')
url = "http://169.254.169.254/latest/meta-data/iam/security-credentials/#{role}"
WebMock.stub_request(:get, url)
.with({headers: request_headers}.compact)
.to_return(status: 200, body: sample_response, headers: {})
end
private
def self.request_headers
{
'Accept': '*/*',
'Accept-Encoding': 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
'User-Agent': 'aws-sdk-ruby3/3.21.2'
}.with_indifferent_access
end
def self.sample_response
{
Code: 'Success',
LastUpdated: '2020-05-08T05:55:23Z',
Type: 'AWS-HMAC',
AccessKeyId: Faker::Lorem.characters(10),
SecretAccessKey: Faker::Lorem.characters(20),
Token: Faker::Lorem.characters(60),
Expiration: '2020-05-08T12:00:45Z'
}.to_json
end
So whenver any rspec uses InstanceCredentials provider you need to stub both of the above defined requests in before block of each of the rspec.
Example:
before do
ApiStub::IamCred.auth
ApiStub::IamCred.role_auth
end
Hope it helps.
Happy Coding
There are some good answers in this thread on how to stub this. Another alternative (a better one?) is to eliminate the need for thE SDK to check for credentials from the instance meta data service. You can do this by providing credentials. These can be fake. A simple way to do this would be to provide credential via the ENV (e.g. ENV['AWS_ACCESS_TOKEN'] and ENV['SECRET_ACCESS_TOKEN']).

Error while upgrading from Rails 3.1 to Rails 3.2

Upgrade steps were performed as mentioned in http://guides.rubyonrails.org/3_2_release_notes.html
Right now I'm on Ruby 1.8.7 and Rails 3.1.x (using system ruby). I installed RVM with ruby 1.8.7 and added to rails 3.2 and then I get the following error:
[app]$ rails console
Faraday: you may want to install system_timer for reliable timeouts
$HOME/src/qbol/tapp/config/environment.rb:16:in `add': undefined method `>' for nil:NilClass (NoMethodError)
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/tagged_logging.rb:55:in `add'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/tagged_logging.rb:61:in `info'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activerecord-3.2.17/lib/active_record/railtie.rb:86
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:36:in `instance_eval'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:36:in `execute_hook'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:26:in `on_load'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:25:in `each'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/lazy_load_hooks.rb:25:in `on_load'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activerecord-3.2.17/lib/active_record/railtie.rb:80
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:30:in `instance_exec'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:30:in `run'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:55:in `run_initializers'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:54:in `each'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/initializable.rb:54:in `run_initializers'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/application.rb:136:in `initialize!'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/railtie/configurable.rb:30:in `send'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/railtie/configurable.rb:30:in `method_missing'
from $HOME/src/app/config/environment.rb:48
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:251:in `require'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:251:in `require'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:236:in `load_dependency'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:251:in `require'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/application.rb:103:in `require_environment!'
from $HOME/.rvm/gems/ruby-1.8.7-p374/gems/railties-3.2.17/lib/rails/commands.rb:40
from script/rails:6:in `require'
from script/rails:6
This is on my development box. Any idea why this is happening?
EDIT: Below is the environment.rb. The error is happening on the last line App::Application.initialize!
# Load the rails application
require File.expand_path('../application', __FILE__)
module ActiveSupport
class BufferedLogger
def self.current_user
Thread.current[:user]
end
def self.current_user=(user)
Thread.current[:user] = user
end
def add(severity, message = nil, progname = nil, &block)
return if #level > severity
message = (message || (block && block.call) || progname).to_s
level = {
0 => "DEBUG",
1 => "INFO ",
2 => "WARN ",
3 => "ERROR",
4 => "FATAL"
}[severity] || "UNKNOWN"
user=BufferedLogger.current_user
if(!user.nil?)
idstr = "uid:#{user.id}"
if !user.current_app_user.nil?
idstr.concat(", acid: #{user.current_app_user.account_id}")
end
else
idstr=""
end
message = "[%s: %s #{idstr}] %s" %
["#{level} pid: #{$$}", Time.now.strftime("%y-%m-%d %H:%M:%S"), message]
message = "#{message}\n" unless message[-1] == ?\n
buffer << message
auto_flush
message
end
end
end
# Initialize the rails application
App::Application.initialize!
The error is happening inside ActiveSupport::BufferedLogger#add method call and you appear to be monkey-patching this class. The internals of ActiveSupport::BufferedLogger have likely changing between the versions of Rails you're using.
Try removing all of the ActiveSupport::BufferedLogger code from your environment.rb to work past this error and get your app running. Then, if you still need the monkeypatch (don't know why), you'd have to rewrite it on top of the newer version of the class.

Sequel gem: Handling invalid date values gracefully

I'm a ruby noob and I'm trying to process some blog posts using Sequel and the data_objects adapter:
DB = Sequel.connect('do:mysql://user:pass#localhost/database')
db[posts_query].each do |post|
puts post
end
But I get Sequel::InvalidValue exception, complaining about the date column:
/usr/lib/ruby/1.9.1/time.rb:202:in `local': ArgumentError: argument out of range (Sequel::InvalidValue)
from /usr/lib/ruby/1.9.1/time.rb:202:in `make_time'
from /usr/lib/ruby/1.9.1/time.rb:271:in `parse'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/core.rb:295:in `string_to_datetime'
I tried to catch the exception:
begin
db[posts_query].each do |post|
puts post
end
rescue Sequel::InvalidValue => e
puts e.inspect
end
but that doesn't help much.
How can I find out which row has the incorrect value?
Also, is there a way to do this iteration, such that I can catch the exception but continue to loop over the remaining rows?
Update:
I switched to the mysql2 adapter and now I can at least see the invalid date:
/var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/adapters/mysql2.rb:154:in `each': Mysql2::Error: Invalid date: 2008-04-00 00:00:15 (Sequel::DatabaseError)
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/adapters/mysql2.rb:154:in `block in fetch_rows'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/adapters/mysql2.rb:89:in `_execute'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:34:in `block in execute'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/database/connecting.rb:236:in `block in synchronize'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/connection_pool/threaded.rb:104:in `hold'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/database/connecting.rb:236:in `synchronize'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:34:in `execute'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/dataset/actions.rb:778:in `execute'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/adapters/mysql2.rb:171:in `execute'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/adapters/mysql2.rb:140:in `fetch_rows'
from /var/lib/gems/1.9.1/gems/sequel-3.42.0/lib/sequel/dataset/actions.rb:154:in `each'
from wordpress_importer.rb:112:in `process'
from wordpress_importer.rb:308:in `<main>'
Can you post more of the backtrace? You need to see what is calling string_to_datetime.
Also, I would recommend against using the do/mysql adapter unless you have specific needs that require it. Use the mysql or mysql2 adapter instead. If the error is being caused by bogus datetimes in your MySQL database, then you may want to use the mysql2 adapter or use the mysql adapter and set DB.convert_invalid_date_time = nil.

Skipping slow websites when looping through an array of URLs using watir-webdriver

I'm trying to loop through an array of websites in Chrome using watir-webdriver, but I always encounter an error on certain websites. Recently, I have had this problem with http://adage.com. The loop will execute perfectly until it reaches http://adage.com and then it will hang until the following error is displayed:
/Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill': Timeout::Error (Timeout::Error)
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1293:in `request'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1286:in `block in request'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:745:in `start'
from /Users/default/.rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/net/http.rb:1284:in `request'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/http/default.rb:82:in `response_for'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/http/default.rb:38:in `request'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/http/common.rb:40:in `call'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/bridge.rb:598:in `raw_execute'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/bridge.rb:576:in `execute'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/remote/bridge.rb:536:in `getActiveElement'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/selenium-webdriver-2.25.0/lib/selenium/webdriver/common/target_locator.rb:60:in `active_element'
from /Users/default/.rvm/gems/ruby-1.9.3-p125/gems/watir-webdriver-0.6.1/lib/watir-webdriver/browser.rb:136:in `send_keys'
from /Users/default/Dropbox/beta_scripts/loop_test.rb:16:in `rescue in <main>'
from /Users/default/Dropbox/beta_scripts/loop_test.rb:11:in `<main>'
I have no idea how to avoid this. I have tried setting timeouts and even sending the ESC key during rescue to stop Chrome from loading the page, but have not had any success. Ultimately, I want to be able to reliably load an array of 500+ websites in succession, but this seems impossible given the likelihood that one of the websites will hang. Is there any way to stop a slow page from loading and move on to the next element in the array?
Below is a shortened version of my code that isolates the problem:
#!/usr/bin/env ruby
require 'watir-webdriver'
b = Watir::Browser.new :chrome
sites = ["twitter.com", "cars.com", "autotrader.com", "rolex.com", "newyorker.com", "adage.com", "theatlantic.com", "pcmag.com"]
sites.each do |uri|
begin
Timeout::timeout(10) do
b.goto uri
end
rescue Timeout::Error => e_time
sleep 5
b.send_keys :escape
p "#{uri} is taking forever to load (#{e_time})"
rescue Exception => e_exception
p e_exception
end
end
b.close
Well I can understand your frustration mate because I have encountered the same when dealing with selenium webdriver. Here it is what you need to do to be 100% sure that your script will run flawless and robust till the end for your 500+ websites.
sites.each do |uri|
!30.times { if ((b.goto uri)rescue false)then break else sleep 1; end }
end
The code above will try to access each website for a maximum of 30sec and then move to the next website.

Resources