Define ActiveRecord connection for given Resque queue - activerecord

Question

Problem Domain
Heroku Cedar stack, with multiple databases. RDS for main database, and Postgres for a second Analytics database. Server runs using the read/write RDS and Postgres databases. Nightly rake task, which are run in a different environment, needs to run a specific Resque queue in a read-only slave of the RDS database.
Postgres DB connection
For the record, all models in the Postgres database include:
module Trove::PostgresConnection
def self.included(base)
base.class_eval do
…set up Postgres database
end
end
end
This works fine, and, being a module injected into each class, does not get squashed by any changes to ActiveRecord::Base.connection
MySQL Connection
Defined using the Heroku RDS plugin. Connection is made to the read/write production database. Unfortunately, this connection is used regardless of environment. Thus, running a rake task on Heroku using RAILS_ENV=analytics rake some:task does not use this connection for ActiveRecord::Base:
analytics:
adapter: mysql2
encoding: utf8
database: dbase
username: uname
password: pword
host: read-only.uri.on.amazonaws.com
reconnect: true
port: 3306
Rather, it uses the connection string provided in the RDS connection:
puts Rails.env
-> 'analytics'
puts SomeModel.connection_config[:host]
-> read-write.uri.on.amazonaws.com
Took me a while to figure that out. Note to self: don't just look at environment, look at database host.
Current Workaround
# Perform an operation using another database connection in ActiveRecord
module SwapConnection
def with_connection_to(config, &block)
ActiveRecord::Base.connection.disconnect!
ActiveRecord::Base.establish_connection(config)
yield
end
end
require 'swap_connection'
class TroveCalculations
#queue = :trove_queue
def self.perform(class_name, id)
SwapConnection.with_connection_to(Rails.env) do
Do something in a given queue
end
end
end
Desired ability
Have something like this in the Procfile
troveworker: env RAILS_ENV=analytics QUEUE=trove_queue bundle exec rake resque:work
which actually uses the analytics database.yml config for that worker. We currently run our server with this Procfile, but it still uses the RDS database.

Answer 1

To expand on my comment on the question, I meant adding a config for your DB the "Heroku way" and then referencing it in your Procfile for the one worker that will process jobs on that queue.
Add a config/environment variable with the DB config you need using a new name:
heroku config:add ANALYTICS_DB=postgres://some_url
And in your Procfile, based on your example of what you want:
troveworker: env DATABASE_URL=$(ANALYTICS_DB) QUEUE=trove_queue \
bundle exec rake resque:work
You'll have to use separate workers for each queue with a different config this way, but the config will be out of the code at least.

Answer 2

I've only played with Heroku, but I thought the database connection info was overridden by the Heroku tools based on environment variables specified by the heroku toolbelt.

Answer 3

The issue here is Heroku generates its own database.yml file: https://devcenter.heroku.com/articles/ruby-support#build-behavior
By using the Amazon RDS addon, Heroku sets a DATABASE_URL environment variable. You can see its contents by running the following from the root of your applications directory:
heroku config
Also, as of Rails 3.2, it will use a DATABASE_URL env var (if set) instead of a database.yml file:
https://github.com/mperham/sidekiq/issues/503#issuecomment-11862427
The simplest workaround might be to:
create an env var called DATABASE_URL_ANALYTICS w/ the Postgres connection string:
heroku config:add DATABASE_URL_ANALYTICS=postgres://xxxxxxxxxxxx
At the beginning of your rake file (before any rails initialization might occur), add:
ENV['DATABASE_URL'] = ENV['DATABASE_URL_ANALYTICS'] if Rails.env.analytics?

Answer 4

Update: Not working. (original answer kept for documentation)
This is how we've solved it:
Procfile:
troveworker: env RAILS_ENV=analytics QUEUE=trove_queue rake trove:worker
lib/tasks/trove.rake:
desc 'Start the Resque workers in the proper environment'
task :worker do
SwapConnection.with_connection_to Rails.env do
Rake::Task['resque:work'].invoke
end
end
This solution solves some other issues for us as well, and works quite nicely. Thanks everyone.