I'm not very experienced in ruby (coding in java on daily basis) and I'm trying to find "the Ruby way" of keeping things like connection pools around. I've standalone Ruby app with multiple threads and I came up with something like below.
Note that MongoClient provided by native ruby driver for Mongo maintains conneciton pool internally, so all I need is to be able to keep one instance of MongoClient around
require 'mongo'
module MongoPool
# module instance var to ensures only one exists
#mongo = nil
def self.lazy_create
#mongo ||= Mongo::MongoClient.new('localhost', 27017, :pool_size => 5, :timeout => 5)
end
# when getting connection lazily create pool by assigning to #mongo only if nil
def connection
MongoPool.lazy_create
end
end
class PeopleRepository
include MongoPool
def random_person
coll = connection['test']['people']
coll.find_one
end
end
# usage
PeopleRepository.new.find_one
I know that works (checked that object_id of #mongo remains the same across several invocations), but is this preferred way to keep things around?
There may be more than one Repositories, so each can incluse MongoPool and use its connections. Are there any drawbacks of the solution above? Are there any other ways I should be aware of?
NOTE: The question is more about how to do things in Ruby way, not about how to do it in general (as I got it working).
You don't really need another gem to do this and actually Mongoid's driver (Moped) doesn't support connection pooling yet anyway.
Similar to the recommendation to use an application level constant in rails, you just need to use a class variable in your headless application so that your MongoClient instance is the same single object/pool instance across all invocations of your application's base class.
For example, you could do something like this:
require 'mongo'
class MyApplication
include Mongo
# creates a single class instance, sets pool size but won't connect until used (lazy)
##client = MongoClient.new('localhost', 27017, :pool_size => 5, :connect => false)
def do_something
##client['my_db']['my_collection].insert({ "foo" => "bar"})
end
end
Simple and very straight-forward. The module approach you used above isn't necessary.
You mentioned Torquebox so I assume you're using JRuby and letting Torquebox manage your application's thread pool for you.
Make sure you're running version 1.8.3 or greater of the mongo ruby driver which includes some major fixes and improvements for running under that kind thread-heavy of setup. We addressed a few thread safety issues and greatly improved concurrency in the connection pool.
Hope that helps.
assuming you are using rails, i would do this:
# config/initializers/mongo.rb
MONGODB = Mongo::MongoClient.new('localhost', 27017, :pool_size => 5, :timeout => 5)
when you are using a library such as mongomapper, there are probably ways to configure pooling and just use it transparently.
have a look at mongo libraries here: http://railscasts.com/episodes/194-mongodb-and-mongomapper
Related
I'm writing a module which will use Sequel ORM. I want to be able to set the DB constant so it's available in the global namespace but I can't figure out a non-hacky way to do so. Here's what I have so far:
require 'sequel'
module BB
class Dal
def initialize(db_url)
self.class.connect(db_url)
end
def self.connect(db_url)
#db = Sequel.connect( db_url )
end
end
BB::Dal.connect(ENV['DATABASE_URL']) if ENV['DATABASE_URL']
end
As you can see I'd like to enable two modes of initialization. Either through the ENV['DATABASE_URL'] variable or through a constructor. What I'd like to do now is replace #db with DB. Any ideas?
You can just kick that out of the module context if you want:
DB = BB::Dal.connect(ENV['DATABASE_URL']) if ENV['DATABASE_URL']
It's worth noting that jamming these things in the root namespace is probably bad form. You might want to have a database connection module to organize these:
module DB
def connection
# Lazy initializer pattern applied here
#connection ||= BB::Dal.connect(ENV['DATABASE_URL'])
end
end
# ...
# Will connect the first time it's called, recycle the connection
# for all subsequent calls.
DB.connection.do_stuff
It really depends on how this will be used. The advantage of a method is it's very easy to upgrade to a connection pool should the occasion arise, whereas a constant is very rigid in terms of function, it must return a singular value with no opportunity to run code first.
I have built a pretty simple REST service in Sinatra, on Rack. It's backed by 3 Tokyo Cabinet/Table datastores, which have connections that need to be opened and closed. I have two model classes written in straight Ruby that currently simply connect, get or put what they need, and then disconnect. Obviously, this isn't going to work long-term.
I also have some Rack middleware like Warden that rely on these model classes.
What's the best way to manage opening and closing the connections? Rack doesn't provide startup/shutdown hooks as I'm aware. I thought about inserting a piece of middleware that provides reference to the TC/TT object in env, but then I'd have to pipe that through Sinatra to the models, which doesn't seem efficient either; and that would only get be a per-request connection to TC. I'd imagine that per-server-instance-lifecycle would be a more appropriate lifespan.
Thanks!
Have you considered using Sinatra's configure blocks to set up your connections?
configure do
Connection.initialize_for_development
end
configure :production do
Connection.initialize_for_production
end
That's a pretty common idiom while using things like DataMapper with Sinatra
Check out the "Configuration" section at http://www.sinatrarb.com/intro
If you have other Rack middleware that depend on these connections (by way of a dependence on your model classes), then I wouldn't put the connection logic in Sinatra -- what happens if you rip out Sinatra and put in another endpoint?
Since you want connection-per-application rather than connection-per-request, you could easily write a middleware that initialized and cleaned up connections (sort of the Guard Idiom as applied to Rack) and install it ahead of any other middleware that need the connections.
class TokyoCabinetConnectionManagerMiddleware
class <<self
attr_accessor :connection
end
def initialize(app)
#app = app
end
def call(env)
open_connection_if_necessary!
#app.call(env)
end
protected
def open_connection_if_necessary!
self.class.connection ||= begin
... initialize the connection ..
add_finalizer_hook!
end
end
def add_finalizer_hook!
at_exit do
begin
TokyoCabinetConnectionManagerMiddleware.connection.close!
rescue WhateverTokyoCabinetCanRaise => e
puts "Error closing Tokyo Cabinet connection. You might have to clean up manually."
end
end
end
end
If you later decide you want connection-per-thread or connection-per-request, you can change this middleware to put the connection in the env Hash, but you'll need to change your models as well. Perhaps this middleware could set a connection variable in each model class instead of storing it internally? In that case, you might want to do more checking about the state of the connection in the at_exit hook because another thread/request might have closed it.
I'm currently developing a framework that basically executes another application, e.g. rails within the context of another ruby program. My initial attempt was simply to boot the app like this:
def load_app!
# Load the rails application
require './config/application'
# Initialize the rails application
#app = App::Application.initialize!
end
Problem here, is that the framework's requires conflict with the loaded application so the initialize! call never works although it would in a normal ruby program.
So my question is, if anyone knows a method to basically scope this calls into a unit that behaves like a blank RVM environment. So basically a behavior like this:
require 'json'
puts JSON.generate({:name => "test"})
blank_environment do
puts JSON.generate({:name => "test"})
#=> uninitialized constant JSON
require 'json'
puts JSON.generate({:name => "test"})
end
It's not done with undefining or unloading the currently loaded constants because I don't know all of them because I'm using gems that have other dependencies again.
So is there a cool way? Or any other way to handle this?
UPDATE:
Just came across an idea. Why is ruby's require method always requiring for the global scope? Wouldn't it be a very nice feature to actually scope the loaded modules under the the current module?
module ScopeA
require 'json' #> adds support for ScopeA::JSON
# due to normal ruby scoping everything can be called like normal in here
JSON.parse("something")
end
# JSON should not be available here
module ScopeB
require 'yaml'
YAML.parse("something") # but no JSON, of course
end
Doesn't something like this exist? include already has to know the constants...
Thanks in advance!
Well, after some more research it really doesn't seem possible the way I need it.
I now implemented a basic version using distributed ruby, which doesn't quite satisfy me:
require 'drb/drb'
URI = "druby://localhost:8787"
# Blank environment
pid = fork do
Signal.trap("INT") { puts "Stopping Server.."; exit }
class Application
def call(env)
[200,{},""]
end
end
DRb.start_service(URI, Application.new)
DRb.thread.join
end
# Working environment
DRb.start_service
app = DRbObject.new_with_uri(URI)
puts app.call({})
Process.kill("INT", pid)
Process.wait
If anyone comes up with a better approach, it's highly appreciated!
I'm trying to upgrade the ActiveRecord gem to the latest 3.1.0 release and seeing a lot of exceptions being raised, I think it's due to how we handle multiple databases.
For each of our databases we specify a separate base class which inherits from ActiveRecord::Base, and call establish_connection in there. There are no cross-database relations. This has worked fine for us up until now.
Having upgraded to ActiveRecord 3.1.0 I'm seeing that it fails with an ActiveRecord::ConnectionNotEstablished exception, when traversing relations (i.e. it will successfully pull a single entity or set of them from the DB, but fails when navigating to a related class).
The top line of the backtrace is C:/Ruby/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/connection_adapters/abstract/connection_pool.rb:410:in 'retrieve_connection', so I dug into this a little. The method is defined as follows:
def retrieve_connection(klass) #:nodoc:
pool = retrieve_connection_pool(klass)
(pool && pool.connection) or raise ConnectionNotEstablished
end
My simple test (puts Customer.first.address) calls retrieve_connection 3 times. Twice with Customer as the klass parameter, and once with ActiveRecord::Base as the parameter - which is when it fails as establish_connection has not been called for ActiveRecord::Base.
To the actual question then - is there a new recommended way of handling multiple database connections in ActiveRecord? If so, what is it?
If not, what could be causing this problem?
I ran into the same issue yesterday while upgrading to ActiveRecord 3.1.0. I can't speak to whether there is a new recommended way of handling multiple database connections in ActiveRecord 3.1, but I did find a way to unblock myself.
It appears a connection must now be established on ActiveRecord::Base in order for it to determine the table name lengths/rules of the adapter. Along with the rest of my connections established in my database initializer, I now also have an ActiveRecord::Base connection established to one of my DBs (it doesn't matter which one).
I'd like to think there's a better solution to be found, but I'm happy to be unblocked for now.
I am using this solution - What I was seeing was that when establish_connection was called in each of the OtherDb classes - there seemed to be alot of overhead reloading table definitions and I would randomly see issues every time the class def was reloaded.
# The idea here is to specify that a given model should use another
# database without having to change the entire inheritance hierarchy
# declare model for table in primary connection
class Bar < ActiveRecord::Base
# assume we have logic here that we don't want to refactor into a module
# but we do want to inherit in OtherDb::Bar
end
module Foo
# base model in Foo namespace - uses another db
class BaseConnection < ActiveRecord::Base
# OtherDb::Title.database contains a db config hash
# This would probably go in the initializers
establish_connection OtherDb::Title.database
end
# module used to override db connection
module OtherDb::Base
def retrieve_connection
# connection_handler.retrieve_connection(self) # normal behavior
connection_handler.retrieve_connection(Foo::BaseConnection) # use db from Foo::BaseConnection
end
end
# Foo::Bar is identical to ::Bar but is in another db
class Bar < ::Bar
extend OtherDb::Base
end
end
I have built a pretty simple REST service in Sinatra, on Rack. It's backed by 3 Tokyo Cabinet/Table datastores, which have connections that need to be opened and closed. I have two model classes written in straight Ruby that currently simply connect, get or put what they need, and then disconnect. Obviously, this isn't going to work long-term.
I also have some Rack middleware like Warden that rely on these model classes.
What's the best way to manage opening and closing the connections? Rack doesn't provide startup/shutdown hooks as I'm aware. I thought about inserting a piece of middleware that provides reference to the TC/TT object in env, but then I'd have to pipe that through Sinatra to the models, which doesn't seem efficient either; and that would only get be a per-request connection to TC. I'd imagine that per-server-instance-lifecycle would be a more appropriate lifespan.
Thanks!
Have you considered using Sinatra's configure blocks to set up your connections?
configure do
Connection.initialize_for_development
end
configure :production do
Connection.initialize_for_production
end
That's a pretty common idiom while using things like DataMapper with Sinatra
Check out the "Configuration" section at http://www.sinatrarb.com/intro
If you have other Rack middleware that depend on these connections (by way of a dependence on your model classes), then I wouldn't put the connection logic in Sinatra -- what happens if you rip out Sinatra and put in another endpoint?
Since you want connection-per-application rather than connection-per-request, you could easily write a middleware that initialized and cleaned up connections (sort of the Guard Idiom as applied to Rack) and install it ahead of any other middleware that need the connections.
class TokyoCabinetConnectionManagerMiddleware
class <<self
attr_accessor :connection
end
def initialize(app)
#app = app
end
def call(env)
open_connection_if_necessary!
#app.call(env)
end
protected
def open_connection_if_necessary!
self.class.connection ||= begin
... initialize the connection ..
add_finalizer_hook!
end
end
def add_finalizer_hook!
at_exit do
begin
TokyoCabinetConnectionManagerMiddleware.connection.close!
rescue WhateverTokyoCabinetCanRaise => e
puts "Error closing Tokyo Cabinet connection. You might have to clean up manually."
end
end
end
end
If you later decide you want connection-per-thread or connection-per-request, you can change this middleware to put the connection in the env Hash, but you'll need to change your models as well. Perhaps this middleware could set a connection variable in each model class instead of storing it internally? In that case, you might want to do more checking about the state of the connection in the at_exit hook because another thread/request might have closed it.