Connection Pool returning an instance of Redis and not ConnectionPool - ruby

I am trying to make my Rails app use Resque for managing workers. But, I would like to continue using the ConnectionPool gem.
I have this in an initializer:
puts ENV["REDISTOGO_URL"]
uri = (not ENV["REDISTOGO_URL"].nil?) ? URI.parse(ENV["REDISTOGO_URL"]) : nil
# at this point, debugger confirms $redis is nil
$redis = ConnectionPool::Wrapper.new(:size => 5, :timeout => 3) {
if uri.nil?
Redis.connect
else
Redis.connect(:host => uri.host, :port => uri.port, :password => uri.password)
end
}
$redis # just put this in here for the debugger
# At this point, $redis is #<Redis:0x007fb1b0036bf0>
# when it should be an instance of ConnectionPool::Wrapper
Does anyone have an idea why $redis would not be returned as an instance ConnectionPool::Wrapper?
I've searched in all the gems source code, nowhere does it set the value of $redis. In ConnectionPool's source code, I did not find anything where it would return an instance of Redis instead of itself.
This only happens when I switched from DelayedJob to Resque. So, it would seem that is the problem. However, I'm at a loss.
I am using Unicorn. Here's that file in config.
worker_processes 2
timeout 30
preload_app true
before_fork do |server, worker|
# Replace with MongoDB or whatever
if defined?(ActiveRecord::Base)
ActiveRecord::Base.connection.disconnect!
Rails.logger.info('Disconnected from ActiveRecord')
end
# If you are using Redis but not Resque, change this
if defined?(Resque)
Resque.redis.quit
Rails.logger.info('Disconnected from Redis')
end
sleep 1
end
after_fork do |server, worker|
# Replace with MongoDB or whatever
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
Rails.logger.info('Connected to ActiveRecord')
end
# If you are using Redis but not Resque, change this
if defined?(Resque)
# Yes, commented the Resque out for debugging, still get the same problem.
#Resque.redis = ENV['REDISTOGO_URL']
Rails.logger.info('Connected to Redis')
end
end
And finally, the Procfile:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: env TERM_CHILD=1 QUEUE=* bundle exec rake resque:work
I'm using foreman in my develop environment.
Any help is greatly appreciated.

From the docs:
You can use ConnectionPool::Wrapper to wrap a single global connection.
I looks like the ConnectionPool::Wrapper is meant to wrap a single connection to Redis as a convenience for migrating large applications from using Redis directly to using ConnectionPools
If you call $redis.with, you get the #with defined by ConnectionPool
To get an actual connection pool, just change your
ConnectionPool::Wrapper.new(:size => 5, :timeout => 3) { #redis logic }
to
ConnectionPool.new(:size => 5, :timeout => 3) { #redis logic }

Internally ConnectionPool::Wrapper creates a normal ConnectionPool object, and uses method_missing to automatically checkout/checkin from that pool whenever any method is called on the wrapper.
This use of method_missing includes calls to inspect, or class, or any number of methods which are normally used to try to look at the object or figure out its type.
require 'connection_pool'
class MyClass
def foo
'bar'
end
end
obj = MyClass.new
obj.respond_to?(:foo) # true
obj.respond_to?(:with) # false
wrapper = ConnectionPool::Wrapper.new { MyClass.new }
wrapper.respond_to?(:foo) # true
wrapper.respond_to?(:with) # also true! 'with' is a method on ConnectionPool::Wrapper
You do have an instance of ConnectionPool::Wrapper, it just is a bit hard to tell.

Related

Optimistic locking in Redis-rb does not seem to be working

Spent hours trying to get a test case working. From my understanding, after the watch block is started, if that key is changed between then and the exec time, the multi block should do nothing.
redis = Redis.new
thread = Thread.new do
redis.watch("test_case") do
sleep 20
redis.multi do |setup|
redis.set "test_case", "foo"
end
end
end
and then during the 20 second sleep in IRB
redis.set "test_case", "bar"
The result is "test_case" => "foo". It should be "bar" because the watch should not allow execution of the multi statement. I have also tested this in redis-cli and the commands work as expected.
Issue was I had a module defining:
def redis
#redis = Redis.new
end
Everytime I called redis, a new object was made.

celluloid-io or eventmachine with mosquitto loops

I'm building a small ruby program to run a connection to a MQTT server and subscribe to a channel. I'm using the mosquitto gem which is just a bridge for libmosquitto C library.
I created a very simple implementation of a program that can run with ruby my_prog.rb:
# Dependencies
require File.expand_path(File.join('..', 'environment'), __FILE__)
# MQTT Application
module Pulsr
class MQTT
attr_reader :host, :port, :alive
def initialize(host = 'iot.eclipse.org', port = 1883, alive = 60)
#client ||= Mosquitto::Client.new SecureRandom.hex(8)
Signal.trap(Signal.list.has_key?('INT') ? 'SIGINT' : 'SIGTERM') do
#client.log 'Shutdown'
shutdown
end
#host = host
#port = port
#alive = alive
start
end
private
def on_connect
Proc.new { |return_code|
#client.log "Connected RC #{return_code}"
#client.subscribe(nil, '/pulsr', Mosquitto::EXACTLY_ONCE)
}
end
def on_disconnect
Proc.new { |return_code| #client.log "Disconnected RC #{return_code}" }
end
def on_subscribe
Proc.new { |message_id, granted_qos| #client.log "Subscribed MID #{message_id} QoS #{granted_qos}" }
end
def on_unsubscribe
Proc.new { |message_id| #client.log "Unsubscribed MID #{message_id}" }
end
def on_message
Proc.new { |message| Pulsr::Workers::TrackingEvent.perform_async message.to_s }
end
def configure
#client.logger = Logger.new(STDOUT)
#client.on_connect &on_connect
#client.on_disconnect &on_disconnect
#client.on_subscribe &on_subscribe
#client.on_unsubscribe &on_unsubscribe
#client.on_message &on_message
end
def connect
#client.connect_async(#host, #port, #alive)
end
def start
#client.loop_start
configure
connect
sleep
end
def shutdown
#client.loop_stop(true)
Process.exit
end
end
end
# MQTT Start
Pulsr::MQTT.new :host => 'iot.eclipse.org', :port => 1883, :alive => 60
I was wondering, if I wanted to use Celluloid or EventMachine to run the loops that the mosquitto gem provides, how would I do it?
The mosquitto gem provides a good documentation and presents a few loop methods that can be used, but I have no clue where to start or how to do it, neither I have ever used EM or Celluloid.
Could anyone help get started with this, I think it could bring some value to the community and it can end up as a open source project, a small addition to the mosquitto gem?
I think it is not that hard.
Mosquitto has a good library.
Yo need to connect these functions:
mosquitto_loop_misc() <-> EventMachine::PeriodicTimer.new
mosquitto_read() <-> EventMachine.watch
mosquitto_write() <-> EventMachine.watch
The em-mqtt gem provides an MQTT protocol implementation for eventmachine. This uses the pure ruby mqtt implementation to process the messages rather than libmosquitto.
If you really have to use the libmosquitto implementation for the parsing via the mosquitto gem then the above delineation would hold. The eventmachine component will be pretty much as is. All the calls to the protocol specific MQTT module would be replaced with the equivalent in libmosquitto. The main problem looks to be that the libmosquitto public API and subsequent Ruby API hides all of this away, down in libmosquitto's own network implementation, which is being replaced with eventmachine, so you would have a lot of hacking to expose the required methods to Ruby before you can get started.

Creating a variable per instance, rather than per request, with Sinatra modular style

I have a Sinatra app, written in modular style, running on Heroku. It uses Redis and I have a limited number (10) of Redis connections. I found that it would often throw errors complaining that it had run out of Redis connections. So I started using connection_pool in the hope that would fix things; a single pool of Redis connections and the app would choose one of those each time, rather than try to create a new connection on each request.
But I'm still getting the same issue. I can do loads of Redis queries on a single query without complaints. But if I reload a single test page, which just does some Redis queries, several times in fairly quick succession, I get the "Redis::CommandError - ERR max number of clients reached" error again.
So I'm assuming, maybe, it's creating a new instance of connection_pool on each request... I don't know. But it's not "pooling" as I would expect it to.
I have this kind of thing:
# myapp.rb
$LOAD_PATH.unshift(File.dirname(__FILE__))
$stdout.sync = true
require 'thin'
require 'myapp/frontend'
MyApp::Frontend.run!
And the Sinatra app:
# myapp/frontend.rb
require 'sinatra/base'
require 'redis'
require 'connection_pool'
require 'uuid'
module MyApp
class Frontend < Sinatra::Base
helpers do
def redis_pool
#redis_pool ||= ConnectionPool.new(:size => 8, :timeout => 5) do
redis_uri = URI.parse(ENV['REDISCLOUD_URL'])
client = ::Redis.new(:host => redis_uri.host,
:port => redis_uri.port,
:password => redis_uri.password)
end
end
end
get '/tester/'
redis_pool.with do |r|
id = UUID.generate
r.hset(:user, id, "Some data")
r.hget(:user, id)
r.hdel(:user, id)
end
p "DONE"
end
end
end
The Procfile looks like:
web: ruby myapp.rb
Any ideas? The current site is pretty low traffic, so this should be possible.
A new instance of #redis_pool is created every time a get request for /tester/ is processed because the helper method redis_pool is called every time.
You can use sinatra's settings helper to initialize a redis connection only once:
config do
redis_uri = URI.parse(ENV['REDISCLOUD_URL'])
set :redis, Redis.new(:host => redis_uri.host,
:port => redis_uri.port,
:password => redis_uri.password)
end
Now the each instance of the app has one redis connection that persists for all requests. Access the setting like so
get '/tester/'
id = UUID.generate
settings.redis.hset(:user, id, "some data")
settings.redis.hget(:user, id)
settings.redis.hdel(:user, id)
p "DONE"
end

How to make rack session cookies httponly?

I am using Ruby and Sinatra to develop an application.
I use
enable :sessions
in order to use the session variable provided by rack. How can I make all of the session cookies be HTTPOnly? Is it like this by default? I can't find any documentation on this.
Instead of enable :sessions:
use Rack::Session::Cookie, {:httponly => true }
I'd suggest using the encrypted_cookie gem instead, it's far more secure. As an example, here's what I'll probably have for a project:
# app/main.rb
module Example
class App < Sinatra::Base # this class in its own file
# stuff here
end
end
# app/config.rb
require "main"
module Example
def self.app #
Rack::Builder.app do
cookie_settings = {
:key => 'usr',
:path => "/",
:expire_after => 86400, # In seconds, 1 day.
:secret => ENV["COOKIE_KEY"], # load this into the environment of the server
:httponly => true
}
cookie_settings.merge!( :secure => true ) if ENV["RACK_ENV"] == "production"
# AES encryption of cookies
use Rack::Session::EncryptedCookie, cookie_settings
# other stuff here
run App
end
end
end
# config.ru
require "app/config"
run Example.app # this in the rackup file
(To clarify why I've laid it out this way - this kind of stucture allows me to split the app up and use it easier in tests by just requiring the app/config.rb. YMMV)

Logging in Sinatra?

I'm having trouble figuring out how to log messages with Sinatra. I'm not looking to log requests, but rather custom messages at certain points in my app. For example, when fetching a URL I would like to log "Fetching #{url}".
Here's what I'd like:
The ability to specify log levels (ex: logger.info("Fetching #{url}"))
In development and testing environments, the messages would be written to the console.
In production, only write out messages matching the current log level.
I'm guessing this can easily be done in config.ru, but I'm not 100% sure which setting I want to enable, and if I have to manually create a Logger object myself (and furthermore, which class of Logger to use: Logger, Rack::Logger, or Rack::CommonLogger).
(I know there are similar questions on StackOverflow, but none seem to directly answer my question. If you can point me to an existing question, I will mark this one as a duplicate).
Sinatra 1.3 will ship with such a logger object, exactly usable as above. You can use edge Sinatra as described in "The Bleeding Edge". Won't be that long until we'll release 1.3, I guess.
To use it with Sinatra 1.2, do something like this:
require 'sinatra'
use Rack::Logger
helpers do
def logger
request.logger
end
end
I personally log in Sinatra via:
require 'sinatra'
require 'sequel'
require 'logger'
class MyApp < Sinatra::Application
configure :production do
set :haml, { :ugly=>true }
set :clean_trace, true
Dir.mkdir('logs') unless File.exist?('logs')
$logger = Logger.new('logs/common.log','weekly')
$logger.level = Logger::WARN
# Spit stdout and stderr to a file during production
# in case something goes wrong
$stdout.reopen("logs/output.log", "w")
$stdout.sync = true
$stderr.reopen($stdout)
end
configure :development do
$logger = Logger.new(STDOUT)
end
end
# Log all DB commands that take more than 0.2s
DB = Sequel.postgres 'mydb', user:'dbuser', password:'dbpass', host:'localhost'
DB << "SET CLIENT_ENCODING TO 'UTF8';"
DB.loggers << $logger if $logger
DB.log_warn_duration = 0.2
If you are using something like unicorn logging or other middleware that tails IO streams, you can easily set up a logger to STDOUT or STDERR
# unicorn.rb
stderr_path "#{app_root}/shared/log/unicorn.stderr.log"
stdout_path "#{app_root}/shared/log/unicorn.stdout.log"
# sinatra_app.rb
set :logger, Logger.new(STDOUT) # STDOUT & STDERR is captured by unicorn
logger.info('some info') # also accessible as App.settings.logger
this allows you to intercept messages at application scope, rather than just having access to logger as request helper
Here's another solution:
module MySinatraAppLogger
extend ActiveSupport::Concern
class << self
def logger_instance
#logger_instance ||= ::Logger.new(log_file).tap do |logger|
::Logger.class_eval { alias :write :'<<' }
logger.level = ::Logger::INFO
end
end
def log_file
#log_file ||= File.new("#{MySinatraApp.settings.root}/log/#{MySinatraApp.settings.environment}.log", 'a+').tap do |log_file|
log_file.sync = true
end
end
end
included do
configure do
enable :logging
use Rack::CommonLogger, MySinatraAppLogger.logger_instance
end
before { env["rack.errors"] = MySinatraAppLogger.log_file }
end
def logger
MySinatraAppLogger.logger_instance
end
end
class MySinatraApp < Sinatra::Base
include MySinatraAppLogger
get '/' do
logger.info params.inspect
end
end
Of course, you can do it without ActiveSupport::Concern by putting the configure and before blocks straight into MySinatraApp, but what I like about this approach is that it's very clean--all logging configuration is totally abstracted out of the main app class.
It's also very easy to spot where you can change it. For instance, the SO asked about making it log to console in development. It's pretty obvious here that all you need to do is a little if-then logic in the log_file method.

Resources