Synchronous logging incurs a large performance penalty as it may block. Is there a standalone Ruby library that does asynchronous logging (log4r doesn't seem to)? Can I modify the standard library logger to log asynchronously? I'm looking for something like log4j's AsyncAppender - but preferably an implementation that makes use of Ruby's code blocks to shift as much work to the background thread as possible.
I know you shouldn't really answer your own question, but it seems everything is easy in ruby:
require 'thread'
require 'singleton'
require 'delegate'
require 'monitor'
class Async
include Singleton
def initialize
#queue = Queue.new
Thread.new { loop { #queue.pop.call } }
end
def run(&blk)
#queue.push blk
end
end
class Work < Delegator
include MonitorMixin
def initialize(&work)
super work; #work, #done, #lock = work, false, new_cond
end
def calc
synchronize {
#result, #done = #work.call, true;
#lock.signal
}
end
def __getobj__
synchronize { #lock.wait_while { !#done } }
#result
end
end
Module.class.class_exec {
def async(*method_names)
method_names.each do |method_name|
original_method = instance_method(method_name)
define_method(method_name) do |*args,&blk|
work = Work.new { original_method.bind(self).call(*args,&blk) }
Async.instance.run { work.calc }
return work
end
end
end
}
And for my logging example:
require 'Logger'
class Logger
async :debug
end
log = Logger.new STDOUT
log.debug "heloo"
As return values work, you can use this for just about anything:
require "test/unit"
class ReturnValues < Test::Unit::TestCase
def do_it
5 + 7
end
async :do_it
def test_simple
assert_equal 10, do_it - 2
end
end
No personal experience with that:
https://github.com/dirs/analogger
The Swiftcore Analogger implements a fast asynchronous logging system
for Ruby programs as well as client library for sending logging messages
to the Analogger process.
Analogger will accept logs from multiple sources and can have multiple
logging destinations. Currently, logging to a file, to STDOUT, or to
STDERR is supported. A future revision may support logging to a
database destination, as well.
Analogger depends on EventMachine (http://rubyforge.org/projects/eventmachine)
to provide the framework for the network communications, though EM is
not used for the client library.
The built in Logger class is already thread safe
Checkout Dunder https://github.com/fonsan/dunder
I created this gem 6 months ago that does just this and more
Related
To be more particular, I'm talking about sentry-raven and sinatra here. I saw examples testing sinatra applications, or middlewares. But I didn't see ones testing if some particular middleware is present. Or should I be testing behavior, not configuration (or how should I call it)?
The important thing (I'd say) is the behaviour, but if you wish to check for middleware there are 2 ways I'd suggest after taking a delve into the Sinatra source (there are possibly much easier/better ways):
The env
In the Sinatra source there's a method that uses the env to check if a middleware is already present:
# Behaves exactly like Rack::CommonLogger with the notable exception that it does nothing,
# if another CommonLogger is already in the middleware chain.
class CommonLogger < Rack::CommonLogger
def call(env)
env['sinatra.commonlogger'] ? #app.call(env) : super
end
You could do the same thing in a route, e.g.
get "/env-keys" do
env.keys.inspect
end
It'll only show you the middleware if it's inserted something in env hash, e.g.
class MyBad
def initialize app, options={}
#app = app
#options = options
end
def call env
#app.call env.merge("mybad" => "I'm sorry!")
end
end
output:
["SERVER_SOFTWARE", "SERVER_NAME", "rack.input", "rack.version", "rack.errors", "rack.multithread", "rack.multiprocess", "rack.run_once", "REQUEST_METHOD", "REQUEST_PATH", "PATH_INFO", "REQUEST_URI", "HTTP_VERSION", "HTTP_HOST", "HTTP_CONNECTION", "HTTP_CACHE_CONTROL", "HTTP_ACCEPT", "HTTP_USER_AGENT", "HTTP_DNT", "HTTP_ACCEPT_ENCODING", "HTTP_ACCEPT_LANGUAGE", "GATEWAY_INTERFACE", "SERVER_PORT", "QUERY_STRING", "SERVER_PROTOCOL", "rack.url_scheme", "SCRIPT_NAME", "REMOTE_ADDR", "async.callback", "async.close", "rack.logger", "mybad", "rack.request.query_string", "rack.request.query_hash", "sinatra.route"]
It's near the end of that list.
The middleware method
There's also a method called middleware in Sinatra::Base:
# Middleware used in this class and all superclasses.
def middleware
if superclass.respond_to?(:middleware)
superclass.middleware + #middleware
else
#middleware
end
end
Call it in the class definition of a modular app and you can get the middlewares in an array:
require 'sinatra/base'
class AnExample < Sinatra::Base
use MyBad
warn "self.middleware = #{self.middleware}"
output:
self.middleware = [[MyBad, [], nil]]
There may be a way to get it from Sinatra::Application, but I haven't looked.
With help from ruby-raven guys, we've got this:
ENV['RACK_ENV'] = 'test'
# the app: start
require 'sinatra'
require 'sentry-raven'
Raven.configure(true) do |config|
config.dsn = '...'
end
use Raven::Rack
get '/' do
'Hello, world!'
end
# the app: end
require 'rspec'
require 'rack/test'
Raven.configure do |config|
logger = ::Logger.new(STDOUT)
logger.level = ::Logger::WARN
config.logger = logger
end
describe 'app' do
include Rack::Test::Methods
def app
#app || Sinatra::Application
end
class TestRavenError < StandardError; end
it 'sends errors to sentry' do
#app = Class.new Sinatra::Application do
get '/' do
raise TestRavenError
end
end
allow(Raven.client).to receive(:send).and_return(true)
begin
get '/'
rescue TestRavenError
end
expect(Raven.client).to have_received(:send)
end
end
Or if raven sending requests is in the way (when tests fail because of raven sending requests, and not because of the underlying error), one can disable them:
Raven.configure(true) do |config|
config.should_send = Proc.new { false }
end
And mock Raven.send_or_skip instead:
...
allow(Raven).to receive(:send_or_skip).and_return(true)
begin
get '/'
rescue TestRavenError
end
expect(Raven).to have_received(:send_or_skip)
...
I'm new to ruby and maybe this is a very simple question..
I'd like to use eventmachine to develop a simulator for my tests.
Following example in documentation I can write something like this:
require 'eventmachine'
class Server< EM::Connection
def receive_data data
send_data data
close_connection_after_writing
end
end
#Note that this will block current thread.
EventMachine.run {
EventMachine.start_server '127.0.0.1','8080', Server
}
But I wonder if there is a way to use an instance of class, something like:
require 'eventmachine'
class Server< EM::Connection
attr_accessor :response
def receive_data data
send_data #response
close_connection_after_writing
end
end
server1 = Server.new
server1.response = "foo"
#Note that this will block current thread.
EventMachine.run {
EventMachine.start_server '127.0.0.1','8080', server1
}
I try to read source code..but it's too hard for me.
I'm surely missing something, but I don't know how to do something like this.
As I say there was something that I was missing.
You can add parameters for class to be instantiated :
class Server< EM::Connection
def initialize par
puts "I'm server number#{par}"
end
def receive_data data
send_data data
close_connection_after_writing
end
end
EventMachine.run {
EventMachine.start_server '127.0.0.1','8080', Server,1
}
EventMachine.run {
EventMachine.start_server '127.0.0.1','8080', Server,2
}
So I will customize instance behaviour with parameters
Currently I'm using puts, but I'm sure that's not the correct answer. How do I correctly setup a logger, inside my gem, to output my internal logging instead of puts?
The most flexible approach for users of your gem is to let them provide a logger rather than setting it up inside the gem. At its simplest this could be
class MyGem
class << self
attr_accessor :logger
end
end
You then use MyGem.logger.info "hello" to log messages from your gem (you might want to wrap it in a utility method that tests whether a logger is set at all)
Users of your gem can then control where messages get logged to (a file, syslog, stdout, etc...)
You can keep the logger in your top-level module. Allow user's to set their own logger but provide a reasonable default for those who don't care to deal with logging. For e.g.
module MyGem
class << self
attr_writer :logger
def logger
#logger ||= Logger.new($stdout).tap do |log|
log.progname = self.name
end
end
end
end
Then, anywhere within your gem code you can access the logger. For e.g.
class MyGem::SomeClass
def some_method
# ...
MyGem.logger.info 'some info'
end
end
References:
Using the Ruby Logger
Logger
A little example:
gem 'log4r'
require 'log4r'
class MyClass
def initialize(name)
#log = Log4r::Logger.new(name)
#Add outputter
#~ log.outputters << Log4r::FileOutputter.new('log_file', :filename => 'mini_example.log', :level => Log4r::ALL )
log.outputters << Log4r::StdoutOutputter.new('log_stdout') #, :level => Log4r::WARN )
#~ log.outputters << Log4r::StderrOutputter.new('log_stderr', :level => Log4r::ERROR)
#log.level = Log4r::INFO
#log.info("Creation")
#~ #log.level = Log4r::OFF
end
attr_reader :log
def myfunction(*par)
#log.debug("myfunction is called")
#log.warn("myfunction: No parameter") if par.empty?
end
end
x = MyClass.new('x')
x.myfunction
y = MyClass.new('y')
y.myfunction
y.myfunction(:a)
y.log.level = Log4r::DEBUG
y.myfunction(:a)
During initialization you create a Logger (#log). In your methods you call the logger.
With #log.level= (or MyClass#log.level=) you can influence, which messages are used.
You can use different outputters (in my example I log to STDOUT). You can also mix outputters (e.g. STDOUT with warnings, each data (including DEBUG) to a file...)
I think the easiest approach is to use it this way
Rails.logger.info "hello"
This below app saves some data to the db and I want to test that it saves properly.
require 'goliath'
class App < Goliath::API
def response(env)
db = EM::Mongo::Connection.new('localhost').db('hello')
db.collection('coll').insert({'identifier => 1'})
[204, {}, {}]
end
end
require 'goliath/test_helper'
Goliath.env = :test
describe App do
include Goliath::TestHelper
it do
with_api(described_class) do
get_request do |req|
db = EM::Mongo::Connection.new('localhost').db('hello')
db.collection('coll').first.callback do |rec|
rec['identifier'].should == 100
end
end
end
end
end
The above spec passes since reactor ends before callback returns. I thought about manually starting a reactor like:
EM.run do
db = EM::Mongo::Connection.new('localhost').db('hello')
db.collection('coll').first.callback do |rec|
rec['identifier'].should == 100
EM.stop
end
end
Though I'm not sure if starting the reactor for every spec is good practice. Help please?
The problem is that when the get_request is setup we add a callback on the request that stops the event loop. So, as soon as your block finishes (which will be before the connection is even created), it will stop the reactor.
I'm not sure the best solution, but a crappy one would be to override:
def hookup_request_callbacks(req, errback, &blk)
req.callback &blk
req.callback { stop }
req.errback &errback if errback
req.errback { stop }
end
in your test class after you include Goliath::TestHelper. Then, I think, you should be able to write your own that just has something like:
def hookup_request_callbacks(req, errback, &blk)
req.callback &blk
req.errback &errback if errback
req.errback { stop }
end
You'll just have to make sure you call stop in your callback from Mongo.
I haven't actually tested this, so let me know if something doesn't work and I can dig in further.
#dj2's solution works great, but I decided instead of use mongo gem in specs, instead of em-mongo. Since mongo blocks, I don't have to worry about Goliath stopping the reactor before database returns results.
I'm having trouble figuring out how to log messages with Sinatra. I'm not looking to log requests, but rather custom messages at certain points in my app. For example, when fetching a URL I would like to log "Fetching #{url}".
Here's what I'd like:
The ability to specify log levels (ex: logger.info("Fetching #{url}"))
In development and testing environments, the messages would be written to the console.
In production, only write out messages matching the current log level.
I'm guessing this can easily be done in config.ru, but I'm not 100% sure which setting I want to enable, and if I have to manually create a Logger object myself (and furthermore, which class of Logger to use: Logger, Rack::Logger, or Rack::CommonLogger).
(I know there are similar questions on StackOverflow, but none seem to directly answer my question. If you can point me to an existing question, I will mark this one as a duplicate).
Sinatra 1.3 will ship with such a logger object, exactly usable as above. You can use edge Sinatra as described in "The Bleeding Edge". Won't be that long until we'll release 1.3, I guess.
To use it with Sinatra 1.2, do something like this:
require 'sinatra'
use Rack::Logger
helpers do
def logger
request.logger
end
end
I personally log in Sinatra via:
require 'sinatra'
require 'sequel'
require 'logger'
class MyApp < Sinatra::Application
configure :production do
set :haml, { :ugly=>true }
set :clean_trace, true
Dir.mkdir('logs') unless File.exist?('logs')
$logger = Logger.new('logs/common.log','weekly')
$logger.level = Logger::WARN
# Spit stdout and stderr to a file during production
# in case something goes wrong
$stdout.reopen("logs/output.log", "w")
$stdout.sync = true
$stderr.reopen($stdout)
end
configure :development do
$logger = Logger.new(STDOUT)
end
end
# Log all DB commands that take more than 0.2s
DB = Sequel.postgres 'mydb', user:'dbuser', password:'dbpass', host:'localhost'
DB << "SET CLIENT_ENCODING TO 'UTF8';"
DB.loggers << $logger if $logger
DB.log_warn_duration = 0.2
If you are using something like unicorn logging or other middleware that tails IO streams, you can easily set up a logger to STDOUT or STDERR
# unicorn.rb
stderr_path "#{app_root}/shared/log/unicorn.stderr.log"
stdout_path "#{app_root}/shared/log/unicorn.stdout.log"
# sinatra_app.rb
set :logger, Logger.new(STDOUT) # STDOUT & STDERR is captured by unicorn
logger.info('some info') # also accessible as App.settings.logger
this allows you to intercept messages at application scope, rather than just having access to logger as request helper
Here's another solution:
module MySinatraAppLogger
extend ActiveSupport::Concern
class << self
def logger_instance
#logger_instance ||= ::Logger.new(log_file).tap do |logger|
::Logger.class_eval { alias :write :'<<' }
logger.level = ::Logger::INFO
end
end
def log_file
#log_file ||= File.new("#{MySinatraApp.settings.root}/log/#{MySinatraApp.settings.environment}.log", 'a+').tap do |log_file|
log_file.sync = true
end
end
end
included do
configure do
enable :logging
use Rack::CommonLogger, MySinatraAppLogger.logger_instance
end
before { env["rack.errors"] = MySinatraAppLogger.log_file }
end
def logger
MySinatraAppLogger.logger_instance
end
end
class MySinatraApp < Sinatra::Base
include MySinatraAppLogger
get '/' do
logger.info params.inspect
end
end
Of course, you can do it without ActiveSupport::Concern by putting the configure and before blocks straight into MySinatraApp, but what I like about this approach is that it's very clean--all logging configuration is totally abstracted out of the main app class.
It's also very easy to spot where you can change it. For instance, the SO asked about making it log to console in development. It's pretty obvious here that all you need to do is a little if-then logic in the log_file method.