I have a running environment with a Rails application, Sidekiq and clockwork mod for scheduling purposes.
I have many different workers, filled with logger.debug and logger.info instructions, and I occasionally need to activate debug logging on some of them to know what's going on.
I like the Sidekiq logger, and I would like to utilize it because it just need a "logger.debug" instruction in the workers to do its job.
What I miss with my current setup is the possibility to activate the DEBUG level for some workers, while leaving the others in standard INFO.
Now in each of my workers I have this initialize method:
class SendMailOnStart
include Sidekiq::Worker
sidekiq_options :retry => false, :queue => :critical
def initialize
logger.level = Logger::INFO
end
.... ...
But if a change the level in one worker, this level will be overwritten by the level specified in the next one - e.g. if two workers are processed together, the second one will "win".
What's the best way to achieve this in an elegant way?
Coming from Java world, I can think only to create a custom logger and putting it in each worker, copying output format used by Sidekiq logger, adding a logger method in each worker like
def logger
logger = MyLogger.new
end
and changing the level when I neeed it in initialize method
Is this the best approach in Ruby?
I had a similar question and I found this thread more useful:
Log to different logger based on call from Sidekiq or Rails
You should be able set the log level for Sidekiq workers specifically in the block mentioned there by altering Rails.logger.
I have no clue if that’s the best approach, but I would do the following. First of all, let’s prepare the function to retrieve the caller’s filename and/or method:
def parse_caller
# magic number 7 below is the amount of calls
# on stack to unwind to get your caller
if /^(?<file>.+?):(?<line>\d+)(?::in `(?<method>.*)')?/ =~ caller(7).first
file = Regexp.last_match[:file]
line = Regexp.last_match[:line].to_i
method = Regexp.last_match[:method]
[file, line, method]
end
end
Then I would override the default formatter of Logger instance, compelling it to check the caller:
logger.formatter = lambda do |severity, datetime, progname, msg|
f,l,m = parse_caller
# HERE GOES YOUR CHECK
if f =~ /…/
…
end
end
I know it looks a weird hack, but it works fine for me.
Related
Sorry for such a big question. I do not have much experience with Rails threads and mutex.
I have a class as follow which is used by different controllers to get the license for each customers.
Customers and their licenses gets added and removed every hour. An api is available to get all customers and their licenses.
I plan to create a rake task to call update_set_customers_licenses, run hourly via a cronjob.
I have following questions:
1) Even with a mutex, currently there is a potential for problem, there is a chance that my rake task can occur while updating. Any idea on how to solve this?
2) My design below writes the json out to a file, this is done is for safety as the api is not that reliable. As can be seen, it is not reading the file back, so in essence the file write is useless. I tried to implement a file read but together with mutex and rake task, it gets really confusing. Any pointers will help here.
class Customer
##customers_to_licenses_hash = nil
##last_updated_at = nil
##mutex = Mutex.new
CUSTOMERS_LICENSES_FILE = "#{Rails.root}/tmp/customers_licenses"
def self.cached_license_with_customer(customer)
Rails.cache.fetch('customer') {self.license_with_customer(customer)}
end
def self.license_with_customer(customer)
##mutex.synchronize do
license = ##customers_to_licenses_hash[customer]
if license
return license
elsif(##customers_to_licenses_hash.nil? || Time.now.utc - ##last_updated_at > 1.hours)
updated = self.update_set_customers_licenses
return ##customers_to_licenses_hash[customer] if updated
else
return nil
end
end
end
def self.update_set_customers_licenses
updated = nil
file_write = File.open(CUSTOMERS_LICENSES_FILE, 'w')
results = self.get_active_customers_licenses
if results
##customers_to_licenses_hash = results
file_write.print(results.to_json)
##last_updated_at = Time.now.utc
updated = true
end
file_write.close
updated
end
def self.get_active_customers_licenses
#http get thru api
#return hash of records
end
end
I'm pretty it's the case that every time rails loads, the environment is "fresh" and has no concept of "state" in between instances. That is to say, a mutex in one ruby instance (the one request to rails) has no effect on a second ruby instance (another request to rails or in this case, a rake task).
If you follow the data upstream, you'll find that the common root of every instance that can be used to synchronize them is the database. You could use transactional blocks or maybe a manual flag you set and unset in the database.
I want to insert name of the method calling the logger methods into my log files. Not the whole stack trace, but the class, method and/or line number would be great.
In any method, one can use caller to get an array of strings, each of which contains the file, line number and method name. I've come up with a pretty awful kludge using regexes and Enumerable#find to try to return the first non-logger stack frame. I guess it works, but if the locations of the logging Ruby files change in a different version or Rails, or I name my files something to do with logs, it will break. Same with if I take a given index from the top of the stack (I did this at first, then refactored one thing and naturally it gave me the wrong frame).
Note that I'm not looking to just log the controller or action, as those can be retrieved easily. Mostly this is for stuff in the lib/ directory.
Isn't there an easy way to do this? I don't want to have to pass in __method__ every time I make a logging statement.
I've looked all over at different solutions for capturing the exact place (file, line number, method name) where I invoke any given logger instance method from within my rails app. To do this, you need to override Logger's format_message method, and a good place to do this is in your rails project's config/environment.rb file.
This is what I've come up with, which is good enough for me ;o)
class Logger
def format_message(severity, timestamp, progname, msg)
line = ''
Kernel.caller.each{|entry|
if (entry.include? Rails.root.to_s)
line = " #{entry.gsub(Rails.root.to_s,'').gsub(/\/(.+)\:in `(.+)'/, "\\1 -> \\2")}"
break
end
}
"[#{timestamp.strftime("%Y%m%d.%H:%M:%S")}] #{severity}#{line}: #{msg}\n"
end
end
Kernel.caller holds an enumerable array of the entire backtrace. If you look at it in its entirety, you'll see most calls are internal inside of a gem somewhere well outside your project. I've found that by looping through the Kernel.caller until I find the first place that includes my Rails.root, I can get the line with the information I want to parse.
Example:
If I call Rails.logger.debug("Streamer class started!") from the start method of my Streamer class, the raw entry would look like this:
/Users/chikoon/www/my_rails_app/lib/streamer.rb:7:in `start'
so by the time it makes it through my formatter, I've got the timestamp, severity mode, the file path, line number, method name, and message:
[20140919.19:23:44] DEBUG lib/streamer.rb:7 -> start: Streamer class started!
I hope that helps get your wheels turning.
How about setting up log_tags to call the __method__?
Blog::Application.configure do
config.log_tags = [lambda { |req| __method__ }]
end
I am getting this error While running this
LoadError: Expected /home/user/Desktop/Tripurari/myapp/app/models/host.rb to define Host##
But every thing on it's place. Can some one tell me what the exact problem is below method.
def self.check_all(keyword)
memo_mutex = Mutex.new
memo = {}
threads = []
name = keyword.keyword
SITES.each do |site_and_options|
threads << Thread.new do
#host = Host.find_or_create_by_name(site)
if keyword.unavailable_usernames.find_by_host_id(#host.id)
memo[#host.name] = true
else
memo[#host.name] = false
end
end
end
threads.each { |t| t.join }
memo
end
The issue is probably caused by the autoloader. If the Host class is not yet loaded when first entering the loop where you create a couple of new threads, it is autoloaded, i.e. Rails searches the loadpath for a file matching the naming conventions and requires it.
This process is not threadsave. In your case, as you are creating servral threads in quick succession, each trying to autoload the global class, you get race conditions and strange things happen. Basically, you have two options for tackling this:
You can explicitly load the model before starting your threads by using require 'host' before starting your loop.
Or you can set config.threadsave! in an initializer. This will (among other things) preload all your classes when starting your server. This is preferred as with this, you avoid a truckload of other difficult to debug concurrency issues. For more information about config.threadsafe!, please refer to the excellent article by Aaron Patterson arguing it should be removed altogether in Rails 4.
Assuming the code you've quoted above is in a model's .rb file, add require_relative "host" to the top of that file.
I am using Mongoid for interacting with MongoDB. In development I usually like to see the logs of what Mongo is doing. However, there is one instance where there is an excessive amount of redundant logging that I simply don't want to see. How can I disable logging in this specific case?
There doesn't appear to be any clean way to control logging in Mongoid on a per collection basis.
However, for your purposes, if you are able to identify the individual calls,
you can turn off logging temporarily by raising the level.
Hope that this is good enough for your purposes.
require 'test_helper'
def dont_log(temp_level = Logger::Severity::UNKNOWN)
logger = Rails.logger
old_level, logger.level = logger.level, temp_level
yield
logger.level = old_level
end
class LogDoTest < ActiveSupport::TestCase
def setup
LogDo.delete_all
LogDont.delete_all
end
test "log do dont" do
LogDo.create(text: 'Log me')
dont_log{ LogDont.create(text: 'Dont log me') }
end
end
Are there any logging frameworks in ruby that allow you to log a specific event type only once?
logger = IdealLogger.new
logger.log(:happy_path, "We reached the happy path") # => logs this message
logger.log(:happy_path, "We reached the happy path yet again") # => Doesn't log this
logger.log(:sad_path, "We've encountered a sad path!") # => logs this message
Also, is there a term for the concept of logging a certain event type only once?
Edit: I'm using Plain Old Ruby Objects, not Rails. I had in mind "once per time the script is run" for "once".
I'm not aware of one, but extending Logger to make your own isn't too difficult. It's essentially implementing caching for your logging, but instead of fetching from the cache and returning it like you would with a normal app, you quash it when it's been cached. Implementation and expiration strategy of this log cache are left as an exercise for the reader.
something like:
class IdealLogger < Logger
def info(event = nil, progname = nil, &block)
super(progname, &block) unless event_is_cached(event)
end
# define debug, warn, error, fatal, and unknown the same way, override others
# as you wish.
end