How to define a common module logging shared among others module - ruby

I am developing a script with a big main function, which I have split in several modules.
What I need is to have access to the log functionality from all of them, this means that the log file has to be opened only once, and the access be shared.
This is what I have:
require 'module_1'
require 'module_2'
require 'module_3'
module Main
Module_1.Function_startup()
Module_2.Function_configuration()
Module_3.Function_self_test()
end
Here is the dirty module for the logger I need available in all the other modules.
Ideally I would like to call it as "logger.error", where "logger" returns the instance of the logger, and "error" is the function call on rlogger as rlogger.error.
require 'logger'
module Logging
#rlogger = nil
def init_logger
if #rlogger.nil?
puts "initializing logger"
file_path = File.join(File.dirname(__FILE__), 'checker.log')
open_mode = File::TRUNC # or File::APPEND
file = File.open(file_path, File::WRONLY | open_mode)
#rlogger = Logger.new(file)
#rlogger.datetime_format = "%Y-%m-%d %H:%M:%S"
#rlogger.formatter = proc do |severity, datetime, progname, msg|
con_msg = ""
if msg.match("ERROR:")
con_msg = msg.color(:red)
elsif msg.match("OK!")
con_msg = msg.color(:green)
else
con_msg = msg
end
puts ">>>#{con_msg}"
# notice that the colors introduce extra non-printable characters
# which are not nice in the log file.
"#{datetime}: #{msg}\n"
end
# Here is the first log entry
#rlogger.info('Initialize') {"#{Time.new.strftime("%H-%M-%S")}: Checker v#{#version}"}
end
end
# returns the logger
def logger
if #rlogger.nil?
puts "requesting nil rlogger"
end
#rlogger
end
end
end

Just after require you can add this piece of code
$FILE_LOG = Logging.create_log(File.expand_path('LoggingFile.log'), Logger::DEBUG)
Explanation of the above line : It is calling a function in Logging Module, to create File , Level of Logging is debug.
Below is the piece of code for Module
module Logging
def self.create_log(output_location level)
log = Logger.new(output_location, 'weekly').tap do |l|
next unless l
l.level = level
l.progname = File.basename($0)
l.datetime_format = DateTime.iso8601
l.formatter = proc { |severity, datetime, progname, msg| "#{severity}: #{datetime} - (#{progname}) : #{msg}\n" }
end
log.level = level if level < log.level
log
end
def self.log(msg, level)
# Here I am just logging only FATAL and DEBUG, similarly you can add in different level of logs
if level == :FATAL
$FILE_LOG.fatal(msg)
elsif level == :DEBUG
$FILE_LOG.debug(msg)
end
end
end
Then in Every method every Ruby file, we can use this logging as follows
Logging.log("Message",:FATAL)

Related

Ruby - Set different log levels for different targets

Wondering how to set different log levels for different targets. Below is my Ruby code that writes a line to both Console and File.
# https://stackoverflow.com/a/6407200
class MultiIO
def initialize(*targets)
#targets = targets
end
def write(*args)
#targets.each do |t|
t.write(*args)
end
end
def close
#targets.each(&:close)
end
end
module Logging
def self.logger(logname, programname, debug = false)
log_file = File.open(logname, "a")
log_file.sync = true
zlogger = Logger.new MultiIO.new(log_file, STDOUT)
zlogger.level = Logger::INFO
zlogger.progname = programname
zlogger.formatter = proc do |serverity, datetime, progname, msg|
"#{datetime.strftime('%Y-%m-%d %I:%M:%S %p %:::z %Z')} - #{serverity} - [#{progname}] | #{msg}\n"
end
zlogger
end
end
I can set the level to Debug if a special env variable is found,
$logger.level = Logger::DEBUG if ENV['enable_debug_logs'] == 'true'
But, not sure how to always write Debug lines to the log file and only Info lines to console.
Does anyone know? Any help is greatly appreciated!
Since the log level is a property of the logger, not of the IO, it sounds to me like what you really need to do is define a MultiLogger rather than a MultiIO. Something along the lines of:
class MultiLogger
attr_reader :default_level, :default_progname, :default_formatter
def initialize(**args)
#default_level = args[:default_level]
#default_progname = args[:default_progname]
#default_formatter = args[:default_formatter]
#loggers = []
Array(args[:loggers]).each { |logger| add_logger(logger) }
end
def add_logger(logger)
logger.level = default_level if default_level
logger.progname = default_progname if default_progname
logger.formatter = default_formatter if default_formatter
#loggers << logger
end
def close
#loggers.map(&:close)
end
Logger::Severity.constants.each do |level|
define_method(level.downcase) do |*args|
#loggers.each { |logger| logger.send(__method__, args) }
end
# These methods are a bit weird in the context of a "multi-logger" with varying levels,
# since they are now returning an `Array`; never a falsey value.
# You may want to define them differently, e.g. `#loggers.all? {...}`, or use a non-predicate method name here.
define_method("#{level.downcase}?".to_sym) do
#loggers.map(&__method__)
end
end
end
# Usage:
log_file = File.open(logname, "a")
log_file.sync = true
file_logger = Logger.new(log_file)
console_logger = Logger.new(STDOUT)
console_logger.level = Logger::INFO
multi_logger = MultiLogger.new(
default_progname: programname,
default_formatter: proc do |severity, datetime, progname, msg|
"#{datetime.strftime('%Y-%m-%d %I:%M:%S %p %:::z %Z')} - #{severity} - [#{progname}] | #{msg}\n"
end,
loggers: [file_logger, console_logger]
)

Sidekiq mechanize overwritten instance

I am building a simple web spider using Sidekiq and Mechanize.
When I run this for one domain, it works fine. When I run it for multiple domains, it fails. I believe the reason is that web_page gets overwritten when instantiated by another Sidekiq worker, but I am not sure if that's true or how to fix it.
# my scrape_search controller's create action searches on google.
def create
#scrape = ScrapeSearch.build(keywords: params[:keywords], profession: params[:profession])
agent = Mechanize.new
scrape_search = agent.get('http://google.com/') do |page|
search_result = page.form...
search_result.css("h3.r").map do |link|
result = link.at_css('a')['href'] # Narrowing down to real search results
#domain = Domain.new(some params)
ScrapeDomainWorker.perform_async(#domain.url, #domain.id, remaining_keywords)
end
end
end
I'm creating a Sidekiq job per domain. Most of the domains I'm looking for should contain just a few pages, so there's no need for sub-jobs per page.
This is my worker:
class ScrapeDomainWorker
include Sidekiq::Worker
...
def perform(domain_url, domain_id, keywords)
#domain = Domain.find(domain_id)
#domain_link = #domain.protocol + '://' + domain_url
#keywords = keywords
# First we scrape the homepage and get the first links
#domain.to_parse = ['/'] # to_parse is an array of PATHS to parse for the domain
mechanize_path('/')
#domain.verified << '/' # verified is an Array field containing valid domain paths
get_paths(#web_page) # Now we should have to_scrape populated with homepage links
#domain.scraped = 1 # Loop counter
while #domain.scraped < 100
#domain.to_parse.each do |path|
#domain.to_parse.delete(path)
#domain.scraped += 1
mechanize_path(path) # We create a Nokogiri HTML doc with mechanize for the valid path
...
get_paths(#web_page) # Fire this to repopulate to_scrape !!!
end
end
#domain.save
end
def mechanize_path(path)
agent = Mechanize.new
begin
#web_page = agent.get(#domain_link + path)
rescue Exception => e
puts "Mechanize Exception for #{path} :: #{e.message}"
end
end
def get_paths(web_page)
paths = web_page.links.map {|link| link.href.gsub((#domain.protocol + '://' + #domain.url), "") } ## This works when I scrape a single domain, but fails with ".gsub for nil" when I scrape a few domains.
paths.uniq.each do |path|
#domain.to_parse << path
end
end
end
This works when I scrape a single domain, but fails with .gsub for nil for web_page when I scrape a few domains.
You can wrap you code in another class, and then create and object of that class within your worker:
class ScrapeDomainWrapper
def initialize(domain_url, domain_id, keywords)
# ...
end
def mechanize_path(path)
# ...
end
def get_paths(web_page)
# ...
end
end
And your worker:
class ScrapeDomainWorker
include Sidekiq::Worker
def perform(domain_url, domain_id, keywords)
ScrapeDomainWrapper.new(domain_url, domain_id, keywords)
end
end
Also, bear in mind that Mechanize::Page#links may be a nil.

Thor::Group do not continue if a condition is not met

I'm converting a generator over from RubiGen and would like to make it so the group of tasks in Thor::Group does not complete if a condition isn't met.
The RubiGen generator looked something like this:
def initialize(runtime_args, runtime_options = {})
super
usage if args.size != 2
#name = args.shift
#site_name=args.shift
check_if_site_exists
extract_options
end
def check_if_site_exists
unless File.directory?(File.join(destination_root,'lib','sites',site_name.underscore))
$stderr.puts "******No such site #{site_name} exists.******"
usage
end
end
So it'd show a usage banner and exit out if the site hadn't been generated yet.
What is the best way to recreate this using thor?
This is my task.
class Page < Thor::Group
include Thor::Actions
source_root File.expand_path('../templates', __FILE__)
argument :name
argument :site_name
argument :subtype, :optional => true
def create_page
check_if_site_exists
page_path = File.join('lib', 'sites', "#{site_name}")
template('page.tt', "#{page_path}/pages/#{name.underscore}_page.rb")
end
def create_spec
base_spec_path = File.join('spec', 'isolation', "#{site_name}")
if subtype.nil?
spec_path = base_spec_path
else
spec_path = File.join("#{base_spec_path}", 'isolation')
end
template('functional_page_spec.tt', "#{spec_path}/#{name.underscore}_page_spec.rb")
end
protected
def check_if_site_exists # :nodoc:
$stderr.puts "#{site_name} does not exist." unless File.directory?(File.join(destination_root,'lib','sites', site_name.underscore))
end
end
after looking through the generators for the spree gem i added a method first that checks for the site and then exits with code 1 if the site is not found after spitting out an error message to the console. The code looks something like this:
def check_if_site_exists
unless File.directory?(path/to/site)
say "site does not exist."
exit 1
end
end

How do I log correctly inside my Ruby gem?

Currently I'm using puts, but I'm sure that's not the correct answer. How do I correctly setup a logger, inside my gem, to output my internal logging instead of puts?
The most flexible approach for users of your gem is to let them provide a logger rather than setting it up inside the gem. At its simplest this could be
class MyGem
class << self
attr_accessor :logger
end
end
You then use MyGem.logger.info "hello" to log messages from your gem (you might want to wrap it in a utility method that tests whether a logger is set at all)
Users of your gem can then control where messages get logged to (a file, syslog, stdout, etc...)
You can keep the logger in your top-level module. Allow user's to set their own logger but provide a reasonable default for those who don't care to deal with logging. For e.g.
module MyGem
class << self
attr_writer :logger
def logger
#logger ||= Logger.new($stdout).tap do |log|
log.progname = self.name
end
end
end
end
Then, anywhere within your gem code you can access the logger. For e.g.
class MyGem::SomeClass
def some_method
# ...
MyGem.logger.info 'some info'
end
end
References:
Using the Ruby Logger
Logger
A little example:
gem 'log4r'
require 'log4r'
class MyClass
def initialize(name)
#log = Log4r::Logger.new(name)
#Add outputter
#~ log.outputters << Log4r::FileOutputter.new('log_file', :filename => 'mini_example.log', :level => Log4r::ALL )
log.outputters << Log4r::StdoutOutputter.new('log_stdout') #, :level => Log4r::WARN )
#~ log.outputters << Log4r::StderrOutputter.new('log_stderr', :level => Log4r::ERROR)
#log.level = Log4r::INFO
#log.info("Creation")
#~ #log.level = Log4r::OFF
end
attr_reader :log
def myfunction(*par)
#log.debug("myfunction is called")
#log.warn("myfunction: No parameter") if par.empty?
end
end
x = MyClass.new('x')
x.myfunction
y = MyClass.new('y')
y.myfunction
y.myfunction(:a)
y.log.level = Log4r::DEBUG
y.myfunction(:a)
During initialization you create a Logger (#log). In your methods you call the logger.
With #log.level= (or MyClass#log.level=) you can influence, which messages are used.
You can use different outputters (in my example I log to STDOUT). You can also mix outputters (e.g. STDOUT with warnings, each data (including DEBUG) to a file...)
I think the easiest approach is to use it this way
Rails.logger.info "hello"

Ruby EventMachine & functions

I'm reading a Redis set within an EventMachine reactor loop using a suitable Redis EM gem ('em-hiredis' in my case) and have to check if some Redis sets contain members in a cascade. My aim is to get the name of the set which is not empty:
require 'eventmachine'
require 'em-hiredis'
def fetch_queue
#redis.scard('todo').callback do |scard_todo|
if scard_todo.zero?
#redis.scard('failed_1').callback do |scard_failed_1|
if scard_failed_1.zero?
#redis.scard('failed_2').callback do |scard_failed_2|
if scard_failed_2.zero?
#redis.scard('failed_3').callback do |scard_failed_3|
if scard_failed_3.zero?
EM.stop
else
queue = 'failed_3'
end
end
else
queue = 'failed_2'
end
end
else
queue = 'failed_1'
end
end
else
queue = 'todo'
end
end
end
EM.run do
#redis = EM::Hiredis.connect "redis://#{HOST}:#{PORT}"
# How to get the value of fetch_queue?
foo = fetch_queue
puts foo
end
My question is: how can I tell EM to return the value of 'queue' in 'fetch_queue' to use it in the reactor loop? a simple "return queue = 'todo'", "return queue = 'failed_1'" etc. in fetch_queue results in "unexpected return (LocalJumpError)" error message.
Please for the love of debugging use some more methods, you wouldn't factor other code like this, would you?
Anyway, this is essentially what you probably want to do, so you can both factor and test your code:
require 'eventmachine'
require 'em-hiredis'
# This is a simple class that represents an extremely simple, linear state
# machine. It just walks the "from" parameter one by one, until it finds a
# non-empty set by that name. When a non-empty set is found, the given callback
# is called with the name of the set.
class Finder
def initialize(redis, from, &callback)
#redis = redis
#from = from.dup
#callback = callback
end
def do_next
# If the from list is empty, we terminate, as we have no more steps
unless #current = #from.shift
EM.stop # or callback.call :error, whatever
end
#redis.scard(#current).callback do |scard|
if scard.zero?
do_next
else
#callback.call #current
end
end
end
alias go do_next
end
EM.run do
#redis = EM::Hiredis.connect "redis://#{HOST}:#{PORT}"
finder = Finder.new(redis, %w[todo failed_1 failed_2 failed_3]) do |name|
puts "Found non-empty set: #{name}"
end
finder.go
end

Resources