Is it possible to dynamically change the LogDevice of Ruby Logger?
If so, it would allow for some un-obtrusive changes to my existing codebase.
Currently Ruby Logger uses StringIO as the LogDevice:
#logDevice = StringIO.new("", "r+")
#log = Logger.new(#logDevice) // a reference to this is used by many objects
// both are instance vars
...
#log.info('some log') // Logging activity
...
// Before program ends, transmit logs to a server
Can LogDevice be dynamically changed to continue logging to a file?
(dynamic change because initially the filename is not known.)
Or if log device cannot be change can the StringIO object start writing to a file?
Instead of doing the above, I could write to a temporary log file, but wanted to check if the above can be done because it would be a less obstrusive change to the existing codebase.
The object you give to the logger just has to implement the 'write' and 'close' methods, so you can easily write your own 'io':
class MyIO
def initialize
#file = nil
#history = StringIO.new "", "w"
end
def file=(filename)
#file = File.open(filename, 'a+')
#file.write #history.string if #history
#history = nil
end
def write(data)
#history.write(data) if #history
#file.write(data) if #file
end
def close
#file.close if #file
end
end
create the logger with an instance of that and keep a reference to the instance. Then, whenever you know the filename, just set it with the 'file=' method.
Related
Our processing server works mainly with TempFiles as it makes things easier on our side: no need to take care of deleting them as they get garbage collected or handle name collisions, etc.
Lately, we are having problems with TempFiles getting GCed too early in the process. Specially with one of our services that will convert a Foo file from a url to some Bar file and upload it to our servers.
For sake of clarity I added bellow a case scenario in order to make discussion easier and have an example at hand.
This workflow does the following:
Get a url as parameter
Download the Foo file as a TempFile
Duplicate it to a new TempFile
Download the related assets to TempFiles
Link the related assets into the local dup TempFile
Convert the Foo to Bar format
Upload it to our server
At times the conversion fail and everything points to the fact that our local Foo file is pointing to related assets that have been created and GCed before the conversion.
My two questions:
Is it possible that my TempFiles get GCed too early? I read about Ruby GCed system it was very conservative to avoid those scenarios.
How can I avoid this from happening? I could try to save all related assets from download_and_replace_uri(node) and passing them as a return to keep it alive while the instance of ConvertService is still existing. But I'm not sure if this would solve it.
myfile.foo
{
"buffers": [
{ "uri": "http://example.com/any_file.jpg" },
{ "uri": "http://example.com/any_file.png" },
{ "uri": "http://example.com/any_file.jpmp3" }
]
}
main.rb
ConvertService.new('http://example.com/myfile.foo')
ConvertService
class ConvertService
def initialize(url)
#url = url
#bar_file = Tempfile.new
end
def call
import_foo
convert_foo
upload_bar
end
private
def import_foo
#foo_file = ImportService.new(#url).call.edited_file
end
def convert_foo
`create-bar "#{#foo_file.path}" "#{#bar_file.path}"`
end
def upload_bar
UploadBarService.new(#bar_file).call
end
end
ImportService
class ImportService
def initialize(url)
#url = url
#edited_file ||= Tempfile.new
end
def call
download
duplicate
replace
end
private
def download
#original = DownloadFileService.new(#url).call.file
end
def duplicate
FileUtils.cp(#original.path, #edited_file.path)
end
def replace
file = File.read(#edited_file.path)
json = JSON.parse(file, symbolize_names: true)
json[:buffers]&.each do |node|
node[:uri] = DownloadFileService.new(node[:uri]).call.file.path
end
write_to_disk(#edited_file.path, json.to_json)
end
end
DownloadFileService
module Helper
class DownloadFileService < ApplicationHelperService
def initialize(url)
#url = url
#file = Tempfile.new
end
def call
uri = URI.parse(#url)
Net::HTTP.start(
uri.host,
uri.port,
use_ssl: uri.scheme == 'https'
) do |http|
response = http.request(Net::HTTP::Get.new(uri.path))
#file.binmode
#file.write(response.body)
#file.flush
end
end
end
end
UploadBarService
module Helper
class UploadBarService < ApplicationHelperService
def initialize(file)
#file = file
end
def call
HTTParty.post('http://example.com/upload', body: { file: #file })
# NOTE: End points returns the url for the uploaded file
end
end
end
Because of the complexity of your code and missing parts which may be obfuscated to us, the simple answer to your problem is to insure that your tempfile instance objects remain in memory throughout the lifecycle in which they are needed, otherwise they will get garbage collected immediately, removing the tempfile from the file system, and will lead to the the missing tempfile state you've encountered.
The Ruby Document for Tempfile states "When a Tempfile object is garbage collected, or when the Ruby interpreter exits, its associated temporary file is automatically deleted."
As per comments, others may find this conversation helpful when running into this problem.
I have a ruby class like this:
require 'logger'
class T
def do_something
log = Logger.new(STDERR)
log.info("Here is an info message")
end
end
And a test script line this:
#!/usr/bin/env ruby
gem "minitest"
require 'minitest/autorun'
require_relative 't'
class TestMailProcessorClasses < Minitest::Test
def test_it
me = T.new
out, err = capture_io do
me.do_something
end
puts "Out is '#{out}'"
puts "err is '#{err}'"
end
end
When I run this test, both out and err are empty strings. I see the message printed on stderr (on the terminal). Is there a way to make Logger and capture_io to play nicely together?
I'm in a straight Ruby environment, not Ruby on Rails.
The magic is to use capture_subprocess_io
out, err = capture_subprocess_io do
do_stuff
end
MiniTest's #capture_io temporarily switches $stdout and $stderr for StringIO objects to capture output written to $stdout or $stderr. But Logger has its own reference to the original standard error stream, which it will write to happily. I think you can consider this a bug or at least a limitation of MiniTest's #capture_io.
In your case, you're creating the Logger inside the block to #capture_io with the argument STDERR. STDERR still points to the original standard error stream, which is why it doesn't work as expected.
Changing STDERR to $stderr (which at that points does point to a StringIO object) works around this problem, but only if the Logger is actually created in the #capture_io block, since outside that block it points to the original standard error stream.
class T
def do_something
log = Logger.new($stderr)
log.info("Here is an info message")
end
end
Documentation of capture_subprocess_io
Basically Leonard's example fleshed out and commented with working code and pointing to the docs.
Captures $stdout and $stderr into strings, using Tempfile to ensure that subprocess IO is captured as well.
out, err = capture_subprocess_io do
system "echo Some info" # echos to standard out
system "echo You did a bad thing 1>&2" # echos to standard error
end
assert_match %r%info%, out
assert_match %r%bad%, err
NOTE: This method is approximately 10x slower than #capture_io so only use it when you need to test the output of a subprocess.
See Documentation
This is an old question, but one way we do this is to mock out the logger with an expects. Something like
logger.expects(:info).with("Here is an info message")
This allows us to assert the code under test without changing how logger works out of the box.
As an example of capture_io, we have a logger implementation to allow us to pass in hashes and output them to json. When we test that implementation we use capture_io. This is possible because we initialize the logger implementation in our subject line with $stdout.
subject { CustomLogging.new(ActiveSupport::Logger.new($stdout)) }
in the test
it 'processes a string message' do
msg = "stuff"
out, err = capture_io do
subject.info(msg)
end
out.must_equal "#{msg}\n"
end
You need to provide a different StringIO object while initializing Logger.new to capture the output, rather than the usual: STDERR which actually points to the console.
I modified the above two files a bit and made into a single file so that you can copy and test easily:
require 'logger'
require 'minitest'
class T
def do_something(io = nil)
io ||= STDERR
log = Logger.new io
log.info("Here is an info message")
end
end
class TestT < Minitest::Test
def test_something
t = T.new
string_io = StringIO.new
t.do_something string_io
puts "Out: #{string_io.string}"
end
end
Minitest.autorun
Explanation:
Method do_something will function normally in all other code when used without the argument.
When a StringIO method is provided, it uses that instead of the typical STDERR thus enabling to capture output like into a file or in this case for testing.
I inherited a tool that is working correctly but when I try to extend it it just fails. Since I am new to ruby and yaml I dont really know what is the reason why this fails...
So I have a class config that looks like this
class Configuration
def self.[] key
##config[key]
end
def self.load name
##config = nil
io = File.open( File.dirname(__FILE__) + "/../../../config/config.yml" )
YAML::load_documents(io) { |doc| ##config = doc[name] }
raise "Could not locate a configuration named \"#{name}\"" unless ##config
end
def self.[]=key, value
##config[key] = value
end
end
end
raise "Please set the A environment variable" unless ENV['A']
Helpers::Configuration.load(ENV['A'])
raise "Please set the D environment variable" unless ENV['D']
Helpers::Configuration.load(ENV['D'])
raise "Please set the P environment variable" unless ENV['P']
Helpers::Configuration.load(ENV['P'])
So I had a first version with the environment variable A that worked fine, then when I want to integrate 2 more environment variables it fails (they are different key/value sets). I did debug it and it looks like when it reads the second key/value it removes the other ones (such as reading the 3rd removes the previous 2, so I end up with ##config with only the 3rd key/value par instead of all the values I need).
It is probably easy to fix this, any idea how?
Thanks!
EDIT:
The config file use to look like:
Test:
position_x: “56”
position_y: “56”
Now I want to make it like
“x56”:
position_x: “56”
“x15”:
position_x: “15”
“y56”:
position_y: “56”
“y15”:
position_y: “15”
My idea is that I set them separately and I don’t need to create all the combinations…
Each time you call load you delete the previous configuration (in the line ##config = nil). If you want the configuration to be a merger of all files you will want to merge the new configuration to the existing configuration rather than overriding it.
Something like this:
def self.load name
##config ||= {}
io = File.open( File.dirname(__FILE__) + "/../../../config/config.yml" )
YAML::load_documents(io) do |doc|
raise "Could not locate a configuration named \"#{name}\"" unless doc[name]
##config.merge!(doc[name])
end
end
Be aware that if the code was written as it has been because the method was called more than once, and the configuration is expected to reset between reads, you will need to explicitly reset the configuration now:
class Configuration
# ...
def reset_configuration
#config = {}
end
end
Helpers::Configuration.reset_configuration
raise "Please set the A environment variable" unless ENV['A']
Helpers::Configuration.load(ENV['A'])
raise "Please set the D environment variable" unless ENV['D']
Helpers::Configuration.load(ENV['D'])
raise "Please set the P environment variable" unless ENV['P']
Helpers::Configuration.load(ENV['P'])
I'd access the YAML using:
YAML::load_file(File.expand_path("../../../config/config.yml", File.dirname(__FILE__)))
expand_path cleans up the '..' chain and returns the cleaned-up version, relative to FILE. For instance:
foo = '/path/to/a/file'
File.expand_path("../config.yml", File.dirname(foo)) # => "/path/to/config.yml"
load_file reads and parses the entire file and returns it.
I'm experimenting with RSpec.
Since I don't like mocks, I would like to emulate a console print using a StringIO object.
So, I want to test that the Logger class writes Welcome to the console. To do so, my idea was to override the puts method used inside Logger from within the spec file, so that nothing actually changes when using Logger elsewhere.
Here's some code:
describe Logger do
Logger.class_eval do
def puts(*args)
???.puts(*args)
end
end
it 'says "Welcome"' do
end
Doing this way, I need to share some StringIO object (which would go where the question marks are now) between the Logger class and the test class.
I found out that when I'm inside RSpec tests, self is an instance of Class. What I thought initially was to do something like this:
Class.class_eval do
attr_accessor :my_io
#my_io = StringIO.new
end
and then replace ??? with Class.my_io.
When I do this, a thousand bells ring in my head telling me it's not a clean way to do this.
What can I do?
PS: I still don't get this:
a = StringIO.new
a.print('a')
a.string # => "a"
a.read # => "" ??? WHY???
a.readlines # => [] ???
Still: StringIO.new('hello').readlines # => ["hello"]
To respond to your last concern, StringIO simulates file behavior. When you write/print to it, the input cursor is positioned after the last thing you wrote. If you write something and want to read it back, you need to reposition yourself (e.g. with rewind, seek, etc.), per http://ruby-doc.org/stdlib-1.9.3/libdoc/stringio/rdoc/StringIO.html
In contrast, StringIO.new('hello') establishes hello as the initial contents of the string while leaving in the position at 0. In any event, the string method just returns the contents, independent of position.
It's not clear why you have an issue with the test double mechanism in RSpec.
That said, your approach for sharing a method works, although:
The fact that self is an anonymous class within RSpec's describe is not really relevant
Instead of using an instance method of Class, you can define your own class and associated class method and "share" that instead, as in the following:
class Foo
def self.bar(arg)
puts(arg)
end
end
describe "Sharing stringio" do
Foo.class_eval do
def self.puts(*args)
MyStringIO.my_io.print(*args)
end
end
class MyStringIO
#my_io = StringIO.new
def self.my_io ; #my_io ; end
end
it 'says "Welcome"' do
Foo.bar("Welcome")
expect(MyStringIO.my_io.string).to eql "Welcome"
end
end
Logger already allows the output device to be specified on construction, so you can easily pass in your StringIO directly without having to redefine anything:
require 'logger'
describe Logger do
let(:my_io) { StringIO.new }
let(:log) { Logger.new(my_io) }
it 'says welcome' do
log.error('Welcome')
expect(my_io.string).to include('ERROR -- : Welcome')
end
end
As other posters have mentioned, it's unclear whether you're intending to test Logger or some code that uses it. In the case of the latter, consider injecting the logger into the client code.
The answers to this SO question also show several ways to share a common Logger between clients.
How this simple task can be done in Ruby?
I have some simple config file
=== config.rb
config = { 'var' => 'val' }
I want to load config file from some method, defined in main.rb file so that the local variables from config.rb became local vars of that method.
Something like this:
=== main.rb
Class App
def loader
load('config.rb') # or smth like that
p config['var'] # => "val"
end
end
I know that i can use global vars in config.rb and then undefine them when done, but i hope there's a ruby way )
The config file.
{ 'var' => 'val' }
Loading the config file
class App
def loader
config = eval(File.open(File.expand_path('~/config.rb')).read)
p config['var']
end
end
As others said, for configuration it's better to use YAML or JSON. To eval a file
binding.eval(File.open(File.expand_path('~/config.rb')).read, "config.rb")
binding.eval(File.read(File.expand_path('~/config.rb')), "config.rb")
This syntax would allow you to see filename in backtraces which is important. See api docs [1].
Updated eval command to avoid FD (file descriptor) leaks. I must have been sleeping or maybe should have been sleeping at that time of the night instead of writing on stackoverflow..
[1] http://www.ruby-doc.org/core-1.9.3/Binding.html
You certainly could hack out a solution using eval and File.read, but the fact this is hard should give you a signal that this is not a ruby-like way to solve the problem you have. Two alternative designs would be using yaml for your config api, or defining a simple dsl.
The YAML case is the easiest, you'd simply have something like this in main.rb:
Class App
def loader
config = YAML.load('config.yml')
p config['var'] # => "val"
end
end
and your config file would look like:
---
var: val
I do NOT recommend doing this except in a controlled environment.
Save a module to a file with a predetermined name that defines an initialize and run_it methods. For this example I used test.rb as the filename:
module Test
##classvar = 'Hello'
def initialize
#who = 'me'
end
def get_who
#who
end
def run_it
print "#{##classvar} #{get_who()}"
end
end
Then write a simple app to load and execute it:
require 'test'
class Foo
include Test
end
END {
Foo.new.run_it
}
# >> Hello me
Just because you can do something doesn't mean you should. I cannot think of a reason I'd do it in production and only show it here as a curiosity and proof-of-concept. Making this available to unknown people would be a good way to get your machine hacked because the code could do anything the owning account could do.
I just had to do a similar thing as I wanted to be able to load a "Ruby DLL" where it returns an anonymous class ( a factory for instances of things ) I created this which keeps track of items already loaded and allows the loaded file to return a value which can be anything - a totally anonymous Class, Module, data etc. It could be a module which you could then "include" in an object after it is loaded and it could could supply a host of "attributes" or methods. you could also add an "unload" item to clear it from the loaded hash and dereference any object it loaded.
module LoadableModule
##loadedByFile_ = {};
def self.load(fileName)
fileName = File.expand_path(fileName);
mod = ##loadedByFile_[fileName];
return mod if mod;
begin
Thread.current[:loadReturn] = nil;
Kernel.load(fileName);
mod = Thread.current[:loadReturn];
##loadedByFile_[fileName] = mod if(mod);
rescue => e
puts(e);
puts(e.backtrace);
mod = nil;
end
Thread.current[:loadReturn] = nil;
mod
end
def self.onLoaded(retVal)
Thread.current[:loadReturn] = retVal;
end
end
inside the loaded file:
LoadableModule.onLoaded("a value to return from the loaded file");