Ruby printing output to file with threads - ruby

I am writing a Ruby script which loops through some hashes and executes the command in the hash, like so
$conf_info = {
"OpenSSH" => {
"type" => "static",
"cmd" => %q[sshd -T],
"msg" => "Testing OpenSSH Configuration",
"res" => $cmd_results,
}
}
I have multiple of these hashes and I loop through all of them, executing each command. I am also using threads. The problem is that I can't output to a file. I have separate threads like this.
threads = []
get_enums($options[:enumerate])
threads << Thread.new {_run_($sys_info , $options[:system] , $enum_sys) }
threads << Thread.new {_run_($net_info , $options[:network] , $enum_net) }
threads.each {|t| t.join}
and I output to file like this
File.open("test_file", "w") do |file|
file.puts __start__
end
but, the file is only filled with contents like this
#<Thread:0x98993fc>
#<Thread:0x9872fcc>
and not the actual output of the program. I would also need the program to output to STDOUT and the file, may someone please tell me what I might be doing wrong?

You can't capture output of __start__ by assigning it to a variable. You can redirect output instead:
With TeeIO (Updated based from this answer too):
class TeeIO
def initialize(*ios)
ios.each{ |e| raise ArgumentError, "Not an IO object: #{e}" if not e.is_a? IO }
#ios = ios
end
def write(data)
#ios.each{ |io| io.write(data) }
end
def close
#ios.each{ |io| io.close }
end
def flush
#ios.each{ |io| io.flush }
end
def method_missing(meth, *args)
# Return first if it's not an IO or else return self.
first = #ios.map{ |io| io.send(meth, *args) }.first
first.is_a? IO ? self : first
end
def respond_to_missing?(meth, include_all)
#ios.all?{ |io| io.respond_to?(meth, include_all) }
end
end
...
real_stdout = $stdout
file = File.open("test_file", "w")
$stdout = TeeIO.new(real_stdout, file)
__start__
$stdout = real_stdout
file.close

Related

ENOENT while trying to read a zipped file

I'm writing a simple program that takes an input string, splits it in words and saves it in it's memory. There's three methods -- to save a string into memory, to load from file and to load from zip archive. Here's the code:
require 'zip'
class Storage
def initialize
#storage = ''
end
def add(string)
words = string.split ','
words.each do |word|
#storage << "#{word},"
end
end
def load_from_file(filename)
File.open filename, 'r' do |f|
f.each { |line| add line }
end
end
def load_from_zip(filename)
Zip::File.open "#{filename}.zip" do |zipfile|
zipfile.each { |entry| load_from_file entry.to_s }
end
end
end
While add and load_from_file methods work perfectly as I expect, load_from_zip returns me the following error each time I try to run it:
storage.rb:39:in `initialize': No such file or directory # rb_sysopen - test.txt (Errno::ENOENT)
Although the file exists in my archive. I would appreciate any suggestion on what I'm doing wrong
Zip::File.open "#{filename}.zip"
does not extract the zip file, it just opens it and shows what's inside.
You cannot call
File.open filename, 'r'
since filename isn't in your filesystem, just inside the .zip file.
You'll need to add a new method :
require 'zip'
class Storage
def initialize
#storage = ''
end
def add(string)
words = string.split ','
words.each do |word|
#storage << "#{word},"
end
end
def load_from_file(filename)
File.open filename, 'r' do |f|
f.each { |line| add line }
end
end
def load_from_zip(filename)
Zip::File.open "#{filename}.zip" do |zipfile|
zipfile.each { |entry| load_from_zipped_file(zipfile,entry)}
end
end
private
def load_from_zipped_file(zipfile, entry)
zipfile.read(entry).lines.each do |line|
add line
end
end
end
s = Storage.new
s.load_from_zip('test')

How to resolve pipe dead-lock in Ruby?

I am trying to emulate UNIX command line pipes in a Ruby-only solution that uses multiple cores. Eventually, the records piped from command to command will be Ruby objects marshaled using msgpack. Unfortunately, the below code hangs after the first dump command. I suspect a pipe deadlock, but I fail to resolve it?
#!/usr/bin/env ruby
require 'parallel'
require 'msgpack'
require 'pp'
class Pipe
def initialize
#commands = []
end
def add(command, options = {})
#commands << Command.new(command, options)
self
end
def run
#commands.each_cons(2) do |c_in, c_out|
reader, writer = IO.pipe
c_out.input = MessagePack::Unpacker.new(reader)
c_in.output = MessagePack::Packer.new(writer)
end
Parallel.each(#commands, in_processes: #commands.size) { |command| command.run }
self
end
class Command
attr_accessor :input, :output
def initialize(command, options)
#command = command
#options = options
#input = nil
#output = nil
end
def run
send #command
end
def cat
#input.each_with_index { |record, i| #output.write(record).flush } if #input
File.open(#options[:input]) do |ios|
ios.each { |record| #output.write(record).flush } if #output
end
end
def dump
#input.each do |record|
puts record
#output.write(record).flush if #output
end
end
end
end
p = Pipe.new
p.add(:cat, input: "foo.tab").add(:dump).add(:cat, input: "table.txt").add(:dump)
p.run

How to create a memory efficient Ruby Pipe class with lazy evaluation?

I would like to create a Pipe class to emulate Unix commands in Ruby in a two step fashion. First step is to compile a pipeline by adding a number of commands, and the second step is to run that pipeline. Here is a mockup:
#!/usr/bin/env ruby
p = Pipe.new
p.add(:cat, input: "table.txt")
p.add(:cut, field: 2)
p.add(:grep, pattern: "foo")
p.add(:puts, output: "result.txt")
p.run
The question is how to code this using lazy evaluation, so that the pipe is processed record by record when run() is called without loading all of the data into memory at any one time?
Take a look at the http://ruby-doc.org/core-2.0.0/Enumerator.html class. The Pipe class will stitch together an Enumerator, e.g. add(:cat, input: 'foo.txt') will create an enumerator which yields lines of foo.txt. add(:grep) will filter it according to regexp etc.
Here's the lazy file reader
require 'benchmark'
def lazy_cat(filename)
e = Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy
end
def cat(filename)
Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
end
lazy = Benchmark.realtime { puts lazy_cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Lazy: #{lazy}"
eager = Benchmark.realtime { puts cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Eager: #{eager}"
Eager version takes 7 seconds for 10 million line file, lazy version takes pretty much no time.
For what I understood you can simply read one line at a time and move this single line thought the pipeline, then write it to the output. Some code:
output = File.new("output.txt")
File.new("input.txt").each do |line|
record = read_record(line)
newrecord = run_pipeline_on_one_record(record)
output.write(dump_record(newrecord))
end
Another much heavier option would be create actual IO blocking pipes and use one thread for each task in the pipeline. This somewhat reassembles what Unix does.
Sample usage with OP's syntax:
class Pipe
def initialize
#actions = []
end
def add(&block)
#actions << block
end
def run(infile, outfile)
output = File.open(outfile, "w")
File.open(infile).each do |line|
line.chomp!
#actions.each {|act| line = act[line] }
output.write(line+"\n")
end
end
end
p = Pipe.new
p.add {|line| line.size.to_s }
p.add {|line| "number of chars: #{line}" }
p.run("in.txt", "out.txt")
Sample in.txt:
aaa
12345
h
Generated out.txt:
number of chars: 3
number of chars: 5
number of chars: 1
This seems to work:
#!/usr/bin/env ruby
require 'pp'
class Pipe
def initialize
#commands = []
end
def add(command, options = {})
#commands << [command, options]
self
end
def run
enum = nil
#commands.each do |command, options|
enum = method(command).call enum, options
end
enum.each {}
enum
end
def to_s
cmd_string = "Pipe.new"
#commands.each do |command, options|
opt_list = []
options.each do |key, value|
if value.is_a? String
opt_list << "#{key}: \"#{value}\""
else
opt_list << "#{key}: #{value}"
end
end
cmd_string << ".add(:#{command}, #{opt_list.join(", ")})"
end
cmd_string << ".run"
end
private
def cat(enum, options)
Enumerator.new do |yielder|
enum.map { |line| yielder << line } if enum
File.open(options[:input]) do |ios|
ios.each { |line| yielder << line }
end
end.lazy
end
def cut(enum, options)
Enumerator.new do |yielder|
enum.each do |line|
fields = line.chomp.split(%r{#{options[:delimiter]}})
yielder << fields[options[:field]]
end
end.lazy
end
def grep(enum, options)
Enumerator.new do |yielder|
enum.each do |line|
yielder << line if line.match(options[:pattern])
end
end.lazy
end
def save(enum, options)
Enumerator.new do |yielder|
File.open(options[:output], 'w') do |ios|
enum.each do |line|
ios.puts line
yielder << line
end
end
end.lazy
end
end
p = Pipe.new
p.add(:cat, input: "table.txt")
p.add(:cut, field: 2, delimiter: ',\s*')
p.add(:grep, pattern: "4")
p.add(:save, output: "result.txt")
p.run
puts p
https://stackoverflow.com/a/20049201/3183101
require 'benchmark'
def lazy_cat(filename)
e = Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy
end
def cat(filename)
Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
end
lazy = Benchmark.realtime { puts lazy_cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Lazy: #{lazy}"
eager = Benchmark.realtime { puts cat("log.txt").map{|s| s.upcase}.take(1).to_a }
puts "Eager: #{eager}"
This could have been simplified to the following, which I think makes the diff between the two methods easier to see.
require 'benchmark'
def cat(filename, evaluation_strategy: :eager)
e = Enumerator.new do |yielder|
f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy if evaluation_strategy == :lazy
end
lazy = Benchmark.realtime { puts cat("log.txt", evaluation_strategy: :lazy).map{ |s|
s.upcase}.take(1).to_a
}
puts "Lazy: #{lazy}"
eager = Benchmark.realtime { puts cat("log.txt", evaluation_strategy: :eager).map{ |s|
s.upcase}.take(1).to_a
}
puts "Eager: #{eager}"
I would have just put this in a comment, but I'm too 'green' here to be permitted to do so. Anyway, the ability to post all of the code I think makes it clearer.
This builds on previous answers, and serves as a warning about a gotcha regarding enumerators. An enumerator that hasn't been exhausted (i.e. raised StopIteration) will not run ensure blocks. That means a construct like File.open { } won't clean up after itself.
Example:
def lazy_cat(filename)
f = nil # visible to the define_singleton_method block
e = Enumerator.new do |yielder|
# Also stored in #f for demonstration purposes only, so we examine it later
#f = f = File.open filename
s = f.gets
while s
yielder.yield s
s = f.gets
end
end
e.lazy.tap do |enum|
# Provide a finish method to close the File
# We can't use def enum.finish because it can't see 'f'
enum.define_singleton_method(:finish) do
f.close
end
end
end
def get_first_line(path)
enum = lazy_cat(path)
enum.take(1).to_a
end
def get_first_line_with_finish(path)
enum = lazy_cat(path)
enum.take(1).to_a
ensure
enum.finish
end
# foo.txt contains:
# abc
# def
# ghi
puts "Without finish"
p get_first_line('foo.txt')
if #f.closed?
puts "OK: handle was closed"
else
puts "FAIL: handle not closed!"
#f.close
end
puts
puts "With finish"
p get_first_line_with_finish('foo.txt')
if #f.closed?
puts "OK: handle was closed"
else
puts "FAIL: handle not closed!"
#f.close
end
Running this produces:
Without finish
["abc\n"]
FAIL: handle not closed!
With finish
["abc\n"]
OK: handle was closed
Note that if you don't provide the finish method, the stream won't be closed, and you'll leak file descriptors. It's possible that GC will close it, but you shouldn't depend on that.

Ruby parallel process in map

Help me plz
How i can implement method pmap for Array like map but in two process
I have code
class Array
def pmap
out = []
each do |e|
out << yield(e)
end
out
end
end
require 'benchmark'
seconds = Benchmark.realtime do
[1, 2, 3].pmap do |x|
sleep x
puts x**x
end
end
puts "work #{seconds} seconds"
In result i must get 3 second for benchmark
To get absolutely 2 forks
You don't absolutely need RPC. Marshal + Pipe should usually work.
class Array
def pmap
first, last = self[0..(self.length/2)], self[(self.length/2+1)..-1]
pipes = [first, last].map do |array|
read, write = IO.pipe
fork do
read.close
message = []
array.each do |item|
message << yield(item)
end
write.write(Marshal.dump message)
write.close
end
write.close
read
end
Process.waitall
first_out, last_out = pipes.map do |read|
Marshal.load(read.read)
end
first_out + last_out
end
end
Edit
Now using fork
Try the parallel gem.
require 'parallel'
class Array
def pmap(&blk)
Parallel.map(self, {:in_processes: 3}, &blk)
end
end

Write Ruby Test::Unit Errors & Fails to File

I am trying to write fails and errors that occur in the test to a log file, so that they don't appear on-screen, but it appears as though errors and assert failures write to STDOUT instead of STDERR. I have been unable to find information on how to redirect this output after hours of googling, and would greatly appreciate help.
Why should the errors not be on stdout?
Up to now I have no prepared solution to suppress the output for specific errors.
If you accept the unchanged output, I have a solution to store errors and failures in a file. It will be no problem to create a 2nd file for Notifications...
gem 'test-unit'
require 'test/unit'
module Test
module Unit
class TestSuite
alias :old_run :run
def run(result, &progress_block)
old_run(result, &progress_block)
File.open('test.log', 'w'){|f|
result.faults.each{|err|
case err
when Test::Unit::Error, Test::Unit::Failure
f << err.long_display
f << "\n===========\n"
#not in log file
when Test::Unit::Pending, Test::Unit::Notification, Test::Unit::Omission
end
}
}
end
end
end
end
class MyTest < Test::Unit::TestCase
def test_1()
assert_equal( 3, 1+1) #failure
end
def test_2()
1 / 0 #force an error
end
def test_3()
notify 'sss'
end
def test_4()
pend "MeineKlasse.new"
end
def test_5
omit 'aaa' if RUBY_VERSION == '1.9.2'
end
def test_5
assert_in_delta( 0.1, 0.00001, 1.0/10)
end
end
Have you tried redirecting the output to StringIO, to write it to a logfile later?
original_stdout = $stdout
original_stderr = $stderr
fake_stdout = StringIO.new
fake_stderr = StringIO.new
$stdout = fake_stdout
$stderr = fake_stderr
Then after you've ran the tests:
$stdout = original_stdout
$stderr = original_stderr
#stdout = fake_stdout.string
#stderr = fake_stderr.string
I'm not really sure this will work though...
I hope I understood yor question correct.
Do you mean something like this:
gem 'test-unit'
require 'test/unit'
class StdOutLogger < IO
def initialize(*)
super
#file = File.open('log.txt', 'w')
#stdout = true
end
#write to stdout and file
def write(s)
$stdout << s
#file << s
end
end
STDOUT = StdOutLogger.new(1)
class MyTest < Test::Unit::TestCase
def test_1()
assert_equal( 2, 1+1)
assert_equal( 2, 4/2)
assert_equal( 1, 3/2)
assert_equal( 1.5, 3/2.0)
end
end
But I would recommend to copy stdout on operation system level
ruby mytest.rb > log.txt
Here is a version to skip between stdout and file output. The output switch is only a interim solution - perhaps it can be made better.
class StdOutLogger < IO
def initialize(*)
super
#file = File.open('log.txt', 'w')
#stdout = true
end
def write(s)
case s
when /\AError:/
#stdout = false #change to file
when /\A(Pending):/
#stdout = true #change to file
end
if #stdout
$stdout << s
else
#file << s
end
end
end
STDOUT = StdOutLogger.new(1)

Resources