How to Control Multithreading from a Console - ruby

Learning to use Ruby Threads for transportability of code between different OS platforms.
The problem is the console is frozen until non_join1 completes which also prevents non_join2 from being started. non_join1 waits at the join command until the threads complete.
The software requires multiple routines running independently. The primary program is a standalone that runs in realtime collecting data. The data collected is written to files. Different programs, using Threads, process the data in parallel. The start/stop and status is controlled from a main console.
What is the best ruby method to launch the separate programs needed to analyze the data files and get status back from the threads?
thanks,
pb
# This is the console that starts up the multiple threads.
#!/usr/bin/ruby
loop do
puts " input a command"
command = gets.chop!
control = case command
when "1" : "1"
when "2" : "2"
end
if control == "1" then
puts `date`+ "routine 1"
puts `./non_join1.rb`
puts `date`
end
if control == "2" then
puts `date` + "routine 2"
`./non_join2.rb`
end
end
#!/usr/bin/ruby
# Example of worker program 1 used to process data files
#file non_join1.rb
x = Thread.new { sleep 0.1; print "xxxxxxxxx"; print "yyyyyyyyyyy"; print "zzzzzzzzzz" }
a = Thread.new { print "aaaaaaaaa"; print "bbbbbbbbbb"; sleep 0.1; print "cccccccc" }
puts " "
(1..10).each {|i| puts i.to_s+" done #{i}"}
x.join
a.join
sleep(30)
#!/usr/bin/ruby
# Example of worker program 2 used to process data files
#file non_join2.rb
x = Thread.new { sleep 0.1; print "xxxxxxxxx"; print "yyyyyyyyyyy"; print "zzzzzzzzzz" }
a = Thread.new { print "aaaaaaaaa"; print "bbbbbbbbbb"; sleep 0.1; print "cccccccc" }
x.join
a.join
$ ./call_ruby.rb
input a command
1
Sat Feb 18 10:36:43 PST 2012
routine 1
aaaaaaaaabbbbbbbbbb
1 done 1
2 done 2
3 done 3
4 done 4
5 done 5
6 done 6
7 done 7
8 done 8
9 done 9
10 done 10
xxxxxxxxxyyyyyyyyyyyzzzzzzzzzzcccccccc
Sat Feb 18 10:37:13 PST 2012
input a command

Instead of executing with `` try this forking (creating a new process), using for example this function:
class Execute
def self.run(command)
pid = fork do
Kernel.exec(command)
end
return pid
end
end
Your code will look like
loop do
puts " input a command"
command = gets.chop!
control = case command
when "1" : "1"
when "2" : "2"
end
if control == "1" then
puts `date`+ "routine 1"
Execute.run("./non_join1.rb")
puts `date`
end
if control == "2" then
puts `date` + "routine 2"
Execute.run("./non_join2.rb")
end
end

Related

Continuously read from STDOUT and STDERR in real time

Say we have a small application (example.rb) which behave like the following ruby code:
#!/usr/bin/env ruby
done = false
out =Thread.new do
cnt = 0
while !done
STDOUT.puts "out #{cnt}"
cnt += 1
sleep 1
end
end
err = Thread.new do
cnt = 0
while !done
STDERR.puts "err #{cnt}"
cnt += 1
sleep 1
end
end
while true
i = STDIN.gets
if i == "q\n"
puts "Quiting"
done = true
break
end
end
out.join
err.join
exit 42
It print something to stdout and to stderr, it must be quited by writing "q\n" to stdin, and when it exit a value is returned in the return code.
Now, I would like to write a small ruby script which can run this program in an external process, where stdout and stdin are captured, and when the external process should be terminated this is done by writing "q\n" to its stdin. This program is called monitor.rb.
This is what I have tried here:
#!/usr/bin/env ruby
require 'open3'
class Monitor
#cmd
attr_accessor :return_code
def initialize cmd
#cmd = cmd
end
def run
#runner_thread = Thread.new do
Open3::popen3(#cmd) do |stdin, stdout, stderr, thread|
puts "#{Time.now} #{#cmd} is running as pid: #{thread.pid}"
stdin.sync = true;
stdout.sync = true;
stderr.sync = true;
#stdin = stdin
t_out = Thread.new do
stdout.readlines do |l|
puts "#{Time.now} STDOUT> #{l}"
end
end
t_err = Thread.new do
stderr.readlines do |l|
puts "#{Time.now} STDERR> #{l}"
end
end
thread.join
t_err.join
t_out.join
#return_code = thread.value
end
end
end
def quit
puts "Quiting"
#stdin.puts "q"
#stdin.close
#runner_thread.join
end
end
mon = Monitor.new "./example.rb"
mon.run
sleep 5
mon.quit
puts "Return code: #{mon.return_code}"
Question 1: What is wrong with my code since the output of the external process is not being printed?
Question 2: Can this be done in a more elegant way, and what would that look like?
The code must be able to run on Linux and portability is not a priority, I uses ruby 2.0.
When run example.rb in a terminal I get:
$ ./example.rb
out 0
err 0
out 1
err 1
out 2
err 2
q
Quiting
When I run the monitor application I get:
$ ./monitor.rb
2013-11-19 14:39:20 +0100 ./example.rb is running as pid: 7228
Quiting
Return code: pid 7228 exit 42
I expected the monitor.rb to print the output from example.rb
Try changing your t_out and t_err threads to use the following code. readlines will read the entire file at once and stdout and stderr will block until your script exits. I think this is why you were not getting any output.
while l = stdout.gets
puts "#{Time.now} STDOUT> #{l}"
end
This should print out to the screen as soon as any output is available.

Double fork and stdin

I wrote this code to run my process in a daemon. The goal is to make this process running even if I close its parent. Now, i would like to be able to write something in its stdin. What should I do ? Here's the code.
def daemonize(cmd, options = {})
rd, wr = IO.pipe
p1 = Process.fork {
Process.setsid
p2 = Process.fork {
$0 = cmd #Name of the command
pidfile = File.new(options[:pid_file], 'w')
pidfile.chmod( 0644 )
pidfile.puts "#{Process.pid}"
pidfile.close
Dir.chdir(ENV["PWD"] = options[:working_dir].to_s) if options[:working_dir]
File.umask 0000
STDIN.reopen '/dev/null'
STDOUT.reopen '/dev/null', 'a'
STDERR.reopen STDOUT
Signal.trap("USR1") do
Console.show 'I just received a USR1', 'warning'
end
::Kernel.exec(*Shellwords.shellwords(cmd)) #Executing the command in the parent process
exit
}
raise 'Fork failed!' if p2 == -1
Process.detach(p2) # divorce p2 from parent process (p1)
rd.close
wr.write p2
wr.close
exit
}
raise 'Fork failed!' if p1 == -1
Process.detach(p1) # divorce p1 from parent process (shell)
wr.close
daemon_id = rd.read.to_i
rd.close
daemon_id
end
Is there a way to reopen stdin in something like a pipe instead of /dev/null in which I would be able to write ?
How about a fifo? In linux, you can use the mkfifo command:
$ mkfifo /tmp/mypipe
Then you can reopen STDIN on that pipe:
STDIN.reopen '/tmp/mypipe'
# Do read-y things
Anything else can write to that pipe:
$ echo "roflcopter" > /tmp/mypipe
allowing that data to be read by the ruby process.
(Update) Caveat with blocking
Since fifos block until there's a read and write (e.g. a read is blocked unless there's a write, and vice-versa), it's best handled with multiple threads. One thread should do the reading, passing the data to a queue, and another should handle that input. Here's an example of that situation:
require 'thread'
input = Queue.new
threads = []
# Read from the fifo and add to an input queue (glorified array)
threads << Thread.new(input) do |ip|
STDIN.reopen 'mypipe'
loop do
if line = STDIN.gets
puts "Read: #{line}"
ip.push line
end
end
end
# Handle the input passed by the reader thread
threads << Thread.new(input) do |ip|
loop do
puts "Ouput: #{ip.pop}"
end
end
threads.map(&:join)

Run code block with Daemons gem

can somebody explain why the following code won't spawn the passed block ?
require 'daemons'
t = Daemons.call do
# This block does not start
File.open('out.log','w') do # code don't get here to open a file
|fw|
10.times {
fw.puts "=>#{rand(100)}"
sleep 1
}
end
end
#t.start # has no effect
10.times {
puts "Running ? #{t.running?}" # prints "Running ? false" 10 times
sleep 1
}
t.stop
puts 'finished'
Ruby 1.9.3p392, x86_64 Linux
You sure you're not trying to run a Thread for concurrent programming?
Here's what a Thread implementation would look like:
f = File.open('out.log', 'w')
t = Thread.new do
10.times {
f.puts "=>#{rand(100)}"
sleep 1
}
end
10.times {
puts "Running ? #{t.alive?}"
sleep 1
}
t.exit
puts 'finished'

Ruby Pause thread

In ruby, is it possible to cause a thread to pause from a different concurrently running thread.
Below is the code that I've written so far. I want the user to be able to type 'pause thread' and the sample500 thread to pause.
#!/usr/bin/env ruby
# Creates a new thread executes the block every intervalSec for durationSec.
def DoEvery(thread, intervalSec, durationSec)
thread = Thread.new do
start = Time.now
timeTakenToComplete = 0
loopCounter = 0
while(timeTakenToComplete < durationSec && loopCounter += 1)
yield
finish = Time.now
timeTakenToComplete = finish - start
sleep(intervalSec*loopCounter - timeTakenToComplete)
end
end
end
# User input loop.
exit = nil
while(!exit)
userInput = gets
case userInput
when "start thread\n"
sample500 = Thread
beginTime = Time.now
DoEvery(sample500, 0.5, 30) {File.open('abc', 'a') {|file| file.write("a\n")}}
when "pause thread\n"
sample500.stop
when "resume thread"
sample500.run
when "exit\n"
exit = TRUE
end
end
Passing Thread object as argument to DoEvery function makes no sense because you immediately overwrite it with Thread.new, check out this modified version:
def DoEvery(intervalSec, durationSec)
thread = Thread.new do
start = Time.now
Thread.current["stop"] = false
timeTakenToComplete = 0
loopCounter = 0
while(timeTakenToComplete < durationSec && loopCounter += 1)
if Thread.current["stop"]
Thread.current["stop"] = false
puts "paused"
Thread.stop
end
yield
finish = Time.now
timeTakenToComplete = finish - start
sleep(intervalSec*loopCounter - timeTakenToComplete)
end
end
thread
end
# User input loop.
exit = nil
while(!exit)
userInput = gets
case userInput
when "start thread\n"
sample500 = DoEvery(0.5, 30) {File.open('abc', 'a') {|file| file.write("a\n")} }
when "pause thread\n"
sample500["stop"] = true
when "resume thread\n"
sample500.run
when "exit\n"
exit = TRUE
end
end
Here DoEvery returns new thread object. Also note that Thread.stop called inside running thread, you can't directly stop one thread from another because it is not safe.
You may be able to better able to accomplish what you are attempting using Ruby Fiber object, and likely achieve better efficiency on the running system.
Fibers are primitives for implementing light weight cooperative
concurrency in Ruby. Basically they are a means of creating code
blocks that can be paused and resumed, much like threads. The main
difference is that they are never preempted and that the scheduling
must be done by the programmer and not the VM.
Keeping in mind the current implementation of MRI Ruby does not offer any concurrent running threads and the best you are able to accomplish is a green threaded program, the following is a nice example:
require "fiber"
f1 = Fiber.new { |f2| f2.resume Fiber.current; while true; puts "A"; f2.transfer; end }
f2 = Fiber.new { |f1| f1.transfer; while true; puts "B"; f1.transfer; end }
f1.resume f2 # =>
# A
# B
# A
# B
# .
# .
# .

Ruby multithreading questions

I've started looking into multi-threading in Ruby.
So basically, I want to create a few threads, and have them all execute, but not display any of the output until the thread has successfully completed.
Example:
#!/usr/bin/env ruby
t1 = Thread.new {
puts "Hello_1"
sleep(5)
puts "Hello_1 after 5 seconds of sleep"
}
t2 = Thread.new {
puts "Hello_2"
sleep(5)
puts "Hello_2 after 5 seconds of sleep"
}
t1.join
t2.join
puts "Hello_3"
sleep(5)
puts "Hello_3 after 5 seconds of sleep"
The first Hello_1 / Hello_2 execute immediately. I wouldn't want any of the output to show until the thread has successfully completed.
Because puts prints to a single output stream (sysout) you can't use it if you want to capture the output each thread.
You will have to use separate buffered stream for each thread, write to that in each thread, and then dump them to sysout when the thread terminates to see the output.
Here is an example of a thread:
t = Thread.new() do
io = StringIO.new
io << "mary"
io.puts "fred"
io.puts "fred"
puts io.string
end
You will have to pass io to every method in the thread.
or have a look at this for creating a module that redirects stdout for a thread.
But in each thread that your start wrap your code with:
Thread.start do
# capture the STDOUT by storing a StringIO in the thread space
Thread.current[:stdout] = StringIO.new
# Do your stuff.. print using puts
puts 'redirected to StringIO'
# print everything before we exit
STDIO.puts Thread.current[:stdout].string
end.join
You can share a buffer but you should 'synchronize' access to it:
buffer = ""
lock = Mutex.new
t1 = Thread.new {
lock.synchronize{buffer << "Hello_1\n"}
sleep(5)
lock.synchronize{buffer << "Hello_1 after 5 seconds of sleep\n"}
}
t2 = Thread.new {
lock.synchronize{buffer << "Hello_2\n"}
sleep(5)
lock.synchronize{buffer << "Hello_2 after 5 seconds of sleep\n"}
}
t1.join
t2.join
puts buffer

Resources