I need to run multiple background threads in a thread pool with timeout.
The scheme is something like:
#!/usr/bin/env ruby
require 'thread'
def foo(&block)
bar(block)
end
def bar(block)
Thread.abort_on_exception=true
#main = Thread.new { block.call }
end
foo {
sleep 1
puts 'test'
}
Why if i run that i get no output? (and no sleep wait?)
The program ends when the main thread ends. You have to wait on the thread created by bar using join:
foo {
sleep 1
puts 'test'
}.join
Try the work_queue gem http://rubygems.org/gems/work_queue/
Related
I have been experimenting multi-threading concept in Ruby for the past a week.
For practising, I am designing a file downloader that makes parallel requests for a collection of URLs. Currently I need to safely shutdown threads when interrupt signal is triggered. I have read the theory of multi-threading and catching a signal at runtime. Yet despite the whole those theoretical knowledge, I still don't have any idea about how to use them in practice.
I am leaving my proof of concept work below, anyhow.
class MultiThread
attr_reader :limit, :threads, :queue
def initialize(limit)
#limit = limit
#threads = []
#queue = Queue.new
end
def add(*args, &block)
queue << [block, args]
end
def invoke
1.upto(limit).each { threads << spawn_thread }
threads.each(&:join)
end
private
def spawn_thread
Thread.new do
Thread.handle_interrupt(RuntimeError => :on_blocking) do
# Nothing to do
end
until queue.empty?
block, args = queue.pop
block&.call(*args)
end
end
end
end
urls = %w[https://example.com]
thread = MultiThread.new(2)
urls.each do |url|
thread.add do
puts "Downloading #{url}..."
sleep 1
end
end
thread.invoke
Yeah, the docs for handle_interrupt are confusing. Try this, which I based on the connection_pool gem used by e.g. puma.
$stdout.sync = true
threads = 3.times.map { |i|
Thread.new {
Thread.handle_interrupt(Exception => :never) do
begin
Thread.handle_interrupt(Exception => :immediate) do
puts "Thread #{i} doing work"
sleep 1000
end
ensure
puts "Thread #{i} cleaning up"
end
end
}
}
Signal.trap("INT") {
puts 'Exiting gracefully'
threads.each { |t|
puts 'killing thread'
t.kill
}
exit
}
threads.each { |t| t.join }
Output:
Thread 1 doing work
Thread 2 doing work
Thread 0 doing work
^CExiting gracefully
killing thread
killing thread
killing thread
Thread 0 cleaning up
Thread 1 cleaning up
Thread 2 cleaning up
I have 3 methods that do something on the file system but each method changes current directory and I can't use Threading for those since it uses same ruby process and I have racing condition. I found Process.spawn (can't use fork on windows) might be solution but I don't know how to spawn 3 process and call method in each of those.
I tried following as a test, but didn't work
#!/usr/bin/env ruby
def something
puts 'printaj'
end
def nextsomething
puts 'vici'
end
pid1 = Process.spawn(something)
pid2 = Process.spawn(nextsomething)
Process.wait pid1
Process.wait pid2
puts 'both finished'
How about threads and mutex?
semaphore = Mutex.new
a = Thread.new {
semaphore.synchronize {
# do something on the file system
}
}
b = Thread.new {
semaphore.synchronize {
# do something on the file system
}
}
With this code you will run only one operation on file system in the same time.
https://ruby-doc.org/core-2.5.0/Mutex.html
My problem is that I don't know how synchronise multiple threads using Ruby. The task is to create six threads and start them immediately. All of them should do some work (for example puts "Thread 1" Hi") one after another in the order I need it to work.
I've tried to work with Mutex, Monitor and Condition Variable, but all of them worked in random order. Could anybody explain how to achieve my goal?
After some time of struggling with Mutex and Condition Variable I've achieved my goal.
This code is a little bit messy, and I intentionally did't use cycles for "clearer view".
cv = ConditionVariable.new
mutex = Mutex.new
mutex2 = Mutex.new
cv2 = ConditionVariable.new
mutex3 = Mutex.new
cv3 = ConditionVariable.new
mutex4 = Mutex.new
cv4 = ConditionVariable.new
mutex5 = Mutex.new
cv5 = ConditionVariable.new
mutex6 = Mutex.new
cv6 = ConditionVariable.new
Thread.new do
mutex.synchronize {
puts 'First: Hi'
cv.wait(mutex)
puts 'First: Bye'
#cv.wait(mutex)
cv.signal
puts 'First: One more time'
}
end
Thread.new do
mutex.synchronize {
puts 'Second: Hi'
cv.signal
cv.wait(mutex)
puts 'Second:Bye'
cv.signal
}
mutex2.synchronize {
puts 'Second: Starting third'
cv2.signal
}
end
Thread.new do
mutex2.synchronize {
cv2.wait(mutex2)
puts 'Third: Hi'
}
mutex3.synchronize {
puts 'Third: Starting forth'
cv3.signal
}
end
Thread.new do
mutex3.synchronize {
cv3.wait(mutex3)
puts 'Forth: Hi'
}
mutex4.synchronize {
puts 'Forth: Starting fifth'
cv4.signal
}
end
Thread.new do
mutex4.synchronize {
cv4.wait(mutex4)
puts 'Fifth: Hi'
}
mutex5.synchronize {
puts 'Fifth: Starting sixth'
cv5.signal
}
end
Thread.new {
mutex5.synchronize {
cv5.wait(mutex5)
puts 'Sixth:Hi'
}
}
sleep 2
Using Queue as a PV Semaphore
You can abuse Queue, using it like a traditional PV Semaphore. To do this, you create an instance of Queue:
require 'thread'
...
sem = Queue.new
When a thread needs to wait, it calls Queue#deq:
# waiting thread
sem.deq
When some other thread wants to unblock the waiting thread, it pushes something (anything) onto the queue:
# another thread that wants to unblock the waiting thread
sem.enq :go
A Worker class
Here's a worker class that uses Queue to synchronize its start and stop:
class Worker
def initialize(worker_number)
#start = Queue.new
Thread.new do
#start.deq
puts "Thread #{worker_number}"
#when_done.call
end
end
def start
#start.enq :start
end
def when_done(&block)
#when_done = block
end
end
When constructed, a worker creates a thread, but that thread then waits on the #start queue. Not until #start is called will the thread unblock.
When done, the thread will execute the block that was called to #when_done. We'll see how this is used in just a moment.
Creating workers
First, let's make sure that if any threads raise an exception, we get to find out about it:
Thread.abort_on_exception = true
We'll need six workers:
workers = (1..6).map { |i| Worker.new(i) }
Telling each worker what to do when it's done
Here's where #when_done comes into play:
workers.each_cons(2) do |w1, w2|
w1.when_done { w2.start }
end
This takes each pair of workers in turn. Each worker except the last is told, that when it finishes, it should start the worker after it. That just leaves the last worker. When it finishes, we want it to notify this thread:
all_done = Queue.new
workers.last.when_done { all_done.enq :done }
Let's Go!
Now all that remains is to start the first thread:
workers.first.start
and wait for the last thread to finish:
all_done.deq
The output:
Thread 1
Thread 2
Thread 3
Thread 4
Thread 5
Thread 6
If you're just getting started with threads, you might want to try something simple. Let the 1st thread sleep for 1 second, the 2nd for 2 seconds, the 3rd for 3 seconds and so on:
$stdout.sync = true
threads = []
(1..6).each do |i|
threads << Thread.new {
sleep i
puts "Hi from thread #{i}"
}
end
threads.each(&:join)
Output (takes 6 seconds because the threads run in parallel):
Hi from thread 1
Hi from thread 2
Hi from thread 3
Hi from thread 4
Hi from thread 5
Hi from thread 6
You can assign each a number, which will denote its place in the queue, and check it to see whose turn it is:
class QueuedWorker
def initialize(mutex, condition_variable, my_turn)
#mutex = mutex
#my_turn = my_turn
#condition_variable = condition_variable
end
def self.turn
#turn ||= 0
end
def self.done
#turn = turn + 1
end
def run
loop do
#mutex.synchronize do
if QueuedWorker.turn == #my_turn
# do actual work
QueuedWorker.done
#condition_variable.signal
return
end
#condition_variable.signal
#condition_variable.wait(#mutex)
end
end
end
end
mutex = Mutex.new
cv = ConditionVariable.new
(0..10).each do |i|
Thread.new do
QueueWorker.new(mutex, cv, i).run
end
end
That being said, the implementation is awkward, since threading are specifically not built for serial work. If you need something to work serially, do it in a single thread.
I'm using RSpec to test the behavior of a simple REPL. The REPL just echoes back whatever the input was, unless the input was "exit", in which case it terminates the loop.
To avoid hanging the test runner, I'm running the REPL method inside a separate thread. To make sure that the code in the thread has executed before I write expectations about it, I've found it necessary to include a brief sleep call. If I remove it, the tests fail intermittently because the expectations are sometimes made before the code in the thread has run.
What is a good way to structure the code and spec such that I can make expectations about the REPL's behavior deterministically, without the need for the sleep hack?
Here is the REPL class and the spec:
class REPL
def initialize(stdin = $stdin, stdout = $stdout)
#stdin = stdin
#stdout = stdout
end
def run
#stdout.puts "Type exit to end the session."
loop do
#stdout.print "$ "
input = #stdin.gets.to_s.chomp.strip
break if input == "exit"
#stdout.puts(input)
end
end
end
describe REPL do
let(:stdin) { StringIO.new }
let(:stdout) { StringIO.new }
let!(:thread) { Thread.new { subject.run } }
subject { described_class.new(stdin, stdout) }
# Removing this before hook causes the examples to fail intermittently
before { sleep 0.01 }
after { thread.kill if thread.alive? }
it "prints a message on how to end the session" do
expect(stdout.string).to match(/end the session/)
end
it "prints a prompt for user input" do
expect(stdout.string).to match(/\$ /)
end
it "echoes input" do
stdin.puts("foo")
stdin.rewind
expect(stdout.string).to match(/foo/)
end
end
Instead of letting :stdout be a StringIO, you could back it by a Queue. Then when you try to read from the queue, your tests will just wait until the REPL pushes something into the queue (aka. writes to stdout).
require 'thread'
class QueueIO
def initialize
#queue = Queue.new
end
def write(str)
#queue.push(str)
end
def puts(str)
write(str + "\n")
end
def read
#queue.pop
end
end
let(:stdout) { QueueIO.new }
I just wrote this up without trying it out, and it may not be robust enough for your needs, but it gets the point across. If you use a data structure to synchronize the two threads like this, then you don't need to sleep at all. Since this removes the non-determinism, you shouldn't see the intermittent failures.
I've used a running? guard for situations like this. You probably can't avoid the sleep entirely, but you can avoid unnecessary sleeps.
First, add a running? method to your REPL class.
class REPL
...
def running?
!!#running
end
def run
#running=true
loop do
...
if input == 'exit
#running = false
break
end
...
end
end
end
Then, in your specs, sleep until the REPL is running:
describe REPL do
...
before { sleep 0.01 until REPL.running? }
...
end
I'm trying to create a simple multithreaded program with jRuby. It needs to start and stop threads based on a specified amount of time e.g. run for five seconds then stop. I'm pretty new to this sort of stuff, so it's probably pretty basic but I can't get it to work.
The relevant code looks like this:
require 'java'
require 'timeout'
require './lib/t1.rb'
require './lib/t2.rb'
class Threads
[...]
def manage_threads
thread2 = T2.new
# Wait for 5 seconds before the thread starts running..
thread2.run(wait_time = 5)
Timeout::timeout(10) do
thread1 = T1.new {}
end
end
class T1 < Thread
def initialize
while super.status != "sleep"
puts "Thread 1"
sleep(1)
end
end
end
class T2
include java.lang.Runnable
def run wait_time
thread = Thread.new do
sleep(wait_time)
loop do
puts "Thread 2"
sleep(1)
end
end
end
def stop_thread(after_run_time)
sleep(after_run_time)
end
end
I have already tried a couple if things, for example:
# Used timeout
Timeout::timeout(10) do
thread1 = T1.new {}
end
# This kinda works, except that it terminates the program and therefore isn't the behavior
# I want.
Does anyone have a suggestion on how to 1. start a thread, run it for a while. 2. Start a new thread, run both thread in parallel. 2. Stop thread 1 but keep running thread 2. Any tips/suggestions would be appreciated.
I think I solved it.
This did the trick:
def run wait_time
thread = Thread.new do
sleep(wait_time)
second_counter = 0
loop do
puts "Thread 2"
second_counter += 1
if second_counter == 15
sleep
end
sleep(1)
end
end
end