My records in a database are already categorized into buckets (0, 1, 2, 3). Rather than applying a function to each record serially, I'd like to open four threads and apply the function to the record in that thread's bucket.
If I run this:
i = 4
i.times do |n|
Thread.new {
puts "opening thread for #{n} degree"
myFunction(n)
}.join
end
I get:
opening thread for 0 degree
opening thread for 1 degree
opening thread for 2 degree
opening thread for 3 degree
with waiting in between each one. It's still going serially.
If I do the same as above, but without join:
i = 4
i.times do |n|
Thread.new {
puts "opening thread for #{n} degree"
myFunction(n)
}
end
I get:
opening thread for 3 degree
opening thread for 2 degreeopening thread for 0 degree
opening thread for 4 degree
which is closer to what I want; it seems they all run simultaneously.
It makes me nervous when my puts statements are printed haphazardly like this. If I don't have the join there, doesn't that mean that whichever thread terminates first, the rest of the script moves on and the other threads terminate early? What should I do?
What SHOULD I be doing here?
You should be joining your threads. Otherwise when the main thread (your script) exits, it takes all unfinished threads with it. The reason why execution is serial in your first case is that you wait for a thread to finish right after you start it (and before you start the next one). First create all threads, then wait on them.
i = 4
threads = i.times.map do |n|
Thread.new {
puts "opening thread for #{n} degree"
myFunction(n)
}
end
threads.each(&:join)
# or
require 'thwait'
ThreadsWait.all_waits(*threads)
You will see further improvements in threading performance if you run the code on JRuby or Rubinius, as their threads are not crippled in any way by some global lock.
Related
Below code
Timeout.timeout(2) do
i = 0
while(true)
i = i + 1
p "test #{i}"
end
end
does not timeout in 2 secs. whereas below similar code timeout in 2 seconds
Timeout.timeout(2) do
i = 0
while(true)
i = i + 1
# p "test #{i}"
end
end
What is the underlying difference? Please help.
I don't know exactly what's going on here and I suspect somebody who understands the underlying C code would be the one to give a complete answer. I have an inkling. The Matz Ruby Interpreter (MRI) has a global thread lock which means only one thread can actually run at any given time. The way threading works is when one thread is waiting on a resource it sleeps and this gives another thread opportunity to run.
Timeout creates a second thread that will sleep for 2 seconds then raise an exception on the current thread enforcing the timeout. We are guaranteed this thread will not run before 2 seconds but not guaranteed exactly when it will run after 2 seconds but usually a few milliseconds or so with some exceptions.
The function p is unique in that it writes directly to std.out. This is where a C programmer may be helpful but it appears to me that its starving the other thread of resources possibly because to throw an exception the second thread needs to own std.out.
p and pp both cause this problem whereas puts does not.
In support of the resource starvation theory the following code works
Timeout.timeout(2) do
i = 0
while(true)
i = i + 1
p "testing timeout #{i}"
sleep 0.001
end
end
I want to start a new thread every x seconds in Ruby, but wasn´t able to figure it out.
Usually the thread execution takes longer then the x seconds, all I managed was something that starts a new thread after the previous one finished.
So I want to start a new thread after x seconds, now matter how many previous threads are still running.
Any ideas?
threads = [] # array of Thread in case you need to do something with all the Threads
# like threads.each { |t| t.join }
1.upto(5) do |n|
threads << Thread.new { puts "Thread #{n}!" }
sleep 1 # or more seconds if need it
end
I need to create 3 threads.
Each thread will print on the screen a collor and sleep for x seconds.
Thread A will print red; Thread B will print yellow; Thread C will print green;
All threads must wait until its their turn to print.
The first thread to print must be Red, after printing, Red will tell Yellow that's its turn to print and so on.
The threads must be able to print multiple times (user specific)
I'm stuck because calling #firstFlag.signal outside a Thread isn't working and the 3 threads aren't working on the right order
How do I make the red Thread go first?
my code so far:
#lock = Mutex.new
#firstFlag = ConditionVariable.new
#secondFlag = ConditionVariable.new
#thirdFlag = ConditionVariable.new
print "Tell me n's vallue:"
#n = gets.to_i
#threads = Array.new
#threads << Thread.new() {
t = Random.rand(1..3)
n = 0
#lock.synchronize {
for i in 0...#n do
#firstFlag.wait(#lock, t)
puts "red : #{t}s"
sleep(t)
#secondFlag.signal
end
}
}
#threads << Thread.new() {
t = Random.rand(1..3)
n = 0
#lock.synchronize {
for i in 0...#n do
#secondFlag.wait(#lock, t)
puts "yellow : #{t}s"
sleep(t)
#thirdFlag.signal
end
}
}
#threads << Thread.new() {
t = Random.rand(1..3)
n = 0
#lock.synchronize {
for i in 0...#n do
#thirdFlag.wait(#lock, t)
puts "green : #{t}s"
sleep(t)
#firstFlag.signal
end
}
}
#threads.each {|t| t.join}
#firstFlag.signal
There are three bugs in your code:
First bug
Your wait calls use a timeout. This means your threads will become de-synchronized from your intended sequence, because the timeout will let each thread slip past your intended wait point.
Solution: change all your wait calls to NOT use a timeout:
#xxxxFlag.wait(#lock)
Second bug
You put your sequence trigger AFTER your Thread.join call in the end. Your join call will never return, and hence the last statement in your code will never be executed, and your thread sequence will never start.
Solution: change the order to signal the sequence start first, and then join the threads:
#firstFlag.signal
#threads.each {|t| t.join}
Third bug
The problem with a wait/signal construction is that it does not buffer the signals.
Therefore you have to ensure all threads are in their wait state before calling signal, otherwise you may encounter a race condition where a thread calls signal before another thread has called wait.
Solution: This a bit harder to solve, although it is possible to solve with Queue. But I propose a complete rethinking of your code instead. See below for the full solution.
Better solution
I think you need to rethink the whole construction, and instead of condition variables just use Queue for everything. Now the code becomes much less brittle, and because Queue itself is thread safe, you do not need any critical sections any more.
The advantage of Queue is that you can use it like a wait/signal construction, but it buffers the signals, which makes everything much simpler in this case.
Now we can rewrite the code:
redq = Queue.new
yellowq = Queue.new
greenq = Queue.new
Then each thread becomes like this:
#threads << Thread.new() {
t = Random.rand(1..3)
n = 0
for i in 0...#n do
redq.pop
puts "red : #{t}s"
sleep(t)
yellowq.push(1)
end
}
And finally to kick off the whole sequence:
redq.push(1)
#threads.each { |t| t.join }
I'd redesign this slightly. Think of your ConditionVariables as flags that a thread uses to say it's done for now, and name them accordingly:
#lock = Mutex.new
#thread_a_done = ConditionVariable.new
#thread_b_done = ConditionVariable.new
#thread_c_done = ConditionVariable.new
Now, thread A signals it's done by doing #thread_a_done.signal, and thread B can wait for that signal, etc. Thread A of course needs to wait until thread C is done, so we get this kind of structure:
#threads << Thread.new() {
t = Random.rand(1..3)
#lock.synchronize {
for i in 0...#n do
#thread_c_done.wait(#lock)
puts "A: red : #{t}s"
sleep(t)
#thread_a_done.signal
end
}
}
A problem here is that you need to make sure that thread A in the first iteration doesn't wait for a flag signal. After all, it's to go first, so it shouldn't wait for anyone else. So modify it to:
#thread_c_done.wait(#lock) unless i == 0
Finally, once you have created your threads, kick them all off by invoking run, then join on each thread (so that your program doesn't exit before the last thread is done):
#threads.each(&:run)
#threads.each(&:join)
Oh btw I'd get rid of the timeouts in your wait as well. You have a hard requirement that they go in order. If you make the signal wait time out you screw that up - threads might still "jump the queue" so to speak.
EDIT as #casper remarked below, this still has a potential race condition: Thread A could call signal before thread B is waiting to receive it, in which case thread B will miss it and just wait indefinitely. A possible way to fix this is to use some form of a CountDownLatch - a shared object that all threads can wait on, which gets released as soon as all threads have signalled that they're ready. The ruby-concurrency gem has an implementation of this, and in fact might have other interesting things to use for more elegant multi-threaded programming.
Sticking with pure ruby though, you could possibly fix this by adding a second Mutex that guards shared access to a boolean flag to indicate the thread is ready.
Ok, thank you guys that answered. I've found a solution:
I've created a fourth thread. Because I found out that calling "#firstFlag.signal" outside a thread doesn't work, because ruby has a "main thread" that sleeps when you "run" other threads.
So, "#firstFlag.signal" calling must be inside a thread so it can be on the same level of the CV.wait
I solved the issue using this:
#threads << Thread.new {
sleep 1
#firstFlag.signal
}
This fourth thread will wait for 1 sec before sending the first signal to red. This only sec seems to be enough for the others thread reach the wait point.
And, I've removed the timeout, as you sugested.
//Edit//
I realized I don't need a fourth Thread, I could just make thread C do the first signal.
I made thread C sleep for 1 sec to wait the other two threads enter in wait state, then it signals red to start and goes to wait too
#threads << Thread.new() {
sleep 1
#redFlag.signal
t = Random.rand(1..3)
n = 0
#lock.synchronize {
for i in 0...#n do
#greenFlag.wait(#lock)
puts "verde : #{t}s"
sleep(t)
#redFlag.signal
n += 1
end
}
}
I am calling the same Ruby function with a number of threads (for example 10 threads). Each thread passes different argument to function.
Example:
def test thread_no
puts "In thread no." + thread_no.to_s
end
num_threads = 6
threads=[]
for thread_no in 1..num_threads
puts "Creating thread no. "+thread_no.to_s
threads << Thread.new{test(thread_no)}
end
threads.each { |thr| thr.join }
Output:
Creating thread no. 1
Creating thread no. 2
Creating thread no. 3
Creating thread no. 4
In thread no.4
Creating thread no. 5
Creating thread no. 6
In thread no.6
In thread no.6
In thread no.6
In thread no.6
In thread no.6
Of course I want to get output: In thread no. 1 (2,3,4,5,6) Can I somehow achieve that this would work?
The problem is the for-loop. In Ruby, it reuses a single variable.
So all blocks of the thread bodies access the same variable. An this variable is 6 at the end of the loop. The thread itself may start only after the loop has ended.
You can solve this by using the each-loops. They are more cleanly implemented, each loop variable exists on its own.
(1..num_threads).each do | thread_no |
puts "Creating thread no. "+thread_no.to_s
threads << Thread.new{test(thread_no)}
end
Unfortunately, for loops in ruby are a source of surprises. So it is best to always use each loops.
Addition:
You an also give Thread.new one or several parameters, and these parameters get passed into the thread body block. This way you can make sure that the block uses no vars outside it's own scope, so it also works with for-loops.
threads << Thread.new(thread_no){|n| test(n) }
#Meier already mentioned the reason why for-end spits out different result than expected.
for loop is language syntax construction, it reuses the same local variable thread_no and thread_no yields 6 because your for loop ends before the last few threads start executing.
In order to get rid of such issue, you can keep a copy of the exact thread_no in an another scope - such as -
def test thread_no
puts "In thread no." + thread_no.to_s
end
num_threads = 6
threads = []
for thread_no in 1..num_threads
threads << -> (thread_no) { Thread.new { test(thread_no) } }. (thread_no)
end
threads.each { |thr| thr.join }
I'm doing what a lot of people probably need to do, processing tasks that have a variable execution time. I have the following proof of concept code:
threads = []
(1...10000).each do |n|
threads << Thread.new do
run_for = rand(10)
puts "Starting thread #{n}(#{run_for})"
time=Time.new
while 1 do
if Time.new - time >= run_for then
break
else
sleep 1
end
end
puts "Ending thread #{n}(#{run_for})"
end
finished_threads = []
while threads.size >= 10 do
threads.each do |t|
finished_threads << t unless t.alive?
end
finished_threads.each do |t|
threads.delete(t)
end
end
end
It doesn't start a new thread until one of the previous threads has dropped off. Does anyone know a better, more elegant way of doing this?
I'd suggest creating a work pool. See http://snippets.dzone.com/posts/show/3276. Then submit all of your variable length work to the pool, and call join to wait for all the threads to complete.
The work_queue gem is the easiest way to perform tasks asynchronously and concurrently in your application.
wq = WorkQueue.new 2 # Limit the maximum number of simultaneous worker threads
(1..10_000).each do
wq.enqueue_b do
# Task
end
end
wq.join # All tasks are complete after this