I have the following code to block until all threads have finished (Gist):
ThreadsWait.all_waits(*threads)
What's the simplest way to set a timeout here, ie kill any threads if they are still running after e.g. 3 seconds?
Thread#join accepts an argument after which it will time out. Try this, for example:
5.times.map do |i|
Thread.new do
1_000_000_000.times { |i| i } # takes more than a second
puts "Finished" # will never print
end
end.each { |t| t.join(1) } # times out after a second
p 'stuff I want to execute after finishing the threads' # will print
If you have some things you want to execute before joining, you can do:
5.times.map do |i|
Thread.new do
1_000_000_000.times { |i| i } # takes more than a second
puts "Finished" # will never print
end
end.each do |thread|
puts 'Stuff I want to do before join' # Will print, multiple times
thread.join(1)
end
Related
I have the following:
text_counter = 0
MAXTEXT_COUNTER = 10
puts "hello, this will start"
loop do
puts "hello"
text_counter += 1
sleep(2)
if text_counter >= MAXTEXT_COUNTER
break
end
end
sleep(7200)
print "ended test"
Once the break has happened, how can I get it to start again from the top?
I'm now thinking I could nest this loop in an until loop with the condition of text_counter == 1000. This would break, then sleep for 2 hours, then start again until it hits 1000.
It looks like you need a loop within a loop where you repeat one N times, the other M times:
MAXTEXT_COUNTER = 10
puts "hello, this will start"
loop do
MAXTEXT_COUNTER.times do
puts "hello"
sleep(2)
end
print "ended test"
sleep(7200)
end
The outer loop is perpetual. The inner one runs a certain number of times and stops using the times method.
You're looking for next
It functions similarly to break, but returns control back to the top of the loop. It's great for creating flat control flow.
For example
0.upto(100) do |i|
if i % 7 == 0
puts "#{i} is a multiple of 7"
next
end
puts i
end
There is a retry keyword which repeats the loop from top, just what you've asked.
Or you can wrap your loop into a method and continuously call that method.
I'm looping through a lot of items and I want to periodically interrupt the loop to save and continue at a later time like this:
begin
big_list.each do |i|
# sensitive stuff
sensitive_method(i)
# other sensitive stuff
end
rescue Interrupt
# finish the current iteration
# then do something else (save)
# don't raise (print done)
end
By sensitive I mean that, if Interrupt is raised in the middle of an iteration, data will be corrupted so I need to guarantee that the iteration finishes before exiting.
Also, if another exception is raised, it should still finish the loop but raise it afterwards
EDIT:
Using the answer by mudasobwa in a test scenario:
while true
result = begin
puts "start"
sleep 1
puts "halfway"
sleep 1
puts "done\n\n"
nil
rescue Exception => e
e
end
case result
when Interrupt
puts "STOPPED"
break
when Exception then raise result
end
end
I get:
start
halfway
done
start
^C: /...
STOPPED
which is my exact problem, I need it to finish the loop (sleep, print halfway, sleep, print done) and only then break out (wrapping the puts, sleep... in a method does not help)
TL;DR: There is no way to continue the execution of the method from inside the middle of it.
big_list.each do |i|
# sensitive stuff
result = begin
sensitive_method(i)
nil
rescue Exception => e
e
end
# other sensitive stuff
case result
when Interrupt
puts "done"
break "done"
when Exception then raise result
end
end
Sidenote: you probably don’t want to rescue the topmost Exception, but some subclass that makes sense to rescue.
To make it possible to finish the chunk of operations:
operations = [
-> { puts "start" },
-> { sleep 1 },
-> { puts "halfway" },
-> { sleep 1 },
-> { puts "done\n\n" }
]
def safe_chunk(operations, index = 0)
result = operations[index..-1].each_with_index(index) do |op, idx|
begin
op.()
rescue Exception => e
safe_chunk(operations, idx) # or idx + 1
break e
end
end
result.is_a?(Array) ? nil : result
end
The Interrupt exception is raised in the main thread. If you use a worker thread to process the list it will never be interrupted. You will need a way to tell the worker thread to terminate though. Rescuing Interrupt in the main thread and setting a flag that's checked by the child can accomplish this.
BigList = (1..100)
def sensitive_method(item)
puts "start #{item}"
sleep 1
puts "halfway #{item}"
sleep 1
puts "done #{item}"
puts
end
#done = false
thread = Thread.new do
begin
BigList.each do |item|
break if #done
sensitive_method item
end
end
end
begin
thread.join
rescue Interrupt
#done = true
thread.join
end
The keyword ensure, used in rescue clauses, is available for situation such as this one, where code must be executed after an exception occurs.
[-1, 0, 1].each do |i|
begin
puts "i=#{i} before exception"
# <additional code>
n = 1/i
rescue ZeroDivisionError => e
puts "Exception: #{e}"
exit
ensure
puts "Just executed 1/#{i}"
# <additional code>
end
end
i=-1 before exception
Just executed 1/-1
i=0 before exception
Exception: divided by 0
Just executed 1/0
Notice that begin/rescue/ensure/end must be inside the loop and that the code after ensure is executed for each i regardless of whether a zero-divide exception occurs.
I have a Worker and Job example, where each Job has an expensive/slow perform method.
If I have 10 Jobs in my #job_table I'd like to work them off in batches of 5, each within their own process.
After the 5 processes (one batch) have exited I'm trying to remove those Jobs from the #job_table with delete_at.
I'm observing something unexpected in my implementation (see code below) though:
jobs:
[#<Job:0x007fd2230082a8 #id=0>,
#<Job:0x007fd223008280 #id=1>,
#<Job:0x007fd223008258 #id=2>,
#<Job:0x007fd223008208 #id=3>,
#<Job:0x007fd2230081e0 #id=4>,
#<Job:0x007fd2230081b8 #id=5>,
#<Job:0x007fd223008190 #id=6>,
#<Job:0x007fd223008168 #id=7>,
#<Job:0x007fd223008140 #id=8>,
#<Job:0x007fd223008118 #id=9>]
This is the #job_table before the first batch is run. I see that Jobs 0-4 have run and exited successfully (omitted output here).
So I'm calling remove_batch_1 and would expect jobs 0-4 to be removed from the #job_table, but this is what I'm observing instead:
jobs:
[#<Job:0x007fd223008280 #id=1>,
#<Job:0x007fd223008208 #id=3>,
#<Job:0x007fd2230081b8 #id=5>,
#<Job:0x007fd223008168 #id=7>,
#<Job:0x007fd223008118 #id=9>]
I've logged the i parameter in the method and it returns 0-4. But it looks like delete_at is removing other jobs (0,2,4,6,8).
I also wrote another method for removing a batch remove_batch_0 which uses slice! and behaves as expected.
BATCH_SIZE = 5 || ENV['BATCH_SIZE']
class Job
def initialize(id)
#id = id
end
def perform
puts "Job #{#id}> Start!"
sleep 1
puts "Job #{#id}> End!"
end
end
class Worker
def initialize
#job_table = []
fill_job_table
work_job_table
end
def fill_job_table
10.times do |i|
#job_table << Job.new(i)
end
end
def work_job_table
until #job_table.empty?
puts "jobs: "
pp #job_table
work_batch
Process.waitall
remove_batch_1
end
end
def work_batch
i = 0
while (i < #job_table.length && i < BATCH_SIZE)
fork { #job_table[i].perform }
i += 1
end
end
def remove_batch_1
i = 0
while (i < #job_table.length && i < BATCH_SIZE)
#job_table.delete_at(i)
i += 1
end
end
def remove_batch_0
#job_table.slice!(0..BATCH_SIZE-1)
end
end
Worker.new
You use delete_at in a while loop. Let's see what happens:
Image you have an array [0,1,2,3,4,5] and you call:
(1..3).each { |i| array.deleted_at(i) }
In the first iteration you will delete the first element from the array, the array will look like this after this step: [1,2,3,4,5] In the next iteration you will delete the second element, what leads to [1,3,4,5]. Then you delete the third: [1,3,5]
You might want to use Array#shift instead:
def remove_batch_1
#job_table.shift(BATCH_SIZE)
end
For this answer I wrote code like:
def show_wait_spinner
dirty = false
spinner = Thread.new{
loop{
print "*"
dirty = true
sleep 0.1
print "\b"
dirty = false
}
}
yield
spinner.kill
print "\b" if dirty
end
print "A"
show_wait_spinner{ sleep rand }
puts "B"
The goal is to ensure that the final output was "AB"—to print a final "\b" if it was not already printed by the thread. That code seems messy to me in Ruby where begin/rescue/ensure exists. So I tried some other implementations of show_wait_spinner; all of them fail to ensure that "AB" is always the output, and never "A*B" or "AB*".
Is there a cleaner, more Ruby-esque way to implement this logic?
Stop at end of loop via Mutex
def show_wait_spinner
stop = false
stopm = Mutex.new
spinner = Thread.new{
loop{
print "*"
sleep 0.1
print "\b"
stopm.synchronize{ break if stop }
}
}
yield
stopm.synchronize{ stop = true }
STDOUT.flush
end
…but my logic must be off, since this always results in "A*B".
Stop at end of loop via Thread-local variable
This second attempt results in sometimes "A*B" being printed, sometimes "AB":
def show_wait_spinner
stop = false
spinner = Thread.new{
Thread.current[:stop] = false
loop{
print "*"
sleep 0.1
print "\b"
stopm.synchronize{ break if Thread.current[:stop] }
}
}
yield
spinner[:stop] = true
STDOUT.flush
end
Kill and Ensure the Thread
def show_wait_spinner
spinner = Thread.new{
dirty = false
begin
loop{
print "*"
dirty = true
sleep 0.1
print "\b"
dirty = false
}
ensure
print "\b" if dirty
end
}
yield
spinner.kill
STDOUT.flush
end
Raise and Rescue the Thread
def show_wait_spinner
spinner = Thread.new{
dirty = false
begin
loop{
print "*"
dirty = true
sleep 0.1
print "\b"
dirty = false
}
rescue
puts "YAY"
print "\b" if dirty
end
}
yield
spinner.raise
STDOUT.flush
end
Instead of killing your thread, why don't you flip a variable that causes it to stop at a pre-defined point? If you let it cycle through and exit at the end of the loop you won't have nearly as much trouble.
For instance:
def show_wait_spinner
running = true
spinner = Thread.new do
while (running) do
print "*"
sleep 0.1
print "\b"
end
end
yield
running = false
spinner.join
end
print "A"
show_wait_spinner{ sleep rand }
puts "B"
When you call Thread#kill you have no idea where the thread is, and the thread isn't given an opportunity to clean up what it's doing. You can always kill the thread if your polite "stop running" request isn't respected.
I prefer your synchronized stop condition approach but you have a couple bugs:
after you set the stop variable, you almost immediately end the
program, so the thing that stops the thread is program exit, not the
conditional test in the loop; use Thread#join to wait for the thread
to exit normally and you'll get the consistent output you want.
break in the synchronize block breaks out of the block, not the
loop.
def show_wait_spinner
stop = false
stopm = Mutex.new
spinner = Thread.new{
loop{
print "*"
sleep 0.1
print "\b"
break if stopm.synchronize{ stop }
}
}
yield
stopm.synchronize{ stop = true }
spinner.join
STDOUT.flush
end
print "A"
show_wait_spinner{ sleep rand }
puts "B"
I would avoid any solution involving Thread#raise and Thread#kill as their behavior can never be predictable and correct, see Charles Nutter's rant about the brokenness of these methods.
The Mutex#synchronize is only necessary for this simple boolean flipping if you really care a lot about the precise timing around the race condition when the parent thread sets the var, which in this example isn't likely, so you could avoid the overhead and just set and read stop normally.
In your mutex example, you need to wait for the Thread to finish before exiting the method. Currently you set stop to true, then exit the method, print B and end your script before the spinner thread is able to wake up and print the last backspace to delete the * character.
Also, the break in stopm.synchronize{ break if stop } only exits the inner block, not the loop, so you need to use catch/throw or something.
However, you don't need the mutex. This works for me in 1.9.3:
def show_wait_spinner
exit = false
spinner = Thread.new{
loop{
print "*"
sleep 0.1
print "\b"
break if exit
}
}
yield
exit = true
spinner.join
end
Adding $stdout.sync = true at the top makes it work in 1.8.7.
Why does this code work (I see the output 1 2 3):
for i in 1..3
Thread.new{
puts i
}
end
However, the following code does not produce the same output (I do not see the output 1 2 3)?
for i in 1..3
Thread.new{
sleep(5)
puts i
}
end
When you hit the end of the script, Ruby exits. If you add sleep 10 after the final loop, you can see the output show up. (Albeit, as 3 each time, because the binding to i reflects the value at the end of processing, and the sleep causes a thread switch back to the loop.)
You might want something like:
threads = []
for i in 1..3
threads << Thread.new {
sleep 5
puts i
}
end
threads.map {|t| t.join }
That will wait for all the threads to terminate before exiting.