I try the Sucker Punch gem to process tasks in parallel. But I found no documentation how to wait for its termination.
require 'sucker_punch'
class SuckerJob
include SuckerPunch::Job
workers 4
def perform(event)
sleep(rand(5))
puts "[#{Thread.current.object_id}] End processing event #{event}."
end
end
10.times { |i| SuckerJob.perform_async(i) }
puts "Shutting down ..."
SuckerPunch::Queue.shutdown_all
puts "Shutdown finished, status: #{SuckerPunch::Queue.stats[SuckerJob.name]}"
# [Ugly] call internal method
SuckerPunch::Queue::QUEUES.fetch_or_store(SuckerJob.name).wait_for_termination(10)
puts "Wait finished, status: #{SuckerPunch::Queue.stats[SuckerJob.name]}"
It seems, that SuckerPunch::Queue.shutdown_all() returns before all tasks are completed.
Shutting down ...
[17487240] End processing event 1.
[17488760] End processing event 0.
[17487240] End processing event 4.
[17488760] End processing event 5.
[17486120] End processing event 2.
[17484940] End processing event 3.
[17487240] End processing event 6.
Shutdown finished, status: {"workers"=>{"total"=>3, "busy"=>3, "idle"=>0}, "jobs"=>{"processed"=>7, "failed"=>0, "enqueued"=>0}}
[17484940] End processing event 9.
[17488760] End processing event 7.
[17486120] End processing event 8.
Wait finished, status: {"workers"=>{"total"=>0, "busy"=>0, "idle"=>0}, "jobs"=>{"processed"=>10, "failed"=>0, "enqueued"=>0}}
How can I wait until all tasks are completed?
You can check status
or check stats you can do
all_stats = SuckerPunch::Queue.stats
then
stats = all_stats[SuckerJob.to_s]
then stats you get now you can see
stats["jobs"]["processed"]
you can check like below
stats["jobs"]["processed"] > 0
stats["jobs"]["failed"] == 0
stats["jobs"]["enqueued"] == 0
I use this:
def wait_for_jobs(job_name:, count:, max_seconds: 100)
Rails.logger.info "Waiting up to #{max_seconds} seconds for #{count} jobs to run"
wait_time = 0
while wait_time < max_seconds
stats = SuckerPunch::Queue.stats[job_name]
processed = stats['jobs']['processed']
break unless processed < count
sleep(1)
wait_time += 1
end
raise StandardError, "Timeout while waiting for #{count} jobs of #{job_name} to have run!" unless wait_time < max_seconds
Rails.logger.info "#{count} jobs took #{wait_time} seconds to run"
end
Related
Ruby 3 introduced Fiber.schedule to dispatch async tasks concurrently.
Similar to what's being asked in this question (which is about threaded concurrency) I would like a way to start multiple concurrent tasks on the fiber scheduler and once they have all been scheduled wait for their combined result, sort of equivalent to Promise.all in JavaScript.
I can come up with this naive way:
require 'async'
def io_work(t)
sleep t
:ok
end
Async do
results = []
[0.1, 0.3, 'cow'].each_with_index do |t, i|
n = i + 1
Fiber.schedule do
puts "Starting fiber #{n}\n"
result = io_work t
puts "Done working for #{t} seconds in fiber #{n}"
results << [n, result]
rescue
puts "Execution failed in fiber #{n}"
results << [n, :error]
end
end
# await combined results
sleep 0.1 until results.size >= 3
puts "Results: #{results}"
end
Is there a simpler construct that will do the same?
Since Async tasks are already scheduled I am not sure you need all of that.
If you just want to wait for all the items to finish you can use an Async::Barrier
Example:
require 'async'
require 'async/barrier'
def io_work(t)
sleep t
:ok
end
Async do
barrier = Async::Barrier.new
results = []
[1, 0.3, 'cow'].each.with_index(1) do |data, idx|
barrier.async do
results << begin
puts "Starting task #{idx}\n"
result = io_work data
puts "Done working for #{data} seconds in task #{idx}"
[idx,result]
rescue
puts "Execution failed in task #{idx}"
[idx, :error]
end
end
end
barrier.wait
puts "Results: #{results}"
end
Based on the sleep values this will output
Starting task 1
Starting task 2
Starting task 3
Execution failed in task 3
Done working for 0.3 seconds in task 2
Done working for 1 seconds in task 1
Results: [[3, :error], [2, :ok], [1, :ok]]
The barrier.wait will wait until all the asynchronous tasks are complete, without it the output would look like
Starting fiber 1
Starting fiber 2
Starting fiber 3
Execution failed in fiber 3
Results: [[3, :error]]
Done working for 0.3 seconds in fiber 2
Done working for 1 seconds in fiber 1
I wasn't too happy with the ergonomics of the solution, so I made the gem fiber-collector to address it.
Disclaimer: I'm describing a library of which I am the author
Example usage in the scenario from the question:
require 'async'
require 'fiber/collector'
def io_work(t)
sleep t
:ok
end
Async do
Fiber::Collector.schedule { io_work(1) }.and { io_work(0.3) }.all
end.wait
# => [:ok, :ok]
Async do
Fiber::Collector.schedule { io_work(1) }.and { io_work(0.3) }.and { io_work('cow') }.all
end.wait
# => raises error
I would like to continously check the table in the DB for the commands to run.
Some commands might take 4minutes to complete, some 10 seconds.
Hence I would like to run them in threads. So every record creates new thread, and after thread is created, record gets removed.
Because the DB lookup + Thread creation will run in an endless loop, how do I get the 'response' from the Thread (thread will issue shell command and get response code which I would like to read) ?
I thought about creating two Threads with endless loop each:
- first for DB lookups + creating new threads
- second for ...somehow reading the threads results and acting upon each response
Or maybe I should use fork, or os spawn a new process?
You can have each thread push its results onto a Queue, then your main thread can read from the Queue. Reading from a Queue is a blocking operation by default, so if there are no results, your code will block and wait on the read.
http://ruby-doc.org/stdlib-2.0.0/libdoc/thread/rdoc/Queue.html
Here is an example:
require 'thread'
jobs = Queue.new
results = Queue.new
thread_pool = []
pool_size = 5
(1..pool_size).each do |i|
thread_pool << Thread.new do
loop do
job = jobs.shift #blocks waiting for a task
break if job == "!NO-MORE-JOBS!"
#Otherwise, do job...
puts "#{i}...."
sleep rand(1..5) #Simulate the time it takes to do a job
results << "thread#{i} finished #{job}" #Push some result from the job onto the Queue
#Go back and get another task from the Queue
end
end
end
#All threads are now blocking waiting for a job...
puts 'db_stuff'
db_stuff = [
'job1',
'job2',
'job3',
'job4',
'job5',
'job6',
'job7',
]
db_stuff.each do |job|
jobs << job
end
#Threads are now attacking the Queue like hungry dogs.
pool_size.times do
jobs << "!NO-MORE-JOBS!"
end
result_count = 0
loop do
result = results.shift
puts "result: #{result}"
result_count +=1
break if result_count == 7
end
I have the following ruby script which kicks off 12 processes which smoke test my application. Some of these can be long running of up to 8 minutes.
pids = []
pids = uris.map do |uri|
print_start(uri)
command = get_http_tests_command("Smoke")
update_http_tests_config( uri )
pid = Process.spawn command
Process.detach pid
pid
end
print( "smoke tests in progress => #{pids} => #{uris}" )
statuses = pids.map do |pid|
puts "waiting for #{pid}"
Process.wait( pid )
$?
end
print("smoke tests finished")
When the process finishes regardless of success I intermittently get an error on the Process.wait( pid ) line. The error is as follows
#<Errno::EINVAL: Invalid argument>
>> SmokeTest.rb:169:in "wait"
>> SmokeTest.rb:169:in "block in run_single"
>> SmokeTest.rb:167:in "map"
>> SmokeTest.rb:167:in "run_single"
I'm not sure what's going on here, any help would be much appreciated. It seems to fail intermittently about 1 in 3 times under 300 seconds. It always fails if it goes over 300 seconds.
My guess is that this exception is thrown if you try to wait on a process which is already dead...
If that is the problem I could suggest using threads to wait for all processes concurrently:
statuses = []
threads = pids.map do |pid|
Thread.new do
puts "waiting for #{pid}"
_, status = Process.wait2( pid )
puts "#{pid} ended with status #{status}"
statuses << status
end
end
threads.each(&:join)
print("smoke tests finished")
I have a Ruby script fetching HTML pages over HTTP using threads:
require "thread"
require "net/http"
q = Queue.new
q << "http://google.com/"
q << "http://rubygems.org/"
q << "http://twitter.com/"
t = Thread.new do
loop do
html = Net::HTTP.get(URI(q.pop))
p html.length
end
end
10.times do
puts t.status
sleep 0.3
end
I'm trying to determine the state of the thread while it is fetching the content from given sources. Here is the output I got:
run
219
sleep
sleep
7255
sleep
sleep
sleep
sleep
sleep
sleep
65446
sleep
The thread is in "sleep" state almost all the time though it's actually working. I understand it's waiting for the HTTP class to retrieve the content. The last "sleep" is different: the thread tried to pop the value from the queue which is empty and switched to "sleep" state until there is something new in the queue.
I want to be able to check what's going on in the thread: Is it working on HTTP or simply waiting for new job to appear?
What is the right way to do it?
The sleep state appears to cover both I/O wait and being blocked in synchronization, so you won't be able to use the thread state to know whether you're processing or waiting. Instead, you could use thread local storage for the thread to communicate that. Use Thread#[]= to store a value, and Thread#[] to get it back.
require "thread"
require "net/http"
q = Queue.new
q << "http://google.com/"
q << "http://rubygems.org/"
q << "http://twitter.com/"
t = Thread.new do
loop do
Thread.current[:status] = 'waiting'
request = q.pop
Thread.current[:status] = 'fetching'
html = Net::HTTP.get(URI(request))
Thread.current[:status] = 'processing'
# Take half a second to process it.
Time.new.tap { |start_time| while Time.now - start_time < 0.5 ; end }
p html.length
end
end
10.times do
puts t[:status]
sleep 0.3
end
I've added a short loop to eat up time. Without it, it's unlikely you'd see "processing" in the output:
219
processing
fetching
processing
7255
fetching
fetching
fetching
62471
processing
waiting
waiting
I was trying to speed up multiple FTP downloads by using threaded FTP connections. My problem is that I always have threads hang. I am looking for a clean way of either telling FTP it needs to retry the ftp transaction, or at least knowing when the FTP connection is hanging.
In the code below I am threading 5/6 separate FTP connections where each thread has a list of files it is expected to download. When the script completes, a few of the threads hang and can not be joined. I am using the variable #last_updated to represent the last successful download time. If the current time + 20 seconds exceeds #last_updated, kill the remaining threads. Is there a better way?
threads = []
max_thread_pool = 5
running_threads = 0
Thread.abort_on_exception = true
existing_file_count = 0
files_downloaded = 0
errors = []
missing_on_the_server = []
#last_updated = Time.now
if ids.length > 0
ids.each_slice(ids.length / max_thread_pool) do |id_set|
threads << Thread.new(id_set) do |t_id_set|
running_threads += 1
thread_num = running_threads
thread_num.freeze
puts "making thread # #{thread_num}"
begin
ftp = Net::FTP.open(#remote_site)
ftp.login(#remote_user, #remote_password)
ftp.binary = true
#ftp.debug_mode = true
ftp.passive = false
rescue
raise "Could not establish FTP connection"
end
t_id_set.each do |id|
#last_updated = Time.now
rmls_path = "/_Photos/0#{id[0,2]}00000/#{id[2,1]}0000/#{id[3,1]}000/#{id}-1.jpg"
local_path = "#{#photos_path}/01/#{id}-1.jpg"
progress += 1
unless File.exist?(local_path)
begin
ftp.getbinaryfile(rmls_path, local_path)
puts "ftp reponse: #{ftp.last_response}"
# find the percentage of progress just for fun
files_downloaded += 1
p = sprintf("%.2f", ((progress.to_f / total) * 100))
puts "Thread # #{thread_num} > %#{p} > #{progress}/#{total} > Got file: #{local_path}"
rescue
errors << "#{thread_num} unable to get file > ftp response: #{ftp.last_response}"
puts errors.last
if ftp.last_response_code.to_i == 550
# Add the missing file to the missing list
missing_on_the_server << errors.last.match(/\d{5,}-\d{1,2}\.jpg/)[0]
end
end
else
puts "found file: #{local_path}"
existing_file_count += 1
end
end
puts "closing FTP connection #{thread_num}"
ftp.close
end # close thread
end
end
# If #last_updated has not been updated on the server in over 20 seconds, wait 3 seconds and check again
while Time.now < #last_updated + 20 do
sleep 3
end
# threads are hanging so joining the threads does not work.
threads.each { |t| t.kill }
The trick for me that worked was to use ruby's Timeout.timeout to ensure the FTP connection was not hanging.
begin
Timeout.timeout(10) do
ftp.getbinaryfile(rmls_path, local_path)
end
# ...
rescue Timeout::Error
errors << "#{thread_num}> File download timed out for: #{rmls_path}"
puts errors.last
rescue
errors << "unable to get file > ftp reponse: #{ftp.last_response}"
# ...
end
Hanging FTP downloads were causing my threads to appear to hang. Now that the threads are no longer hanging, I can use the more proper way of dealing with threads:
threads.each { |t| t.join }
rather than the ugly:
# If #last_updated has not been updated on the server in over 20 seconds, wait 3 seconds and check again
while Time.now < #last_updated + 20 do
sleep 3
end
# threads are hanging so joining the threads does not work.
threads.each { |t| t.kill }