spawning multiple parallel tasks - ruby

Question

I have M tasks to process and N parallel processing resources available (think worker threads on Heroko, or EC2 instances), where M >> N.
I could roll my own system, but it seems likely there's already a debugged package or gem for this: what do you recommend? (Now that I think about it, I could torture Delayed::Job into doing this.)
The tasks can be written just about any language -- even a shell script will do the job. The 'mother ship' is Ruby On Rails with a PostgreSQL database. The basic idea is that when a resource is ready to process a task, it asks the mother ship for the next un-processed task in the queue and starts processing it. If the job fails, it is re-tried a few times before giving up. The results can go into flat files or be written into the PostgreSQL database.
(And, no, this is not for generating spam. I'm researching degree distribution of several large social networks.)

Answer 1

I think it is a delayed_job https://github.com/collectiveidea/delayed_job or resque https://github.com/defunkt/resque job, as you said.

Answer 2

This would be rolling your own, but but if your parallel task are not resource intensive, it is a reasonably quick solution. On the other hand, if they are resource intensive, you'll want to implement something much more robust.
You could start each thread with Process::fork (if the process is in ruby), or Process::exec, or Process::spawn (if the process is in something else). Then use Process::waitall for the sub-processes to complete.
Below, I used a Hash to hold the functions themselves as well as the PID's. This could definitely be improved on.
# define the sub-processes
sleep_2_fail = lambda { sleep 2; exit -1; }
sleep_2_pass = lambda { sleep 2; exit 0; }
sleep_1_pass = lambda { sleep 1; exit 0; }
sleep_3_fail = lambda { sleep 3; exit -1; }
# use a hash to store the lambda's and their PID's
sub_processes = Hash.new
# add the sub_processes to the hash
# key = PID
# value = lambda (can use to be re-called later on)
sub_processes.merge! ({ Process::fork { sleep_2_fail.call } => sleep_2_fail })
sub_processes.merge! ({ Process::fork { sleep_2_pass.call } => sleep_2_pass })
sub_processes.merge! ({ Process::fork { sleep_1_pass.call } => sleep_1_pass })
sub_processes.merge! ({ Process::fork { sleep_3_fail.call } => sleep_3_fail })
# starting time of the loop
start = Time.now
# use a while loop to wait at most 10 seconds or until
# the results are empty (no sub-processes)
while ((results = Process.waitall).count > 0 && Time.now - start < 10) do
results.each do |pid, status|
if status != 0
# again add the { PID => lambda } to the hash
sub_processes.merge! ( { Process::fork { sub_processes[pid].call } => sub_processes[pid] } )
end
# delete the original entry
sub_processes.delete pid
end
end
The ruby-doc on waitall is helpful.

Answer 3

It sounds like you want a job processor. Look at Gearman http://gearman.org/
Fairly language agnostic.
And here's the ruby Gem info http://gearmanhq.com/help/tutorials/ruby/getting_started/