Running multiple ruby processes (data import) - ruby

Question

I have a script that imports data from XML files in a folder ~/xml/. Currently it runs sequentially, but it's beginning to take too long as the number of import files increases.
I'd like to run multiple copies of the script in parallel but I can envisage there being problems whereby both scripts start processing the same file, how would you get around this, considering that the scripts are essentially ignorant of each other's existence?
There isn't a problem with database concurrency as each import file is for a different database.

Answer 1

You don't have anything arbitrating between the scripts, or doling out the work, and you need it.
You say the files are for different databases. How do the scripts know which database? Can't you preprocess the queued files and rename them by appending something to the name? Or, have a script that determines which data goes where and then pass the names to sub-scripts that do the loading?
I'd do the later, and would probably fork the jobs but threads can do it too. Forking has some advantages but threads are easier to debug.
You don't specify enough about your system to give you code that will slide in, but this is a general idea of what to do using threads:
require 'thread'
file_queue = Queue.new
Dir['./*'].each { |f| file_queue << f }
consumers = []
2.times do |worker|
consumers << Thread.new do
loop do
break if file_queue.empty?
data_file = file_queue.pop
puts "Worker #{ worker } reading #{ data_file }. Queue size: #{ 1 + file_queue.length }\n"
num_lines = 0
File.foreach(data_file) do |li|
num_lines += 1
end
puts "Worker #{ worker } says #{ data_file } contained #{ num_lines } lines.\n"
end
end
end
consumers.each { |c| c.join }
Which, after running, shows this in the console:
Worker 1 reading ./blank.yaml. Queue size: 28
Worker 0 reading ./build_links_to_test_files.rake. Queue size: 27
Worker 0 says ./build_links_to_test_files.rake contained 68 lines.
Worker 0 reading ./call_cgi.rb. Queue size: 26
Worker 1 says ./blank.yaml contained 3 lines.
Worker 1 reading ./cgi.rb. Queue size: 25
Worker 0 says ./call_cgi.rb contained 11 lines.
Worker 1 says ./cgi.rb contained 10 lines.
Worker 0 reading ./client.rb. Queue size: 24
Worker 1 reading ./curl_test.sh. Queue size: 23
Worker 0 says ./client.rb contained 19 lines.
Worker 0 reading ./curl_test_all_post_vars.sh. Queue size: 22
That's been trimmed down, but you get the idea.
Ruby's Queue class is the key. It's like an array with icing slathered on it, which arbitrates access to the queue. Think of it this way: "consumers", i.e., Threads, put a flag in the air to receive permission to access the queue. When given that permission, they can pop or shift or modify the queue. Once they're done, the permission is given to the next thread with its flag up.
I use pop instead of shift for esoteric reasons but, if your files have to be loaded in a certain order, sort them before they're added to the queue so that order is set, then use shift.
We want to store the number of threads running so we can join them later. This lets the threads complete their tasks before the mother script ends.