I'm working with 6 csv files that each contain the attributes of an object. I can read them one at a time, but the idea of splitting each one to a thread to do in parallel is very appealing.
I've created a database object (no relational DBs or ORMs allowed) that has an array for each of the objects it is holding. I've tried the following to make each CSV open and initialize concurrently, but have seen no impact on speed.
threads = []
CLASS_FILES.each do |klass, filename|
threads << Thread.new do
file_to_objects(klass, filename)
end
end
threads.each {|thread| thread.join}
update
end
def self.load(filename)
CSV.open("data/#{filename}", CSV_OPTIONS)
end
def self.file_to_objects(klass, filename)
file = load(filename)
method_name = filename.sub("s.csv","")
file.each do |line|
instance = klass.new(line.to_hash)
Database.instance.send("#{method_name}") << instance
end
end
How can I speed things up in ruby (MRI 1.9.3)? Is this a good case for Rubinius?
Even though Ruby 1.9.3 uses native threads in order to implement concurrency, it has a global interpreter lock which makes sure only one thread executes at a time.
Therefore, nothing really runs in parallel in C Ruby. I know that JRuby imposes no internal lock on any thread, so try using it to run your code, if possible.
This answer by Jörg W Mittag has a more in-depth look at the threading models of the several Ruby implementations. It isn't clear to me whether Rubinius is fit for the job, but I'd give it a try.
Related
I have some processing to do involving a third party API, and I was planning to use a CSV file as a backlog of things to do.
Example
Task to do Resulting file
#1 data/1.json
#2 data/2.json
#3
So, #1 and #2 are already done. I want to work on #3, and save the CSV file as soon as data/3.json is completed.
As the task is unstable and error prone, I want to save progress after each task in the CSV file.
I've written this script in Ruby, it's working well, but as tasks are numerous (> 100k), it's written couple Megabytes to disk each time a task is processed. The whole thing. It seems a good way to kill my HD:
class CSVResolver
require 'csv'
attr_accessor :csv_path
def initialize csv_path:
self.csv_path = csv_path
end
def resolve
csv = CSV.read(csv_path)
csv.each_with_index do |row, index|
next if row[1] # Don't do anything if we've already processed this task, and got a JSON data
json = very_expensive_task_and_error_prone
row[1] = "/data/#{index}.json"
File.write row[1], JSON.pretty_generate(json)
csv[index] = row
CSV.open(csv_path, "wb") do |old_csv|
csv.each do |row|
old_csv << row
end
end
resolve
end
end
end
Is there any way to improve on this, like making the write to CSV file atomic?
I'd use an embedded database for this purpose, such as SQLite or LevelDB.
Unlike a regular database, you'll still get many of the benefits of a CSV file, ie it can be stored in a single file/folder and without any server or permissioning hassle. At the same time, you'll get the benefit of better I/O characteristic than reading and writing a monolithic file upon each update ... the library should be smart enough to be able to index records, minimise changes, and store things in memory while buffering output.
For data persistence you would be, in most cases, best served to select a tool designed for the job, a database. You've already named enough of a reason to not use the hand spun CSV design as it is memory inefficient and proposes more problems then it likely solves. Also, depending on the amount of data you need to process via the 3rd part API, you may want to handle multi-threaded processes where reading/writing to a single file won't work.
You might wanna checkout https://github.com/jeremyevans/sequel
I'll try to extend a title of my question. I work on ruby project. I have to process a big amount of data (around 120000) stored in CSV files. I have to read this data, process and put in DB. Now it takes couple days. I have to make this much faster. The problem is that sometimes during processing I get some anomalies and I have to repeat whole import process. I decided that more important is to improve performance instead of looking for bug using small amount of data. For now I stick to CSV files. I decided to benchmark processing script to find bottle necks and improve loading data from CSV. I see following steps:
Benchmark and fix the most problematic bottle necks
Maybe split loading from CSV and processing. For example create separate table and load data there. In next step load this data, process and put in right table.
Introduce threads to load data from CSV
For now I use standard ruby CSV library. Do you recommend some better gem?
If some of you are familiar in similar problem It would be happy to get to know you opinion.
Edit:
Database: postgrees
System: linux
I haven't had the opportunity to test it myself but refently I crossed this article, seems to do the job.
https://infinum.co/the-capsized-eight/articles/how-to-efficiently-process-large-excel-files-using-ruby
You'll have to adapt to use CSV instead of XLSX.
For future reference if the site would stop here the code.
It works by writing BATCH_IMPORT_SIZE records at the database at the same time, should give a huge profit.
class ExcelDataParser
def initialize(file_path)
#file_path = file_path
#records = []
#counter = 1
end
BATCH_IMPORT_SIZE = 1000
def call
rows.each do |row|
increment_counter
records << build_new_record(row)
import_records if reached_batch_import_size? || reached_end_of_file?
end
end
private
attr_reader :file_path, :records
attr_accessor :counter
def book
#book ||= Creek::Book.new(file_path)
end
# in this example, we assume that the
# content is in the first Excel sheet
def rows
#rows ||= book.sheets.first.rows
end
def increment_counter
self.counter += 1
end
def row_count
#row_count ||= rows.count
end
def build_new_record(row)
# only build a new record without saving it
RecordModel.new(...)
end
def import_records
# save multiple records using activerecord-import gem
RecordModel.import(records)
# clear records array
records.clear
end
def reached_batch_import_size?
(counter % BATCH_IMPORT_SIZE).zero?
end
def reached_end_of_file?
counter == row_count
end
end
https://infinum.co/the-capsized-eight/articles/how-to-efficiently-process-large-excel-files-using-ruby
Make sure your Ruby script can process multiple files given as parameters, i.e. that you can run it like this:
script.rb abc.csv def.csv xyz.csv
Then parallelise it using GNU Parallel like this to keep all your CPU cores busy:
find . -name \*.csv -print0 | parallel -0 -X script.rb
The -X passes as many CSV files as possible to your Ruby job without exceeding maximum length of command line. You can add in -j 8 after parallel if you want GNU Parallel to run say 8 jobs at a time, and you can use --eta to get the estimated arrival/finish time:
find . -name \*.csv -print0 | parallel -0 -X -j 8 --eta script.rb
By default, GNU Parallel will run as many jobs in parallel as you have CPU cores.
There are a few ways of going about it. I personally would recommend SmarterCSV, which makes it much faster and easier to process CSVs using Array of Hashes. You should definitely split up the work if possible, perhaps making a queue of files to process and do it in batches with use of Redis
According to GIL implementation in Ruby (MRI), the code below must fail by printing a message more than one time. But it doesn't, it always print it one time:
class Sheep
def initialize
#shorn = false
end
def shorn?
#shorn
end
def shorn!
puts "shearing..."
#shorn = true
end
end
s = Sheep.new
55.times.map do
Thread.new { s.shorn! unless s.shorn? }
end.each(&:join)
How come?
$ ruby --version
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin13.0]
It depends a bit on which exact ruby version you use (which differ in the way they schedule threads). On my system it depends a bit on the overall system load and how fast the terminal feels, but on Ruby 2.0.00p481 I get between 1 and 55 lines of output, on Ruby 1.8.7, I consistently get only one line.
It should be noted here that Ruby 2.0 and higher uses actual OS threads (albeit still with a GIL) while Ruby 1.8 uses internal green threads with its own scheduling. It might be very well possible that older ruby versions schedule threads more granular.
In any case, you should not rely on any incidentally thread scheduling behavior. This is not part of any documented behavior and things will change on different systems as as Ruby matures. You should always ensure that you use shared data structures safely when using threads.
I use Ruby version ruby 2.1.5p273 and I suppose your slightly different Ruby version should yield similar results.
I have different results every time I run the program.
I tried with one core enabled and fore cores enabled. I don't see a difference. It is not thread safe, as you expected.
Otherwise the only answer I can come up with is that your program is too fast/lightweight, so that the interpreter does not think of thread switching too often.
I have only one suggestion in this case. A trick you could use to give the interpreter a hint that maybe she could switch threads. You could use the sleep function.
In your example I would put it just before the race condition:
def shorn!
sleep 0.0001
puts "shearing..."
#shorn = true
end
If you'd like to have more info about the GIL I can recommend Jesse Storimer's Nobody understands the GIL
If you'd like to read more about Ruby and concurrency I can recommend Dotan Nahum's Pragmatic Concurrency with Ruby
The trick I suggested was mentioned in this answer
As others have mentioned, the GIL's behavior is not documented and is totally implementation-dependent. You shouldn't rely on any expectations about its scheduling behavior.
A more detailed (and also more general) answer, however, is that the scheduler switches execution between threads to make sure that no single thread blocks the process. This switch is called a context switch or more specifically a thread switch.
When the context switch occurs, the current thread's execution is paused and another thread's execution is resumed. If it's a brand new thread that's being "resumed," then it means that the new thread's execution starts from the beginning.
In the case of your program, each new thread begins with
s.shorn?
as it evaluates unless s.shorn?. At this point, #shorn == false and s.shorn? evaluates to false. So then the thread runs:
s.shorn!
The first command in #shorn! that gets run is:
puts "shearing..."
What happens next depends on the thread scheduler:
If the scheduler decides to let the current thread continue executing, then the next command that gets executed is #shorn = true. Then the thread ends, the scheduler starts the next thread, unless s.shorn? evaluates to true, and the thread stops. This behavior repeats in a loop until there are no more threads left.
If the scheduler decides to switch to another thread, then it will pause execution right before #shorn = true and start running the same code as before from the beginning. That means that #shorn == false when the new thread starts, and so puts "shearing..." will execute again.
As you can see, it all depends on when the scheduler decides to perform a context switch.
But what about the GIL?
The GIL is a horribly misunderstood part of MRI Ruby. There are plenty of resources out there to explain how the GIL works, but in this case the most important thing that you should know is that the GIL doesn't guarantee that each thread will run sequentially.
Instead, the GIL merely guarantees that most core Ruby methods that are implemented in C (for example, Array#<<) won't be interrupted by a context switch until they are finished. In the case of puts "shearing...", I haven't looked at the code for puts, but probably the GIL guarantees that no other thread will run until the currently running thread finishes executing puts.
As for why when you ran your code under MRI 1.8.7 it only displayed shearing... once, that doesn't necessarily have anything to do with green vs. native threads. The better answer is that it was a coincidence. The more precise answer is that in your case, for some reason the scheduler decided to interrupt the first thread after running #shorn = true. This behavior may possibly have been due to green threads in the sense that maybe your native scheduler interrupts more frequently than Ruby's scheduler (hence the "more granular" suggestion in one of the answers below), but that's not necessarily true. It could also have been a fluke.
Multithreading in Ruby is really easy to mess up. Hence why Matz recommends sticking to forking processes, which is memory-inefficient but removes the burden of managing threads. Another approach for larger projects would be to use a library like Celluloid, which abstracts away Ruby's thread safety mechanisms. For a small example like this, however, a simple mutex would do:
semaphore = Mutex.new
s = Sheep.new
55.times.map {
Thread.new {
semaphore.synchronize do
s.shorn! unless s.shorn?
end
}
}.each(&:join)
I am writing a program that loads data from four XML files into four different data structures. It has methods like this:
def loadFirst(year)
File.open("games_#{year}.xml",'r') do |f|
doc = REXML::Document.new f
...
end
end
def loadSecond(year)
File.open("teams_#{year}.xml",'r') do |f|
doc = REXML::Document.new f
...
end
end
etc...
I originally just used one thread and loaded one file after another:
def loadData(year)
time = Time.now
loadFirst(year)
loadSecond(year)
loadThird(year)
loadFourth(year)
puts Time.now - time
end
Then I realized that I should be using multiple threads. My expectation was that loading from each file on a separate thread would be very close to four times as fast as doing it all sequentially (I have a MacBook Pro with an i7 processor):
def loadData(year)
time = Time.now
t1 = Thread.start{loadFirst(year)}
t2 = Thread.start{loadSecond(year)}
t3 = Thread.start{loadThird(year)}
loadFourth(year)
t1.join
t2.join
t3.join
puts Time.now - time
end
What I found was that the version using multiple threads is actually slower than the other. How can this possibly be? The difference is around 20 seconds with each taking around 2 to 3 minutes.
There are no shared resources between the threads. Each opens a different data file and loads data into a different data structure than the others.
I think (but I'm not sure) the problem is that you are reading (using multiple threads) contents placed on the same disk, so all your threads can't run simultaneously because they wait for IO (disk).
Some days ago I had to do a similar thing (but fetching data from network) and the difference between sequential vs threads was huge.
A possible solution could be to load all file content instead of load it like you did in your code. In your code you read contents line by line. If you load all the content and then process it you should be able to perform much better (because threads should not wait for IO)
It's impossible to give a conclusive answer to why your parallel problem is slower than the sequential one without a lot more information, but one possibility is:
With the sequential program, your disk seeks to the first file, reads it all out, seeks to the 2nd file, reads it all out, and so on.
With the parallel program, the disk head keeps moving back and forth trying to service I/O requests from all 4 threads.
I don't know if there's any way to measure disk seek time on your system: if so, you could confirm whether this hypothesis is true.
I've been playing around with threads but I keep running into a problem where the treads seem to just die or stop.
Whats going on here? And how do I get round it?
I've included the code, but didn't paste it here as I think this problem is more fundamental to ruby.
source code
thanks.
Edit
Ruby 1.8, MacOS (snow leopard)
It looks like you're forgetting to add your new Thread objects to your threads object.
3.times do |t|
threads << Thread.new { word_list.process }
end
Your threads.each {|t| t.join} is working on an empty array, and so is ignoring the threads you did create. Make the change and it should wait.
Edit: I meant to << onto the array, not set it equal.
If you've got threads randomly "going away", make sure you've set Thread.abort_on_exception to true. That'll stop the interpreter if an uncaught exception reaches the top-level in a background thread (via SystemExit, so don't rescue Exception or they'll get swallowed), and can be immensely useful in tracking down random bugs.