Optimise large CSV whilst using #group_by & each_with_object - ruby

Question

I have a CSV file with 498,766 rows. The contents of the CSV are pulled remotely and stuffed into a tempfile. Once I have the tempfile I group by a specific column and then go through each of the objects and create a new hash.
report = ::RestClient::Request.execute(
url: report_url,
method: :get,
headers: {Authorization: basic_auth.to_s}
)
#file = ::Tempfile.new(["#{report_run.result.filename}", ".csv"])
#file.write(report.body.force_encoding("UTF-8"))
#file.rewind
time = Benchmark.realtime do
::CSV.foreach(#file, headers: true)
.group_by { |fee| fee['charge_id'] }
.each_with_object({}) { |key, hash| hash[key.first] = key.last.sum { |fee| fee['total_amount'].to_f}.round(2) };
end
end
Benchmarking the above it takes about 52 seconds which seems relatively long to me. Is there any further optimisations that I can make here ?
For added clarity the CSV I'm looking at contains columns: charge_id and total_amount. It is possible for there to be multiple rows with the same charge_id and as such I consolidate them and then sum the total value. A better representation of what the CSV rows would look like is something like:
#
# Note this is a dummy representation of CSV data that would come back from
# doing ::CSV.foreach(#file, headers: true)
#
csv_data = [
OpenStruct.new(charge_id: 'ch_1G79Pi4Kqv3kyKfABfXoXycx', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Pi4Kqv3kyKfABfXoXycx', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Pi4Kqv3kyKfABfXoXycx', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Pi4Kqv3kyKfABfXoXycx', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Pi4Kqv3kyKfABfXoXycx', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Pi4Kqv3kyKfABfXoXycx', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Pi4Kqv3kyKfABfXoXycx', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Xt4Kqv3kyKfAnBz9ZJGJ', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Yu4Kqv3kyKfA7CnwoNEo', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79ZQ4Kqv3kyKfAYZMLs8tW', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Ze4Kqv3kyKfAmNbovTjO', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Zs4Kqv3kyKfA38s1yVmq', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79Zy4Kqv3kyKfA99Arn1Lh', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79b04Kqv3kyKfA8uYHL0DY', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79bS4Kqv3kyKfAAWxowFGO', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79dS4Kqv3kyKfADejRhlbZ', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79gM4Kqv3kyKfA30s5NTAj', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79hc4Kqv3kyKfAxJWbu8Ny', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79j64Kqv3kyKfATjAI1JcC', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79jk4Kqv3kyKfAKYdakMAk', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79k64Kqv3kyKfAXmpONrNI', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79le4Kqv3kyKfAJMzltr6U', total_amount: 10.0),
OpenStruct.new(charge_id: 'ch_1G79lu4Kqv3kyKfAdHG5Qw6r', total_amount: 10.0)
].group_by { |fee| fee['charge_id'] }.each_with_object({}) { |key, hash| hash[key.first] = key.last.sum { |fee| fee['total_amount'].to_f}.round(2) }
#=>
{"ch_1G79Pi4Kqv3kyKfABfXoXycx"=>70.0,
"ch_1G79Xt4Kqv3kyKfAnBz9ZJGJ"=>10.0,
"ch_1G79Yu4Kqv3kyKfA7CnwoNEo"=>10.0,
"ch_1G79ZQ4Kqv3kyKfAYZMLs8tW"=>10.0,
"ch_1G79Ze4Kqv3kyKfAmNbovTjO"=>10.0,
"ch_1G79Zs4Kqv3kyKfA38s1yVmq"=>10.0,
"ch_1G79Zy4Kqv3kyKfA99Arn1Lh"=>10.0,
"ch_1G79b04Kqv3kyKfA8uYHL0DY"=>10.0,
"ch_1G79bS4Kqv3kyKfAAWxowFGO"=>10.0,
"ch_1G79dS4Kqv3kyKfADejRhlbZ"=>10.0,
"ch_1G79gM4Kqv3kyKfA30s5NTAj"=>10.0,
"ch_1G79hc4Kqv3kyKfAxJWbu8Ny"=>10.0,
"ch_1G79j64Kqv3kyKfATjAI1JcC"=>10.0,
"ch_1G79jk4Kqv3kyKfAKYdakMAk"=>10.0,
"ch_1G79k64Kqv3kyKfAXmpONrNI"=>10.0,
"ch_1G79le4Kqv3kyKfAJMzltr6U"=>10.0,
"ch_1G79lu4Kqv3kyKfAdHG5Qw6r"=>10.0}

Answer 1

A more direct way to compute the desired hash from csv_data follows. Because it requires a single pass through the array, I expect it will speed things up but have not done a benchmark.
require 'ostruct'
csv_data.each_with_object(Hash.new(0)) do |os,h|
h[os[:charge_id]] += os[:total_amount]
end
#=> {"ch_1G79Pi4Kqv3kyKfABfXoXycx"=>70.0,
# "ch_1G79Xt4Kqv3kyKfAnBz9ZJGJ"=>10.0,
# "ch_1G79Yu4Kqv3kyKfA7CnwoNEo"=>10.0,
# "ch_1G79ZQ4Kqv3kyKfAYZMLs8tW"=>10.0,
# "ch_1G79Ze4Kqv3kyKfAmNbovTjO"=>10.0,
# "ch_1G79Zs4Kqv3kyKfA38s1yVmq"=>10.0,
# "ch_1G79Zy4Kqv3kyKfA99Arn1Lh"=>10.0,
# "ch_1G79b04Kqv3kyKfA8uYHL0DY"=>10.0,
# "ch_1G79bS4Kqv3kyKfAAWxowFGO"=>10.0,
# "ch_1G79dS4Kqv3kyKfADejRhlbZ"=>10.0,
# "ch_1G79gM4Kqv3kyKfA30s5NTAj"=>10.0,
# "ch_1G79hc4Kqv3kyKfAxJWbu8Ny"=>10.0,
# "ch_1G79j64Kqv3kyKfATjAI1JcC"=>10.0,
# "ch_1G79jk4Kqv3kyKfAKYdakMAk"=>10.0,
# "ch_1G79k64Kqv3kyKfAXmpONrNI"=>10.0,
# "ch_1G79le4Kqv3kyKfAJMzltr6U"=>10.0,
# "ch_1G79lu4Kqv3kyKfAdHG5Qw6r"=>10.0}
See the doc for the version of Hash::new that takes an argument called the default value.
If the data is received from a remote source a line at a time one could do the processing on the fly, while receiving the data, by writing something like the following.
CSV.foreach(#file, headers: true).
with_object(Hash.new(0)) do |csv,h|
# <your processing to produce `os`, a line of csv_data>
h[os[:charge_id]] += os[:total_amount]
end
If this could be done it would have to be benchmarked to see it if actually improved performance.
For readers unfamiliar with this form of Hash::new, suppose
h = Hash.new(0)
making h's default value zero. All that means is that if h does not have a key k h[k] returns zero, which I'll write
h[k] #=> 0
Let's add a key-value pair: h[:dog] = 1. Then
h #=> { :dog=>1 }
and
h[:dog] #=> 1
Since h does not have a key :cat
h[:cat] #=> 0
Suppose now we write
h[:dog] += 1
That's the same as
h[:dog] = h[:dog] + 1
which equals
h[:dog] = 1 + 1 #=> 2
Similarly,
h[:cat] += 1
means
h[:cat] = h[:cat] + 1
= 0 + 1
= 1
because h[:cat] on the right (the method Hash#[], as contrasted with the method Hash#[]= on the left) returns zero. At this point
h #=> { :dog=>2, :cat=>1 }
When a hash is defined in this way it is sometimes called a counting hash. It's effectively the same as
h = {}
[1,3,1,2,2].each do |n|
h[n] = 0 unless h.key?(n)
h[n] += 1
end
h #=> {1=>2, 3=>1, 2=>2}

Answer 2

You're doing two passes through the data, one to do the grouping (group_by) and one to accumulate the sums. Here's an example showing a single pass that does both at once along with your original. I included benchmarking.
From my tests, the one-pass method is almost 100% faster. Your mileage may vary. Also, note that I removed header info when reading the data in my method. This further reduces processing overhead and memory manipulation.
require 'csv'
require 'benchmark'
filename = './data.csv'
def one_pass(filename)
file = File.open(filename, 'r')
csv = CSV.new(file)
headers = csv.shift # get rid of headers
results = Hash.new(0)
csv.each do |row|
charge_id, total_amount = row
results[charge_id] += total_amount.to_f
end
file.close
return results
end
def with_group_by(filename)
file = File.open(filename, 'r')
results = CSV.foreach(file, headers: true)
.group_by { |fee| fee['charge_id'] }
.each_with_object({}) { |key, hash| hash[key.first] = key.last.sum { |fee| fee['total_amount'].to_f}.round(2) }
file.close
return results
end
o_results = nil
g_results = nil
time = Benchmark.realtime do
o_results = one_pass filename
end
puts "one_pass: #{time}"
time = Benchmark.realtime do
g_results = with_group_by filename
end
puts "with_group_by: #{time}"
puts "o_results == g_results: #{o_results == g_results}"
My benchmarking results with a file that has 56k lines:
one_pass: 0.24479200004134327
with_group_by: 0.4725199999520555
o_results == g_results: true