Find duplicates in array of hashes on specific keys - ruby

Question

I have an array of hashes (CSV rows, actually) and I need to find and keep all the rows that match two specific keys (user, section). Here is a sample of the data:
[
{ user: 1, role: "staff", section: 123 },
{ user: 2, role: "staff", section: 456 },
{ user: 3, role: "staff", section: 123 },
{ user: 1, role: "exec", section: 123 },
{ user: 2, role: "exec", section: 456 },
{ user: 3, role: "staff", section: 789 }
]
So what I would need to return is an array that contained only the rows where the same user/section combo appears more than once, like so:
[
{ user: 1, role: "staff", section: 123 },
{ user: 1, role: "exec", section: 123 },
{ user: 2, role: "staff", section: 456 },
{ user: 2, role: "exec", section: 456 }
]
The double loop solution I'm trying looks like this:
enrollments.each_with_index do |a, ai|
enrollments.each_with_index do |b, bi|
next if ai == bi
duplicates << b if a[2] == b[2] && a[6] == b[6]
end
end
but since the CSV is 145K rows it's taking forever.
How can I more efficiently get the output I need?

Answer 1

In terms of efficiency you might want to try this:
grouped = csv_arr.group_by{|row| [row[:user],row[:section]]}
filtered = grouped.values.select { |a| a.size > 1 }.flatten
The first statement groups the records by the :user and :section keys. the result is:
{[1, 123]=>[{:user=>1, :role=>"staff", :section=>123}, {:user=>1, :role=>"exec", :section=>123}],
[2, 456]=>[{:user=>2, :role=>"staff", :section=>456}, {:user=>2, :role=>"exec", :section=>456}],
[3, 123]=>[{:user=>3, :role=>"staff", :section=>123}],
[3, 789]=>[{:user=>3, :role=>"staff", :section=>789}]}
The second statement only selects the values of the groups with more than one member and then it flattens the result to give you:
[{:user=>1, :role=>"staff", :section=>123},
{:user=>1, :role=>"exec", :section=>123},
{:user=>2, :role=>"staff", :section=>456},
{:user=>2, :role=>"exec", :section=>456}]
This could improve the speed of your operation, but memory wise I can't say what the effect would be with a large input, because it would depend on your machine, resources and the size of file

Answer 2

To do this check in memory you don't need a double loop, you can keep an array of unique values and check each new csv line against it:
found = []
unique_enrollments = []
CSV.foreach('/path/to/csv') do |row|
# do whatever you're doing to parse this row into the hash you show in your question:
# => { user: 1, role: "staff", section: 123 }
# you might have to do `next if row.header_row?` if the first row is the header
enrollment = parse_row_into_enrollment_hash(row)
unique_tuple = [enrollment[:user], enrollment[:section]]
unless found.include? unique_tuple
found << unique_tuple
unique_enrollments << enrollment
end
end
Now you have unique_enrollments. With this approach you parse the CSV line by line so you're not keeping the whole thing in memory. Then building a smaller array of unique tuples made of the user and section with which you'll use for your uniqueness check and also building up the array of unique rows.
You can further optimize this by not saving the unique_enrollments in a big array but rather just building your Model and saving it to the db:
unless found.include? unique_tuple
found << unique_tuple
Enrollment.create enrollment
end
With the above tweak you'll be able to save on memory by not keeping a big array of enrollments. Although the drawback would be that if something blows up you won't be able to rollback. For example, had we done the former and kept an array of unique_enrollments at the end you can do:
Enrollment.transaction do
unique_enrollments.each &:save!
end
And now you have the ability to rollback if any of those saves blow up. Also, wrapping a bunch of db calls in a single transaction is much faster. I'd go with this approach.
Edit: Using the array of unique_enrollments you can iterate over these at the end and create a new CSV:
CSV.open('path/to/new/csv') do |csv|
csv << ['user', 'role', 'staff'] # write the header
unique_enrollments.each do |enrollment|
csv << enrollment.values # just the values not the keys
end
end