Merge CSV files same unique ID with Ruby - ruby

Question

I have two CSV files
file1.csv
username;userid;full_name;follower_count;following_count;media_count;email;category
helloworld;1234;data3;data4;data5;data6;data7;data8
file2.csv
username;owner_id;owner_profile_pic_url;media_url;tagged_brand_username
helloworld;1234;data3b;data4b;data5b
I need the following output file using Ruby with blank if file1.csv username is not found in file2.csv (e.g. row 2).
output.csv
username;userid;full_name;follower_count;following_count;media_count;email;category;owner_profile_pic_url;media_url;tagged_brand_username
helloworld;1234;data3;data4;data5;data6;data7;data8;data3b;data4b;data5b
helloworld;1234;data3;data4;data5;data6;data7;data8;;;
Currently I'm doing it using a Excel vlookup function.
Thanks

Answer 1

There's a lot to unpack in this script. Essentially you need to read both CSV files into a hash, merge file2 into file1, and write it back to a CSV.
require "csv"
dict = Hash.new
options = { col_sep: ";", headers: true}
# read file1
CSV.foreach("file1.csv", options) do |row|
row = row.to_h
user = "#{row['username']+row['userid']}"
dict[user] = row
end
# read file2
CSV.foreach("file2.csv", options) do |row|
row = row.to_h
user = "#{row['username']+row['owner_id']}"
row.delete('owner_id')
dict[user] = row.merge(dict[user]) if dict[user]
end
# turn hash into rows
rows = [['username','userid','full_name','follower_count','following_count','media_count','email','category','owner_profile_pic_url','media_url','tagged_brand_username']]
dict.each do |key, value|
row = rows[0].map{|h| value[h] || "" }
rows.push(row)
end
# write to csv
File.write("output.csv", rows.map{|r| r.to_csv(col_sep: ";") }.join)
This covers both when there is a match and no username match in file1.
# file1.csv
username;userid;full_name;follower_count;following_count;media_count;email;category
helloworld;1234;data3;data4;data5;data6;data7;data8
goodbyeworld;5678;data3;data4;data5;data6;data7;file2.csv
# file2.csv
username;owner_id;owner_profile_pic_url;media_url;tagged_brand_username
helloworld;1234;data3b;data4b;data5b
# output.csv
username;userid;full_name;follower_count;following_count;media_count;email;category;owner_profile_pic_url;media_url;tagged_brand_username
helloworld;1234;data3;data4;data5;data6;data7;data8;data3b;data4b;data5b
goodbyeworld;5678;data3;data4;data5;data6;data7;data8;"";"";""
As mentioned, the fact that there is two lines with the same ID in output.csv is very confusing. Next time just add an extra row showing what happens if there's no match. While this is a good question, we have guidelines on how to write an excellent question.

Answer 2

There are two existing CSV input files and we wish to create one CSV output file:
FNAME1 = 'file1.csv'
FNAME2 = 'file2.csv'
FILE_OUT = 'output.csv'
Let's first create the two input files.
File.write(FNAME1, "username;userid;full_name;follower_count;following_count;media_count;email;category\nhelloworld;1234;data3;data4;data5;data6;data7;data8\n")
#=> 136
File.write(FNAME2, "username;owner_id;owner_profile_pic_url;media_url;tagged_brand_username\nhelloworld;1234;data3b;data4b;data5b\n")
#=> 109
Now go through the steps to read those files, manipulate their contents and write the output file.
require 'csv'
First read both input files and save their contents in variables.
def read_csv(fname)
CSV.read(fname, col_sep: ';', headers: true)
end
csv1 = read_csv(FNAME1)
#=> #<CSV::Table mode:col_or_row row_count:2>
csv2 = read_csv(FNAME2)
#=> #<CSV::Table mode:col_or_row row_count:2>
Note:
csv1.to_a
#=> [["username", "userid", "full_name", "follower_count", "following_count",
# "media_count", "email", "category"],
# ["helloworld", "1234", "data3", "data4", "data5",
# "data6", "data7", "data8"]]
csv2.to_a
#=> [["username", "owner_id", "owner_profile_pic_url", "media_url", "tagged_brand_username"],
# ["helloworld", "1234", "data3b", "data4b", "data5b"]]
As you see, these are ordinary arrays, so if we wished we could at this point forget they came from CSV files and use standard Ruby methods to create the desired output file.
Now see if the values of "username" are the same in both files:
username1 = csv1['username'].first
#=> "helloworld"
username2 = csv2['username'].first
#=> "helloworld"
csv1['username'] creates an array of all values in the "helloworld" column. Here that is simply ["helloworld"]; hence .first. Same for csv2, of course.
If username1 == username2 #=> false we perform an action that I am not clear about, then quit. Henceforth, I assume the two usernames are equal.
Read the headers of both files into arrays.
headers1 = csv1.headers
#=> ["username", "userid", "full_name", "follower_count", "following_count",
# "media_count", "email", "category"]
headers2 = csv2.headers
#=> ["username", "owner_id", "owner_profile_pic_url", "media_url",
# "tagged_brand_username"]
The output file is to contain all the columns in headers1 and all the columns in headers2 with the exception of "username" and "owner_id" in headers2, so let's next get rid of those headers in headers2:
headers2 -= ["username", "owner_id"]
#=> ["owner_profile_pic_url", "media_url", "tagged_brand_username"]
Next retrieve the values of the headers in the first file:
values1 = headers1.flat_map { |h| csv1[h] }
#=> ["helloworld", "1234", "data3", "data4", "data5", "data6", "data7", "data8"]
and the values of the remaining headers in the second file:
values2 = headers2.flat_map { |h| csv2[h] }
#=> ["data3b", "data4b", "data5b"]
We will modify values2 below so we need to save its current size:
values2_size = values2.size
#=> i
The first line in the output file after the header line is to contain the values:
values1 += values2
#=> ["helloworld", "1234", "data3", "data4", "data5", "data6", "data7", "data8",
# "data3b", "data4b", "data5b"]
and the second line is to contain:
values2 = values1 - values2
#=> ["helloworld", "1234", "data3", "data4", "data5", "data6", "data7", "data8",
plus values2_size #=> 3 empty fields.
We could use CSV methods to write this to file, but there is really no advantage in doing so over using regular file methods. We can simply write the following string to file.
str = [(headers1 + headers2).join(';'),
values1.join(';'),
values2.join(';') + ';' * values2_size
].join("\n")
puts str
username;userid;full_name;follower_count;following_count;media_count;email;category;owner_profile_pic_url;media_url;tagged_brand_username
helloworld;1234;data3;data4;data5;data6;data7;data8;data3b;data4b;data5b
helloworld;1234;data3;data4;data5;data6;data7;data8;;;
Let's do it.
File.write(FILE_OUT, str)
#=> 265
Note that, if a and b are arrays, a += b and a -= b expand to a = a + b and a = a - b, respectively. The CSV methods I've used are documented here.
I will leave it to the OP to combine the operations I've discussed into a method.