reformat csv in ruby by manipulating rows and columns - ruby

Question

I have to work with a csv file that is not directly useable for my needs of generating a simple chart. I need to manipulate the file into something "cleaner" and am running into issues and unsure if my overall strategy is correct, as I'm just learning to parse through files with ruby.... My issues here mainly related to me looking for data that is offset from where I've found or haven't found matches. After I find a line which meets criteria, I need to read info from 2 lines after it and manipulate some of it (move something from the last column to the second).
Here's the original csv file:
component
quantity header,design1,design2,design3,Ref,Units
quantity type,#,#,#,ref#,unit value
component
quantity header,design1,design2,design3,Ref,Units
quantity type,#,#,#,ref#,unit value
component
quantity header,design1,design2,design3,Ref,Units
quantity type,#,#,#,ref#,unit value
Desired output:
Component Header,Quantity type Header,Units Header,design1 header,design2 header,design3 header,Ref header
component,quantity type,unit value,#,#,#,n/a
component,quantity type,unit value,#,#,#,n/a
component,quantity type,unit value,#,#,#,n/a
component,quantity type,unit value,#,#,#,n/a
component,quantity type,unit value,#,#,#,n/a
My ruby script at the moment:
require 'csv'
f = File.new("sp.csv")
o = CSV.open('output.csv', 'w')
f.each_line do |l| #iterate through each line
data = l.split
if l !~ /,/ #if the line does not contain a comma it is a component
o << [data,f.gets] #start writing data, f.gets skips next line but need to skip 2 and split the line to manipulate columns
else
o << ['comma'] #just me testing that I can find lines with commas
end
end
f.gets skips the next line and the documentation isn't clear to me how to use it to skip 2. After that I THINK I can split that line by commas and manipulate row data with array[column]. Aside from this offset issue I'm also unsure if my general approach is a good strategy
EDIT
Here's some lines from the real file.... I'll work through the answers provided and see if I can make it all work. The idea I've had is to read and write line by line, vs. converting the whole file to an array and then reading and writing. My thought is that when these files get big, and they do, it'll take less memory doing it line by line.
THANKS for the help, I'll work through answers and get back to you.
DCB
Result Quantity,BL::BL,BL::BL_DCB-noHeat,DC1::DC1,DC2::DC2,noHS::noHS,20mmHS::20mmHS,Reference,Units
Avg Temperature,82.915,69.226,78.35,78.383,86.6,85.763,N/A,Celsius
RCB
Result Quantity,BL::BL,BL::BL_DCB-noHeat,DC1::DC1,DC2::DC2,noHS::noHS,20mmHS::20mmHS,Reference,Units
Avg Temperature,76.557,68.779,74.705,74.739,80.22,79.397,N/A,Celsius
Antenna
Result Quantity,BL::BL,BL::BL_DCB-noHeat,DC1::DC1,DC2::DC2,noHS::noHS,20mmHS::20mmHS,Reference,Units
Avg Temperature,69.988,65.045,69.203,69.238,73.567,72.777,N/A,Celsius
PCBA_fiberTray
Result Quantity,BL::BL,BL::BL_DCB-noHeat,DC1::DC1,DC2::DC2,noHS::noHS,20mmHS::20mmHS,Reference,Units
Avg Temperature,66.651,65.904,66.513,66.551,72.516,70.47,N/A,Celsius
EDIT 2
Using some regexp from answers below I developed a line by line strategy to parse through this. I'll post it as an answer for completeness.
Thanks for helping out and exposing me to methods to develop a solution

Answer 1

How about slicing it into groups of 3 lines:
File.read("sp.csv").split("\n").each_slice(3) do |slice|
o << [slice[0], *slice[2].split(',')]
end

Answer 2

I created a CSV file based on the sample, called "test.csv".
Starting with this code:
data = File.readlines('test.csv').slice_before(/^component/)
I get an enumerator back. If I look at the data that enumerator will return I get:
pp data.to_a
[["component\n",
"quantity header,design1,design2,design3,Ref,Units\n",
"quantity type,#,#,#,ref#,unit value\n"],
["component\n",
"quantity header,design1,design2,design3,Ref,Units\n",
"quantity type,#,#,#,ref#,unit value\n"],
["component\n",
"quantity header,design1,design2,design3,Ref,Units\n",
"quantity type,#,#,#,ref#,unit value\n"]]
That's an array of arrays, broken into sub-arrays on the "component" line. I suspect the values aren't reflecting reality, but without a more accurate sample... well, GIGO.
If the "component" line isn't really a bunch of repeating "component" lines, and doesn't have any commas, you can use this instead:
data = File.readlines('test.csv').slice_before(/\A[^,]+\Z/)
or:
data = File.readlines('test.csv').slice_before(/^[^,]+$/)
The result will be the same with the current samples.
If you need a more complex regex you can substitute that, for instance:
/^(?:#{ Regexp.union(%w[component1 component2]).source })$/i
Which returns a pattern that will find any words in the %w[] array:
/^(?:component1|component2)$/i
From there we can walk the data array and clean out all the extraneous headers using:
data.map{ |a| a[2..-1] }.flatten
Which returns something like:
[
"quantity type,#,#,#,ref#,unit value\n",
"quantity type,#,#,#,ref#,unit value\n",
"quantity type,#,#,#,ref#,unit value\n"
]
That can be iterated and passed to CSV to be parsed into arrays if needed:
data.map{ |a| a[2..-1].map{ |r| CSV.parse(r) }.flatten }
[
["quantity type", "#", "#", "#", "ref#", "unit value"],
["quantity type", "#", "#", "#", "ref#", "unit value"],
["quantity type", "#", "#", "#", "ref#", "unit value"]
]
That's all background to get you thinking how you can tear apart the CSV data.
Using this code:
data.flat_map { |ary|
component = ary[0].strip
ary[2..-1].map{ |a|
data = CSV.parse(a).flatten
[
component,
data.shift,
data.pop,
*data[0..-2]
]
}
}
Returns:
[
["component", "quantity type", "unit value", "#", "#", "#"],
["component", "quantity type", "unit value", "#", "#", "#"],
["component", "quantity type", "unit value", "#", "#", "#"]
]
The only thing left to do is create the header you want to use, and pass the returned data back into CSV to let it generate the output file. You should be able to get there from here using the CSV documentation.
Edit:
Based on the actual data, here's a version of the code with a minor tweak, along with its output:
require 'csv'
require 'pp'
data = File.readlines('test.csv').slice_before(/^[^,]+$/)
pp data.flat_map { |ary|
component = ary[0].strip
ary[2..-1].map{ |a|
record = CSV.parse(a).flatten
[
component,
record.shift,
record.pop,
*record[0..-2]
]
}
}
Which looks like:
[["DCB",
"Avg Temperature",
"Celsius",
"82.915",
"69.226",
"78.35",
"78.383",
"86.6",
"85.763"],
["RCB",
"Avg Temperature",
"Celsius",
"76.557",
"68.779",
"74.705",
"74.739",
"80.22",
"79.397"],
["Antenna",
"Avg Temperature",
"Celsius",
"69.988",
"65.045",
"69.203",
"69.238",
"73.567",
"72.777"],
["PCBA_fiberTray",
"Avg Temperature",
"Celsius",
"66.651",
"65.904",
"66.513",
"66.551",
"72.516",
"70.47"]]

Answer 3

Code I'm using which creates csv file with everything manipulated... Thanks to those that contributed some help.
require 'csv'
file_in = File.new('sp1.csv')
file_out = CSV.open('output.csv', 'w')
header = []
row = []
file_in.each_line do |line|
case line
when /^[^,]+$/ #Find a component (line with no comma)
comp_header = file_in.gets.split(',') #header is after component and is split into an arry
if header.empty? #header
header.push("Component", comp_header[0], comp_header[-1].strip)
comp_header[1..-3].each do |h|
header.push(h)
end
file_out << header
end
#comp = line.to_s.strip
next
when /,/ #when a row had commas
puts #comp
vals = line.split(',') #split up into vals array
row.push(#comp, vals[0], vals[-1].strip) #add quantity and unit to row array
vals[1..-3].each do |v| #for values (excluding quanity, units, reference info)
row.push(v) #add values to row array
end
end
file_out << row #write the current row to csv file
row = [] #reset the row array to move on to the next component set
end