CSV parsing, newline/linebreak issues - ruby

Question

I'm trying to create a parser for multiple CSV files, that will eventually output to another CSV file in Excel-compatible format. The CSV files are exported by a commercial tool that takes a Firewall configuration and gives us a report of any issues it finds.
So far I have figured out how to read a directory of files in, look for certain values, determine the type of device I have and then spit it out to screen or to a CSV, but only if each line has single cell entries. If the source IP 'cell' (or any other) contains more than one IP, separated by a newline, the output breaks on that newline and pushes the remainder onto the next line.
The code I have so far is:
require 'csv'
require 'pp'
nipperfiles = Dir.glob(ARGV[0] + '/*.csv')
def allcsv(nipperfiles)
filearray = []
nipperfiles.each do |csv|
filearray << csv
end
filearray
end
def devicetype(filelist)
filelist.each do |f|
CSV.foreach(f, :headers => true, :force_quotes => true) do |row|
if row["Table"] =~ /audit device list/ && row["Device"] =~ /Cisco/
return "Cisco"
elsif row["Table"] =~ /audit device list/ && row["Device"] =~ /Dell/
return "Sonicwall"
elsif row["Table"] =~ /audit device list/ && row["Device"] =~ /Juniper/
return "Juniper"
end
end
end
end
def adminservices(device, filelist)
administrative = []
filelist.each do |f|
CSV.foreach(f, :headers => true, :col_sep => ",", :force_quotes => true, :encoding => Encoding::UTF_8) do |row|
if row["Table"] =~ /administrative service rule/
if row["Dst Port"] != "Any" and row["Service"] != "[Host] Any"
if device == "Cisco"
administrative << row["Table"] + ',' + row["Rule"] + ',' + row["Protocol"] + ',' + row["Source"] + ',' + row["Destination"] + ',' + row["Dst Port"]
elsif device == "Sonicwall"
administrative << row["Table"] + ',' + row["Rule"] + ',' + row["Source"] + ',' + row["Destination"] + ',' + row["Service"]
elsif device == "Juniper"
administrative << row["Table"] + ',' + row["Rule"] + ',' + row["Source"] + ',' + row["Destination"] + ',' + row["Service"]
end
end
end
end
end
administrative
end
def writecsv(admin)
finalcsv = File.new("randomstorm.csv", "w+")
finalcsv.puts("Administrative Services Table:\n", admin, "\r\n")
finalcsv.close
end
filelist = allcsv(nipperfiles)
device = devicetype(filelist)
adminservices(device, filelist)
admin = adminservices(device, filelist)
writecsv(admin)
Is there a way to get it to ignore the newlines that are inside cells, or is my code complete balls and needs to be started again?
I have tried writing a CSV file with the CSV library, but the results are the same and I figured this code was slightly clearer for demonstrating the issue.
I can sanitise an input file if it would help.

Answer 1

newlines are OK inside of fields as long they are quoted:
CSV.parse("1,\"2\n\n\",3")
=> [["1", "2\n\n", "3"]]
Try writing directly to a string or a file like in the documentation which will ensure your fields with newlines are quoted:
def writecsv(admin)
csv_string = CSV.generate do |csv|
admin.each { |row| csv << row }
end
finalcsv = File.new("randomstorm.csv", "w+")
finalcsv.puts("Administrative Services Table:\n", csv_string, "\r\n")
finalcsv.close
end
Also ensure you are writing your fields as an array inside of adminservices():
administrative << [row["Table"], row["Rule"], row["Protocol"], row["Source"], row["Destination"], row["Dst Port"]]