Reading in a fixed number of pipe delimited fields per row? - ruby

Question

I have a bunch of pipe-delimited files that weren't properly escaped for carriage returns when generated, and so I cant use the CR or newline characters to delimit the rows. I DO know however that each record has to have exactly 7 fields.
Splitting the fields is easy with the CSV library in Ruby 1.9 setting the 'col_sep' argument, but the 'row_sep' argument cannot be set because I have newlines within the fields.
Is there a way to parse a pipe-delimited file using a fixed number of fields as the row delimiter?
Thanks!

Answer 1

Here's one way of doing it:
Build a sample string of seven words, with an embedded new-line in the
middle of the string. There are three lines worth.
text = (["now is the\ntime for all good"] * 3).join(' ').gsub(' ', '|')
puts text
# >> now|is|the
# >> time|for|all|good|now|is|the
# >> time|for|all|good|now|is|the
# >> time|for|all|good
Process like this:
lines = []
chunks = text.gsub("\n", '|').split('|')
while (chunks.any?)
lines << chunks.slice!(0, 7).join(' ')
end
puts lines
# >> now is the time for all good
# >> now is the time for all good
# >> now is the time for all good
So, that shows we can rebuild the rows.
Pretending that the words are actually columns from the pipe-delimited file we can make the code do the real thing by taking out the .join(' '):
while (chunks.any?)
lines << chunks.slice!(0, 7)
end
ap lines
# >> [
# >> [0] [
# >> [0] "now",
# >> [1] "is",
# >> [2] "the",
# >> [3] "time",
# >> [4] "for",
# >> [5] "all",
# >> [6] "good"
# >> ],
# >> [1] [
# >> [0] "now",
# >> [1] "is",
# >> [2] "the",
# >> [3] "time",
# >> [4] "for",
# >> [5] "all",
# >> [6] "good"
# >> ],
# >> [2] [
# >> [0] "now",
# >> [1] "is",
# >> [2] "the",
# >> [3] "time",
# >> [4] "for",
# >> [5] "all",
# >> [6] "good"
# >> ]
# >> ]

Answer 2

Say for instance you wanted to parse all charities in the IRS txt file that is pipe delimited.
Say you had a model called Charity that had all the same fields as your pipe delimited file.
class Charity < ActiveRecord::Base
# http://apps.irs.gov/app/eos/forwardToPub78DownloadLayout.do
# http://apps.irs.gov/app/eos/forwardToPub78Download.do
attr_accessible :city, :country, :deductibility_status, :deductibility_status_description, :ein, :legal_name, :state
end
You can make a rake task called import.rake
namespace :import do
desc "Import Pipe Delimted IRS 5013c Data "
task :irs_data => :environment do
require 'csv'
txt_file_path = 'db/irs_5013cs.txt'
results = File.open(txt_file_path).readlines do |line|
line = line.split('|').each_slice(7)
end
# Order Field Notes
# 1 EIN Required
# 2 Legal Name Optional
# 3 City Optional
# 4 State Optional
# 5 Deductibility Status Optional
# 6 Country Optional - If Country is null, then Country is assumed to be United States
# 7 Deductibility Status Description Optional
results.each do |row|
row = row.split('|').each_slice(7).to_a.first
#ID,Category,Sub Category,State Standard
Charity.create!({
:ein => row[0],
:legal_name => row[1],
:city => row[2],
:state => row[3],
:deductibility_status => row[4],
:country => row[5],
:deductibility_status_description => row[6]
})
end
end
end
finally you can run this import by typing following on command line from your rails app
rake import:irs_data

Answer 3

Here's one idea, use a regex:
#!/opt/local/bin/ruby
fp = File.open("pipe_delim.txt")
r1 = /.*?\|.*?\|.*?\|.*?\|.*?\|.*?\|.*?\|/m
results = fp.gets.scan(r1)
results.each do |result|
puts result
end
This regex seems to trip up on newlines within a field, but I'm sure you could tweak it to work properly.

Answer 4

Just a thought, but the cucumber testing gem has a Cucumber::Ast::Table class you could use to process this file.
Cucumber::Ast::Table.new(File.read(file))
Then I think it's the rows method you can use to read it out.

Answer 5

Try using String#split and Enumerable#each_slice:
result = []
text.split('|').each_slice(7) { |record| result << record }