Interpreting this raw text - a strategy? - ruby

Question

I have this raw text:
________________________________________________________________________________________________________________________________
Pos Car Competitor/Team Driver Vehicle Cap CL Laps Race.Time Fastest...Lap
1 6 Jason Clements Jason Clements BMW M3 3200 10 9:48.5710 3 0:57.3228*
2 42 David Skillender David Skillender Holden VS Commodore 6000 10 9:55.6866 2 0:57.9409
3 37 Bruce Cook Bruce Cook Ford Escort 3759 10 9:56.4388 4 0:58.3359
4 18 Troy Marinelli Troy Marinelli Nissan Silvia 3396 10 9:56.7758 2 0:58.4443
5 75 Anthony Gilbertson Anthony Gilbertson BMW M3 3200 10 10:02.5842 3 0:58.9336
6 26 Trent Purcell Trent Purcell Mazda RX7 2354 10 10:07.6285 4 0:59.0546
7 12 Scott Hunter Scott Hunter Toyota Corolla 2000 10 10:11.3722 5 0:59.8921
8 91 Graeme Wilkinson Graeme Wilkinson Ford Escort 2000 10 10:13.4114 5 1:00.2175
9 7 Justin Wade Justin Wade BMW M3 4000 10 10:18.2020 9 1:00.8969
10 55 Greg Craig Grag Craig Toyota Corolla 1840 10 10:18.9956 7 1:00.7905
11 46 Kyle Orgam-Moore Kyle Organ-Moore Holden VS Commodore 6000 10 10:30.0179 3 1:01.6741
12 39 Uptiles Strathpine Trent Spencer BMW Mini Cooper S 1500 10 10:40.1436 2 1:02.2728
13 177 Mark Hyde Mark Hyde Ford Escort 1993 10 10:49.5920 2 1:03.8069
14 34 Peter Draheim Peter Draheim Mazda RX3 2600 10 10:50.8159 10 1:03.4396
15 5 Scott Douglas Scott Douglas Datsun 1200 1998 9 9:48.7808 3 1:01.5371
16 72 Paul Redman Paul Redman Ford Focus 2lt 9 10:11.3707 2 1:05.8729
17 8 Matthew Speakman Matthew Speakman Toyota Celica 1600 9 10:16.3159 3 1:05.9117
18 74 Lucas Easton Lucas Easton Toyota Celica 1600 9 10:16.8050 6 1:06.0748
19 77 Dean Fuller Dean Fuller Mitsubishi Sigma 2600 9 10:25.2877 3 1:07.3991
20 16 Brett Batterby Brett Batterby Toyota Corolla 1600 9 10:29.9127 4 1:07.8420
21 95 Ross Hurford Ross Hurford Toyota Corolla 1600 8 9:57.5297 2 1:12.2672
DNF 13 Charles Wright Charles Wright BMW 325i 2700 9 9:47.9888 7 1:03.2808
DNF 20 Shane Satchwell Shane Satchwell Datsun 1200 Coupe 1998 1 1:05.9100 1 1:05.9100
Fastest Lap Av.Speed Is 152kph, Race Av.Speed Is 148kph
R=under lap record by greatest margin, r=under lap record, *=fastest lap time
________________________________________________________________________________________________________________________________
Issue# 2 - Printed Sat May 26 15:43:31 2012 Timing System By NATSOFT [phone] www.natsoft.com.au/results
Amended
I need to parse it into an object with the obvious Position, Car, Driver etc fields. The issue is I have no idea on what sort of strategy to use. If I split it on whitespace, I would end up with a list like so:
["1", "6", "Jason", "Clements", "Jason", "Clements", "BMW", "M3", "3200", "10", "9:48.5710", "3", "0:57.3228*"]
Can you see the issue. I cannot just interpret this list, because people may have just 1 name, or 3 words in a name, or many different words in a car. It makes it impossible to just reference the list using indexes alone.
What about using the offsets defined by the column names? I can't quite see how that could be used though.
Edit: So the current algorithm I am using works like this:
Split the text on new line giving a collection of lines.
Find the common whitespace characters FURTHEST RIGHT on each line. I.e. the positions (indexes) on each line where every other
line contains whitespace. EG:
Split the lines based on those common characters.
Trim the lines
Several issues exist:
If the names contain the same lengths like so:
Jason Adams
Bobby Sacka
Jerry Louis
Then it will interpret that as two separate items: (["Jason" "Adams", "Bobby", "Sacka", "Jerry", "Louis"]).
Whereas if they all differed like so:
Dominic Bou
Bob Adams
Jerry Seinfeld
Then it would correctly split on the last 'd' in Seinfeld (and thus we'd get a collection of three names(["Dominic Bou", "Bob Adams", "Jerry Seinfeld"]).
It's also quite brittle. I am looking for a nicer solution.

Answer 1

This is not a good case for regex, you really want to discover the format and then unpack the lines:
lines = str.split "\n"
# you know the field names so you can use them to find the column positions
fields = ['Pos', 'Car', 'Competitor/Team', 'Driver', 'Vehicle', 'Cap', 'CL Laps', 'Race.Time', 'Fastest...Lap']
header = lines.shift until header =~ /^Pos/
positions = fields.map{|f| header.index f}
# use that to construct an unpack format string
format = 1.upto(positions.length-1).map{|x| "A#{positions[x] - positions[x-1]}"}.join
# A4A5A31A25A21A6A12A10
lines.each do |line|
next unless line =~ /^(\d|DNF)/ # skip lines you're not interested in
data = line.unpack(format).map{|x| x.strip}
puts data.join(', ')
# or better yet...
car = Hash[fields.zip data]
puts car['Driver']
end

Answer 2

http://blog.ryanwood.com/past/2009/6/12/slither-a-dsl-for-parsing-fixed-width-text-files this may solve your problem.
here are few more examples and github.
Hope this helps!

Answer 3

I think it is easy enough to just use the fixed width on each line.
#!/usr/bin/env ruby
# ruby parsing_winner.rb winners_list.txt
args = ARGV
puts "ruby parsing_winner.rb winners_list.txt " if args.empty?
winner_file = open args.shift
array_of_race_results, array_of_race_results_array = [], []
class RaceResult
attr_accessor :position, :car, :team, :driver, :vehicle, :cap, :cl_laps, :race_time, :fastest, :fastest_lap
def initialize(position, car, team, driver, vehicle, cap, cl_laps, race_time, fastest, fastest_lap)
#position = position
#car = car
#team = team
#driver = driver
#vehicle = vehicle
#cap = cap
#cl_laps = cl_laps
#race_time = race_time
#fastest = fastest
#fastest_lap = fastest_lap
end
def to_a
# ["1", "6", "Jason", "Clements", "Jason", "Clements", "BMW", "M3", "3200", "10", "9:48.5710", "3", "0:57.3228*"]
[position, car, team, driver, vehicle, cap, cl_laps, race_time, fastest, fastest_lap]
end
end
# Pos Car Competitor/Team Driver Vehicle Cap CL Laps Race.Time Fastest...Lap
# 1 6 Jason Clements Jason Clements BMW M3 3200 10 9:48.5710 3 0:57.3228*
# 2 42 David Skillender David Skillender Holden VS Commodore 6000 10 9:55.6866 2 0:57.9409
# etc...
winner_file.each_line do |line|
next if line[/^____/] || line[/^\w{4,}|^\s|^Pos/] || line[0..3][/\=/]
position = line[0..3].strip
car = line[4..8].strip
team = line[9..39].strip
driver = line[40..64].strip
vehicle = line[65..85].strip
cap = line[86..91].strip
cl_laps = line[92..101].strip
race_time = line[102..113].strip
fastest = line[114..116].strip
fastest_lap = line[117..-1].strip
racer = RaceResult.new(position, car, team, driver, vehicle, cap, cl_laps, race_time, fastest, fastest_lap)
array_of_race_results << racer
array_of_race_results_array << racer.to_a
end
puts "Race Results Objects: #{array_of_race_results}"
puts "Race Results: #{array_of_race_results_array.inspect}"
Output =>
Race Results Objects: [#<RaceResult:0x007fcc4a84b7c8 #position="1", #car="6", #team="Jason Clements", #driver="Jason Clements", #vehicle="BMW M3", #cap="3200", #cl_laps="10", #race_time="9:48.5710", #fastest="3", #fastest_lap="0:57.3228*">, #<RaceResult:0x007fcc4a84aa08 #position="2", #car="42", #team="David Skillender", #driver="David Skillender", #vehicle="Holden VS Commodore", #cap="6000", #cl_laps="10", #race_time="9:55.6866", #fastest="2", #fastest_lap="0:57.9409">, #<RaceResult:0x007fcc4a849ce8 #position="3", #car="37", #team="Bruce Cook", #driver="Bruce Cook", #vehicle="Ford Escort", #cap="3759", #cl_laps="10", #race_time="9:56.4388", #fastest="4", #fastest_lap="0:58.3359">, #<RaceResult:0x007fcc4a8491f8 #position="4", #car="18", #team="Troy Marinelli", #driver="Troy Marinelli", #vehicle="Nissan Silvia", #cap="3396", #cl_laps="10", #race_time="9:56.7758", #fastest="2", #fastest_lap="0:58.4443">, #<RaceResult:0x007fcc4b091ab8 #position="5", #car="75", #team="Anthony Gilbertson", #driver="Anthony Gilbertson", #vehicle="BMW M3", #cap="3200", #cl_laps="10", #race_time="10:02.5842", #fastest="3", #fastest_lap="0:58.9336">, #<RaceResult:0x007fcc4b0916a8 #position="6", #car="26", #team="Trent Purcell", #driver="Trent Purcell", #vehicle="Mazda RX7", #cap="2354", #cl_laps="10", #race_time="10:07.6285", #fastest="4", #fastest_lap="0:59.0546">, #<RaceResult:0x007fcc4b091298 #position="7", #car="12", #team="Scott Hunter", #driver="Scott Hunter", #vehicle="Toyota Corolla", #cap="2000", #cl_laps="10", #race_time="10:11.3722", #fastest="5", #fastest_lap="0:59.8921">, #<RaceResult:0x007fcc4b090e88 #position="8", #car="91", #team="Graeme Wilkinson", #driver="Graeme Wilkinson", #vehicle="Ford Escort", #cap="2000", #cl_laps="10", #race_time="10:13.4114", #fastest="5", #fastest_lap="1:00.2175">, #<RaceResult:0x007fcc4b090a78 #position="9", #car="7", #team="Justin Wade", #driver="Justin Wade", #vehicle="BMW M3", #cap="4000", #cl_laps="10", #race_time="10:18.2020", #fastest="9", #fastest_lap="1:00.8969">, #<RaceResult:0x007fcc4b090668 #position="10", #car="55", #team="Greg Craig", #driver="Grag Craig", #vehicle="Toyota Corolla", #cap="1840", #cl_laps="10", #race_time="10:18.9956", #fastest="7", #fastest_lap="1:00.7905">, #<RaceResult:0x007fcc4b090258 #position="11", #car="46", #team="Kyle Orgam-Moore", #driver="Kyle Organ-Moore", #vehicle="Holden VS Commodore", #cap="6000", #cl_laps="10", #race_time="10:30.0179", #fastest="3", #fastest_lap="1:01.6741">, #<RaceResult:0x007fcc4b08fe48 #position="12", #car="39", #team="Uptiles Strathpine", #driver="Trent Spencer", #vehicle="BMW Mini Cooper S", #cap="1500", #cl_laps="10", #race_time="10:40.1436", #fastest="2", #fastest_lap="1:02.2728">, #<RaceResult:0x007fcc4b08fa38 #position="13", #car="177", #team="Mark Hyde", #driver="Mark Hyde", #vehicle="Ford Escort", #cap="1993", #cl_laps="10", #race_time="10:49.5920", #fastest="2", #fastest_lap="1:03.8069">, #<RaceResult:0x007fcc4b08f628 #position="14", #car="34", #team="Peter Draheim", #driver="Peter Draheim", #vehicle="Mazda RX3", #cap="2600", #cl_laps="10", #race_time="10:50.8159", #fastest="10", #fastest_lap="1:03.4396">, #<RaceResult:0x007fcc4b08f218 #position="15", #car="5", #team="Scott Douglas", #driver="Scott Douglas", #vehicle="Datsun 1200", #cap="1998", #cl_laps="9", #race_time="9:48.7808", #fastest="3", #fastest_lap="1:01.5371">, #<RaceResult:0x007fcc4b08ee08 #position="16", #car="72", #team="Paul Redman", #driver="Paul Redman", #vehicle="Ford Focus", #cap="2lt", #cl_laps="9", #race_time="10:11.3707", #fastest="2", #fastest_lap="1:05.8729">, #<RaceResult:0x007fcc4b08e9f8 #position="17", #car="8", #team="Matthew Speakman", #driver="Matthew Speakman", #vehicle="Toyota Celica", #cap="1600", #cl_laps="9", #race_time="10:16.3159", #fastest="3", #fastest_lap="1:05.9117">, #<RaceResult:0x007fcc4b08e5e8 #position="18", #car="74", #team="Lucas Easton", #driver="Lucas Easton", #vehicle="Toyota Celica", #cap="1600", #cl_laps="9", #race_time="10:16.8050", #fastest="6", #fastest_lap="1:06.0748">, #<RaceResult:0x007fcc4b08e1d8 #position="19", #car="77", #team="Dean Fuller", #driver="Dean Fuller", #vehicle="Mitsubishi Sigma", #cap="2600", #cl_laps="9", #race_time="10:25.2877", #fastest="3", #fastest_lap="1:07.3991">, #<RaceResult:0x007fcc4b08ddc8 #position="20", #car="16", #team="Brett Batterby", #driver="Brett Batterby", #vehicle="Toyota Corolla", #cap="1600", #cl_laps="9", #race_time="10:29.9127", #fastest="4", #fastest_lap="1:07.8420">, #<RaceResult:0x007fcc4a848348 #position="21", #car="95", #team="Ross Hurford", #driver="Ross Hurford", #vehicle="Toyota Corolla", #cap="1600", #cl_laps="8", #race_time="9:57.5297", #fastest="2", #fastest_lap="1:12.2672">, #<RaceResult:0x007fcc4a847948 #position="DNF", #car="13", #team="Charles Wright", #driver="Charles Wright", #vehicle="BMW 325i", #cap="2700", #cl_laps="9", #race_time="9:47.9888", #fastest="7", #fastest_lap="1:03.2808">, #<RaceResult:0x007fcc4a847010 #position="DNF", #car="20", #team="Shane Satchwell", #driver="Shane Satchwell", #vehicle="Datsun 1200 Coupe", #cap="1998", #cl_laps="1", #race_time="1:05.9100", #fastest="1", #fastest_lap="1:05.9100">]
Race Results: [["1", "6", "Jason Clements", "Jason Clements", "BMW M3", "3200", "10", "9:48.5710", "3", "0:57.3228*"], ["2", "42", "David Skillender", "David Skillender", "Holden VS Commodore", "6000", "10", "9:55.6866", "2", "0:57.9409"], ["3", "37", "Bruce Cook", "Bruce Cook", "Ford Escort", "3759", "10", "9:56.4388", "4", "0:58.3359"], ["4", "18", "Troy Marinelli", "Troy Marinelli", "Nissan Silvia", "3396", "10", "9:56.7758", "2", "0:58.4443"], ["5", "75", "Anthony Gilbertson", "Anthony Gilbertson", "BMW M3", "3200", "10", "10:02.5842", "3", "0:58.9336"], ["6", "26", "Trent Purcell", "Trent Purcell", "Mazda RX7", "2354", "10", "10:07.6285", "4", "0:59.0546"], ["7", "12", "Scott Hunter", "Scott Hunter", "Toyota Corolla", "2000", "10", "10:11.3722", "5", "0:59.8921"], ["8", "91", "Graeme Wilkinson", "Graeme Wilkinson", "Ford Escort", "2000", "10", "10:13.4114", "5", "1:00.2175"], ["9", "7", "Justin Wade", "Justin Wade", "BMW M3", "4000", "10", "10:18.2020", "9", "1:00.8969"], ["10", "55", "Greg Craig", "Grag Craig", "Toyota Corolla", "1840", "10", "10:18.9956", "7", "1:00.7905"], ["11", "46", "Kyle Orgam-Moore", "Kyle Organ-Moore", "Holden VS Commodore", "6000", "10", "10:30.0179", "3", "1:01.6741"], ["12", "39", "Uptiles Strathpine", "Trent Spencer", "BMW Mini Cooper S", "1500", "10", "10:40.1436", "2", "1:02.2728"], ["13", "177", "Mark Hyde", "Mark Hyde", "Ford Escort", "1993", "10", "10:49.5920", "2", "1:03.8069"], ["14", "34", "Peter Draheim", "Peter Draheim", "Mazda RX3", "2600", "10", "10:50.8159", "10", "1:03.4396"], ["15", "5", "Scott Douglas", "Scott Douglas", "Datsun 1200", "1998", "9", "9:48.7808", "3", "1:01.5371"], ["16", "72", "Paul Redman", "Paul Redman", "Ford Focus", "2lt", "9", "10:11.3707", "2", "1:05.8729"], ["17", "8", "Matthew Speakman", "Matthew Speakman", "Toyota Celica", "1600", "9", "10:16.3159", "3", "1:05.9117"], ["18", "74", "Lucas Easton", "Lucas Easton", "Toyota Celica", "1600", "9", "10:16.8050", "6", "1:06.0748"], ["19", "77", "Dean Fuller", "Dean Fuller", "Mitsubishi Sigma", "2600", "9", "10:25.2877", "3", "1:07.3991"], ["20", "16", "Brett Batterby", "Brett Batterby", "Toyota Corolla", "1600", "9", "10:29.9127", "4", "1:07.8420"], ["21", "95", "Ross Hurford", "Ross Hurford", "Toyota Corolla", "1600", "8", "9:57.5297", "2", "1:12.2672"], ["DNF", "13", "Charles Wright", "Charles Wright", "BMW 325i", "2700", "9", "9:47.9888", "7", "1:03.2808"], ["DNF", "20", "Shane Satchwell", "Shane Satchwell", "Datsun 1200 Coupe", "1998", "1", "1:05.9100", "1", "1:05.9100"]]

Answer 4

You can use the fixed_width gem.
Your given file can be parsed with the following code:
require 'fixed_width'
require 'pp'
FixedWidth.define :cars do |d|
d.head do |head|
head.trap { |line| line !~ /\d/ }
end
d.body do |body|
body.trap { |line| line =~ /^(\d|DNF)/ }
body.column :pos, 4
body.column :car, 5
body.column :competitor, 31
body.column :driver, 25
body.column :vehicle, 21
body.column :cap, 5
body.column :cl_laps, 11
body.column :race_time, 11
body.column :fast_lap_no, 4
body.column :fast_lap_time, 10
end
end
pp FixedWidth.parse(File.open("races.txt"), :cars)
The trap method identifies the lines in each section. I used regex:
The head regex looks for lines that don't contain a digit.
The body regex looks for lines starting with a digit or "DNF"
Each section must include the line immediately after the last. The column definitions simply identify the number of columns to grab. The library strips whitespace for you. If you wanted to produce a fixed-width file, you can add alignment parameters, but it doesn't appear you will need that.
The result is a hash that starts like this:
{:head=>[{}, {}, {}],
:body=>
[{:pos=>"1",
:car=>"6",
:competitor=>"Jason Clements",
:driver=>"Jason Clements",
:vehicle=>"BMW M3",
:cap=>"3200",
:cl_laps=>"10",
:race_time=>"9:48.5710",
:fast_lap_no=>"3",
:fast_lap_time=>"0:57.3228"},
{:pos=>"2",
:car=>"42",
:competitor=>"David Skillender",
:driver=>"David Skillender",
:vehicle=>"Holden VS Commodore",
:cap=>"6000",
:cl_laps=>"10",
:race_time=>"9:55.6866",
:fast_lap_no=>"2",
:fast_lap_time=>"0:57.9409"},

Answer 5

Depending on how consistent the formatting is, you can probably use regex for this.
Here is a sample regex that works for the current data - may need to be tweaked depending on precise rules, but it gives the idea:
^
# Pos
(\d+|DNF)
\s+
#Car
(\d+)
\s+
# Team
([\w-]+(?: [\w-]+)+)
\s+
# Driver
([\w-]+(?: [\w-]+)+)
\s+
# Vehicle
([\w-]+(?: ?[\w-]+)+)
\s+
# Cap
(\d{4}|\dlt)
\s+
# CL Laps
(\d+)
\s+
# Race.Time
(\d+:\d+\.\d+)
\s+
# Fastest Lap
(\d+)
\s+
# Fastest Lap Time
(\d+:\d+\.\d+\*?)
\s*
$

Answer 6

If you can verify that the whitespace is space characters rather than tabs, and that overlong text is always truncated to fit the column structure, then I'd hard-code the slice boundaries:
parsed = [rawLine[0:3],rawLine[4:7],rawLine[9:38], ...etc... ]
Depending on the data source, this may be brittle (if, for instance every run has different column widths).
If the header row is always the same, you could extract the slice boundaries by searching for the known words of the header row.

Answer 7

Alright, I gotchu:
Edit: I forgot to mention, its assuming you've stored your input text in the variable input_string
# Choose a delimeter that is unlikely to occure
DELIM = '|||'
# DRY -> extend String
class String
def split_on_spaces(min_spaces = 1)
self.strip.gsub(/\s{#{min_spaces},}/, DELIM).split(DELIM)
end
end
# just get the data lines
lines = input_string.split("\n")
lines = lines[2...(lines.length - 4)].delete_if { |line|
line.empty?
}
# Grab all the entries into a nice 2-d array
entries = lines.map { |line|
[
line[0..8].split_on_spaces,
line[9..85].split_on_spaces(3).map{ |string|
string.gsub(/\s+/, ' ') # replace whitespace with 1 space
},
line[85...line.length].split_on_spaces(2)
].flatten
}
# BONUS
# Make nice hashes
keys = [:pos, :car, :team, :driver, :vehicle, :cap, :cl_laps, :race_time, :fastest_lap]
objects = entries.map { |entry|
Hash[keys.zip entry]
}
Outputs:
entries # =>
["1", "6", "Jason Clements", "Jason Clements", "BMW M3", "3200", "10", "9:48.5710", "3 0:57.3228*"]
["2", "42", "David Skillender", "David Skillender", "Holden VS Commodore", "6000", "10", "9:55.6866", "2 0:57.9409"]
...
# all of length 9, no extra spaces
And in case arrays just dont cut it
objects # =>
{:pos=>"1", :car=>"6", :team=>"Jason Clements", :driver=>"Jason Clements", :vehicle=>"BMW M3", :cap=>"3200", :cl_laps=>"10", :race_time=>"9:48.5710", :fastest_lap=>"3 0:57.3228*"}
{:pos=>"2", :car=>"42", :team=>"David Skillender", :driver=>"David Skillender", :vehicle=>"Holden VS Commodore", :cap=>"6000", :cl_laps=>"10", :race_time=>"9:55.6866", :fastest_lap=>"2 0:57.9409"}
...
I leave refactoring it into nice functions to you.

Answer 8

Unless there's a clear rule on how the columns are separated, you can't really do it.
The approach you have is good, assuming you know that each column value is properly indented to the column title.
Another approach could be to group words that are separated by exactly one space together (from the text you provided, I can see that this rule also holds).

Answer 9

Assuming the text will always be spaced the same, you could split the string based on position, then strip away extra spaces around each part. For example, in python:
pos=row[0:3].strip()
car=row[4:7].strip()
and so on. Alternately, you could define a regular expression to capture each part:
([:alnum:]+)\s([:num:]+)\s(([:alpha:]+ )+)\s(([:alpha:]+ )+)\s(([:alpha:]* )+)\s
and so on. (The exact syntax depends on your regexp grammar.) Note that the car regexp needs to handle the added spaces.

Answer 10

I'm not going to code this, but one way that definitely works for the above data set is by parsing it by white space and then assigning elements this way:
someArray = array of strings that were split by white space
Pos = someArray[0]
Car = someArray[1]
Competitor/Team = someArray[2] + " " + someArray[3]
Driver = someArray[4] + " " + someArray[5]
Vehicle = someArray[6] + " " + ... + " " + someArray[someArray.length - 6]
Cap = someArray[someArray.length - 5]
CL Laps = someArray[someArray.length - 4]
Race.Time = someArray[someArray.length - 3]
Fastest...Lap = someArray[someArray.length - 2] + " " + someArray[someArray.length - 1]
The vehicle part can be done by some sort of for or while loop.