Unique frequency of occurence - ruby

Question

For a project for class, we are supposed to take a published paper and create an algorithm to create a list of all words in the unit of text while excluding the stop words. I am trying to produce a list of all unique words (in the entire text) along with their frequency of occurrence. This is the algorithm that I created for one line of the text:
x = l[125] #Selecting specific line in the text
p = Array.new() # Assign new array to variable p
p = x.split # Split the array
for i in (0...p.length)
if(p[i] != "the" and p[i] != "to" and p[i] != "union" and p[i] != "political")
print p[i] + " "
end
end
puts
The output of this program is one sentence (from line 125) excluding the stop words. Should I use bubble sort? How would I modify it to sort strings of equal length (or is that irrelevant)?

Answer 1

I'd say you have a good start, considering you are new to Ruby. You asked if you should use a bubble sort. I guess you're thinking of grouping multiple occurrences of a word, then go through the array to count them. That would work, but there are a couple of other approaches that are easier and more 'Ruby-like'. (By that I mean they make use of powerful features of the language and at the same time are more natural.)
Let's focus on counting the unique words in a single line. Once you can do that, you should be able to easily generalize that for multiple lines.
First Method: Use a Hash
The first approach is to use a hash. h = {} creates a new empty one. The hash's keys will be words and its values will be the number of times each word is present in the line. For example, if the word "cat" appears 9 times, we will have h["cat"] = 9, just what you need. To construct this hash, we see if each word w in the line is already in hash. It is in the hash if
h[w] != nil
If it is, we increment the word count:
h[w] = h[w] + 1
or just
h[w] += 1
If it's not in the hash, we add the word to the hash like this:
h[w] = 1
That means we can do this:
if h[w]
h[w] += 1
else
h[w] = 1
end
Note that here if h[w] is the same as if h[w] != nil.
Actually, we can use a trick to make this even easier. If we create the hash like this:
h = Hash.new(0)
then any key we add without a value will be assigned a default value of zero. That way we don't have to check if the word is already in the hash; we simply write
h[w] += 1
If w is not in the hash, h[w] will add it and initialize it to 0, then += 1 will increment it to 1. Cool, eh?
Let's put all this together. Suppose
line = "the quick brown fox jumped over the lazy brown fox"
We convert this string to an array with the String#split method:
arr = line.split # => ["the", "quick", "brown", "fox", "jumped", \
"over", "the", "lazy", "brown", "fox"]
then
h = Hash.new(0)
arr.each {|w| h[w] += 1}
h # => {"the"=>2, "quick"=>1, "brown"=>2, "fox"=>2, "jumped"=>1, "over"=>1, "lazy"=>1}
We're done!
Second Method: use the Enumerable#group_by method
Whenever you want to group elements of an array, hash or other collection, the group_by method should come to mind.
To apply group_by to the quick, brown fox array, we provide a block that contains the grouping criterion, which in this case is simply the words themselves. This produces a hash:
g = arr.group_by {|e| e}
# => {"the"=>["the", "the"], "quick"=>["quick"], "brown"=>["brown", "brown"], \
# "fox"=>["fox", "fox"], "jumped"=>["jumped"], "over"=>["over"], "lazy"=>["lazy"]}
The next thing to do is convert the hash values to the number of occurrences of the word (e.g., convert ["the", "the"] to 2). To do this, we can create a new empty hash h, and add hash pairs to it:
h = {}
g.each {|k,v| h[k] = v.size}
h # => {"the"=>2, "quick"=>1, "brown"=>2, "fox"=>2, "jumped"=>1, "over"=>1, "lazy"=>1
One More Thing
You have this code snippet:
if(p[i] != "the" and p[i] != "to" and p[i] != "union" and p[i] != "political")
print p[i] + " "
end
Here are a couple of ways you could make this a little cleaner, both using the hash h above.
First Way
skip_words = %w[the to union political] # => ["the", "to", "union", "political"]
h.each {|k,v| (print v + ' ') unless skip_words.include?(k)}
Second Way
h.each |k,v|
case k
when "the", "to", "union", "political"
next
else
puts "The word '#{k}' appears #{v} times."
end
end
Edit to address your comment. Try this:
p = "The quick brown fox jumped over the quick grey fox".split
freqs = Hash.new(0)
p.each {|w| freqs[w] += 1}
sorted_freqs = freqs.sort_by {|k,v| -v}
sorted_freqs.each {|word, freq| puts word+' '+freq.to_s}
=>
quick 2
fox 2
jumped 1
The 1
brown 1
over 1
the 1
grey 1
Normally, ypu would not sort a hash; rather you'd first convert it to an array:
sorted_freqs = freqs.to_a.sort_by {|x,y| v}.reverse
or
sorted_freqs = freqs.to_a.sort_by {|x,y| -v}
Now sorted_freqs is an array, rather than a hash. The last line stays the same. In general, it's best not to rely on a hash's order. In fact, before Ruby version 1.9.2, hashes were not ordered. If order is important, use an array or convert a hash to array.
Having said that, you can sort smallest-to-largest on the hash values, or (as I have done), sort largest-to-smallest on the negative of the hash values. Note that there is no Enumerable#reverse or Hash#reverse. Alternatively (always many ways to skin a cat with Ruby), you could sort on v and then use Enumerable#reverse_each:
sorted_freqs.reverse_each {|word, freq| puts word+' '+freq.to_s}
Lastly, you could eliminate the temporary variable sorted_freqs (needed because there is no Enumerable#sort_by! method), by chaining the last two statements:
freqs.sort_by {|k,v| -v}.each {|word, freq| puts word+' '+freq.to_s}

Answer 2

You should really look into Ruby's enumerable class. you very seldom do for x in y in ruby.
word_list = ["the", "to", "union", "political"]
l[125].split.each do |word|
print word + " " unless word_list.include?(word)
end
In order to count, sort and all that stuff look into the group_by method and perhaps the sort_by method of arrays.