Extracting unique words - ruby

Question

I need to take a file name and an integer N, and return the first N unique words in the file given. Let us say that input.txt has this content:
I like pancakes in my breakfast. Also, I like pancakes in my dinner.
The output of running this with N = 13 could be
I
like
pancakes
in
my
breakfast.
Also,
dinner.
I know how to open the file and read line by line, but beyond that, I don't know how to take the unique words out if the lines.

Answer 1

Let's first create a test file.
str =<<END
We like pancakes for breakfast,
but we know others like waffles.
END
FName = 'temp'
File.write(FName, str)
#=> 65 (characters written)
We need to return an array containing the first nbr_unique unique words from the file named file, so let's write a method that will do that.
def unique_words(fname, nbr_unique)
<code needed here>
end
You need to add unique words to an array that will be returned by this method, so let's begin by creating an empty array and then return that array at the end of the method.
def unique_words(fname, nbr_unique)
arr = []
<code needed here>
arr
end
You know how to read a file line-by-line, so let's do that, using the class method IO::foreach1.
def unique_words(fname, nbr_unique)
arr = []
File.foreach(fname) do |line|
<code need here to process line>
end
arr
end
The block variable line equals "We like pancakes for breakfast,\n" after the first line is read. Firstly, the newline character needs to be removed. Examine the methods of the class
String to see if one can be used to do that.
The second line contains the word "we". I assume "We" and "we" are not to be regarded as unique words. This is usually handled by converting all characters of a string to either all lowercase or all uppercase. You can do this to each line or to each word (after words have been extracted from a line). Again, look for a suitable method in the class String for doing this.
Next you need to extract words from each line. Once again, look for a String method for doing that.
Next we need to determine if, say, "like" (or "LIKE") is to be added to the array arr. Look at the instance methods for the class Array for a suitable method. If it is added we need to see if arr now contains nbr_unique words. If it does we don't need to read any more lines of the file, so we need to break out of foreach's block (perhaps use the keyword break).
There's one more thing we need to take care of. The first line contains "breakfast,", the second, "waffles.". We obviously don't want the words returned to contain punctuation. There are two ways to do that. The first is to remove the punctuation, the second is to accept only letters.
Given a string that contains punctuation (a line or a word) we can create a second string that equals the original string with the punctuation removed. One way to do that is to use the method String#tr. Suppose the string is "breakfast,". Then
"breakfast,".tr(".,?!;:'", "") #=> "breakfast"
To only accept letters we could use any of the following regular expressions (all return "breakfast"):
"breakfast,".gsub(/[a-zA-Z]+/, "")
"breakfast,".gsub(/[a-z]+/i, "")
"breakfast,".gsub(/[[:alphaa:]]+/, "")
"breakfast,".gsub(/\p{L}+/, "")
The first two work with ASCII characters only. The third (POSIX) and fourth work (\p{} construct) with Unicode (search within Regexp).
Note that it is more efficient to remove punctuation from a line before words are extracted.
Extra credit: use Enumerator#with_object
Whenever you see an object (here arr) initialized to be be empty, manipulated and then returned at the end of a method, you should consider using the method Enumerator#with_object or (more commonly), Enumerable#each_with_object. Both of these return the object referred to in the method name.
The method IO::foreach returns an enumerator (an instance of the class Enumerator) when it does not have a block (see doc). We therefore could write
def unique_words(fname, nbr_unique)
File.foreach(fname).with_object([]) do |line, arr|
<code need here to process line>
end
end
We have eliminated two lines (arr = [] and arr), but have also confined arr's scope to the block. This is not a big deal but is the Ruby way.
More extra credit: use methods of the class Set
Suppose we wrote the following.
require 'set'
def unique_words(fname, nbr_unique)
File.foreach(fname).with_object(Set.new) do |line, set|
<code need here to process line>
end.to_a
end
When we extract the word "we" from the second line we need to check if it should be added to the set. Since sets have unique elements we can just try to do it. We won't be able to do that because set will already contain that word from the first line of the file. A handy method for doing that is Set#add?:
set.add?("we")
#=> nil
Here the method returns nil, meaning the set already contains that word. It also tells us that we don't need to check if the set now contains nbr_unique words. Had we been able to add the word to the set, set (with the added word) would be returned.
The block returns the value of set (a set). The method Set#to_a converts that set to an array, which is returned by the method.
1 Notice that I've invoked the class method IO::foreach by writing File.foreach(fname)... below. This is permissible because File is a subclass of IO (File.superclass #=> IO). I could have instead written IO.foreach(fname)..., but it is more common to use File as the receiver.