File comparison Ran out of memory - ruby

Question

Hi i am using this code - but for files > 8 Million rows of lines - 2 files passed as text input, the memory runs out. How can i compare both text files which are more than 30 Million lines maybe.
fileA1 = ARGV[0]
fileA2 = ARGV[1]
if ARGV.length != 2
raise 'Send Two files pls'
end
cmd = "sort #{fileA1} > Sorted.txt"
`#{cmd}`
aFile = File.open("Sorted.txt", "r");
bFile = File.open(fileA2, "r").readlines;
fileR = File.open("result.txt", "w")
p aFile.class
p bFile.class
p bFile.length
aFile.each do |e|
if(! bFile.include?(e) )
p 'Able to get differences:' + e.to_s
fileR.write('Does not Include:' + e)
end
end
additional coding i tried without luck.
counterA = counterB = 0
aFile = File.open("Sample1 - Copy.txt", "r");
bFile = File.open("Sample2.txt", "r");
file1lines = aFile.readlines
file2lines = bFile.readlines
file1lines.each do |e|
if(!file2lines.include?(e))
puts e
else
p "Files include these lines:"
end
end
stopTime = Time.now

Answer 1

As a starting point, I would use the diff Unix command (available on Windows as part of Cygwin, etc), and see if that addresses your need:
#!/usr/bin/env ruby
raise "Syntax is comp_files file1 file2" unless ARGV.length == 2
file1, file2 = ARGV
`sort #{file1} > file1_sorted.txt`
`sort #{file2} > file2_sorted.txt`
`diff file1_sorted.txt file2_sorted.txt 2>&1 > diff.txt`
puts 'Created diff.txt.' # After running the script, view it w/less, etc.
Here is a similar script that uses temporary files that are automatically deleted before exiting:
#!/usr/bin/env ruby
raise "Syntax is comp_files file1 file2" unless ARGV.length == 2
require 'tempfile'
input_file1, input_file2 = ARGV
sorted_file1 = Tempfile.new('comp_files_sorted_1').path
sorted_file2 = Tempfile.new('comp_files_sorted_2').path
puts [sorted_file1, sorted_file2]
`sort #{input_file1} > #{sorted_file1}`
`sort #{input_file2} > #{sorted_file2}`
`diff #{sorted_file1} #{sorted_file2} 2>&1 > diff.txt`
puts 'Created diff.txt.' # After running the script, view it w/less, etc.
# The code below can be used to create sample input files
# File.write('input1.txt', (('a'..'j').to_a.shuffle + %w(s y)).join("\n"))
# File.write('input2.txt', (('a'..'j').to_a.shuffle + %w(s t z)).join("\n"))

Answer 2

I believe your problem is with readlines. This method will read the entire file and return a string. Since your file is huge, you will risk running out of memory.
To work with large files, don't read the entire contents at once, but read in pieces as needed.
Also, your algorithm has another problem since the comparison really checks whether all lines in aFile are included in bFile without actually checking for order at all. I'm not sure if that is indeed your intent.
If you really want to compare line by line and if order matters, then your comparison should be line-by-line and you don't have to read the entire file into a string. Use the gets method instead, which by default returns the next line in a file or nil at EOF.
Something like this:
aFile.each do |e|
if e != bFile.gets
p 'Able to get differences:' + e.to_s
fileR.write('Does not Include:' + e)
end
end
On the other hand, if you really want to find if all lines in a are in b, regardless of order, you can do a nested loop, where for each line in a, you iterate all lines of b. Make sure to return on first match to speedy things up since this will be a really expensive operation, but the include call is also expensive so it's probably a tie IMO with the exception of the file IO overhead.

Answer 3

Here is a script that will analyze 2 text files, reporting the first difference, or a difference in the number of lines, or success.
NOTE: THE CODE HERE IS TRUNCATED. PLEASE GO TO https://gist.github.com/keithrbennett/1d043fdf7b685d9692f0181ad68c6307 FOR THE COMPLETE SCRIPT!
#!/usr/bin/env ruby
raise "Syntax is first_diff file1 file2" unless ARGV.size == 2
FILE1, FILE2 = ARGV
ENUM1 = File.new(FILE1).to_enum
ENUM2 = File.new(FILE2).to_enum
def build_unequal_error_message(line_num, line1, line2)
"Difference found at line #{line_num}:
#{FILE1}: #{line1}
#{FILE2}: #{line2}"
end
def build_unequal_line_count_error_message(line_count, file_exhausted)
"All lines up to line #{line_count} were identical, " \
"but #{file_exhausted} has no more text lines."
end
def get_line(file_enumerator)
file_enumerator.next.chomp
end
def has_next(enumerator)
begin
enumerator.peek
true
rescue StopIteration
false
end
end
# Returns an analysis of the results in the form of a string
# if a compare error occurred, else returns nil.
def error_text_or_nil
line_num = 0
loop do
has1 = has_next(ENUM1)
has2 = has_next(ENUM2)
case
when has1 && has2
line1 = get_line(ENUM1)
line2 = get_line(ENUM2)
if line1 != line2
return build_unequal_error_message(line_num, line1, line2)
end
when !has1 && !has2
return nil # if both have no more values, we're done
else # only 1 enum has been exhausted
exhausted_file = has1 ? FILE2 : FILE1
not_exhausted_file = exhausted_file == FILE1 ? FILE2 : FILE1
return build_unequal_line_count_error_message(line_num, exhausted_file)
end
line_num += 1
end
puts "Lines processed successfully: #{line_num}"
end
result = error_text_or_nil
if result
puts result
exit -1
else
puts "Compare successful"
exit 0
end