Ruby chunk and compare two large files - ruby

Question

Looking for direction on how to chunk and compare two large text files using ruby. Any help is appreciated. Something like 100 lines at a time.
tried:
file(file1).foreach.each_slice(100) do |lines|
pp lines
end
getting confused how to include the second file to this loop.

Answer 1

CHUNK_SIZE = 256 # bytes
def same? path1, path2
return false unless [path1, path2].map { |f| File.size f }.reduce &:==
f1, f2 = [path1, path2].map { |f| File.new f }
loop do
s1, s2 = [f1, f2].map { |f| f.read(CHUNK_SIZE) }
break false if s1 != s2
break true if s1.nil? || s1.length < CHUNK_SIZE
end
ensure
[f1, f2].each &:close
end
UPD: credits for fixed typo and file size comparison goes to #tadman.

Answer 2

Just "Process two files at the same time in Ruby" and compare by chunks, like this:
f1 = File.open('file1.txt', 'r')
f2 = File.open('file2.txt', 'r')
f1.each_slice(10).zip(f2.each_slice(10)).each do |line1, line2|
return false unless line1 == line2
end
return true
Or, as suggested by #meagar (in this case line by line):
f1.each_line.zip(f2.each_line).all? { |a,b| a == b }
This will return true if files identical.

Answer 3

Just compare those files line by line:
def same_file?(path1, path2)
file1 = File.open(path1)
file2 = File.open(path2)
return true if File.absolute_path(path1) == File.absolute_path(path2)
return false unless file1.size == file2.size
enum1 = file1.each
enum2 = file2.each
loop do
# It's a mystery that the loop really ends
# when any of the 2 files has nothing to read
return false unless enum1.next == enum2.next
end
return true
ensure
file1.close
file2.close
end
I did my homework and found in the Kernel#loop documentation:
StopIteration raised in the block breaks the loop. In this case, loop returns the "result" value stored in the exception.
And, in the Enumerator#next documentation:
When the position reached at the end, StopIteration is raised.
So the mystery is no longer a mystery for me.

Answer 4

Here's another one, the approach is similar to mudasobwa's answer:
def same?(file_1, file_2)
return true if File.identical?(file_1, file_2)
return false unless File.size(file_1) == File.size(file_2)
buf_size = 2 ** 15 # 32 K
buf_1 = ''
buf_2 = ''
File.open(file_1) do |f1|
File.open(file_2) do |f2|
while f1.read(buf_size, buf_1) && f2.read(buf_size, buf_2)
return false unless buf_1 == buf_2
end
end
end
true
end
In the first two lines perform quick checks for identical files (e.g. hard and soft links) and same size using File.identical? and File.size.
File.open opens each file in read-only mode. The while loop then keeps calling read to read 32K chunks from each file into the buffers buf_1 and buf_2 until EOF. If the buffers differ, false is returned. Otherwise, i.e. without encountering any differences, true is returned.

Answer 5

To determine if two files have the exact same content, without comparing the actual content of the same chunk of each file, you can use a checksum function that turns the data into a hash string in a deterministic way. And while you have to read the contents to checksum it, you can get checksums for each slice, and end up with an array of checksums for each file.
You can then compare the collection of checksums. If the two files have the exact same content, the two collections will be equal.
require 'digest/md5'
hashes1 = File.foreach('./path_to_file').each_slice(100).map do |slice|
Digest::MD5.hexdigest(slice)
end
hashes2 = File.read('./path_to_duplicate').each_slice(100).map do |slice|
Digest::MD5.hexdigest(slice)
end
hashes1.join == hashes2.join
#=> true, meaning the two files contain the same content

Answer 6

Benchmark time
(Matt's answer is not included because I couldn't get it working)
Results 1 KB file size (N = 10000)
user system total real
aetherus 0.510000 0.300000 0.810000 ( 0.823201)
meagar 0.350000 0.160000 0.510000 ( 0.512755)
mudasobwa 0.290000 0.200000 0.490000 ( 0.500831)
stefan 0.150000 0.160000 0.310000 ( 0.312743)
yevgeniy_anfilofyev 0.320000 0.170000 0.490000 ( 0.497157)
Results 1 MB file size (N = 100)
user system total real
aetherus 1.540000 0.110000 1.650000 ( 1.667937)
meagar 1.170000 0.130000 1.300000 ( 1.310278)
mudasobwa 1.470000 0.830000 2.300000 ( 2.313481)
stefan 0.010000 0.030000 0.040000 ( 0.045577)
yevgeniy_anfilofyev 0.570000 0.100000 0.670000 ( 0.677226)
Results 1 GB file size (N = 1)
user system total real
aetherus 15.570000 0.920000 16.490000 ( 16.525826)
meagar 24.170000 1.910000 26.080000 ( 26.190057)
mudasobwa 16.260000 8.160000 24.420000 ( 24.471977)
stefan 0.120000 0.330000 0.450000 ( 0.443074)
yevgeniy_anfilofyev 12.940000 1.310000 14.250000 ( 14.295736)
Notes
mudasobwa's code runs significantly faster with larger CHUNK_SIZE
with identical chunk sizes, stefan's code seems to be ~2x faster than mudasobwa's code
"fastest" chunk size is somewhere between 16 K and 512 K
I couldn't use fruity because the 1 GB test would have taken too long
Code
def aetherus_same?(f1, f2)
enum1 = f1.each
enum2 = f2.each
loop do
return false unless enum1.next == enum2.next
end
return true
end
def meagar_same?(f1, f2)
f1.each_line.zip(f2.each_line).all? { |a,b| a == b }
end
CHUNK_SIZE = 256 # bytes
def mudasobwa_same?(f1, f2)
loop do
s1, s2 = [f1, f2].map { |f| f.read(CHUNK_SIZE) }
break false if s1 != s2
break true if s1.nil? || s1.length < CHUNK_SIZE
end
end
def stefan_same?(f1, f2)
buf_size = 2 ** 15 # 32 K
buf_1 = ''
buf_2 = ''
while f1.read(buf_size, buf_1) && f2.read(buf_size, buf_2)
return false unless buf_1 == buf_2
end
true
end
def yevgeniy_anfilofyev_same?(f1, f2)
f1.each_slice(10).zip(f2.each_slice(10)).each do |line1, line2|
return false unless line1 == line2
end
return true
end
FILE1 = ARGV[0]
FILE2 = ARGV[1]
N = ARGV[2].to_i
def with_files
File.open(FILE1) { |f1| File.open(FILE2) { |f2| yield f1, f2 } }
end
require 'benchmark'
Benchmark.bm(19) do |x|
x.report('aetherus') { N.times { with_files { |f1, f2| aetherus_same?(f1, f2) } } }
x.report('meagar') { N.times { with_files { |f1, f2| meagar_same?(f1, f2) } } }
x.report('mudasobwa') { N.times { with_files { |f1, f2| mudasobwa_same?(f1, f2) } } }
x.report('stefan') { N.times { with_files { |f1, f2| stefan_same?(f1, f2) } } }
x.report('yevgeniy_anfilofyev') { N.times { with_files { |f1, f2| yevgeniy_anfilofyev_same?(f1, f2) } } }
end