Regular expression - Ruby vs Perl - ruby

Question

I noticed some extreme delays in my Ruby (1.9) scripts and after some digging it boiled down to regular expression matching. I'm using the following test scripts in Perl and in Ruby:
Perl:
$fname = shift(#ARGV);
open(FILE, "<$fname" );
while (<FILE>) {
if ( /(.*?) \|.*?SENDING REQUEST.*?TID=(.*?),/ ) {
print "$1: $2\n";
}
}
Ruby:
f = File.open( ARGV.shift )
while ( line = f.gets )
if /(.*?) \|.*?SENDING REQUEST.*?TID=(.*?),/.match(line)
puts "#{$1}: #{$2}"
end
end
I use the same input for both scripts, a file with only 44290 lines.
The timing for each one is:
Perl:
xenofon#cpm:~/bin/local/project$ time ./try.pl input >/dev/null
real 0m0.049s
user 0m0.040s
sys 0m0.000s
Ruby:
xenofon#cpm:~/bin/local/project$ time ./try.rb input >/dev/null
real 1m5.106s
user 1m4.910s
sys 0m0.010s
I guess I'm doing something awfully stupid, any suggestions?
Thank you

Answer 1

regex = Regexp.new(/(.*?) \|.*?SENDING REQUEST.*?TID=(.*?),/)
f = File.open( ARGV.shift ).each do |line|
if regex .match(line)
puts "#{$1}: #{$2}"
end
end
Or
regex = Regexp.new(/(.*?) \|.*?SENDING REQUEST.*?TID=(.*?),/)
f = File.open( ARGV.shift )
f.each_line do |line|
if regex.match(line)
puts "#{$1}: #{$2}"
end

Answer 2

One possible difference is the amount of backtracking being performed. Perl might do a better job of pruning the search tree when backtracking (i.e. noticing when part of a pattern can't possibly match). Its regex engine is highly optimised.
First, adding a leading «^» could make a huge difference. If the pattern doesn't match starting at position 0, it's not going to match at starting position 1 either! So don't try to match at position 1.
Along the same lines, «.*?» isn't as limiting as you might think, and replacing each instance of it with a more limiting pattern could prevent a lot of backtracking.
Why don't you try:
/
^
(.*?) [ ]\|
(?:(?!SENDING[ ]REQUEST).)* SENDING[ ]REQUEST
(?:(?!TID=).)* TID=
([^,]*) ,
/x
(Not sure if it was safe to replace the first «.*?» with «[^|]», so I didn't.)
(At least for patterns that match a single string, (?:(?!PAT).) is to PAT as [^CHAR] is to CHAR.)
Using /s could possibly speed things up if «.» is allowed to match newlines, but I think it's pretty minor.
Using «\space» instead of «[space]» to match a space under /x might be slightly faster in Ruby. (They're the same in recent versions of Perl.) I used the latter because it's far more readable.

Answer 3

From the perlretut chapter: Using regular expressions in Perl section - "Search and replace"
(Even though the regular expression appears in a loop, Perl is smart enough to compile it only once.)
I don't know Ruby very good, but I suspect that it does compile the regex in each cycle.
(Try the code from LaGrandMere's answer to verfiy it).

Answer 4

Try using the (?>re) Extension. See Ruby-Documentation for Details, here a Quote:
This construct [..] inhibits backtracking, which can be a
performance enhancement. For example, the pattern /a.*b.*a/ takes
exponential time when matched against a string containing an a
followed by a number of bs, but with no trailing a. However,
this can be avoided by using a nested regular expression
/a(?>.*b).*a/.
File.open(ARGV.shift) do |f|
while line = f.gets
if /(.*?)(?> \|.*?SENDING REQUEST.*?TID=)(.*?),/.match(line)
puts "#{$1}: #{$2}"
end
end
end

Answer 5

Ruby:
File.open(ARGV.shift).each do |line|
if line =~ /(.*?) \|.*?SENDING REQUEST.*?TID=(.*?),/
puts "#{$1}: #{$2}"
end
end
Change match method to =~ operator. It is faster because:
(Ruby has Benchmark. I don't know your file content so I randomly typed something)
require 'benchmark'
def bm(n)
Benchmark.bm do |x|
x.report{n.times{"asdfajdfaklsdjfklajdklfj".match(/fa/)}}
x.report{n.times{"asdfajdfaklsdjfklajdklfj" =~ /fa/}}
x.report{n.times{/fa/.match("asdfajdfaklsdjfklajdklfj")}}
end
end
bm(100000)
Output report:
user system total real
0.141000 0.000000 0.141000 ( 0.140564)
0.047000 0.000000 0.047000 ( 0.046855)
0.125000 0.000000 0.125000 ( 0.124945)
The middle one is using =~. It takes less than 1/3 of others. Other two are using match method. So, use =~ in your code.

Answer 6

Regular expression matching is time-consuming compared to other forms of matching. Since you are expecting a long, static string in the middle of your matching lines, try filtering out lines that don't include that string by using relatively-cheap string operations. That should result in less that needs to go through regular expression parsing (depending on what your input looks like, of course).
f = File.open( ARGV.shift )
my_re = Regexp.new(/(.*?) \|.*?SENDING REQUEST.*?TID=(.*?),/)
while ( line = f.gets )
continue if line.index('SENDING REQUEST') == nil
if my_re.match(line)
puts "#{$1}: #{$2}"
end
end
f.close()
I haven't benchmarked this particular version since I don't have your input data. I have had success doing things like this in the past, though, especially with lengthy logfiles where pre-filtering can eliminate the vast majority of the input without running any regular expressions.