Segmentation fault in bash script using find, grep, sed - bash

Question

I have a script that searches through a very large number of files, and uses sed to substitute a multiple line pattern. The script is iterative, and it works fine on some iterations but sometimes it causes a segmentation fault.
This is what the script is doing:
Search for files that DON'T contain the string X
Out of these files, search the ones that CONTAIN the string Y
Iterate the returned file list with a for-loop
If the file contents match pattern A, replace pattern A with A_TAG
The same for patterns B,C,D (a file can contain only one of A,B,C,D)
Patterns A,B,C,D are multiline, and they are replaced with two lines. X and Y are single line.
Here's the script. I apologise for the long lines, but I decided not to edit them since they're regex. I did however shorten the regex by replacing strings with "pattern" - the replaced contents are NOT the same in every regex, but they don't have any special characters so I don't think the actual contents are relevant to this question. Besides, the regex has been shown to work so you probably don't need to fully understand it..
#!/bin/sh
STRING_A="Pattern(\n|.)*Pattern\.\""
A_TAG="\$STRING:A$"
STRING_B="(Pattern(\n|.)*)?(Pattern(\n|.)*)?Pattern(\n|.)*Pattern(\n|.)*Pattern\.((\n|.)*will be met\: http\:\/\/www.foo\.org\/example\/temp\.html\.\n)?"
B_TAG="\$STRING:B$"
STRING_C="(Pattern(\n|.)*)?Pattern(\n|.)*http\:\/\/www\.foo\.org\/bar\/old-foobar\/file\-2\.1\.html\.((\n|.)*Pattern.*Pattern)?"
C_TAG="\$STRING:C$"
STRING_D="(Pattern(\n|.)*)?(Pattern(\n|.)*http\:\/\/www\.foo\.org\/bar\/old-foobar\/file\-2\.1\.html.*|Pattern(\n|.)*Pattern)((\n|.)*http\:\/\/www\.some-site\.org/\.)?"
D_TAG="\$STRING:D$"
## params: #1 file, #2 PATTERN, #3 TAG
multil_sed()
{
echo "In multil_sed"
# -n = silent, -r = extended regex, -i = inline changes
sed -nr '
# Sed has a hold buffer that we can use to "keep text in memory".
# Here we copy the line to the buffer if it is the first line of the file,
# or append it if it is not
1h
1!H
# We must first save all lines until the nth line to the hold buffer,
# then we can search for our pattern
60 {
# Then we must use the pattern buffer. Pattern buffer holds text that
# is up for modification. With g we can hopy the hold buffer into the pattern space
g
# Now we can just use the substitution command as we normally would. Use # as a delimiter
s#([ \t:#*;/".\\-]*)'"$2"'#\1'"$3"'\
\1$QT_END_LICENSE$#Ig
# Finally print what we did
p
}
' $1 > $1.foo;
echo "Done"
}
for p in $(find . -type f -not -iwholename '*.git*' -exec grep -iL '.*STRING_X.*' {} \; | xargs grep -il -E '.*STRING_Y.*')
do
echo
echo "####################"
echo "Working on file" $p
#Find A
if pcregrep -qiM "$STRING_A" "$p";
then
echo "A"
multil_sed "$p" "$STRING_A" "$A_TAG"
#Find B
elif pcregrep -qiM "$STRING_B" "$p";
then
echo "B"
multil_sed "$p" "$STRING_B" "$B_TAG"
#Find C
elif pcregrep -qiM "$STRING_C" "$p";
then
echo "C"
multil_sed "$p" "$STRING_C" "$C_TAG"
#Find D
elif pcregrep -qiM "$STRING_D" "$p";
then
echo "D"
multil_sed "$p" "$STRING_D" "$D_TAG"
else
echo "No match found"
fi
echo "####################"
done
I should probably note that C is essentially a longer version of D, that has some extra contents before the common part.
What happens is that for some iterations this works ok..
####################
Working on file ./src/listing.txt
A
In multil_sed
Done
####################
and sometimes it doesn't.
####################
Working on file ./src/web/page.html
/home/tekaukor/code/project/tag_adder.sh: line 54: 16904 Segmentation fault (core dumped) pcregrep -qiM "$STRING_A" "$p"
No match found
####################
It's not dependent on which pattern is being searched.
####################
Working on file ./src/test/formatter_test.cpp
/home/tekaukor/code/project/tag_adder.sh: line 54: 18051 Segmentation fault (core dumped) pcregrep -qiM "$STRING_B" "$p"
/home/tekaukor/code/project/tag_adder.sh: line 54: 18053 Segmentation fault (core dumped) pcregrep -qiM "$STRING_C" "$p"
/home/tekaukor/code/project/tag_adder.sh: line 54: 18055 Segmentation fault (core dumped) pcregrep -qiM "$STRING_D" "$p"
No match found
####################
Line 54 points to the line "for p in $(find . -type f -not -iwholename '.git' -exec grep...".
My guess is that sed is causing a buffer overflow, but I haven't found a way to ascertain or fix this.

Answer 1

Bash isn't great about locating the source of a fault in a compound statement so
Line 54 points to the line for p in $(find . -type f ....
is misleading as the error could be anywhere in that for statement block. The error message
Segmentation fault (core dumped) pcregrep -qiM "$STRING_D" "$p"
is much more accurate. And likely the cause of the fault is the -M flag combined with unbounded patterns like (.|\n)* As the pcregrep man page notes:
-M, --multiline
Allow patterns to match more than one line. When this option is given, patterns may usefully contain literal newline characters and internal occurrences of ^ and $ characters. The output for any one match may consist of more than one line. When this option is set, the PCRE library is called in "multiline" mode. There is a limit to the number of lines that can be matched, imposed by the way that pcregrep buffers the input file as it scans it. However, pcregrep ensures that at least 8K characters or the rest of the document (whichever is the shorter) are available for forward matching, and similarly the previous 8K characters (or all the previous characters, if fewer than 8K) are guaranteed to be available for lookbehind assertions.
with emphasis mine. The single pattern fragment .* or (.|\n)* can literally match an entire file, so yes, it will fill up its lookahead buffer not just to the next literal (e.g. http) but until it finds the last such literal, because by default regular expressions seek the longest conforming match.

Answer 2

UPDATE #2: So apparently sed doesn't support non greedy matching, which makes part of my answer invalid. There are ways around this, but I will not include them here as it's far removed from the original question. The answer to this question is using the --disable-stack-for-recursion flag as described below.
The answer by msw helped me in the right direction.
First I changed the regex to be lazy instead of greedy. By default regex is greedy, which (as msw stated) means that a multiline expression with "PATTERN(.|\n)*TEXT" will search through the whole file. By adding "?" after quantifiers (* -> *?) I made the regez lazy, which means that the "(.|\n)*?" in "PATTERN(.|\n)*?TEXT" will stop expanding at the first TEXT.
I also made the optional parts lazy (? -> ??), though I'm not sure if this was necessary.
However this was not enough. I also had to configure pcregrep to use heap instead of stack memory. I downloaded pcre and configured using the flag --disable-stack-for-recursion. Note that using heap is much slower, so you shouldn't do this if you don't have to.
I'm including a step-by-step in case anyone wonders here with the same problem. Note that I'm still a linux newb and there's a high chance that I made something unnecessary and/or stupid. The instructions are based on http://www.mail-archive.com/pcre-dev#exim.org/msg00817.html and http://www.linuxfromscratch.org/blfs/view/svn/general/pcre.html
Download pcre from http://downloads.sourceforge.net/pcre/pcre-8.33.tar.bz2
tar jxf pre-8.33.tar.bz2
cd pcre-8.33
./configure --prefix=/usr --docdir=/usr/share/doc/pcre-8.33 --enable-utf --enable-unicode-properties --enable-pcregrep-libz2 --disable-static --disable-stack-for-recursion
make
sudo make install
There are some additional steps in the provided guide, but I didn't have to do them.
UPDATE: Making the optional elements lazy (? -> ??) is a mistake, as then they will not be included in the matched pattern if possible.