Number of dict words containing each substring - algorithm

Question

I chanced upon a problem which requires user to count the number of matching words for N substrings in a list of K strings (each string has length M) for the following constraints:
0 < length of substring <= 100
0 < M <= 100
0 < N <= 10,000
0 < K <= 10,000
For example:
substring se for { serpent, lose, last } will yield 2.
Given the enormous bounds of this input, checking all K strings for each substring will be too expensive. KMP would not work for this reason. Preprocessing the strings with suffix tree would be the next better option. But I can't possibly create suffix tree for each word because it would again lead to the above problem. Even if I try to join all the words together, the problem is that I would not be able to detect substrings in the same word (e.g. substring s for {stools, sat} will yield 3).
Is there a more efficient way of solving this problem?

Answer 1

An approach can be to index each chars of the string you want to search in.
"serpent" will give :
s > {0}
e > {1, 4}
r > {2}
p > {3}
n > {5}
t > {6}
From that, for each word you search in this string, look in the dictionary for its first and last char.
If they are found, you'll need to look into the corresponding indexes to find a pair that can match the string length.
Once found, you can do a full string comparison but only in this case.
The code can be like this (c#):
static void Main( string[] args )
{
List<string> words = new List<string>() { "se", "s", "pen", "oo", "st" };
List<int> scores = new List<int>( words.Select( w => 0 ) );
List<string> strings = new List<string>() { "serpent", "lose", "last", "stools", "sat" };
foreach ( var s in strings )
{
var indexes = MakeIndexes( s );
for ( int i = 0 ; i < words.Count ; i++ )
{
scores[i] += Score( words[i], s, indexes );
}
}
}
static int Score( string word, string s, Dictionary<char, List<int>> indexes )
{
int firstPos = 0, lastPos = word.Length - 1;
char first = word[firstPos];
char last = word[lastPos];
List<int> firstIndexes;
if ( indexes.TryGetValue( first, out firstIndexes ) )
{
if ( firstPos == lastPos )
return 1;
else
{
List<int> lastIndexes;
if ( indexes.TryGetValue( last, out lastIndexes ) )
{
int fiPos = 0, liPos = 0;
while ( fiPos < firstIndexes.Count && liPos < lastIndexes.Count )
{
int fi = firstIndexes[fiPos], li = lastIndexes[liPos];
int len = li - fi;
if ( len < lastPos )
liPos++;
else if ( len == lastPos )
{
if ( FullCompare( word, s, fi ) )
return 1;
fiPos++;
liPos++;
}
else
fiPos++;
}
}
}
}
return 0;
}
static bool FullCompare( string word, string s, int from )
{
for ( int i = 0 ; i < word.Length ; i++ )
if ( word[i] != s[i + from] )
return false;
return true;
}
static Dictionary<char, List<int>> MakeIndexes( string s )
{
Dictionary<char, List<int>> result = new Dictionary<char, List<int>>();
for ( int i = 0 ; i < s.Length ; i++ )
{
char c = s[i];
List<int> indexes;
if ( result.TryGetValue( c, out indexes ) == false )
{
indexes = new List<int>();
result.Add( c, indexes );
}
indexes.Add( i );
}
return result;
}