Fuzzy Problems in Solr Filter Query - performance

Question

It would be grateful if somebody can help me with my problem. I have this query:
select?q=city:Frankfurt am Main~&fq=street:Gerhart-Hauptmann-Str.~
This is not working for me. I want to use fuzzy search to catch some user input mistakes.
Here is what I want:
Frankfurt am Main should be searched completely in the field city with fuzzy search
Gerhart-Hauptmann-Str. should be converted into three terms with fuzzy search.
Debug output of what I get actually:
"debug": {
"rawquerystring": "city:Frankfurt am Main~",
"querystring": "city:Frankfurt am Main~",
"parsedquery": city:frankfurt text:am text:Main~2",
"parsedquery_toString": "city:frankfurt text:am text:Main~2",
"explain": {...},
"QParser": "LuceneQParser",
"filter_queries": [
"street:Gerhart-Hauptmann-Str.~"
],
"parsed_filter_queries": [
"street:gerhart-hauptmann-str.~2"
],
I (think) I want this output:
"debug": {
"rawquerystring": "city:Frankfurt am Main~",
"querystring": "city:Frankfurt am Main~",
"parsedquery": city:frankfurt~2 city:am~2 text:Main~2",
"parsedquery_toString": "city:frankfurt~2 city:am~2 text:Main~2",
"explain": {...},
"QParser": "LuceneQParser",
"filter_queries": [
"street:Gerhart-Hauptmann-Str.~"
],
"parsed_filter_queries": [
# My analyser converts Str. to strasse
"street:gerhart~2 street:hauptmann~2 strasse~2"
],
The definition of the fields in the schema.xml
<field name="city" type="admin_name" indexed="true" stored="true" />
<field name="street" type="street_name" indexed="true" stored="true" multiValued="false"/>
<fieldType name="admin_name" class="solr.TextField" >
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.SynonymFilterFactory" synonyms="lang/synonyms_de_admin.txt"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
</analyzer>
</fieldType>
<fieldType name="street_name" class="solr.TextField" >
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
<!-- The StartEndSynonymFilter replaces synonyms which
are at the start or the end of an term. The types
START_SYNONYM or END_SYNONYM will be set. -->
<filter class="my.StartEndSynonymFilterFactory" synonyms="lang/synonyms_de_street.txt"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
</analyzer>
</fieldType>
Is this somehow possible?
If you need additional information to answer, please leave a hint in a comment.

Answer 1

Tokenizing on Hyphens
Have a look at the WordDelimiterFilterFactory:
https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.WordDelimiterFilterFactory
Applying Fuzzy to every single term
DISCLAIMER: I have not yet used fuzzy search in my SOLR setups.
You might have to be careful with tokenizing the city names and applying the fuzzy search to every single token. Your example "Frankfurt am Main" would in this case apply fuzzy search to "am", as well. Please try with parenthesis: (Frankfurt am Main)~ whether this gets you the intended result.
However, in case of names (city or streets) I'm not sure you should be even tokenizing them. Maybe storing them as one case insensitive token and applying the fuzzy search like this "Frankfurt am Main"~ (with quotes in the query) is actually what you need.
Nevertheless, you should try and get it to work in the way you have described it. Then look at the query results. And (maybe in parallel) setup an index where you store the city and street names as single tokens (KeywordTokenizer with lower casing and ascii folding, e.g.) and apply fuzzy search to them as single terms. I would guess that the results will be sharper. But best - try it out and compare.
In addition, I would suggest to try out the (extended or not) DisMax Handler for input without even caring to differentiate between cities and streets on the input side: https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser
With the dismax handler processing the input, you can allow the user to input search terms very freely (like having a single search field where cities and streets can be input in random order and format).