Elasticsearch succeeds for partial match and fails for exact match - elasticsearch

Question

I have a value in the "handle" field of my users index. Let's say the exact string of this value is "exactstring". My query will return the correct result for "exactstrin" but it will fail (return nothing) for "exactstring". N-grams, like "actstr", also return the correct result. WTF?
{
"query": {
"bool": {
"must": [{
"multi_match": {
"fields": ["handle","name","bio"],
"prefix_length": 2,
"query": "exacthandle",
"fuzziness": "AUTO"
}
}],
"should" : [{
"multi_match": {
"fields": ["handle^6", "name^3", "bio"],
"query": "exacthandle"
}
}]
}
},
"size": 100,
"from": 0
}
Here's my settings:
"settings": {
"analysis": {
"analyzer": {
"search_term_analyzer": {
"type": "custom",
"stopwords": "_none_",
"filter": [
"standard",
"lowercase",
"asciifolding",
"no_stop"
],
"tokenizer": "whitespace"
},
"ngram_token_analyzer": {
"type": "custom",
"stopwords": "_none_",
"filter": [
"standard",
"lowercase",
"asciifolding",
"no_stop",
"ngram_filter"
],
"tokenizer": "whitespace"
}
},
"filter": {
"no_stop": {
"type": "stop",
"stopwords": "_none_"
},
"ngram_filter": {
"type": "nGram",
"min_gram": "2",
"max_gram": "9"
}
}
}
}
And my mappings:
"handle": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"bio": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}

Answer 1

This must be happening because you have set less value for max_gram(Maximum length of characters in a gram). When you are querying for exactstring, then the tokens generated for exactstring, will have a maximum token of length 9 (i.e exactstri). Due to this, the query will match on exactstri and not on exactstring.
You need to specify the analyzer of the field. When mapping an
index, you can use the analyzer mapping parameter to specify an
analyzer for each text field.If none of the analyzers is specified,
then the standard analyzer is used.
You also need to increase the max_gram to at least 11 for ngram_token_analyzer in your index setting. And also set "max_ngram_diff": "20". The tokens generated (which you can check using Analyze API) will have exactstrin as well as exactstring and now your query will match on both exactstrin and exactstring
Index Mapping:
{
"settings": {
"analysis": {
"analyzer": {
"search_term_analyzer": {
"type": "custom",
"stopwords": "_none_",
"filter": [
"lowercase",
"asciifolding",
"no_stop"
],
"tokenizer": "whitespace"
},
"ngram_token_analyzer": {
"type": "custom",
"stopwords": "_none_",
"filter": [
"lowercase",
"asciifolding",
"no_stop",
"ngram_filter"
],
"tokenizer": "whitespace"
}
},
"filter": {
"no_stop": {
"type": "stop",
"stopwords": "_none_"
},
"ngram_filter": {
"type": "nGram",
"min_gram": "2",
"max_gram": "11" <-- note this
}
}
},
"max_ngram_diff": 20
},
"mappings": {
"properties": {
"handle": {
"type": "text",
"analyzer": "ngram_token_analyzer"
}
}
}
}
Index Data:
{
"handle":"exactstring"
}
Search Query:
{
"query": {
"bool": {
"must": [
{
"multi_match": {
"fields": [
"handle",
"name",
"bio"
],
"prefix_length": 2,
"query": "exactstring",
"fuzziness": "AUTO"
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "64957062",
"_type": "_doc",
"_id": "1",
"_score": 0.6292809,
"_source": {
"handle": "exactstring"
}
}
]

Answer 2

I ended up using Multi-fields to solve this. Querying the field "handle.text" now returns exact string matches of arbitrary length.
I was also able to create ngram and edge_ngram filters with different min and max grams and then use them simultaneously.
The mapping looks like this now:
"handle": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
},
"text": {
"type": "text"
},
"edge_ngram": {
"type": "text",
"analyzer": "edge_ngram_token_analyzer"
},
"ngram": {
"type": "text",
"analyzer": "ngram_token_analyzer"
}
}
}
The multi-match query can now simultaneously use different analyzers on the same field (the big difference between the must and should queries comes down to fuzziness, the bio field tends to contain a lot of small tokens and fuzziness produces a lot of bad hits):
{
"query": {
"bool": {
"must" : [{
"multi_match": {
"fields": [
"handle.text^6",
"handle.edge_ngram^4",
"handle.ngram^2",
"bio.text^6",
"bio.keyword^4",
"name.text^6",
"name.keyword^5",
"name.edge_ngram^2",
"name.ngram^3"
],
"query": "exactstring",
"prefix_length": 2
}
}],
"should" : [{
"multi_match": {
"fields": [
"handle.text^6",
"handle.edge_ngram^2",
"handle.ngram^1",
"name.text^6",
"name.keyword^5",
"name.edge_ngram^2",
"name.ngram^3"
],
"query": "exactstring",
"prefix_length": 2,
"fuzziness": "AUTO"
}
}]
}
},
"size": 100,
"from": 0
}
For completeness, here are the settings of my index:
"settings": {
"analysis": {
"analyzer": {
"search_term_analyzer": {
"type": "custom",
"stopwords": "_none_",
"filter": [
"standard",
"lowercase",
"asciifolding",
"no_stop"
],
"tokenizer": "whitespace"
},
"ngram_token_analyzer": {
"type": "custom",
"stopwords": "_none_",
"filter": [
"standard",
"lowercase",
"asciifolding",
"no_stop",
"ngram_filter"
],
"tokenizer": "whitespace"
},
"edge_ngram_token_analyzer": {
"type": "custom",
"stopwords": "_none_",
"filter": [
"standard",
"lowercase",
"asciifolding",
"no_stop",
"edge_ngram_filter"
],
"tokenizer": "whitespace"
}
},
"filter": {
"no_stop": {
"type": "stop",
"stopwords": "_none_"
},
"ngram_filter": {
"type": "nGram",
"min_gram": "3",
"max_gram": "12"
},
"edge_ngram_filter": {
"type": "edgeNGram",
"min_gram": "3",
"max_gram": "50"
}
}
},
"max_ngram_diff": 50
}
h/t to #bhavya for max_ngram_diff