elasticsearch - number of searches affects revelance? - elasticsearch

Question

I have the following mapping:
POST music
{
"settings": {
"analysis": {
"filter": {
"nGram_filter": {
"type": "nGram",
"min_gram": 2,
"max_gram": 20,
"token_chars": [
"letter",
"digit",
"punctuation",
"symbol"
]
}
},
"analyzer": {
"nGram_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"asciifolding",
"nGram_filter"
]
},
"whitespace_analyzer": {
"type": "custom",
"tokenizer": "whitespace",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"song": {
"properties": {
"song_field": {
"type": "string",
"analyzer": "nGram_analyzer",
"search_analyzer": "whitespace_analyzer"
}
}
}
}
}
I've inserted two docs:
POST music/song
{
"song_field" : "Premeditiated murder"
}
POST music/song
{
"song_field" : "Premeditiated"
}
Here is the query:
POST music/song/_search
{
"size": 10,
"query": {
"match": {
"song_field": {
"query": "Premeditiated murd",
"fuzziness": 2
}
}
}
}
Response:
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0.78730416,
"hits": [
{
"_index": "music",
"_type": "song",
"_id": "AVUf6XK1ancUpEdFLdz8",
"_score": 0.78730416,
"_source": {
"song_field": "Premeditiated"
}
},
{
"_index": "music",
"_type": "song",
"_id": "AVUfUbocancUpEdFLdUf",
"_score": 0.668494,
"_source": {
"song_field": "Premeditiated murder"
}
}
]
}
}
I have two questions:
Why does Premeditiated score is higher ? How can I get a resonable correction + auto-complete?
Does searching the same document over and over again affects default es score ?

Answer 1

You get wrong response because sorting by relevance is broken for very small set of data when you have multiple shareds. Relevance is calculated for each shared and then results from each shared are merged and return so your "Premeditiated" has bigger relevance in one shared. This is a common issue and is well described here: https://www.elastic.co/guide/en/elasticsearch/guide/current/relevance-is-broken.html
There are two ways to solve this issue you can use:
1. number_of_shards option =1 during defining index mapping
2. add the following information to your search query: search_type=dfs_query_then_fetch
After using one of the above options you will get a result you want.
Regarding your second question: every time when you search scoring is calculated. Even if you are searching the same document over and over again the scoring is calculated and _score result is always the same. If you want to read more how scoring works then you need to read "Controlling relevance" chapter https://www.elastic.co/guide/en/elasticsearch/guide/current/controlling-relevance.html. You can always add explain property to your query to see how scroing was calculated https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-explain.html.
P.S
Great that you provided your JSONs but there is a wrong field inside query it should be song_field instead of song_field_1. Additionaly your response doesn’t fit to data stored inside type (look at _source field in the respown) but it doesn't matter here:P.