How to make minimum_should_match work with nested mapping? - elasticsearch

Question

I have an issue regarding ElasticSearch and More like this query.
Having mapping:
{
"directory.v1": {
"mappings": {
"profile.event": {
"properties": {
"event": {
"properties": {
"naics": {
"type": "nested",
"properties": {
"type": {
"type": "keyword"
},
"value": {
"type": "keyword"
}
}
}
}
},
"user_id": {
"type": "long"
}
}
}
}
}
}
and document (A) as a source and document (B) to be found with more like this query (for A)
Profile A (used as source):
{
"_index": "directory.v1",
"_type": "profile.event",
"_id": "83731111.559",
"_score": 1,
"_source": {
"user_id": 8373,
"event": {
"naics": [
{
"value": 331,
"type": "naics"
},
{
"value": 74,
"type": "naics"
},
{
"value": 938,
"type": "naics"
},
{
"value": 2048,
"type": "naics"
},
{
"value": 939,
"type": "naics"
},
{
"value": 2049,
"type": "naics"
},
{
"value": 940,
"type": "naics"
},
{
"value": 2050,
"type": "naics"
},
{
"value": 941,
"type": "naics"
},
{
"value": 2051,
"type": "naics"
},
{
"value": 942,
"type": "naics"
},
{
"value": 2052,
"type": "naics"
},
{
"value": 943,
"type": "naics"
},
{
"value": 2053,
"type": "naics"
},
{
"value": 944,
"type": "naics"
},
{
"value": 2054,
"type": "naics"
},
{
"value": 945,
"type": "naics"
},
{
"value": 2055,
"type": "naics"
},
{
"value": 473,
"type": "naics"
},
{
"value": 128,
"type": "naics"
},
{
"value": 10,
"type": "naics"
},
{
"value": 1242,
"type": "naics"
},
{
"value": 472,
"type": "naics"
},
{
"value": 1241,
"type": "naics"
}
]
}
}
}
Profile B:
{
"_index": "directory.v1",
"_type": "profile.event",
"_id": "46124111.559",
"_score": 1,
"_source": {
"user_id": 46124,
"event": {
"naics": [
{
"value": 331,
"type": "naics"
},
{
"value": 74,
"type": "naics"
},
{
"value": 938,
"type": "naics"
},
{
"value": 2048,
"type": "naics"
},
{
"value": 939,
"type": "naics"
},
{
"value": 2049,
"type": "naics"
},
{
"value": 940,
"type": "naics"
},
{
"value": 2050,
"type": "naics"
},
{
"value": 941,
"type": "naics"
},
{
"value": 2051,
"type": "naics"
},
{
"value": 942,
"type": "naics"
},
{
"value": 2052,
"type": "naics"
},
{
"value": 943,
"type": "naics"
},
{
"value": 2053,
"type": "naics"
},
{
"value": 944,
"type": "naics"
},
{
"value": 2054,
"type": "naics"
},
{
"value": 945,
"type": "naics"
},
{
"value": 2055,
"type": "naics"
}
]
}
}
}
where B doc has all elements (naics) included in A document.
So that I really do not understand why for query:
{
"query": {
"nested": {
"path": "event.naics",
"query": {
"more_like_this": {
"like": [
{
"_id": "83731111.559",
"_type": "profile.event"
}
],
"fields": [
"event.naics.value"
],
"min_term_freq": 1,
"min_doc_freq": 1,
"minimum_should_match": "8%"
}
}
}
}
}
I have results!!
but when I increase min_should_match >= 9% it does not match at all and I get no results.
Also tried to do something like this which gets me some results up to 11%
{
"query": {
"nested": {
"path": "event.naics",
"query": {
"more_like_this": {
"like": [
{
"_id": "83731111.559",
"_type": "profile.event"
}
],
"fields": [
"event.naics.*"
],
"min_term_freq": 1,
"min_doc_freq": 1,
"minimum_should_match": "11%"
}
}
}
}
}
And termvecor for source document is:
{
"_index": "directory.v1",
"_type": "profile.event",
"_id": "83731111.559",
"_version": 5,
"found": true,
"took": 0,
"term_vectors": {}
}

Answer 1

If you get the term vector for document "A" for field event.naics.value you will see you have 24 terms in total each with term frequency 1.
So when you do 8% match that will be rounded down to 1 clause of the 24 generated should clauses, so you get a match. But 9% of 24 will round to 2 clauses should match which is no bueno as each of your nested document has only one value.
For calculation details you can see the bottom of this page
https://github.com/elastic/elasticsearch/blob/99f88f15c5febbca2d13b5b5fda27b844153bf1a/server/src/main/java/org/elasticsearch/common/lucene/search/Queries.java
And morelikethis source is here
https://github.com/elastic/elasticsearch/blob/46a79127edfb0cc93b7580624010ff81ca0cb2f4/server/src/main/java/org/elasticsearch/common/lucene/search/MoreLikeThisQuery.java
Term vector
POST /directory.v1/profile.event/83731111.559/_termvectors
{
"fields":["event.naics.value"],
"offsets" : false,
"payloads" : false,
"positions" : false,
"term_statistics" : true,
"field_statistics" : true
}