inner_hits does not work for nested filters when searching multiple indices - elasticsearch

Question

Elasticsearch version
Version: 6.2.2
I'm trying to search multiple indices which both have nested objects. To keep it simple I will use this example. In the query I use "inner_hits" property.
PUT /sales
{
"mappings": {
"_doc" : {
"properties" : {
"tags" : { "type" : "keyword" },
"comments" : {
"type" : "nested",
"properties" : {
"username" : { "type" : "keyword" },
"comment" : { "type" : "text" }
}
}
}
}
}
}
PUT /sales/_doc/1?refresh
{
"tags": ["car", "auto"],
"comments": [
{"username": "baddriver007", "comment": "This car could have better brakes"},
{"username": "dr_who", "comment": "Where's the autopilot? Can't find it"},
{"username": "ilovemotorbikes", "comment": "This car has two extra wheels"}
]
}
PUT /markets
{
"mappings": {
"_doc" : {
"properties" : {
"name" : { "type" : "keyword" },
"products" : {
"type" : "nested",
"properties" : {
"name" : { "type" : "keyword" },
"sku" : { "type" : "text" }
}
}
}
}
}
}
POST /sales,markets/_search
{
"query": {
"bool": {
"should": [
{
"bool": {
"should": [
{
"nested": {
"path": "comments",
"inner_hits": {
},
"ignore_unmapped": true,
"query": {
"match_all": {}
}
}
}]
}
},
{
"bool": {
"should": [
{
"nested": {
"path": "products",
"inner_hits": {
},
"ignore_unmapped": true,
"query": {
"match_all": {}
}
}
}
]
}
}
]
}
}
}
So this query gives an error
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
},
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "markets",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
}
},
{
"shard": 0,
"index": "sales",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
}
]
},
"status": 500
}
But when I add "ignore_unmapped": true inside each "inner_hits": { "ignore_unmapped": true } everything works fine. This is not implemented in NEST .net library.
Is it right to use "ignore_unmapped" inside "inner_hits", because I didn't find this as "inner_hits" property in the documentation ?
Is there any other solution in NEST for this to be done.
UPDATE:
I tried using operator overloading queries and I got
Func<SearchDescriptor<object>, ISearchRequest> search = s => s.Type<object>()
.Index(Indices.Index("sales", "markets"))
.AllTypes()
.Explain(false)
.Query(q => (q
.Nested(n => n
.IgnoreUnmapped(true)
.Path(Infer.Field<SaleDocument>(f => f.Comments))
.InnerHits(ih => ih
.Size(1)
)
.Query(q1 => q1
.MatchAll()
)
) && +q.Terms(t => t
.Field("_index")
.Terms(new[] { "sales" })
)
) || (q
.Nested(n => n
.IgnoreUnmapped(true)
.Path(Infer.Field<MarketDocument>(f => f.Products))
.InnerHits(ih => ih
.Size(1)
)
.Query(q1 => q1
.MatchAll()
)
) && +q.Terms(t => t
.Field("_index")
.Terms(new[] { "markets" })
)
)
);
This code created query
POST /sales,markets/_search
{
"from": 0,
"size": 10,
"explain": false,
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"nested": {
"query": {
"match_all": {}
},
"path": "comments",
"inner_hits": {
"size": 1
},
"ignore_unmapped": true
}
}
],
"filter": [
{
"terms": {
"_index": [
"sales"
]
}
}
]
}
},
{
"bool": {
"must": [
{
"nested": {
"query": {
"match_all": {}
},
"path": "products",
"inner_hits": {
"size": 1
},
"ignore_unmapped": true
}
}
],
"filter": [
{
"terms": {
"_index": [
"markets"
]
}
}
]
}
}
]
}
}
}
Which again gives error
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
},
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "markets",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
}
},
{
"shard": 0,
"index": "sales",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
}
]
},
"status": 500
}

Answer 1

As you say, it looks like inner_hits property is missing within NEST; I'll open an issue to add this for the next release now.
ignore_unmapped is the way to handle this when needing inner_hits. If you didn't need inner_hits, you could combine each nested query with a term query on the "_index" metadata field that targets the respective index name in each case, such that the nested query is run only against the indices that contain the target field.

Answer 2

This issue has been fixed in both places and will be available in some of the future releases of NEST or ElasticSearch.
NEST .net library by adding IgnoreUnmapped() method in InnerHits.
https://github.com/elastic/elasticsearch-net/issues/3132
Elastic Search by inherit ignore_unmapped in inner_hits from nested query ignore_unmapped property
https://github.com/elastic/elasticsearch/issues/29071