Elasticsearch 6.1 multi index search with nested fields issue - elasticsearch

I run a multi index search (elasticsearch 6.1.1) in 2 indexes with nested field,
"uniqueID" is a nested field that exist only in person index.
"pobox" is a nested field that exist only in adress index
I am getting error:
"index": "adress", "[nested] failed to find nested object under path [uniqueID]"
"index": "person", "[nested] nested object under path [pobox] is not of nested type"
In my query I search in person index for field uniqueID, why I am getting error for pobox field that exist only in adress index. Same for search in adress index it look for uniqueId field that exist only in person index
POST http://locahost:9200/person,adress/_search
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"terms": {
"_index": [
"person"
]
}
},
{
"nested": {
"path": "uniqueID",
"query": {
"span_near": {
"clauses": [
{
"span_term": {
"uniqueID.uniqueID.auto": "1"
}
}
],
"slop": 3,
"in_order": true
}
}
}
}
]
}
},
{
"bool": {
"must": [
{
"terms": {
"_index": [
"adress"
]
}
},
{
"nested": {
"path": "pobox",
"query": {
"span_near": {
"clauses": [
{
"span_term": {
"pobox.pobox.auto": "1"
}
}
],
"slop": 3,
"in_order": true
}
}
}
}
]
}
}
]
}
}
}
Error
{
"error": {
"root_cause": [
{
"type": "query_shard_exception",
"reason": "failed to create query: { my_query }",
"index_uuid": "9Z0W-P9ZS02kJ7WmOKHPVQ",
"index": "adress"
},
{
"type": "query_shard_exception",
"reason": "failed to create query: { my_query }",
"index_uuid": "EHoxKGhdSmKoYdNgsylotw",
"index": "person"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "adress",
"node": "AEhiq0wvQTGh468sSmDN5g",
"reason": {
"type": "query_shard_exception",
"reason": "failed to create query: { my_query }",
"index_uuid": "9Z0W-P9ZS02kJ7WmOKHPVQ",
"index": "adress",
"caused_by": {
"type": "illegal_state_exception",
"reason": "[nested] failed to find nested object under path [uniqueID]"
}
}
},
{
"shard": 0,
"index": "person",
"node": "AEhiq0wvQTGh468sSmDN5g",
"reason": {
"type": "query_shard_exception",
"reason": "failed to create query: { my_query }",
"index_uuid": "EHoxKGhdSmKoYdNgsylotw",
"index": "person",
"caused_by": {
"type": "illegal_state_exception",
"reason": "[nested] nested object under path [pobox] is not of nested type"
}
}
}
]
},
"status": 400
}

Well, the nested structure you try to query on one index doesn't exist on the other, so what do you expect?
IMO you should split the queries up and use _msearch if you always need those together.

Related

inner_hits does not work for nested filters when searching multiple indices

Elasticsearch version
Version: 6.2.2
I'm trying to search multiple indices which both have nested objects. To keep it simple I will use this example. In the query I use "inner_hits" property.
PUT /sales
{
"mappings": {
"_doc" : {
"properties" : {
"tags" : { "type" : "keyword" },
"comments" : {
"type" : "nested",
"properties" : {
"username" : { "type" : "keyword" },
"comment" : { "type" : "text" }
}
}
}
}
}
}
PUT /sales/_doc/1?refresh
{
"tags": ["car", "auto"],
"comments": [
{"username": "baddriver007", "comment": "This car could have better brakes"},
{"username": "dr_who", "comment": "Where's the autopilot? Can't find it"},
{"username": "ilovemotorbikes", "comment": "This car has two extra wheels"}
]
}
PUT /markets
{
"mappings": {
"_doc" : {
"properties" : {
"name" : { "type" : "keyword" },
"products" : {
"type" : "nested",
"properties" : {
"name" : { "type" : "keyword" },
"sku" : { "type" : "text" }
}
}
}
}
}
}
POST /sales,markets/_search
{
"query": {
"bool": {
"should": [
{
"bool": {
"should": [
{
"nested": {
"path": "comments",
"inner_hits": {
},
"ignore_unmapped": true,
"query": {
"match_all": {}
}
}
}]
}
},
{
"bool": {
"should": [
{
"nested": {
"path": "products",
"inner_hits": {
},
"ignore_unmapped": true,
"query": {
"match_all": {}
}
}
}
]
}
}
]
}
}
}
So this query gives an error
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
},
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "markets",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
}
},
{
"shard": 0,
"index": "sales",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
}
]
},
"status": 500
}
But when I add "ignore_unmapped": true inside each "inner_hits": { "ignore_unmapped": true } everything works fine. This is not implemented in NEST .net library.
Is it right to use "ignore_unmapped" inside "inner_hits", because I didn't find this as "inner_hits" property in the documentation ?
Is there any other solution in NEST for this to be done.
UPDATE:
I tried using operator overloading queries and I got
Func<SearchDescriptor<object>, ISearchRequest> search = s => s.Type<object>()
.Index(Indices.Index("sales", "markets"))
.AllTypes()
.Explain(false)
.Query(q => (q
.Nested(n => n
.IgnoreUnmapped(true)
.Path(Infer.Field<SaleDocument>(f => f.Comments))
.InnerHits(ih => ih
.Size(1)
)
.Query(q1 => q1
.MatchAll()
)
) && +q.Terms(t => t
.Field("_index")
.Terms(new[] { "sales" })
)
) || (q
.Nested(n => n
.IgnoreUnmapped(true)
.Path(Infer.Field<MarketDocument>(f => f.Products))
.InnerHits(ih => ih
.Size(1)
)
.Query(q1 => q1
.MatchAll()
)
) && +q.Terms(t => t
.Field("_index")
.Terms(new[] { "markets" })
)
)
);
This code created query
POST /sales,markets/_search
{
"from": 0,
"size": 10,
"explain": false,
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"nested": {
"query": {
"match_all": {}
},
"path": "comments",
"inner_hits": {
"size": 1
},
"ignore_unmapped": true
}
}
],
"filter": [
{
"terms": {
"_index": [
"sales"
]
}
}
]
}
},
{
"bool": {
"must": [
{
"nested": {
"query": {
"match_all": {}
},
"path": "products",
"inner_hits": {
"size": 1
},
"ignore_unmapped": true
}
}
],
"filter": [
{
"terms": {
"_index": [
"markets"
]
}
}
]
}
}
]
}
}
}
Which again gives error
{
"error": {
"root_cause": [
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
},
{
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "markets",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [comments]"
}
},
{
"shard": 0,
"index": "sales",
"node": "-22psoQNRLa8_Y9GeHBXaw",
"reason": {
"type": "illegal_state_exception",
"reason": "[match_all] no mapping found for type [products]"
}
}
]
},
"status": 500
}
As you say, it looks like inner_hits property is missing within NEST; I'll open an issue to add this for the next release now.
ignore_unmapped is the way to handle this when needing inner_hits. If you didn't need inner_hits, you could combine each nested query with a term query on the "_index" metadata field that targets the respective index name in each case, such that the nested query is run only against the indices that contain the target field.
This issue has been fixed in both places and will be available in some of the future releases of NEST or ElasticSearch.
NEST .net library by adding IgnoreUnmapped() method in InnerHits.
https://github.com/elastic/elasticsearch-net/issues/3132
Elastic Search by inherit ignore_unmapped in inner_hits from nested query ignore_unmapped property
https://github.com/elastic/elasticsearch/issues/29071

update a particular field of elasticsearch document

Hi I am trying to update documents a elasticsearch which meets specific criteria. I am using google sense(chrome extension) for making request. The request that I am making is as shown below:
GET styling_rules2/product_line_filters/_update
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{"term":{"product_line_attribute": "brand"}}
],
"minimum_should_match": 1
}
},
"filter": {
"term": {
"product_line_name": "women_skirts"
}
}
}
},
"script" : "ctx._source.brand=brands"
}
sample document is as shown below:
{
"product_line_attribute_db_path": "product_filter.brand",
"product_line_attribute": "brand",
"product_line_name": "women_skirts",
"product_line_attribute_value_list": [
"vero moda",
"faballey",
"only",
"rider republic",
"dorothy perkins"
]
}
desired result: update all the document which has product_line_attribute="brand" and product_line_name="women_skirts" to product_line_attribute="brands".
problem: I am getting the error as follows:
{
"error": {
"root_cause": [
{
"type": "search_parse_exception",
"reason": "failed to parse search source. unknown search element [script]",
"line": 18,
"col": 4
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "styling_rules2",
"node": "2ijp1pXwT46FN4on4-JPlg",
"reason": {
"type": "search_parse_exception",
"reason": "failed to parse search source. unknown search element [script]",
"line": 18,
"col": 4
}
}
]
},
"status": 400
}
thanks in advance!
You should use the _update_by_query endpoint and not _update. Also the script section is not correct, which is probably why you're getting a class_cast_exception.
Try this instead:
POST styling_rules2/product_line_filters/_update_by_query
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"term": {
"product_line_attribute": "brand"
}
}
],
"minimum_should_match": 1
}
},
"filter": {
"term": {
"product_line_name": "women_skirts"
}
}
}
},
"script": {
"inline": "ctx._source.brand=brands"
}
}

Error in executing groovy script in elasticsearch

below is my query, I want to change score calculation using function_score feature:
{
"size": 1,
"query":{
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {
"messageText": "car"
}
}
]
}
},
"script_score" : {
"script" : "doc['time_views'].values[doc['time_views'].values.length-1]"
}
,
"boost_mode": "replace"
}
},
"from": 0
}
but I got this error response
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "failed to run inline script [doc['time_views'].values[doc['time_views'].values.length-1]] using lang [groovy]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "datacollection",
"node": "TWeZV3R6Rq-WYQ2YIHjILQ",
"reason": {
"type": "script_exception",
"reason": "failed to run inline script [doc['time_views'].values[doc['time_views'].values.length-1]] using lang [groovy]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [time_views] in mapping with types [message]"
}
}
}
]
},
"status": 500
}
some solutions says using quotation in "doc['time_views']" causes the problem when query has been send from command prompt tools. I don't know why!
I don't use any command prompt tools. I create the query in java code directly
EDIT
this is my index mapping:
"mappings": {
"message": {
"properties": {
"text": {
"type": "string"
},
"time_views": {
"type": "nested",
"properties": {
"backupTimestamp": {
"type": "long"
},
"views": {
"type": "integer"
}
}
}
}
}
}
}
I want to use "views" of last item of "time_views". so I try below scripts too, but each of them throw different error:
"doc['time_views.views'].values[doc['time_views.views'].values.length-1]"
error: java.util.ArrayList cannot be cast to java.lang.Number
"doc['time_views.views'].values[doc['time_views.views'].values.size()-1]"
error: failed to run inline script [doc['time_views.views'].values[doc['time_views.views'].values.size()-1]] using lang [groovy]
"doc['time_views'].values[doc['time_views'].values.size()-1].views"
error: failed to run inline script [doc['time_views'].values[doc['time_views'].values.size()-1].views] using lang [groovy]"
I'm really new in elasticsearch and groovy language. I didn't care about that "time_views" is nested Object, also I don't know syntax of groovy exactly, after some affort I found my mistakes and the solution:
{
"size": 1,
"query":{
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {
"messageText": "car"
}
}
]
}
},
"script_score" : {
"script" : "doc['time_views.views'].values.get(doc['time_views.views'].values.size()-1)"
}
,
"boost_mode": "replace"
}
},
"from": 0
}
It's work as I expected

ElasticSearch - failed to parse search source. expected field name but got [START_OBJECT]

I can't figure out what's wrong in my ES query.
I want to filter on a specific field, and also sort by other field.
Request:
GET /_search
{
"query" : {
"term": {
"_type" : "monitor"
},
"filtered" : {
"filter" : { "term" : { "ProcessName" : "myProc" }}
}
},
"sort": { "TraceDateTime": { "order": "desc", "ignore_unmapped": "true" }}
}
Response:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "failed to parse search source. expected field name but got [START_OBJECT]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": ".kibana",
"node": "94RPDCjhQh6eoTe6XoRmSg",
"reason": {
"type": "parse_exception",
"reason": "failed to parse search source. expected field name but got [START_OBJECT]"
}
}
]
},
"status": 400
}
You have a syntax error in your query, you need to enclose both of your term queries inside a bool/must compound query, it needs to be like this:
POST /_search
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"ProcessName": "myProc"
}
},
{
"term": {
"_type": "monitor"
}
}
]
}
}
}
},
"sort": {
"TraceDateTime": {
"order": "desc",
"ignore_unmapped": "true"
}
}
}
PS: Always use POST when sending a payload in your query.

elasticsearch how to include many values for the same field in the match phrase

This is my query
GET blablabla/_search
{
"query": {
"bool": {
"must": [
{"match" : {"source" : "balblabla"}},
{"match" :{"blablablab" : "JBR"}},
{"match": {"city" : "blab bla"}},
{"match" : {"something": ["Balcony" , "Gym"]}}
]
}
}
}
I am getting this error:
{
"error": {
"root_cause": [
{
"type": "query_parsing_exception",
"reason": "[match] query parsed in simplified form, with direct field name, but included more options than just the field name, possibly use its 'options' form, with 'query' element?",
"index": "index name goes here",
"line": 8,
"col": 35
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "index name goes here",
"node": "4Qjq5UGPSZO2Qtg-ECI_mQ",
"reason": {
"type": "query_parsing_exception",
"reason": "[match] query parsed in simplified form, with direct field name, but included more options than just the field name, possibly use its 'options' form, with 'query' element?",
"index": "index name goes here",
"line": 8,
"col": 35
}
}
]
},
"status": 400
}
When I remove this line
{"match" : {"something": ["Balcony" , "Gym"]}}
It works fine, but why does that line causes a parsing error, since arrays are fine in json?
My elasticsearch version is 2.2
The error comes from the parser, you must specified with an "OR" when passing multiple needles.
to fix your case you must replace
{
"match": {
"something": [
"Balcony",
"Gym"
]
}
}
WITH
{
"match": {
"something": "Balcony OR Gym"
}
}
so at the end it should look like this:
{
"query": {
"bool": {
"must": [
{
"match": {
"source": "balblabla"
}
},
{
"match": {
"blablablab": "JBR"
}
},
{
"match": {
"city": "blab bla"
}
},
{
"match": {
"something": "Balcony OR Gym"
}
}
]
}
}
}
hope this will help you and point you in the right path :)
regards,
Daniel

Resources