Can I ignore "failed to find nested object under path" in ElasticSearch if I get results? - elasticsearch

We have an index with mapping that includes nested fields. In our Java class these fields are lists of objects, and sometimes the lists can be empty (so in the json structure we get e.g {... "some_nested_field": [], ...}.
When we run a query we do get results as expected, but also an error:
"failures": [
{
"shard": 0,
"index": ".kibana",
"node": "ZoEuUdkORpuBSNs7gqiv1Q",
"reason": {
"type": "query_shard_exception",
"reason": """
failed to create query: {
"nested" : {
"query" : {
"bool" : {
"must" : [
{
"match" : {
"foobar.name" : {
"query" : "brlo",
"operator" : "OR",
"prefix_length" : 0,
"max_expansions" : 50,
"fuzzy_transpositions" : true,
"lenient" : false,
"zero_terms_query" : "NONE",
"auto_generate_synonyms_phrase_query" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"path" : "foobar",
"ignore_unmapped" : false,
"score_mode" : "avg",
"boost" : 1.0
}
}
""",
"index_uuid": "xrFCunLNSv6AER_KwNMHSA",
"index": ".kibana",
"caused_by": {
"type": "illegal_state_exception",
"reason": "[nested] failed to find nested object under path [foobar]"
}
}
}
Can I assume that this error is caused by records with empty lists, and ignore it? Or does this indicate an internal error and possibly missing results from my query? Is there a way to avoid this error?
UPDATE:
This is an example of the query we're executing:
GET /_search
{
"query": {
"nested": {
"path": "mynested",
"query": {
"bool": {
"should" : [
{ "match" : { "mynested.name": "foo" } },
{ "match" : { "mynested.description": "bar" } },
{ "match" : { "mynested.category": "baz" } }
],
"minimum_should_match" : 1
}
}
}
}
}
The response from ES reports 10 successful shards and one failure:
{
"took": 889,
"timed_out": false,
"_shards": {
"total": 11,
"successful": 10,
"skipped": 0,
"failed": 1,
"failures": [...]
And we do get hits back:
"hits": {
"total": 234450,
"max_score": 11.092936,
"hits": [ ...

Looks like you have Kibana installed. In the error message it says that it can't find nested under path foobar of index .kibana, which is the one that Kibana uses:
"index_uuid": "xrFCunLNSv6AER_KwNMHSA",
"index": ".kibana",
"caused_by": {
"type": "illegal_state_exception",
"reason": "[nested] failed to find nested object under path [foobar]"
}
When doing simple GET /_search all Elasticsearch indexes are queried, also .kibana, which is probably not what you wanted.
To ignore this particular index you can use Multiple Indices search capability and do a query like:
GET /*,-.kibana/_search
Hope that helps!

Related

Elastic Search shows "Unknown key for a START_OBJECT" exception

I am sending the following query to elastic search in order to get data which are within the range of the values between the from and to:
{
"range" : {
"variables.value.long" : {
"from" : -1.0E19,
"to" : 9.1E18,
"include_lower" : true,
"include_upper" : true,
"boost" : 1.0
}.
}
}
Despite that elastic search throws the following error:
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "Unknown key for a START_OBJECT in [range].",
"line": 2,
"col": 13
}
],
"type": "parsing_exception",
"reason": "Unknown key for a START_OBJECT in [range].",
"line": 2,
"col": 13
},
"status": 400
}
Does anybody know what this error means and why I am getting it?
There is some lack of context here like your mappings or the full query you are running, but this is how a range query should look for your document.
Create index
PUT test_andromachiii
{
"mappings": {
"properties": {
"variables": {
"properties": {
"values": {
"properties": {
"long": {
"type": "double"
}
}
}
}
}
}
}
}
Index document
POST test_andromachiii/_doc
{
"variables": {
"values": {
"long": 9.1E18
}
}
}
Run Query
POST test_andromachiii/_search
{
"query": {
"range": {
"variables.values.long": {
"lte": -1.0E19,
"gte": 9.1E18,
"boost": 1
}
}
}
}
Note lte means lower or equals to, gte greater or equals to.
Response
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "test_andromachiii",
"_type" : "_doc",
"_id" : "gtGj73cBbr4pOF0Is9my",
"_score" : 1.0,
"_source" : {
"variables" : {
"values" : {
"long" : 9.1E18
}
}
}
}
]
}
}
It looks like you're using version <0.90.4. If that's the case, simply wrap your range in a parent query object:
{
"query":{
"range":{
"variables.value.long":{
"from":-1.0E19,
"to":9.1E18,
"include_lower":true,
"include_upper":true,
"boost":1.0
}
}
}
}
If you're using any newer version than that, note that:
The from, to, include_lower and include_upper parameters have been deprecated in 0.90.4 in favour of gt, gte, lt, and lte.
This error is saying (somewhat cryptically) that you have a key range with an Object value, in a place where that key isn't recognised.
The specific cause here is that your range needs to be part of a higher query key such as (i.e.) the bool query, not part of the main.
Credit: https://discuss.elastic.co/t/unknown-key-for-a-start-object-in-should/140008/3

Elasticsearch geospatial queries returning no hits

I'm using Kibana to look at a geospatial dataset in Elasticsearch for a feature currently under development. There is a index of positions which contains field "loc.coordinates", which is a geo_point, and has as data as such:
loc.coordinates 25.906958000000003, 51.776407000000006
However when I run the following query I get no results:
Query
GET /positions/_search
{
"query": {
"bool" : {
"must" : {
"match_all" : {}
},
"filter" : {
"geo_distance" : {
"distance" : "2000km",
"loc.coordinates" : {
"lat" : 25,
"lon" : 51
}
}
}
}
}
}
Response
{
"took": 12,
"timed_out": false,
"_shards": {
"total": 6,
"successful": 6,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
I'm trying to understand why this is, as there are over 250,000 datapoints in the index, and I'm getting no hits regardless of how big the search area is. When I look in the position index mapping I see the following:
"loc": {
"type": "nested",
"properties": {
"coordinates": {
"type": "geo_point"
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
I'm new to Elasticsearch and have been making my way through the documentation, but so far I don't see why my geo queries aren't working as expected. What am I doing wrong?
Your loc field is of type nested, so you need to query that field accordingly with a nested query:
GET /positions/_search
{
"query": {
"bool" : {
"filter" : {
"nested": {
"path": "loc",
"query": {
"geo_distance" : {
"distance" : "2000km",
"loc.coordinates" : {
"lat" : 25,
"lon" : 51
}
}
}
}
}
}
}
}

Invalid aggregation name. Aggregation names must be alpha-numeric and can only contain '_' and '-'"

I'm trying to use bucket selection with over an aggregation whose name contains numerical characters. Invalid aggregation name [secondagg_sum_[filters_equals_100]]. Aggregation names must be alpha-numeric and can only contain '_' and '-'" error has been occurred for me. I realized that aggregations with numeric characters has been solved but I think there is still an error for bucket selection. I asked the question on elastic search forum but nobody is replied. Is there anybody how to solve this problem?
PS: I'm generating my aggregation name dynamically so I need to use numeric characters.
Error:
{
"error" : {
"root_cause" : [
{
"type" : "parsing_exception",
"reason" : "Invalid aggregation name [secondagg_sum_[filters_equals_100]]. Aggregation names must be alpha-numeric and can only contain '_' and '-'",
"line" : 1,
"col" : 318
}
],
"type" : "parsing_exception",
"reason" : "Invalid aggregation name [secondagg_sum_[filters_equals_100]]. Aggregation names must be alpha-numeric and can only contain '_' and '-'",
"line" : 1,
"col" : 318
},
"status" : 400
}
My query:
{
"size": 0,
"query": {
"bool": {
"disable_coord": false,
"adjust_pure_negative": true,
"boost": 1.0
}
},
"aggregations": {
"first_agg": {
"terms": {
"field": "firstproperty",
"size": 2147483647,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": {
"_term": "asc"
}
},
"aggregations": {
"secondagg_sum_[filters_equals_100]": {
"sum": {
"field": "secondproperty"
}
},
"agg_values": {
"bucket_selector": {
"buckets_path": {
"total": "secondagg_sum_[filters_equals_100]"
},
"script": {
"inline": "params.total > 100",
"lang": "painless"
},
"gap_policy": "skip"
}
}
}
}
}
}

elastic search 5 - how to query Object datatype and nested array of json

I want to query against nested data already loaded into Elasticsearch 5 but every query returns nothing. The data is of object datatype and nested array of json.
This the nested datatype ie team_members array of json:
[{
"id": 6,
"name": "mike",
"priority": 1
}, {
"id": 7,
"name": "james",
"priority": 2
}]
This object datatype ie the availability_slot json:
{
"monday": {
"on": true,
"end_time": "15",
"start_time": "9",
"end_time_unit": "pm",
"start_time_unit": "am",
"events_starts_every": 10
}
}
This is my elasticsearch mapping:
{
"meetings_development_20170716013030509": {
"mappings": {
"meeting": {
"properties": {
"account": {"type": "integer"},
"availability_slot": {
"properties": {
"monday": {
"properties": {
"end_time": {"type": "text"},
"end_time_unit": {"type": "text"},
"events_starts_every": {
"type":"integer"
},
"on": {"type": "boolean"},
"start_time": {"type": "text"},
"start_time_unit": {
"type": "text"
}
}
}
}
},
"team_members": {
"type": "nested",
"properties": {
"id": {"type": "integer"},
"name": {"type": "text"},
"priority": {"type": "integer"}
}
}
}
}
}
}
}
I have two queries which are failing for different reasons:
query 1
This query returns a count of zero despite the records existing in elasticsearch, I discovered the queries are failing because of the filter:
curl -u elastic:changeme http://172.19.0.4:9200/meetings_development/_search?pretty -d '{"query":{"nested":{"path":"team_members","score_mode":"avg","query":{"bool":{"must":[{"match":{"team_members.name":"mike"}},{"match":{"team_members.priority":1}}],"filter":[{"match":{"account":1}}]}}}}}'
This returns zero result:
{
"took" : 8,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
query 1 without filter
Thesame query from above without the filter works:
curl -u elastic:changeme http://172.19.0.4:9200/meetings_development/_search?pretty -d '{"query":{"nested":{"path":"team_members","score_mode":"avg","query":{"bool":{"must":[{"match":{"team_members.name":"mike"}},{"match":{"team_members.priority":1}}]}}}}}'
The query above returns 3 hits:
{
"took" : 312,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 2.1451323,
"hits" : [{**results available here**} ]
}
}
query 2 for the object datatype
curl -u elastic:changeme http://172.19.0.4:9200/meetings_development/_search?pretty -d '{"query":{"bool":{"must":{"match":{"availability_slot.start_time":1}},"filter":[{"match":{"account":1}}]}}}'
The query returns a hit of zero but the data is in elasticsearch:
{
"took" : 172,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
How do I get both queries to work filtering by account. Thanks
This elasticsearch guide link was very helpful in coming up with the correct elasticsearch queries shown below:
query 1 for the nested array of json
{
"query" => {
"bool": {
"must": [
{
"match": {
"name": "sales call"
}
},
{"nested" => {
"path" => "team_members",
"score_mode" => "avg",
"query" => {
"bool" => {
"must" => {
"match" => {"team_members.name" => "mike"}
}
}
}
}
}
],
"filter": {
"term": {
"account": 1
}
}
},
}
}
Just pass the query to elastic search like this:
curl http://172.19.0.4:9200/meetings_development/_search?pretty -d '{"query":{"bool":{"must":[{"match":{"name":"sales call"}},{"nested":{"path":"team_members","score_mode":"avg","query":{"bool":{"must":{"match":{"team_members.name":"mike"}}}}}}],"filter":{"term":{"account":1}}}}}'
correct syntax for query 2 for the object datatype ie json
{
"query": {
"bool": {
"must": {
"match": {'availability_slot.monday.start_time' => '9'}
},
"filter": [{
"match": {'account': 1}
}]
}
}
}
You the pass this to elasticsearch like this:
curl http://172.19.0.4:9200/meetings_development/_search?pretty -d '{"query":{"bool":{"must":{"match":{"availability_slot.monday.start_time":"9"}},"filter":[{"match":{"account":1}}]}}}'

ElasticSearch search for part of url

I'm working with ElasticSearch 5 and can't find a solution for the following:
I want to search for a string with slashes (part of a url) in a document. But it won't return matching documents.
I've read something that strings with slashes are splitted by ES and that's not what I want for this field. I've tried to set "not_analyzed" on the field with a mapping, but I can't seem to get it to work somehow.
"Create index":
Put http://localhost:9200/test
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"type1" : {
"properties" : {
"field1" : { "type" : "text","index": "not_analyzed" }
}
}
}
}
"Add document":POST http://localhost:9200/test/type1/
{
"field1" : "this/is/a/url/test"
}
"Search document" POST http://localhost:9200/test/type1/_search
{
"size" : 1000,
"query" : {
"bool" : {
"must" : [{
"term" : {
"field1" : {
"value" : "this/is/a/url/test"
}
}
}
]
}
}
}
Response:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
"The mapping response": GET http://localhost:9200/test/_mapping?pretty
{
"test": {
"mappings": {
"type1": {
"properties": {
"field1": {
"type": "text"
}
}
}
}
}
}
Using a term query for getting an exact match is correct. However, your initial mapping is wrong.
"type" : "text", "index": "not_analyzed"
should be this instead
"type": "keyword"
(Note: The keyword type in ES5 is equivalent to a not_analyzed string in ES 2.x)
You need to delete your index and re-create it with the corrected mapping. Then your term query will work.
I suspect what you need is a Match query, not a Terms query. Terms is looking for a single "term"/word and is not breaking down your request with an analyzer.
{
"size" : 1000,
"query" : {
"bool" : {
"must" : [{
"match" : {
"field1" : "this/is/a/url/test"
}
}
]
}
}
}

Resources