_update_by_query + script do not work correctly,error:Trying to create too many scroll contexts - elasticsearch

Elasticsearch version: 7.6.2
JVM:13.0.2
OS version:centeros7
This is my code
POST recommend_index/_update_by_query
{
"script": {
"source": "ctx._source.rec_doctor_id = 1"
},
"query": {
"bool": {
"must": [{
"terms": {
"id": ["22222"]
}
}]
}
}
}
This code does not return the result correctly,The error message is
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting."
}
],
"type": "search_phase_execution_exception",
"reason": "Partial shards failure",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 1,
"index": "recommend_index",
"node": "XXX",
"reason": {
"type": "exception",
"reason": "Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting."
}
}
]
},
"status": 500
}
I'm sure the current scroll is 0
When I replace _UPDATE_BY_QUERY with _UPDATE, it updates normally
No change has been made in ES since last Friday, and suddenly an error is reported
No configuration changes have been made to the ES server

follow-up:
I set the search.max_open_scroll_context parameter to 5000 and found nothing to do with it.
I looked up the 7.6.2 release and found that someone was having the same problem as me.Link on this
here #71354 #56202
I guess this is due to scrolling triggering the 7.6.2 bug.I restarted the cluster node without upgrading and found that it worked!!

Related

How to create a mutlitype index in Elasticsearch?

In several pages in Elasticsearch documentation is mentioned how to query a multi-type index.
But I failed to create one at the first place.
Here is my minimal example (on a Elasticsearch 6.x server):
PUT /myindex
{
"settings" : {
"number_of_shards" : 1
}
}
PUT /myindex/people/123
{
"first name": "John",
"last name": "Doe"
}
PUT /myindex/dog/456
{
"name": "Rex"
}
Index creation and fist insert did well, but at the dog type insert attempt:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Rejecting mapping update to [myindex] as the final mapping would have more than 1 type: [people, dog]"
}
],
"type": "illegal_argument_exception",
"reason": "Rejecting mapping update to [myindex] as the final mapping would have more than 1 type: [people, dog]"
},
"status": 400
}
But this is exactly what I'm trying to do, buddy! Having "more than 1 type" in my index.
Do you know what I have to change in my calls to achieve this?
Many thanks.
Multiple mapping types are not supported from Elastic 6.0.0 onwards. See breaking changes for details.
You can still effectively use multiple types by implementing your own custom type field.
For example:
{
"mappings": {
"doc": {
"properties": {
"type": {
"type": "keyword"
},
"first_name": {
"type": "text"
},
"last_name": {
"type": "text"
}
}
}
}
}
This is described in removal of types.

Can an Elasticsearch rollup job dynamically create indexes like Logstash does?

I am currently testing out the new rollup APIs in Elasticsearch 6.3 and am wondering if there is any way to configure the rollup job to dynamically create an index based on timestamp like Logstash does when ingesting data? The use case is to try and roll up large amounts of time series network performance reporting data and I'm worried that even an hourly rollup will create a huge index to manage so am looking to split it to have one index for each day's hourly rollup.
Current rollup job config:
{
"index_pattern": "dxs-raw-*",
"rollup_index": "dxs-hourly-%{+YYYY.MM.dd}",
"cron": "* */15 * * * ?",
"page_size": 1000,
"groups": {
"date_histogram": {
"field": "#timestamp",
"interval": "1h",
"delay": "12h"
},
"terms": {
"fields": ["ci_id.keyword", "client_id.keyword", "element_name.keyword", "measurement.keyword", "source_management_platform.keyword", "unit.keyword"]
}
},
"metrics": [
{
"field": "value",
"metrics": ["min", "max", "avg"]
}
]
}
Error seen when PUTting job via Kibana DevTools console:
{
"error": {
"root_cause": [
{
"type": "invalid_index_name_exception",
"reason": "Invalid index name [dxs-hourly-%{+YYYY.MM.dd}], must be lowercase",
"index_uuid": "_na_",
"index": "dxs-hourly-%{+YYYY.MM.dd}"
}
],
"type": "runtime_exception",
"reason": "runtime_exception: Could not create index for rollup job [dxs-hourly]",
"caused_by": {
"type": "invalid_index_name_exception",
"reason": "Invalid index name [dxs-hourly-%{+YYYY.MM.dd}], must be lowercase",
"index_uuid": "_na_",
"index": "dxs-hourly-%{+YYYY.MM.dd}"
}
},
"status": 500
}
As of version 6.4 this is not possible but a new enhancement has been raised here
When the final solution is released I will update this answer with the implementation we have.

ElasticSearch returns 404 while multi-index, multi-type search

We have a requirement where in we need to query across multiple indices as follows
We are using ElasticSearch 5.1.1.
http://localhost:9200/index1,index2,index3/type1,type2/_search
query:
{
"query": {
"multi_match": {
"query": "data",
"fields": ["status"]
}
}
}
However we may not know in advance if the index is present or not , we get following error if either of above indices is not present.
{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "index3",
"index_uuid": "_na_",
"index": "index3"
}
],
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "index3",
"index_uuid": "_na_",
"index": "index3"
},
"status": 404
}
One obvious way is to check if index is already present or not but I would like to avoid that extra call.
Note: At least 1 index will always be present
Is it possible to avoid this Exception ?
Thanks in advance !!
"ignore_unavailable" is the solution for this. Pass this as a query parameter in search url.
Exa. http://localhost:9200/index1,index2/type/_search?ignore_unavailable
This will not give 404 even if either of the indices are not present

Conditional update in Elasticsearch: "Invalid op [none]"

Elasticsearch 2.3. I am firing this query to update every doc in an index based on a condition.
{
"query": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
},
"script": {
"inline": "if (ctx._source.url.endsWith('a=6')) ctx.op = 'none' else ctx._source.url = ctx._source.url + '&b=3'"
}
}
As stated here, I am using ctx.op = 'none' to avoid updating the docs that do not match the condition.
I am getting
Invalid op [none]
Full error:
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Invalid op [none]"
}
],
"type": "illegal_argument_exception",
"reason": "Invalid op [none]"
},
"status": 400
Seems to be so simple that I am pretty lost. Thanks for your help.
The correct operation is noop, not none.
From the documentation:
Just as in Update API you can set ctx.op = "noop" if your script decides that it doesn’t have to make any changes. That will cause _update_by_query to omit that document from its updates.
PR created: https://github.com/elastic/elasticsearch/pull/24613

Parsing exception when creating index and mapping at once

I am receiving an exception when trying to create an index, along with a mapping. I am issuing a PUT to my local ElasticSearch instance (v. 5.1.1) http://127.0.0.1:9200/indexname with the following body
{
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
},
"mappings": {
"examplemapping": {
"properties": {
"titel": {
"type": "text",
"index": false
},
"body": {
"type": "text"
},
"room": {
"type": "text",
"index": false
},
"link": {
"type": "text",
"index": false
}
}
}
}
}
I receive the following error
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "No handler for type [text] declared on field [body]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [examplemapping]: No handler for type [text] declared on field [body]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "No handler for type [text] declared on field [body]"
}
},
"status": 400
}
From the documentation on index creation it should be possible to create an index, and create one or more mappings at the same time:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
I have read https://www.elastic.co/guide/en/elasticsearch/reference/current/string.html and believe that I am correctly using the new datatype, but the exception suggests otherwise.
Any help is greatly appreciated.
Resolution
Thanks to a comment by Val i was pointed in the right direction. Indeed I was not using version 5.1.1, but version 2.4.3
So why the confusion? Well, I have been running both versions (not at once), and startet and stopped them using the respective bat scripts:
call es-2.4.3/bin/service.bat start
call es-5.1.1/bin/elasticsearch-service.bat start
It seems that even though I have been running the latter, it was still ES2.4.3 that was started. This is probably caused by the logic inside the bat script.
Going forward I will keep in mind to check the version response from the service itself, and I'm gonna have to find a proper setup to run multiple versions of ElasticSearch.
Thanks for the answers.
I tried your settings on ElasticSeach 5.0.0 and it worked fine, output of GET indexname :
{
"indexname": {
"aliases": {},
"mappings": {
"examplemapping": {
"properties": {
"body": {
"type": "text"
},
"link": {
"type": "text",
"index": false
},
"room": {
"type": "text",
"index": false
},
"titel": {
"type": "text",
"index": false
}
}
}
},
"settings": {
"index": {
"creation_date": "1488892255496",
"number_of_shards": "1",
"number_of_replicas": "1",
"uuid": "GugRGgllQbCadCTj5oq4ow",
"version": {
"created": "5000099"
},
"provided_name": "testo"
}
}
}
}
Also please note that I would definitely recommend that you set a different value of "number_of_shards": "1" as a rule of thumb consider that Elasticsearch is allocating 1 thread per shard, and thus the bigger your shard will become, the slower the text search will be. Now also bear in mind that some overhead comes from allocating more shards, so don't "overallocate". See this post and this onefor more details.
Thanks to a comment by Val i was pointed in the right direction. Indeed I was not using version 5.1.1, but version 2.4.3
So why the confusion? Well, I have been running both versions (not at once), and startet and stopped them using the respective bat scripts:
call es-2.4.3/bin/service.bat start
call es-5.1.1/bin/elasticsearch-service.bat start
It seems that even though I have been running the latter, it was still ES2.4.3 that was started. This is probably caused by the logic inside the bat script.
Going forward I will keep in mind to check the version response from the service itself, and I'm gonna have to find a proper setup to run multiple versions of ElasticSearch.
Thanks to all who pitched in!

Resources