Getting error index.max_inner_result_window during rolling upgrade of ES from 5.6.10 to 6.8.10 - elasticsearch

I have 2 data nodes and 3 master nodes in an ES cluster. I was doing a rolling upgrade as ES suggested moving from 5.6.10 to 6.8.10.
As there should be zero downtime, I was testing that and getting one error.
I have upgraded the 1 data node and do basic search testing. It is working fine. When I have upgraded 2nd node search is breaking with the below Error.
java.lang.IllegalArgumentException: Top hits result window is too large, the top hits aggregator [top]'s from + size must be less than or equal to: [100] but was [999]. This limit can be set by changing the [index.max_inner_result_window] index level setting.
index.max_inner_result_window -- This property was introduced in the 6.X version, and the master node is still on 5.6.10. So what will be the solution with 0 downtimes?
Note: My indexing is stopped completely. My 2 data nodes are now on 6.8.10 and master nodes are on 5.6.
Thanks

1 - Change the parameter on current indexes:
curl -X PUT "http://localhost:9200/_all/_settings?pretty" -H 'Content-Type: application/json' -d'
{
"index.max_inner_result_window": "2147483647"
}
'
2 - Create a template to further indexes:
curl -X PUT "http://localhost:9200/_index_template/template_max_inner_result?pretty" -H 'Content-Type: application/json' -d'
{
"index_patterns": ["*"],
"template": {
"settings": {
"index":{
"max_inner_result_window": 2147483647
}
}
}
}
'

Related

Restore elasticsearch cluster onto another cluster

Hello i have 3 node elasticsearch cluster ( source ) and i have snapshot called
snapshot-1 which taken from source cluster
and i have another 6 node elasticsearch cluster ( destination ) cluster
and when i restore my destinatition cluster from snapshot-1 using this command
curl -X POST -u elastic:321 "192.168.2.15:9200/snapshot/snapshot_repository/snapshot-1/_restore?pretty" -H 'Content-Type: application/json' -d'
> {
> "indices": "*",
> "ignore_unavailable": true,
> "include_global_state": false,
> "rename_pattern": ".security(.+)",
> "rename_replacement": "delete_$1",
> "include_aliases": false
> }
> '
{
and i got this error
"error" : {
"root_cause" : [
{
"type" : "snapshot_restore_exception",
"reason" : "[snapshot:snapshot-1 yjg/mHsYhycHQsKiEhWVhBywxQ] cannot restore index [.ilm-history-0003] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name"
}
so as you can see the index .ilm-history-0003 already exists in the cluster, but how can i do rename replacement for security,.ilm,.slm,.transfrom indices using only 1 rename_pattern?
like this one
"rename_pattern": ".security(.+)",
From my experiences the rename pattern doesn't need to be super fancy because you will probably
a) delete the index (as your renaming pattern suggests) or
b) reindex data from the restored index to new indices. In this case the naming of the restored index is insignificant.
So this is what I would suggest:
Use the following renaming pattern to include all indices. Again, from my experience, your first aim is to get the old data restored. After that you have to manage the reindexing etc.
POST /_snapshot/REPOSITORY_NAME/SNAPSHOT_NAME/_restore
{
"indices": "*",
"ignore_unavailable": true,
"include_aliases": false,
"include_global_state": false,
"rename_pattern": "(.+)",
"rename_replacement": "restored_$1"
}
This will prepend restored_ to the actual index name resulting in the following restored indices:
restored_security
restored_.ilm*
restored_.slm*
restored_.transfrom*
I hope I could help you.
solve it using this way
curl -X POST -u elastic:321 "192.168.2.15:9200/snapshot/snapshot_repository/snapshot-1/_restore?pretty" -H 'Content-Type: application/json' -d'
with response:
{
"indices": "*,-.slm*,-,ilm*,-.transfrom*,-security*",
"ignore_unavailable": true,
"include_global_state": false,
"include_aliases": false
}

Backup and restore some records of an elasticsearch index

I wish to take a backup of some records(eg latest 1 million records only) of an Elasticsearch index and restore this backup on a different machine. It would be better if this could be done using available/built-in Elasticsearch features.
I've tried Elasticsearch snapshot and restore (following code), but looks like it takes a backup of the whole index, and not selective records.
curl -H 'Content-Type: application/json' -X PUT "localhost:9200/_snapshot/es_data_dump?pretty=true" -d '
{
"type": "fs",
"settings": {
"compress" : true,
"location": "es_data_dump"
}
}'
curl -H 'Content-Type: application/json' -X PUT "localhost:9200/_snapshot/es_data_dump/snapshot1?wait_for_completion=true&pretty=true" -d '
{
"indices" : "index_name",
"type": "fs",
"settings": {
"compress" : true,
"location": "es_data_dump"
}
}'
The format of backup could be anything, as long as it can be successfully restored on a different machine.
you can use _reinex API. it can take any query. after reindex, you have a new index as backup, which contains requested records. easily copy it where ever you want.
complete information is here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
In the end, I fetched the required data using python driver because that is what I found the easiest for the given use case.
For that, I ran an Elasticsearch query and stored its response in a file in newline-separated format and then I later restored data from it using another python script. A maximum of 10000 entries are returned this way along with the scroll ID to be used to fetch next 10000 entries and so on.
es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)
page = es.search(index=['ct_analytics'], body={'size': 10000, 'query': _query, 'stored_fields': '*'}, scroll='5m')
while len(page['hits']['hits']) > 0:
es_data = page['hits']['hits'] #Store this as you like
scrollId = page['_scroll_id']
page = es.scroll(scroll_id=scrollId, scroll='5m')

How to compress with 'best_compression' elasticsearch data

How can I compress all elasticsearch data (existing one as well as new data) with the "best_compression" option?
I know since 5.00 version I can't put "index.codec: best_compression" in the elasticsearch.yml file. I've read the log which indicates that it's deccaped and I should use
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{"index.codec" : "best_compression"}'
But when used I'm given the following error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Can't update non dynamic settings [[index.codec]] for open indices [[logstash-dns-2018.07.30/xHq6UfgsSD2M1dBZhV3cOg], [logstash-2018.07.27/7U7uUsEORFqXtJtrk4KvDw], [logstash-dns-2018.07.27/Xbx15QXOQ5KJAK7iop_54Q], [logstash-http-2018.07.27/q0Rs65a3TjW4NJfcljUHEw], [logstash-flow-2018.07.30/0Erbh2TcRgmFJLMLr8Ka8w], [logstash-2018.07.30/boOd8BdrQV2QoziKaZ_2lw], [logstash-alert-2018.07.27/o5yqwdNqR5yAcbJ-HCNVHw], [logstash-alert-2018.07.30/pp6ZWKLISECVzUiCDDeydQ], [logstash-tls-2018.07.30/rZi6KfC7RtqOVjUt7CCqDQ], [logstash-ssh-2018.07.27/wKi-p6slSqO0-vbwRqS1ZA], [.kibana/XaFQRcEXTW6jLUCmBijzKQ], [logstash-tls-2018.07.27/hbiXYCzjRumh3ND6up9vNw], [logstash-flow-2018.07.27/XfspJr1TS4y6MnCgAmRq1g], [logstash-fileinfo-2018.07.27/9VWyBHsqRmO4QsnN-gdt_w], [logstash-http-2018.07.30/U9JO9Cp-QQO7gvRNoHt7FQ], [logstash-fileinfo-2018.07.30/nlwHeDOsQ3ii8CLxcgE3Ag]]"}],"type":"illegal_argument_exception","reason":"Can't update non dynamic settings [[index.codec]] for open indices [[logstash-dns-2018.07.30/xHq6UfgsSD2M1dBZhV3cOg], [logstash-2018.07.27/7U7uUsEORFqXtJtrk4KvDw], [logstash-dns-2018.07.27/Xbx15QXOQ5KJAK7iop_54Q], [logstash-http-2018.07.27/q0Rs65a3TjW4NJfcljUHEw], [logstash-flow-2018.07.30/0Erbh2TcRgmFJLMLr8Ka8w], [logstash-2018.07.30/boOd8BdrQV2QoziKaZ_2lw], [logstash-alert-2018.07.27/o5yqwdNqR5yAcbJ-HCNVHw], [logstash-alert-2018.07.30/pp6ZWKLISECVzUiCDDeydQ], [logstash-tls-2018.07.30/rZi6KfC7RtqOVjUt7CCqDQ], [logstash-ssh-2018.07.27/wKi-p6slSqO0-vbwRqS1ZA], [.kibana/XaFQRcEXTW6jLUCmBijzKQ], [logstash-tls-2018.07.27/hbiXYCzjRumh3ND6up9vNw], [logstash-flow-2018.07.27/XfspJr1TS4y6MnCgAmRq1g], [logstash-fileinfo-2018.07.27/9VWyBHsqRmO4QsnN-gdt_w], [logstash-http-2018.07.30/U9JO9Cp-QQO7gvRNoHt7FQ], [logstash-fileinfo-2018.07.30/nlwHeDOsQ3ii8CLxcgE3Ag]]"},"status":400}
Solved:
Close all indices:
http://localhost:9200/_all/_close'
Apply best_compression to all
curl -XPUT 'http://localhost:9200/_all/_settings' -d '{"index.codec" : "best_compression"}'
Open all indices:
curl -XPOST 'http://localhost:9200/_all/_open'

Laravel Scout with Elastic search not working

I tried
Using Elastic search with Laravel scout with packages
"laravel/scout": "^1.0",
"tamayo/laravel-scout-elastic": "^1.0"
Ran Elasticsearch server in localhost:9200 and created index and gave necessary config's,
added searchable trait's to the model,
and imported data to index like
php artisan scout:import "App\story"
Imported [App\story] models up to ID: 4
All [App\story] records have been imported.
But when I do a search it returns an empty array
story::search("the")->get()
=> Illuminate\Database\Eloquent\Collection {#754
all: [],
}
when I do curl also it shows like,
// http://localhost:9200/author/_search?pretty=true&q=*:*
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": [
]
}
}
When I adding the record without index in ES the model throws an error like index not found. But after adding data and all, it seems empty. Did I miss anything?
The whole same works fine with algolia.
Set QUEUE=sync or you could turn off queue on config/scout.php.
Had the same issue: https://github.com/ErickTamayo/laravel-scout-elastic/issues/43
I had the same issue.
Delete your index in Elasticsearch and run:
php artisan scout:import App\\story
Let scout create it.
In case you are using elastic search, check the error log - I had:
sudo su
tail -f /var/log/elasticsearch/elasticsearch.log
high disk watermark [90%] exceeded on [minbqqKpRV-umA0DPxkuww][mihai-MS-7A72][/var/lib/elasticsearch/nodes/0] free: 28.9gb[6.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
IF that`s the case,disable the disk check,at least for testing
curl -XPUT -H "Content-Type: application/json" \
http://localhost:9200/_all/_settings \
-d '{"index.blocks.read_only_allow_delete": false}'
Or make those changes permanent in elasticsearch.yml and restart the service.

How to really delete document of a certain type in elasticsearch

I would like to delete all of my document of _type=varnish-request on my elasticsearch.
I installed the delete by query plugin (https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/plugins-delete-by-query.html)
I did DELETE http://localhost:9200/logstash*/_query
{
"query": {
"bool": {
"must": [
{ "match": {"_type":"varnish-request"}},
{ "match": {"_index":"logstash-2016.02.05"}}
]
}
}
}
And it's OK
{"took":2265842,"timed_out":false,"_indices":{"_all":{"found":3062614,"deleted":3062614,"missing":0,"failed":0},"logstash-2016.02.05":
{"found":3062614,"deleted":3062614,"missing":0,"failed":0}},"failures":[]}
curl http://localhost:9200/_cat/indices | sort
Before the clean
yellow open logstash-2016.02.05 5 1 4618245 0 4.1gb 4.1gb
After the clean
yellow open logstash-2016.02.05 5 1 1555631 3062605 4.1gb 4.1gb
The whole point is to 'light' my ES server by removing useless data. But here I see that the index size is still the same.
I already check Delete documents of type in Elasticsearch but no luck
I try with elasticsearch: how to free store size after deleting documents
POST http://localhost:9200/logstash-2016.02.05/_forcemerge
{"_shards":{"total":10,"successful":5,"failed":0}}
But still
yellow open logstash-2016.02.05 5 1 1555631 3062605 4.1gb 4.1gb
The first step is correct. Now you simply need to call _optimize (or _forcemerge if you're using ES 2.1+) by enabling only_expunge_deletes. This will delete the segments with deleted documents and free some space.
curl -XPOST 'http://localhost:9200/_optimize?only_expunge_deletes=true'
or
curl -XPOST 'http://localhost:9200/_forcemerge?only_expunge_deletes=true'

Resources