I changed one of my mappings so I started a reindex using kopf. How can I tell when the reindex is done?
I've tried searching the output of the following:
curl -GET http://example-elastic:9200/logstash-2015.03.31/_status?pretty
curl -GET http://example-elastic:9200/_cat/indices
curl -GET http://example-elastic:9200/_stats/indices
Update
I started the reindex with the kopf plugin. (Removed incorrect indexing command from question)
Update2
I figured out how to see the last index date, but it seems to me like there is a better way to tell if it is currently being indexed.
Works on OSX
FOO=logstash-2015.03.31
curl -s -XGET "http://example-elastic:9200/l$FOO?pretty" | grep creation_date | sed s'/[^0-9]//g' | date -j
Wed Apr 8 16:42:23 MDT 2015
Update3
I installed a plugin to give more feedback of how long an index takes.
elasticsearch-reindex -f http://example-elastic:9200/logstash-2015.03.22/string -t http://example-elastic:9200
It looks like the indexing procedure takes a long time. 5 minutes in, and it is still at 2%
...
reindexing [=-----------------------------] 165200/8492680(2%) 158.1 7971.1s
reindexing [=-----------------------------] 165300/8492680(2%) 158.5 7984.9s
reindexing [=-----------------------------] 165400/8492680(2%) 158.5 7981.0s
Is there a way to see this progress from elasticsearch?
Take a look at the cat api and in particular, the output of /_cat/nodes?help. You'll probably want to monitor the indexing ops with:
curl -XGET 'http://localhost:9200/_cat/nodes?v&h=indexing.index_current'
Related
I'm using Windows 10 and I'm getting
Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]
How can I resolve this? I don't mind if I lose data since it only runs locally.
Aside from the answers mentioned above, you can also try increasing the shards until you try to rearchitect the nodes
curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'
Besides, the following can be useful the should be proceeded with CAUTION ofcourse
Get total number of unassigned shards in cluster
curl -XGET -u elasticuser:yourpassword http://localhost:9200/_cluster/health\?pretty | grep unassigned_shards
USE WITH CAUTION
To DELETE the unassigned shards in a cluster (USE WITH CAUTION)
curl -XGET -u elasticuser:yourpassword http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} #(USE WITH CAUTION) | xargs -i curl -XDELETE -u elasticuser:yourpassword "http://localhost:9200/{}"
You are reaching the limit cluster.max_shards_per_node. Add more data node or reduce the number of shards in cluster.
If you don't mind about the data loss, delete old indicies.
The easy way is do it from the GUI Kibana > Management > DevTools, then to get all indicies:
GET /_cat/indices/
You can delete within a pattern like below:
DELETE /<index-name>
e.g.:
DELETE /logstash-2020-10*
You probably have too many shards per node.
May I suggest you look at the following resources about sizing:
https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing
I have the problem that this query doesnt bring the information for the host fancyserver01, there are connections from other machines.
In fact, this query against a elastic5 instance does not match only one host.
curl -XGET -u "USER:PASSWORD" 'https://elasticserver:9200/connbeat-*/_search?_source=local_ip,local_port,remote_ip,remote_port' -d '{"query":{"bool":{"must":[{"query_string":{"analyze_wildcard":"true","query":"beat.hostname":"fancyserver01"}},{"range":{"#timestamp":{"from":"now-24h","to":"now"}}}]}}, "from":0,"size":5000000,"sort":[]})'
any idea where is my problem here.
The Problem is the "-" char, wich brings me the data of fancyserver and live01 instead of fancyserver-live01
"beat.hostname":"fancyserver-live01"
and i have this quote like this
"beat.hostname:\""fancyserver-live01\""
To get the structure of an Elasticsearch index via CLI, we can do:
curl -u myuser:p4ssw0rd -XGET "https://myeshost:9200/myindexname"
Is there a way to get the structure (or other information) about a Kibana index pattern, or get the list of all Kibana index patterns that have been created? I haven't found information about this on the documentation.
There is a way to retrieve all Kibana index-patterns using the command below:
GET .kibana/_search?size=100&q=type:"index-pattern"
Note: if you have more than 100 index-patterns, you might want to increase the size.
Using the functions _stats or _settings:
curl -u myuser:p4ssw0rd -XGET "https://myeshost:9200/myindexname/_stats"
curl -u myuser:p4ssw0rd -XGET "https://myeshost:9200/myindexname/_settings"
Reference:
https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-stats.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.3/indices-get-settings.html
I am trying to delete a type in elastic search using curl script in bat file
ECHO Running Curl Script
curl -XDELETE "http://localhost/testing/" -d''
pause
The response that i got was No handler found for uri . I looked into documentation of Elastic Search and it says to use delete by query https://www.elastic.co/guide/en/elasticsearch/reference/5.0/docs-delete-by-query.html
How can i modify the my curl script to use this new api for ES 2.3
Thanks
If you want to use the delete-by-query API to delete all documents of a given type, you can do it like this:
curl -XDELETE "http://localhost/testing/_query?q=_type:typename"
However, you're better off deleting the index and recreating it so you can modify the mapping type as you see fit.
curl -XDELETE "http://localhost/testing/"
curl -XPUT "http://localhost/testing/" -d '{"settings": {...}, "mappings": {...}}'
Elasticsearch: 2.1
Timezone: +0800
Index: nginx-2015.12.18
Current Time: 2015-12-18 16:08
According to doc following query will resolve to index nginx-2015.12.18
curl -XGET 'localhost:9200/<nginx-{now/d}>/_search'
But, in reality it gives
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"<nginx-now","index":"<nginx-now"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"<nginx-now","index":"<nginx-now"},"status":404}
curl 'localhost:9200/_cat/indices?v'
⋊> ~ curl 'localhost:9200/_cat/indices?v' 16:31:35
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open nginx 5 1 18 0 120.7kb 120.7kb
yellow open wx-data-2015.12.18 5 1 2 0 9.6kb 9.6kb
yellow open nginx-2015.12.18 5 1 5 0 33kb 33kb
As it turns out, this is a known issue, This is what #javanna said,
we had initially treated this as a bug and fixed it, but afterwards we
found out that the fix introduced a regression (#14177), which made us
take a step back. Any slash that should not be used as a path
separator in a uri should be properly escaped, and it is wrong to try
and make distinctions between the different slashes on the server side
depending on what surrounds them, that heuristic is not going to fly
(in fact it didn't :) )
We are going to revert the initial fix then, the solution to this
problem is to escape the '/' in the url like this: curl -XGET
'localhost:9200/<logstash-{now%2Fd}>/animals/1'. That is going to work
and didn't require any fix in the first place.
I also had to escape the {}, This worked for me on ES 2.0
curl -XGET 'localhost:9200/<nginx-\{now%2Fd\}>/_search'
Does this help?