In some of my indices, I'm doing "index.blocks.read_only_allow_delete": true by using the PUT /index/_settings API call. But after around 10 seconds, the setting disappears and the index is writable again.
I'm wondering if this can be a bug in ES, as in version 6.8 a change was made to reset this setting automatically when a node whose disk had gone over the flooding stage, was again below the normal thresholds.
I'm experiencing that odd behaviour in ES 7.9. What I expected is that, if ES changed the setting to true because of the watermarks, then it could reset it to false later. But if an operator changes the setting to true manually, then ES was going to respect that setting.
These are the docs where I read about that behaviour:
Controls the flood stage watermark, which defaults to 95%. Elasticsearch enforces a read-only index block ( index.blocks.read_only_allow_delete ) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark.
Cross-posted here.
I ended up using index.blocks.read_only instead, as this one is not updated by ElasticSearch automatically.
Related
I'm trying to measure Elasticsearch performance for some queries and make a benchmark. I'm looking for a way to completely disable the cache. I've already tried some ways but always my first request takes longer than the next queries. So I think even I disabled the cache, at some level it is still working! I've tried this:
1- GET my_index/_search?request_cache=false
2- POST /my_index/_cache/clear
3-
PUT /my_index/_settings
{ "index.requests.cache.enable": false }
I think you are disabling the elasticsearch cache correctly. the problem is that there is filesystem cache memory at OS level that you cannot disable it easily. every files and segment that have been read from harddisk will be cached in memory until new segment and files arrived. you can check this cache via free -g command and buffer column.
you can clear memory page cache via below command:
# sync; echo 3 > /proc/sys/vm/drop_caches
Installed Kibana and Elasticsearch on my macOS Catalina via Brew (not sudo), but I'm not able to install the sample data sets. Anyone have any idea why I'm getting this Forbidden error and how to resolve? The error message is on the bottom right of the picture
go to the conf of elasticsearch uncomment and fill the path.logs line with the right path
Check if you have enough(>90%) disk space available
A good way to look for reason of any error is logs if available :)
I was trying to load Sample Data (eCommerce orders, flight , web logs) in to my kibana. I was getting some error. Logs shown below
elasticsearch.log
[o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [/Users/xyz/Installs/ElasticSearch/elasticsearch-7.9.3/data/nodes/0] free: 15.1gb[6.4%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
My MAC has total 250 GB space, I freed up extra 20 GB, then it worked. Please check if you have enough memory (more than 90% should be available)
go to management -> index pattern -> create index pattern.
Looks like for some reason I hit the threshold where the indexes are locked because of no more disk space and so I had to unlock the indexes manually
https://discuss.elastic.co/t/unable-to-create-index-pattern-from-kibana/167184
I am indexing a large amount of daily data ~160GB per index into elasticsearch. I am facing this case where I need to update almost all the docs in the indices with a small amount of data(~16GB) which is of the format
id1,data1
id1,data2
id2,data1
id2,data2
id2,data3
.
.
.
My update operations start happening at 16000 lines per second and in over 5 minutes it comes down to 1000 lines per second and doesnt go up after that. The update process for this 16GB of data is currently longer than the time it takes for my entire indexing of 160GB to happen
My conf file for the update operation currently looks as follows
output
{
elasticsearch {
action => "update"
doc_as_upsert => true
hosts => ["host1","host2","host3","host4"]
index => "logstash-2017-08-1"
document_id => "%{uniqueid}"
document_type => "daily"
retry_on_conflict => 2
flush_size => 1000
}
}
The optimizations I have done to speed up indexing in my cluster based on the suggestions here https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html are
Setting "indices.store.throttle.type" : "none"
Index "refresh_interval" : "-1"
I am running my cluster on 4 instances of the d2.8xlarge EC2 instances. I have allocated 30GB of heap to each nodes.
While the update is happening barely any cpu is used and the load is very less as well.
Despite everything the update is extremely slow. Is there something very obvious that I am missing that is causing this issue? While looking at the threadpool data I find that the number of threads working on bulk operations are constantly high.
Any help on this issue would be really helpful
Thanks in advance
There are a couple of rule-outs to try here.
Memory Pressure
With 244GB of RAM, this is not terribly likely, but you can still check it out. Find the jstat command in the JDK for your platform, though there are visual tools for some of them. You want to check both your Logstash JVM and the ElasticSearch JVMs.
jstat -gcutil -h7 {PID of JVM} 2s
This will give you a readout of the various memory pools, garbage collection counts, and GC timings for that JVM as it works. It will update every 2 seconds, and print headers every 7 lines. Spending excessive time in the FCT is a sign that you're underallocated for HEAP.
I/O Pressure
The d2.8xlarge is a dense-storage instance, and may not be great for a highly random, small-block workload. If you're on a Unix platform, top will tell you how much time you're spending in IOWAIT state. If it's high, your storage isn't up to the workload you're sending it.
If that's the case, you may want to consider provisioned IOP EBS instances rather than the instance-local stuff. Or, if your stuff will fit, consider an instance in the i3 family of high I/O instances instead.
Logstash version
You don't say what version of Logstash you're using. Being StackOverflow, you're likely to be using 5.2. If that's the case, this isn't a rule-out.
But, if you're using something in the 2.x series, you may want to set the -w flag to 1 at first, and work your way up. Yes, that's single-threading this. But the ElasticSearch output has some concurrency issues in the 2.x series that are largely fixed in the 5.x series.
With elasticsearch version 6.0 we had an exactly same issue of slow updates on aws and the culprit was slow I/O. Same data was upserting on a local test stack completely fine but once in cloud on ec2 stack, everything was dying after an initial burst of speedy inserts lasting only for few minutes.
Local test stack was a low-spec server in terms of memory and cpu but contained SSDs.
s3 stack was EBS volumes with default gp2 300 IOPS.
Converting the volumes to type io1 with 3000 IOPS solved the issue and everything got back on track.
I am using amazon aws elasticsearch service version 6.0 . I need heavy write/insert from serials of json file to the elasticsearch for 10 billion items . The elasticsearch-py bulk write speed is really slow most of time and occasionally high speed write . i tried all kinds of methods , such as split json file to smaller pieces , multiprocess read json files , parallel_bulk insert into elasticsearch , nothing works . Finally , after I upgraded io1 EBS volume , everything goes smoothly with 10000 write IOPS .
I recently ran into a problem with elasticsearch, versions 1.0.1, 1.2.2, 1.2.4, and 1.4.1.
When I try to get a document by ID GET http://localhost:9200/thing/otherthing/700254a4-4e72-46b9-adeb-d498159652cb It will return the document half the time, and the other half I will get a "found" : false error. (These switch off literally every other time, I do a get and it works, do another get and it doesn't).
These documents have no custom routing.
I have tried completely uninstalling elasticsearch and removing all files related to it, then re installing from the official repo to no avail, and google doesn't give me any similar problems or ideas on how to solve this.
The only thing I would think of that would cause a repeatable failure like this would be unassigned shards/replica sets which contain this information.
Do you know how many replica sets you have?
I believe the read is round-robin, so if you only have 2 replicas of the data (1 master + a replica set), and 1 has become unassigned (after being written to), then you might see a failure like this.
I have been having a few jobs failing due to OutOfMemory and GC overhead limit exceeded errors. To counter the former I tried setting SET mapred.child.java.opts="-Xmx3G"; at the start of the hive script**.
Basically any time I add this option to the script, the MR jobs that get scheduled(for the first of several queries in the script) are 'killed' right away.
Any thoughts on how to rectify this? Are there any other params that need to be tinkered with in conjunction with max heap space(eg. io.sort.mb)? Any help would be most appreciated.
FWIW, I am using hive-0.7.0 with hadoop-0.20.2. The default setting for max heap size in our cluster is 1200M.
TIA.
** - Some other alternatives that were tried(with comical effect but no discernible change in outcome):
SET mapred.child.java.opts="-Xmx3G";
SET mapred.child.java.opts="-server -Xmx3072M";
SET mapred.map.child.java.opts ="-server -Xmx3072M";
SET mapred.reduce.child.java.opts ="-server -Xmx3072M";
SET mapred.child.java.opts="-Xmx2G";
Update 1: It is possible that it's not necessarily anything to do with setting heap size. Tinkering with mapred.child.java.opts in any way is causing the same outcome. For example setting it thusly, SET mapred.child.java.opts="-XX:+UseConcMarkSweepGC"; is having the same result of MR jobs getting killed right away. Or even setting it explicitly in the script to what is the 'cluster default' causes this.
Update 2: Added a pastebin of a grep of JobTracker logs here.
Figured this would end up being something trivial/inane and it was in the end.
Setting mapred.child.java.opts thusly:
SET mapred.child.java.opts="-Xmx4G -XX:+UseConcMarkSweepGC";
is unacceptable. But this seem to go through fine:
SET mapred.child.java.opts=-Xmx4G -XX:+UseConcMarkSweepGC; (minus the double-quotes)
sigh. Having better debug options/error messages would have been nice.
Two other guards can restrict task memory usage. Both are designed for admins to enforce QoS, so if you're not one of the admins on the cluster, you may be unable to change them.
The first is the ulimit, which can be set directly in the node OS, or by setting mapred.child.ulimit.
The second is a pair of cluster-wide mapred.cluster.max.*.memory.mb properties that enforce memory usage by comparing job settings mapred.job.map.memory.mb and mapred.job.reduce.memory.mb against those cluster-wide limits.