I have a elasticsearch cluster set up with node query cache enabled, I have set the size of the cache to be 2gb, but I am not completely sure how the LRU caching policy works in this case.
I have a query context run against the elasticsearch index and i expect the result to be cached, so that when there is request for the same query context again - there should be increase in the hit_count, but this is not the behavior i see in ES.
These are the stats of my query_cache
memory_size_in_bytes: 7176480,
total_count: 36605,
hit_count: 15657,
miss_count: 20948,
cache_size: 130,
cache_count: 130,
evictions: 0
Even though the memory_size_in_bytes is not reached its max. The result of the query context is not completely cached and when the same query context is fired against the elasticsearch index i see miss counts stats getting increased rather than hit counts.
Can anyone please explain about how the node query caching works in ES.
Related
By default, the requests cache will only cache the results of search requests where size=0, so it will not cache hits, but it will cache hits.total, aggregations, and suggestions.
I do not understand the part where stated: "size=0".
What is the the size context/meaning here?
Does it mean that results cache will
cache only for empty results?
cache page 1 only (default 10 results I think)?
No, size param is useful if you want to fetch results different than 10, as default size is 10, so if you are using a search query for which you need to fetch lets suppose 1000 results, than you specify size param to 1000, without this you will get only top 10 search results, sorted on their score in descending order.
size=0, in shard request cache, is that it will not cache the exact results(ie number of documents with their score) but only cache the metadata like total number of results(which is hits.total) and other things.
I have a Flink job that's bulk writing/upserting a few thousands docs per second onto Elasticsearch. Every time I query it takes ~10-20 seconds to get a response.
I have second index that's exactly the same and equally as full on the same cluster but writes are now turned down to 0 on this index. When I query it it takes milliseconds to get a response.
I.e. with writes off queries take milliseconds. With writes on queries take 10-20 seconds.
CPU utilization ~10%, JVM mem pressure ~70%. ES 7.8.
It would appear then that writes to shards are somehow slowing the reads down. This is odd considering with "profile": true it's giving me query timings on the order of milliseconds yet took (total request time) is in seconds like I'm seeing.
My question is why might this be happening, and how can I optimize it?
(I did think of maybe I could have read replica nodes, but ES doesn't support a read replica node type https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles )
Not sure, what was your original refresh_interval value, but the default is 1 second which you set explicitly and it makes a difference on what documents are returned in search results.
Refresh makes documents to be written to segments which is used by search queries and without refresh you will get the obsolete(for existing docs) also new documents written after refresh will not be available in search results.
But it doesn't make difference in the performance of the search queries and in your case while indexing is also happening, search queries are taking more time which you need to debug more(see the CPU, memory, node stats(queue size of search and index queue)) etc.
By the way, the replica is related to shard and not the node, and you can easily increase the replica of an index dynamically to improve the search performance.
PUT /my-index-000001/_settings
{
"index" : {
"number_of_replicas" : 2 // no of replica you want to set.
}
}
Setting refresh_interval to 1s seems to have fixed the issue. If anyone has an explanation as to why I'd appreciate it.
curl -X PUT "host/index/_settings?pretty&human=true" -H 'Content-Type: application/json' -d'
{
"index" : {
"refresh_interval" : "1s"
}
}
Edit:
The refresh rate changes dynamically based on read load unless it's explicitly set.
By default, Elasticsearch periodically refreshes indices every second, but only on indices that have received one search request or more in the last 30 seconds. You can change this default interval using the index.refresh_interval setting.
https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-refresh.html#refresh-api-desc
Curious if there is some way to check if document ID is part of a large (million+ results) Elasticsearch query/filter.
Essentially I’ll have a group of related document ID’s and only want to return them if they are part of a larger query. Hoping to do database side. Theoretically seemed possible since ES has to cache stuff related to large scrolls.
It's a interesting use-case but you need to understand that Elasticsearch(ES) doesn't return all the matching documents ids in the search result and return by default only the 10 documents in the response, which can be changed by the size parameter.
And if you increase the size param and have millions of matching docs in your query then ES query performance would be very bad and it might bring even entire cluster down if you frequently fire such queries(in absence of circuit breaker) so be cautious about it.
You are right that, ES cache the stuff, but again that if you try to cache huge amount of data and that is getting invalidate very frequent then you will not get the required performance benefits, so better do the benchmark against it.
You are already on the correct path to use, scroll API to iterate on millions on search result, just see below points to improve further.
First get the count of search result, this is included in default search response with eq or greater value which will give you idea that how many search results you have based on which you can give size param for subsequent calls to see if your id is present or not.
See if you effectively utilize the filters context in your query, which is by default cached at ES.
Benchmark your some heavy scroll API calls with your data.
Refer this thread to fine tune your cluster and index configuration to optimize ES response further.
I am trying to search and fetch the documents from Elasticsearch but in some cases, I am not getting the updated documents. By updated I mean, we update the documents periodically in Elasticsearch. The documents in ElasticSearch are updated at an interval of 30 seconds, and the number of documents could range from 10-100 Thousand. I am aware that the update is generally a slow process in Elasticsearch.
I am suspecting it is happening because Elasticsearch though accepted the documents but the documents were not available for searching. Hence I have the following questions:
Is there a way to measure the time between indexing and the documents being available for search? There is setting in Elasticsearch which can log more information in Elasticsearch logs?
Is there a setting in Elasticsearch which enables logging whenever the merge operation happens?
Any other suggestion to help in optimizing the performance?
Thanks in advance for your help.
By default the refresh_interval parameter is set to 1 second, so unless you changed this parameter each update will be searchable after maximum 1 second.
If you want to make the results searchable as soon as you have performed the update operation you can use the refresh parameter.
Using refresh=wait_for the endpoint will respond once a refresh has occured. If you use refresh=true a refresh operation will be triggered. Be careful using refresh=true if you have many update since it can impact performances.
When running curl -XPOST node1:9200/_cache/clear
I expected a large amount of cache to disappear in each node.
But when looking at bigdesk, cache size doesn't seem to drop a lot.
Shouldn't it drop a lot??
The main problem from that is that each node has generally very high cache size and it's the reason for exceptions like below:
ElasticsearchException[org.elasticsearch.common.breaker.CircuitBreakingException: Data too large, data for field [ts] would be larger than limit of [24108466176/22.4gb]]
Another thing is I don't see any cache eviction generally(not after clear_cache only), there are no properties for cache eviction set in settings(perhaps some should be set?). The only property set is fielddata.breaker.limit: 75%
What can be done for handling better cache in elasticseach?
ElasticSearch version: 1.3.5