Optimize Elasticsearch Writes slowing down Reads - elasticsearch

Question

I have a Flink job that's bulk writing/upserting a few thousands docs per second onto Elasticsearch. Every time I query it takes ~10-20 seconds to get a response.
I have second index that's exactly the same and equally as full on the same cluster but writes are now turned down to 0 on this index. When I query it it takes milliseconds to get a response.
I.e. with writes off queries take milliseconds. With writes on queries take 10-20 seconds.
CPU utilization ~10%, JVM mem pressure ~70%. ES 7.8.
It would appear then that writes to shards are somehow slowing the reads down. This is odd considering with "profile": true it's giving me query timings on the order of milliseconds yet took (total request time) is in seconds like I'm seeing.
My question is why might this be happening, and how can I optimize it?
(I did think of maybe I could have read replica nodes, but ES doesn't support a read replica node type https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles )

Answer 1

Not sure, what was your original refresh_interval value, but the default is 1 second which you set explicitly and it makes a difference on what documents are returned in search results.
Refresh makes documents to be written to segments which is used by search queries and without refresh you will get the obsolete(for existing docs) also new documents written after refresh will not be available in search results.
But it doesn't make difference in the performance of the search queries and in your case while indexing is also happening, search queries are taking more time which you need to debug more(see the CPU, memory, node stats(queue size of search and index queue)) etc.
By the way, the replica is related to shard and not the node, and you can easily increase the replica of an index dynamically to improve the search performance.
PUT /my-index-000001/_settings
{
"index" : {
"number_of_replicas" : 2 // no of replica you want to set.
}
}

Answer 2

Setting refresh_interval to 1s seems to have fixed the issue. If anyone has an explanation as to why I'd appreciate it.
curl -X PUT "host/index/_settings?pretty&human=true" -H 'Content-Type: application/json' -d'
{
"index" : {
"refresh_interval" : "1s"
}
}
Edit:
The refresh rate changes dynamically based on read load unless it's explicitly set.
By default, Elasticsearch periodically refreshes indices every second, but only on indices that have received one search request or more in the last 30 seconds. You can change this default interval using the index.refresh_interval setting.
https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-refresh.html#refresh-api-desc