ElasticSearch search perfomance - performance

Question

We have a two node cluster (VM in a private cloud, 64GB of RAM, 8 core CPU each node, CentOS), a few small indices ( ~1 mil documents) and one big index with ~220 mil docs (2 shards, 170GB of space). 24GB of memory is allocated to elastic search on each box.
Document structure:
{
'article_id': {
'index': 'not_analyzed',
'store': 'yes',
'type': 'long'
},
'feed_id': {
'index': 'not_analyzed',
'store': 'yes',
'type': 'string'
},
'title': {
'index': 'analyzed',
'type': 'string'
},
'content': {
'index': 'analyzed',
'type': 'string'
},
'lang': {
'index': 'not_analyzed',
'type': 'string'
}
}
It takes about 1-2 seconds to run the following query:
{
"query" : {
"multi_match" : {
"query" : "some search term",
"fields" : [ "title", "content" ],
"type": "phrase_prefix"
}
},
"size": 20,
"fields" :["article_id", "feed_id"]
}
Are we hitting hardware limits at this point or are there ways to optimize the query or data structure to increase performance?
Thanks in advance!

Answer 1

It's possible you are hitting the limits of your hardware, but there are a few things you can do to your query first to help optimize it.
Max Expansions
The first thing I would do is limit max_expansions. The way the prefix-queries work is by generating a list of prefixes that match the last token in your query. In your search query "some search term", the last token "term" would be expanded using "term" as the prefix seed. You may generate a list like this:
term
terms
terminate
terminator
termite
The prefix expansion process runs through your posting list looking for any word which matches the seed prefix. By default, this list is unbounded, which means you can generate a very large list of expansions.
The second phase rewrites your original query into a series of term queries using the expansions. The bigger the expansion list, the more terms are evaluated against your index and a corresponding decrease in speed.
If you limit the expansion process to something reasonable, you can maintain speed and still usually get good prefix matching:
{
"query" : {
"multi_match" : {
"query" : "some search term",
"fields" : [ "title", "content" ],
"type": "phrase_prefix",
"max_expansions" : 100
}
},
"size": 20,
"fields" :["article_id", "feed_id"],
}
You'll have to play with how many expansions you want. It is a tradeoff between speed and recall.
Filtering
In general, the other thing you can add is filtering. If there is some type of criteria you can filter on, you can potentially drastically improve speed. Currently, your query is executing against the entire index (250m documents), which is a lot to evaluate. If you can add filter that cuts that number down, you can see much improved latency.
At the end of the day, the fewer documents which the query evaluates, the faster the query will run. Filters decrease the number of docs that a query will see, are cached, operate very quickly, etc etc.
Your situation may not have any applicable filters, but if it does, they can really help!
File System Caching
This advice is entirely dependent on the rest of the system. If you aren't fully utilizing your heap (24gb) because you are doing simple search and filtering (e.g. not faceting / geo / heavy sorts / scripts) you may be able to reallocate your heap to the file system cache.
For example, if your max heap usage peaks at 12gb, it may make sense to decrease heap size down to 15gb. The extra 10gb that you freed will go back to the OS and help cache segments, which will help boost search performance simply by the fact that more operations are diskless.