Elasticsearch NEST: specifying Id explicitly seems to cause inconsistent search scores - elasticsearch

Question

I have a model class that looks like this:
public class MySearchDocument
{
public string ID { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public int DBID { get; set; }
}
We always use bulk indexing. By default our searches do a relatively simple multi_match with more weight given to ID and Name, like this:
{
"query": {
"multi_match": {
"query": "burger",
"fields": [
"ID^1.2",
"Name^1.1",
"Description"
],
"auto_generate_synonyms_phrase_query": true
}
}
}
I was previously just relying on Id inference, allowing Elasticsearch to use my ID property for its Id purposes, but for a few reasons it has become preferable to use DBID as the Id property in Elasticsearch. I tried this 3 different ways, separately and in combo:
Explicitly when bulk indexing: new BulkIndexOperation<MySearchDocument>(d) { Id = d.DBID }
In the ConnectionSettings using DefaultMappingFor<MySearchDocument>(d => d.IdProperty(p => p.DBID))
Using an attribute on MySearchDocument: [ElasticsearchType(IdProperty = nameof(DBID))]
Any and all of these seem to work as expected; the _id field in the indexed documents are being set to my DBID property. However, in my integration tests, search results are anything but expected. Specifically, I have a test that:
Creates a new index from scratch.
Populates it with a handful of MySearchDocuments
Issues a Refresh on the index just to make sure it's ready.
Issues a search.
Asserts that the results come back in the expected order.
With Id inference, this test consistently passes. When switching the Id field using any or all of the techniques above, it passes maybe half the time. Looking at the raw results, the correct documents are always returned, but the _score often varies for the same document from test run to test run. Sometimes the varying score is the one associated with the document whose ID field matches the search term, other times it's the score of a different document.
I've tried coding the test to run repeatedly and in parallel. I've tried waiting several seconds after issuing Refresh, just to be sure the index is ready. None of these make a difference - the test passes consistently with Id inference, and is consistently inconsistent without. I know nothing in this world is truly random, so I feel like I must be missing something here. Let me know if more details would be helpful. Thanks in advance.

Answer 1

Search relevancy scores are calculated per shard, and a hashing algorithm on the value of _id determines into which primary shard a given document will be indexed.
It sounds like you may be seeing the effects of this when indexing a small sample of documents across N > 1 primary shards; in this case, the local relevancy scores may be different enough to manifest in some odd looking _scores returned. With a larger set of documents and even distribution, differences in local shard scores diminish.
There are a couple of approaches that you can take to overcome this for testing purposes:
Use a single primary shard
or
Use dfs_query_then_fetch when making the search request. This tells Elasticsearch to take the local relevancy scores first in order to calculate global relevancy scores, then use global scores for _score. There is a slight overhead to using dfs_query_then_fetch.
Take a look also at the section "Relevance is Broken!" from the Elasticsearch Definitive guide; although the guide refers to Elasticsearch 2.x, much of it is still very much relevant for later versions.