Is it possible to add a new similarity metric to an existing index in Elasticsearch? - elasticsearch

Question

Let's say there is an existing index with a customized BM25 similarity metric like this:
{
"settings": {
"index": {
"similarity": {
"BM25_v1": {
"type": "BM25",
"b": 1.0
}
},
"number_of_replicas": 0,
"number_of_shards": 3,
"refresh_interval": "120s"
}
}
}
And this similarity metric is used for two fields:
{
'some_field': {
'type': 'text',
'norms': 'true',
'similarity': 'BM25_v1'
},
'another_field': {
'type': 'text',
'norms': 'true',
'similarity': 'BM25_v1'
},
}
Now, I was wondering if it's possible to add another similarity metric (BM25_v2) to the same index and use this new metric for the another_field, like this:
"index": {
"similarity": {
# The existing metric, not changed.
"BM25_v1": {
"type": "BM25",
"b": 1.0
},
# The new similarity metric for this index.
"BM25_v2": {
"type": "BM25",
"b": 0.0
}
}
}
# ... and use the new metric for one of the fields:
{
'some_field': {
'type': 'text',
'norms': 'true',
'similarity': 'BM25_v1' # This field uses the same old metric.
},
'another_field': {
'type': 'text',
'norms': 'true',
'similarity': 'BM25_v2' # The new metric is used for this field.
},
}
I couldn't find any example for this scenario in the documentation, so I wasn't sure if this is possible at all.
Update: I have already seen this old still-open issue which concerns with dynamic update of similarity metrics in Elasticsearch. But it is not completely clear from that discussion what is and isn't possible. Also there have been some attempts for achieving some level of similarity update; but I think it's not documented (e.g. it is possible to change the parameters of an existing similarity metric, say b or k1 in an existing BM25-based metric).

Answer 1

TLDR;
I believe you can't.
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Mapper for [title] conflicts with existing mapper:\n\tCannot update parameter [similarity] from [my_similarity] to [my_similarity_v2]"
}
],
"type" : "illegal_argument_exception",
"reason" : "Mapper for [title] conflicts with existing mapper:\n\tCannot update parameter [similarity] from [my_similarity] to [my_similarity_v2]"
},
"status" : 400
}
If you want to, I believe you will have to create a new field and re-index the data.
To reproduce
PUT /70973345
{
"settings": {
"index": {
"similarity": {
"my_similarity": {
"type": "BM25",
"b": 1.0
}
}
}
}
}
PUT /70973345/_mapping
{
"properties" : {
"title" : { "type" : "text", "similarity" : "my_similarity" }
}
}
We insert some dummy data, and retrieve it.
POST /70973345/_doc
{
"title": "I love rock'n roll"
}
POST /70973345/_doc
{
"title": "I love pasta al'arabita"
}
POST /70973345/_doc
{
"title": "pasta rock's"
}
GET /70973345/_search?explain=true
{
"query": {
"match": {
"title": "pasta"
}
}
}
If we try to update it the settings without closing, we get an error.
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Can't update non dynamic settings ...."
}
],
"type" : "illegal_argument_exception",
"reason" : "Can't update non dynamic settings ...."
},
"status" : 400
}
POST /70973345/_close?wait_for_active_shards=0
PUT /70973345/_settings
{
"index": {
"similarity": {
"my_similarity": {
"type": "BM25",
"b": 1.0
},
"my_similarity_v2": {
"type": "BM25",
"b": 0
}
}
}
}
The update works fine, BUT :
PUT /70973345/_mapping
{
"properties": {
"title": {
"type": "text",
"similarity": "my_similarity_v2"
}
}
}
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Mapper for [title] conflicts with existing mapper:\n\tCannot update parameter [similarity] from [my_similarity] to [my_similarity_v2]"
}
],
"type" : "illegal_argument_exception",
"reason" : "Mapper for [title] conflicts with existing mapper:\n\tCannot update parameter [similarity] from [my_similarity] to [my_similarity_v2]"
},
"status" : 400
}
It will not work, regardless of the open/close status of the index.
Which makes me believe this is not possible. you might need to re-index into a new indice the existing data.