Updating Documents in Elasticsearch is not Applying the Custom Analyzer to the Fields Data - elasticsearch

Question

I have a field that has a custom analyzer on it that is to put the data into lowercase.
The analyzer is defined as:
"analysis" : {
"analyzer" : {
"custom_keyword_analyzer" : {
"filter" : [
"lowercase"
],
"type" : "custom",
"tokenizer" : "keyword"
}
}
}
With the mapping on the field like:
"Field" : {
"type" : "text",
"fields" : {
"raw" : {
"type" : "keyword"
}
},
"copy_to" : [
"all_field"
],
"analyzer" : "custom_keyword_analyzer",
"fielddata" : true
}
When creating documents normally with data in Field the analyzer is working correctly. Field has the data in lowercase and Field.raw has the original un-analyzed data in it.
However if the documents are created without anything in Field but are later updated, the analyzer is not used, Field has the un-analyzed data in it and Field.raw is empty.
I have tried manually scripting bulk updates in python, and also using _update_by_query to perform the updates. In no cases can I get the analyser to work on the updated data.

Answer 1

I don't know how you are verifying that your updated doc is not having the analyzer impact, below is complete example to show you, it works and how you can check it.
Index mapping according to your def
{
"settings": {
"analysis": {
"analyzer": {
"custom_keyword_analyzer": {
"filter": [
"lowercase"
],
"type": "custom",
"tokenizer": "keyword"
}
}
}
},
"mappings": {
"properties": {
"title": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
},
"copy_to": [
"all_field"
],
"analyzer": "custom_keyword_analyzer",
"fielddata": true
}
}
}
}
Index sample doc
{
"title" : "Hello world"
}
Check the analyzed value in inverted index
use _search endpoint with below query
{
"docvalue_fields": [
"title",
"title.raw"
],
"query": {
"term": {
"_id": 1
}
}
}
Result of above query
"_source": {
"title": "Hello world" // actual indexed content
},
"fields": {
"title.raw": [
"Hello world" // same as keyword analyzer
],
"title": [
"hello world" // notice lowercased `h`.
]
}
}
Now update the doc, by using the PUT API
{
"title" : "Hello world Updated" // note `U` in `Updated`
}
And again use the same _search query
"_source": {
"title": "Hello world Updated"
},
"fields": {
"title.raw": [
"Hello world Updated"
],
"title": [
"hello world updated" // note lowercase
]
}
As you can see, even after updating the document analyzer impact is present and that how it works and can be verified, it's very core functionality and can't be broken and maybe you are missing something while verifying and above method should give you some way to identify your mistake