Copy analyzed text into another field - elasticsearch

Question

We are trying to build a WordCloud over four Text-Fields. Each field has its own Stop Analyzer.
For example TextFr with a French Stop Analyzer, TextDe with a German Stop Analyzer. The analyzed result should be copied into another field called WordCloudText on which the aggregations takes place.
Do you have any advice how to do this? Is this even possible?
Thanks for your help

Answer 1

I don't think there is a way to copy the analyzed output of a field, only the values (unanalyzed) of a field. Probably the easiest way to achieve this is to define your own analyzer that filters all of four languages. Something like this:
PUT stackoverflow
{
"settings": {
"analysis": {
"filter": {
"english_stop": {
"type": "stop",
"stopwords": "_english_"
},
"dutch_stop": {
"type": "stop",
"stopwords": "_dutch_"
}
},
"analyzer": {
"eng_stop": {
"type": "stop",
"stopwords": "_english_"
},
"dutch_stop": {
"type": "stop",
"stopwords": "_dutch_"
},
"all_lang_stop": {
"tokenizer": "lowercase",
"filter": [
"english_stop",
"dutch_stop"
]
}
}
}
},
"mappings": {
"record": {
"properties": {
"field": {
"type": "keyword",
"fields": {
"english": {"type": "text", "analyzer": "eng_stop" },
"dutch": {"type": "text", "analyzer": "dutch_stop" },
"word_cloud": {"type": "text", "analyzer": "all_lang_stop"}
}
}
}
}
}
}
The key is the custom analyzer called all_lang_stop that combines multiples top filters. Then you use a multi-field to have your data automatically copied to into each type of stop analyzer.
Alternatively, if your text is already separated into different fields by language, you can use the copy_to directive on each individual language field to copy it into the word_cloud field. Note that copy_to copies the input value, not the output value of the analyzer, so you still need the combined analyzer. Something like this:
"mappings": {
"record": {
"properties": {
"english": {"type": "text", "analyzer": "eng_stop", copy_to: "word_cloud"},
"dutch": {"type": "text", "analyzer": "dutch_stop", copy_to: "word_cloud"},
"word_cloud": {"type": "text", "analyzer": "all_lang_stop"}
}
}
}