Elasticsearch modify asciifolding - elasticsearch

Question

ASCII Folding Token Filter folds "Ə"/"ə"(U+018F / U+0259) characters to "A"/"a". I need to modify or add fold to "E"/"e". char_filter doesn't help and doesn't preserve original
Add analyzer:
curl -XPUT 'localshot:9200/myix/_settings?pretty' -H 'Content-Type: application/json' -d'
{
"analysis" : {
"analyzer" : {
"default" : {
"tokenizer" : "standard",
"filter" : ["standard", "my_ascii_folding"]
}
},
"filter" : {
"my_ascii_folding" : {
"type" : "asciifolding",
"preserve_original" : true
}
}
}
}
'
Test result:
http://localhost:9200/myix/_analyze?text=üöğıəçşi_ÜÖĞIƏÇŞİ&filter=my_ascii_folding
{
"tokens": [
{
"token": "uogiacsi_UOGIACSI",
"start_offset": 0,
"end_offset": 17,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "üöğıəçşi_ÜÖĞIƏÇŞİ",
"start_offset": 0,
"end_offset": 17,
"type": "<ALPHANUM>",
"position": 0
}
]
}

Answer 1

When looking at Lucene's ASCIIFoldingFilter.java source file, it doesn indeed seem like Ə gets folded into an E and not a A. Even the ICU folding filter which is asciifolding on steroids, does the same folding.
However, there's an interesting discussion on the subject and it seems that given the pronunciation it should be folded into an a and not a e:
A quick search on English or French Wikipedia, where it currently gets folded, shows that it gets folded to an a! I would have expected an e based on orthography, but a makes sense in terms of pronunciation (in English, at least).
Someone else even thinks that neither a nor e makes sense:
That seems like a really bad decision. I don't think ə should fold to either of a or e.
Anyway, I don't think there is a way except using a char_filter or extending the ASCIIFoldingFilter and bundling it into an ES analysis plugin yourself.