Elasticsearch support for traditional chinese - elasticsearch

Question

I am trying to index and search Chinese into Elasticsearch. By using Smart Chinese Analysis (elasticsearch-analysis-smartcn) plugin I have managed to search characters and words for both simplified and traditional chinese. I have tried to insert the same text in both simplified and traditional chinese, but the search returns only one result (depending on how the search is performed); since the text is the same I would expect both results to be returned. I have read here that in order to support traditional chinese I must also install the STConvert Analysis (elasticsearch-analysis-stconvert) plugin. Can anyone provide a working example that uses these two plugins? (or an alternative method that achieves the same result)
The test index is created as
{
"settings":{
"analysis":{
"analyzer":{
"chinese":{
"type":"smartcn"
}
}
}
},
"mappings":{
"testType":{
"properties":{
"message":{
"store":"yes",
"type":"string",
"index":"analyzed",
"analyzer":"chinese"
},
"documentText": {
"store":"compress",
"type":"string",
"index":"analyzed",
"analyzer":"chinese",
"termVector":"with_positions_offsets"
}
}
}
}
}
and the two requests with the same text in simplified-traditional are
{
"message": "汉字",
"documentText": "制造器官的噴墨打印機 這是一種制造人體器官的裝置。這種裝置是利用打印機噴射生物 細胞、 生長激素、凝膠體，形成三維的生物活體組織。凝膠體主要是為細胞提供生長的平台，之后逐步形成所想要的器官或組織。這項技術可以人工方式制造心臟、肝臟、腎臟。這項研究已經取得了一定進展，目前正在研究如何將供應營養的血管印出來。這個創意目前已經得到了佳能等大公司的贊助"
}
{
"message": "汉字",
"documentText": "制造器官的喷墨打印机 这是一种制造人体器官的装置。这种装置是利用打印机喷射生物 细胞、 生长激素、凝胶体，形成叁维的生物活体组织。凝胶体主要是为细胞提供生长的平台，之后逐步形成所想要的器官或组织。这项技术可以人工方式制造心脏、肝脏、肾脏。这项研究已经取得了一定进展，目前正在研究如何将供应营养的血管印出来。这个创意目前已经得到了佳能等大公司的赞助"
}
Finally, a sample search that I want to return two results is
{
"query":{
"query_string":{
"query":"documentText : 制造器官的喷墨打印机",
"default_operator":"AND"
}
}
}

Answer 1

After many attempts I found a configuration that works. I did not manage to make smartcn work with stconvert plugin, so I used the cjk analyzer of elasticsearch, with an addition of icu_tokenizer instead. By using t2s and s2t as filters, each character is stored in both forms, traditional and simplified.
{
"settings":{
"analysis":{
"filter": {
"english_stop": {
"type": "stop",
"stopwords": "_english_"
},
"t2s_convert": {
"type": "stconvert",
"delimiter": ",",
"convert_type": "t2s"
},
"s2t_convert": {
"type": "stconvert",
"delimiter": ",",
"convert_type": "s2t"
}
},
"analyzer": {
"my_cjk": {
"tokenizer": "icu_tokenizer",
"filter": [
"cjk_width",
"lowercase",
"cjk_bigram",
"english_stop",
"t2s_convert",
"s2t_convert"
]
}
}
}
},
"mappings":{
"testType":{
"properties":{
"message":{
"store":"yes",
"type":"string",
"index":"analyzed",
"analyzer":"my_cjk"
},
"documentText": {
"store":"compress",
"type":"string",
"index":"analyzed",
"analyzer":"my_cjk",
"termVector":"with_positions_offsets"
}
}
}
}
}