Comma in Elasticsearch synonyms - elasticsearch

Question

Some of the sentences I want to define as syonyms have commas in them, for example:
"Employment Information" and "Your Activity, Your Job" are synonyms.
However if I define them in the following way the result is not what I envisioned, since "," has a special meaning in the Elasticsearch format:
Employment Information=>Your Activity, Your Job
Is the only solution for me to use WordNet synonym format in this case or perhaps I can just ignore the comma entirely and take it out?

Answer 1

I dont think comma will be an issue, If you are using standard analyzer then it will remove comma. This is my test setup
"filter": {
"my_synonym_filter": {
"type": "synonym",
"synonyms": [
"employment information=>your activity your job"
]
}
},
"analyzer": {
"my_synonyms": {
"tokenizer": "standard",
"filter": [
"lowercase",
"my_synonym_filter"
]
}
}
It is good to use lowercase filter to avoid case sensitive issues, so now this query
GET my_index/_analyze?text=employment Information&analyzer=my_synonyms
will give you following tokens
{
"tokens": [
{
"token": "your",
"start_offset": 0,
"end_offset": 10,
"type": "SYNONYM",
"position": 1
},
{
"token": "activity",
"start_offset": 11,
"end_offset": 22,
"type": "SYNONYM",
"position": 2
},
{
"token": "your",
"start_offset": 11,
"end_offset": 22,
"type": "SYNONYM",
"position": 3
},
{
"token": "job",
"start_offset": 11,
"end_offset": 22,
"type": "SYNONYM",
"position": 4
}
]
}
There is a gotcha with multiword synonym, if you analyze the output of
GET my_index/_analyze?text=employment Information is useful&analyzer=my_synonyms, you will get unexpected results like this
{
"tokens": [
{
"token": "your",
"start_offset": 0,
"end_offset": 10,
"type": "SYNONYM",
"position": 1
},
{
"token": "activity",
"start_offset": 11,
"end_offset": 22,
"type": "SYNONYM",
"position": 2
},
{
"token": "is",
"start_offset": 23,
"end_offset": 25,
"type": "<ALPHANUM>",
"position": 3
},
{
"token": "your",
"start_offset": 23,
"end_offset": 25,
"type": "SYNONYM",
"position": 3
},
{
"token": "useful",
"start_offset": 26,
"end_offset": 32,
"type": "<ALPHANUM>",
"position": 4
},
{
"token": "job",
"start_offset": 26,
"end_offset": 32,
"type": "SYNONYM",
"position": 4
}
]
}
You can solve this issue with simple contraction, write synonyms like this
"synonyms": [
"employment information,your activity your job=>sentence1"
]
If you are using keyword analyzer then you could use pattern replace char filter to remove comma.