How to implement space ignored search across a document with multiple fields in Elasticsearch - elasticsearch

Question

I'm trying to implement a space agnostic product catalog search solution using Elasticsearch for a chemical oriented product database. The use case is as follows:
Consider a chemical with the name: Dimethyl Sulphoxide
Some manufacturers label it as Dimethylsulphoxide and some as Dimethyl Sulphoxide. So I could have two Item entries in my ES document as follows
Item 1: {"item_name":"Dimethyl Sulphoxide", "brand":"Merck"}
Item 2: {"item_name":"Dimethylsulphoxide","brand":"Spectrochem"}
Now ideally, If the user enters either string (i.e Dimethyl sulphoxide or Dimethylsulphoxide), I want both the documents to be displayed in the hits.
To achieve this I'm doing two things:
1)At index time, I'm currently running the item_name field through a custom analyzer that consists of the following flow:
Tokenizing with keyword, then filtering with lowercase, then filtering with word joiner(with catenate_all), then filtering with an edge_ngram filter.
So the string "Dimethyl Sulphoxide" becomes ("Dimethyl","Sulphoxide") then ("dimethyl","sulphoxide"), then ("dimethyl","sulphoxide","dimethylsulphoxide"), then ("d","di","dim",dime"....."dimethyl","s","su","sul"....,"sulphoxide","d","di"......,"dimethylsulphoxide")
I'm also running the other fields in the product document, such as the brand field with the same analyzer.
At query time, I'm running the query search string via a similar analyzer without the edge_ngram. So a query string of "Dimethyl Sul" will become ("Dimethyl","Sul") then ("dimethyl","sul") then ("dimethyl","sul","dimethylsul") by specifying a custom search analyzer for each field at index time.
Now I'm able to surface both the results when the user searches a string with or without space, but this approach is coming in the way of my other use cases.
Consider the second use case where the user should also be able to search for an item by the name + the brand and other fields, all in one search box. For example, A user could search for one of the items above by entering, "dimethyl spectrochem" or "sulphoxide merck".
To allow this, I have tried using a multi_match with type as cross fields query and a combined_fields query, both with the operator as AND. But this combination of word_joiner in query string with cross_fields/combined_fields is giving me undesired results for my second use case.
When a user enters, "dimethyl spectrochem", the query search analyzer generates three tokens ("dimethyl","spectrochem" and "dimethylspectrochem") and when these are passed to the cross_fields/combined_fields it essentially generates a query as :
+("item_name":"dimethyl","brand":"dimethyl")
+("item_name":"spectrochem","brand":"spectrochem")
+("item_name":"dimethylspectrochem","brand":"dimethylspectrochem")
Given the way how cross_field works , it looks for a document where all the three queryStrings are present in either field. Since it's unable to find "dimethylspectrochem" in a single field it returns zero results.
Is there a way I can satisfy both use cases?
The mapping that I have specified during index creation is below
curl -XPUT http://localhost:9200/test-item-summary-5 -H 'Content-Type: application/json' -d'
{
"settings":{
"analysis":{
"tokenizer":{
"whitespace":{
"type":"whitespace"
},
"keyword":{
"type":"keyword"
}
},
"filter":{
"lowercase":{
"type":"lowercase"
},
"shingle_word_joiner":{
"type":"shingle",
"token_separator":""
},
"word_joiner":{
"type":"word_delimiter_graph",
"catenate_all":true,
"split_on_numerics":false,
"stem_english_possessive":false
},
"edge_ngram_filter":{
"type":"edge_ngram",
"min_gram":1,
"max_gram":20,
"token_chars":[
"letter",
"digit"
]
}
},
"analyzer":{
"whitespaceWithEdgeNGram":{
"tokenizer":"keyword",
"filter":[
"lowercase",
"word_joiner",
"edge_ngram_filter"
]
},
"spaceIgnoredWithLowerCase":{
"tokenizer":"keyword",
"char_filter":[
"dash_char_filter"
],
"filter":[
"lowercase",
"word_joiner"
]
},
"shingleSearchAnalyzer":{
"tokenizer":"whitespace",
"char_filter":[
"dash_char_filter"
],
"filter":[
"lowercase",
"shingle_word_joiner"
]
},
"whitespaceWithLowerCase":{
"tokenizer":"whitespace",
"char_filter":[
"dash_char_filter"
],
"filter":[
"lowercase"
]
}
},
"char_filter":{
"dash_char_filter":{
"type":"mapping",
"mappings":[
"- => ",
", => ",
". => ",
"( => ",
") => ",
"? => ",
"! => ",
": => ",
"; => ",
"_ => ",
"% => ",
"& => ",
"+ => ",
"\" => ",
"\/ => ",
"\\[ => ",
"\\] => ",
"* => ",
"\u0027 => "
]
}
}
}
},
"mappings":{
"properties":{
"item_code":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"mfr_item_code":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"brand":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"name":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"short_name":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"alias":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"attrs":{
"type":"nested",
"properties":{
"name":{
"type":"text",
"index":"false"
},
"value":{
"type":"text",
"copy_to":"item:attrs:value",
"index":"false"
},
"primaryAttribute":{
"type":"boolean",
"index":"false"
}
}
},
"variant_summaries":{
"type":"nested",
"properties":{
"item_code":{
"type":"text",
"index":"false"
},
"variant_code":{
"type":"text",
"copy_to":"variant:variant_code",
"index":"false"
},
"mfr_item_code":{
"type":"text",
"index":"false"
},
"mfr_variant_code":{
"type":"text",
"copy_to":"variant:mfr_variant_code",
"index":"false"
},
"brand":{
"type":"text",
"index":"false"
},
"unit":{
"type":"text",
"copy_to":"variant:unit",
"index":"false"
},
"unit_mag":{
"type":"float",
"copy_to":"variant:unit",
"index":"false"
},
"primary_alternate_unit":{
"type":"nested",
"properties":{
"unit":{
"type":"text",
"copy_to":"variant:unit",
"index":"false"
},
"unit_mag":{
"type":"float",
"copy_to":"variant:unit",
"index":"false"
}
}
},
"attrs":{
"type":"nested",
"properties":{
"name":{
"type":"text",
"index":"false"
},
"value":{
"type":"text",
"copy_to":"variant:attrs:value",
"index":"false"
},
"primaryAttribute":{
"type":"boolean",
"index":"false"
}
}
},
"image":{
"type":"text",
"index":"false"
},
"in_stock":{
"type":"boolean",
"index":"false"
}
}
},
"added_by":{
"type":"text",
"index":"false"
},
"modified_by":{
"type":"text",
"index":"false"
},
"created_on":{
"type":"date",
"index":"false"
},
"updated_on":{
"type":"date",
"index":"false"
},
"is_deleted":{
"type":"boolean",
"index":"false"
},
"variant:variant_code":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"variant:mfr_variant_code":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"variant:attrs:value":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"variant:unit":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
},
"item:attrs:value":{
"type":"text",
"analyzer":"whitespaceWithEdgeNGram",
"search_analyzer":"shingleSearchAnalyzer"
}
}
}
}'
Any suggestions on implementing a space ignored search across multiple fields would be highly appreciated.