elasticsearch: giving score to ngrams - elasticsearch

Question

I am trying to use elasticsearch to do a name search matching using ngrams,
The technique I am trying to implement is as follow:
input: a name that needs to be match to the db.
output: all potential name matching from my db of names.
The way I try to do that is as follow, I split the name to ngrams with length of 3-5.
I then collect all the names that match those ngrams from the db.
Then I go over the ngrams and sort them by there reverse frequency,
meaning that common ngrams will get the lowest score.
for example, if I use it on a company name like "my company inc" I will give the "inc" ngram the lowest score because inc appears in a lot of company names.
The way I calculate the score is by doing: 1/(count appearences of the ngram in all my db), that way I will have the "strongest" ngrams as the ones that appear the least.
I implemented this in a python script, but I want to use the power of elastic to do the same for me,
I know about the ngram tokenizer, but is there a way to tell him to do the score I do?
As far as I know, when I do a matching now, it will score the result by how much of the ngrams in the query match the ngrams in the word he has in the db
this is the mapping I use:
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "ngram",
"min_gram": 3,
"max_gram": 5,
"token_chars": ["letter", "digit"]
}
}
}
},
"mappings": {
"names": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256,
}
},
"analyzer": "my_analyzer"
},
"id": {
"type": "long"
}
}
}
}
}
this is the query I do:
GET /names/_search
{
"query": {
"match" : { "name" : "my company inc"}
}
}

Answer 1

The query that you would want to use is this:
{
"query": {
"common": {
"name": {
"query": "my company inc",
"cutoff_frequency": 0.001
}
}
}
}
Common terms query returns the relevance score based only on important terms (important nGrams) i.e. terms with less frequency. Here, the words that have a document frequency greater than 0.1% will be considered as common words and will not affect the relevance score.
Alternatively, if you already have a predefined list of stopwords (inc, pvt, ltd), then you can always use a custom stop words filter in your analyzer to filter them out for generating hits.
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer",
"filter": [
"custom_stop_token_filter"
]
}
},
"tokenizer": {
"my_tokenizer": {
"type": "ngram",
"min_gram": 3,
"max_gram": 5,
"token_chars": ["letter", "digit"]
}
},
"filter": {
"custom_stop_token_filter": {
"type": "stop",
"stopwords": [
"inc",
"pvt",
"ltd"
]
}
}
}
}
}
For more info:
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/query-dsl-common-terms-query.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stop-tokenfilter.html