get all similar documents in the entire index - elasticsearch

Question

Is there a way to find documents that match query, but the query has no specific values.
For example, I have index person with mapping:
{
"properties": {
"fullname": {
"type": "text"
},
"email": {
"type": "keyword"
}
}
}
I have a query to find similar persons:
{
"query": {
"bool": {
"must": [
{
"match": {
"fullname": "Foo Bar"
}
},
{
"term": {
"email": "foobar#gmail.com"
}
}
]
}
}
}
It works for finding similar persons for specific person.
Is there a way to get all similar persons between each other in the index? Maybe some kind of aggregation?
It may be helpful to set up an alarm when there are some new similar documents.

Answer 1

First off, defining what's similar is arbitrary -- but you may want to look into fuzzy match queries.
Secondly, when you query using term on a keyword field, your results will be restricted to exact matches -- somewhat defeating the purpose of similar persons.
Finally, aggregations operate on concrete values so once you've found your similar persons using the match query, you can aggregate in multiple ways but you've 'lost' the fuzziness aspect, and rightly so.
Side note: when you intend to aggregate on text fields like fullname, you can either set fielddata: true on that field or add another subfield with the keyword mapping like so:
...
"fullname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
...
In concrete terms, then, after ditching the term query, we can proceed as follows:
GET similar/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"fullname": "Foo Bar"
}
}
]
}
},
"aggs": {
"by_email": {
"terms": {
"field": "email"
}
},
"by_name": {
"terms": {
"field": "fullname.keyword"
},
"aggs": {
"by_email": {
"terms": {
"field": "email"
}
}
}
}
}
}
The by_email aggregation gives us the top 10 emails associated with the persons matching the query, ordered by the number of occurrences of those emails. I suspect this won't help because emails are ... unique ;)
The by_name aggregation is more useful -- there may be lots of people called "Foo Bar" and the sub-aggregation, also called by_email will give you their emails.
Alerting is an entirely different topic -- feel free to ask another question.