Querying after aggregation in elasticsearch? - elasticsearch

Question

We have a database of contacts outside of elasticsearch. Every of these contacts has many dynamic attributes (gender:male,yearOfBirth:1985,carColor:blue,etc).
We wanted to integrate elasticsearch to our setup and we decided to index by attributes for scalability. So an example of document in elasticsearch would be like this:
{
contactId:"XYZ",
attribute:"gender",
value:"male"
}
That way, we can add unlimited attributes for any contacts without having to reindex any documents.
Our problem comes when it's time to search within those documents. We want to be able to list contacts passing attribute definitions to elastic search i.e. (be able to list contacts that are male AND have blue cars AND etc.)
So we would want to do something like
Aggregate documents by contactId
write the query for the attributes needs
Paginate the results
We came up with something like this.
{
query: {
bool: {
should: [
{
bool: {
must: [
{ match: { attribute: "gender" }},
{ match: { value: "f" }},
],
},
},
{
bool: {
must: [
{ match: { attribute: "carColor" }},
{ match: { value: "blue" }},
],
},
}
],
minimum_should_match: 2,
},
},
aggs: {
contacts: {
composite: {
size: 15,
sources: [
{
contactId: {
terms: {
field: "contactId",
}
}
}
],
},
},
},
}
But we can really get to the result we want.
Anyone has any idea of what we do wrong and/or how we could improve this query.
Thanks a lot !

Answer 1

The main issue that you're having is that ElasticSearch can't join data. I can imagine that your query don't match anything because ElasticSearch applies criteria to each document one by one. Since the attribute-value pairs are stored in individual document, a multi-criteria query like below can't match a single document:
(attribute:gender AND value:male) AND (attribute:carColor AND value:blue)
Going forward, one option is to get all the documents that match the given criteria and then aggregate. I guess, this is what you're trying to achieve.
Assuming the data:
attribute | contactId | value
---------------+---------------+---------------
gender |XYZ |male
carColor |XYZ |blue
gender |ABS |female
pet |ABS |tiger
carColor |ABS |red
pet |XYZ |dog
gender |XXX |female
carColor |XXX |blue
With the following query:
{
"size": 0,
"query": {
"query_string": {
"query": "(attribute:gender AND value:male) OR (attribute:carColor AND value:blue)" // Lets read them as (C1) or (C2)
}
},
"aggs": {
"contacts": {
"terms": {
"field": "contactId",
"order": {
"_count": "asc"
},
"size": 5
},
"aggs": {
"attribute": {
"terms": {
"field": "attribute",
"order": {
"_count": "asc"
},
"size": 5
},
"aggs": {
"attribute_value": {
"terms": {
"field": "value",
"order": {
"_count": "asc"
},
"size": 5
}
}
}
}
}
}
}
}
Results in:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"contacts" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "XXX",
"doc_count" : 1,
"attribute" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "carColor",
"doc_count" : 1,
"attribute_value" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "blue",
"doc_count" : 1
}
]
}
}
]
}
},
{
"key" : "XYZ",
"doc_count" : 2,
"attribute" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "carColor",
"doc_count" : 1,
"attribute_value" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "blue",
"doc_count" : 1
}
]
}
},
{
"key" : "gender",
"doc_count" : 1,
"attribute_value" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "male",
"doc_count" : 1
}
]
}
}
]
}
}
]
}
}
This way, you can get a top-level grouping by the "contactId". The caveats are
The nested-aggregations are needed to join the data (perhaps the only way to join data in ES dynamically)
Result buckets will contain any grouping which can be made from records that either match (C1) or (C2). The buckets then need to be post-filtered to remove buckets that don't match all the given criteria (in above example buckets with single match need to be dropped).
For N-many criteria, the nesting will have a depth N and will become painfully-slow with a N>10
Finally, I'd suggest that re-indexing is probably not that big of a deal, especially if your original source already has these fields together in a single document/row. The pros for this approach are:
Querying ES would be cheaper (implementation & computation wise)
Updating the document in ElasticSearch is as simple as indexing it the first time.
In conclusion, the question boils down to this: Is the pain worth it? In other words: is the cost of re-indexing the documents more or less than the cost of writing & maintaining these queries in the first place (plus the query-time computation cost that follows).
PS: And yeah, you can just paste the query part in Kibana & use the data-table visualization to generate the nested-aggregation query. This way, you can also get an idea of when the queries start to become really slow as your increase your nesting level (adding more columns in data-table does that for you).
adios

Answer 2

We decided to reindex our whole dataset and put all attribute documents as child of certain documents that represents our contacts. That way we can do easy querying without complex aggregations this way.
We have to specify the parameter min_children to the has_child because we want both should operator to exists within our contacts.
{
size: 10,
from: 1,
query: {
has_child: {
type: "attributedocument",
query: {
bool: {
should: [
{
bool: {
must: [
{ match: { attribute: "gender" } },
{ match: { value: "f"} },
],
},
},
{
bool: {
must: [
{ match: { attribute: "carColor" } },
{ match: { value: "Blue" } },
],
},
},
],
},
},
min_children: "2",
},
},
}