bucket Terms aggregation Elasticsearch - elasticsearch

Question

elasticsearch version
{
"name" : "abc-Inspiron-5521",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "2vLvphpURJOtfAZSGDDX5w",
"version" : {
"number" : "7.10.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
Document mapping
"user_data" : {
"aliases" : { },
"mappings" : {
"properties" : {
"experience" : {
"properties" : {
"brand" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"brand_segment" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"company" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"duration" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"property_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"real_estate_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
Document structure is right, please make changes if there is mismatch in parenthesis accordingly.
document sample
{
"_index" : "user_data",
"_type" : "_doc",
"_id" : "dONuEXgBU9vYaZRqY8Jo",
"_score" : 1.0,
"_source" : {
"experience" : [
{
"brand" : "Hilton",
"company" : "Hilton LLC",
"brand_segment" : "Luxury",
"property_type" : "All-Inclusive",
"duration" : "2 years",
"real_estate_type" : "Institutional"
},
{
"brand" : "Mantis",
"company" : "Accor LLC",
"brand_segment" : "Upper-Upscale",
"property_type" : "Condo",
"duration" : "2 years",
"real_estate_type" : "Family Office"
},
{
"brand" : "Marriott",
"company" : "Marriott LLC",
"brand_segment" : "Independent",
"property_type" : "Convention",
"duration" : "2 years",
"real_estate_type" : "Family Office"
}
]
}
}
my term aggregation query on brand_segment
GET user_data/_search
{
"aggs": {
"experience": {
"terms": { "field": "experience.brand_segment" }
}
}
}
Now I have 2 problems while making term aggregation
While executing term aggregation on 'brand_segment', the value 'Upper-Upscale' is suppose to be considered as single unit and count is to be made according but currently I am getting it as:
Second concern is if I want to count number of times brand_segment value is 'Luxury' or any value, but currently from above query I am getting count of number of documents in which Luxury occurs, not the number of times in all documents Luxury occurs. (multiple occurrences are getting counted as single for 1 document as of now).
wrong result
"aggregations" : {
"experience" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "independent",
"doc_count" : 15
},
{
"key" : "luxury",
"doc_count" : 15
},
{
"key" : "upper",
"doc_count" : 14
},
{
"key" : "upscale",
"doc_count" : 14
}
]
}
}
Desired output should have Upper-Upscale as one value. I have taken multiple sample documents hence this result.
feel free to use this as sample document for creating index
{
"id": 1,
"name": "abcs",
"source": "csv_status",
"profile_complition": "70%",
"creation_date": "2020-04-02",
"current_position": [
{
"position": "Financial Reporting",
"position_category": "Finance",
"position_level": 2
}
],
"seeking_position": [
{
"position": "Financial Planning and Analysis",
"position_category": "Finance",
"position_level": 3
}
],
"last_updation_date": "2021-02-02",
"experience": [
{
"brand": "Hilton",
"company": "Hilton LLC",
"brand_segment": "Luxury",
"property_type": "All-Inclusive",
"duration": "2 years",
"real_estate_type": "Institutional"
},
{
"brand": "Accor",
"company": "Accor LLC",
"brand_segment": "Luxury",
"property_type": "Condo",
"duration": "2 years",
"real_estate_type": "Family Office"
},
{
"brand": "Marriott",
"company": "Marriott LLC",
"brand_segment": "Independent",
"property_type": "Convention",
"duration": "2 years",
"real_estate_type": "Family Office"
}
]
}
other occurrences in brand_segment = ['Economy', 'Upscale', 'Midscale', 'Upper-Upscale', 'Luxury', 'Independent', 'Extended Stay']
PS: all brand_segment are desired to be considered as single entity ('Upper-Upscale' is not desired as 'Upper', 'Upscale'. Same for 'Extended Stay')
Let me know if further clarification required.

Answer 1

For the first issue, you need to make your aggregation on the keyword subfield:
GET user_data/_search
{
"aggs": {
"experience": {
"terms": { "field": "experience.brand_segment.keyword" }
}
}
}
To solve the second issue, you need to make your experience field nested, which means your mapping needs to look as follows:
"user_data" : {
"aliases" : { },
"mappings" : {
"properties" : {
"experience" : {
"type": "nested", <--- add this
"properties" : {
"brand" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},