Extracting and aggregating text values from a field with Elasticsearch - elasticsearch

Question

Im new to ElasticSearch, and my first mission is as follow:
Here is the beginning of my query:
"_source": "pageVisitsValues",
"query": {
"filtered": {
"filter": {
"and": {
"filters": [
{
"term": {
"blockSize": 30
}
},
{
"range": {
"pageViews": {
"from": 1,
"to": null,
"include_lower": true,
"include_upper": true
}
}
},
{
"bool": {
"should": {
"bool": {
"must": {
"regexp": {
"pageVisitsValues": {
"value": ".*utm_medium=.*"
}
}
}
}
}
}
}
]
}
}
}
},
"aggregations": {
"Utm_term1": {
"terms": {
"field": "pageVisitsValues",
"size": 50,
"order": {
"_count": "desc"
}
}
},
now here is part of the result:
"hits": { -
"total": 204223,
"max_score": 1,
"hits": [ -
{ -
"_index": "my_index",
"_type": "user",
"_id": "AX45WhAzfmq",
"_score": 1,
"_source": { -
"pageVisitsValues": [ -
"www.test.com/search?&q=bullion deals&utm_source=iterable&utm_medium=email&utm_campaign=weeklydeals"
]
}
},
{ -
"_index": "8767827_all_25216812968619266m",
"_type": "user",
"_id": "AX45vDJEamKugpqPWxqJ",
"_score": 1,
"_source": { -
"pageVisitsValues": [ -
"www.mytestsite.com/search?&q=indian",
"www.mytestsite.com/search?&q=indianhead cents&rows=80&view=grid&version=v2&start=160",
"www.mytestsite.com/product/24158/1867-indianvg",
"www.mytestsite.com/search?q=&x=11&y=7&silver&page=1&utm_source=bing&utm_medium=cpc&utm_campaign=cpa&utm_term=testx&utm_content=test",
"www.mytestsite.com/product/233871/flying-eagle-indian-head-cent-1856-1909",
"www.mytestsite.com/search?&q=silver eagles bu"
]
}
},
{ -
"_index": "my_index",
"_type": "user",
"_id": "userID",
"_score": 1,
"_source": { -
"pageVisitsValues": [ -
"www.mytestsite.com/product/1/bu-random-year?utm_source=criteo&utm_medium=email&utm_campaign=prospecting&dclid=cpyz4_j54fqcfcdbpaqd0dkkiw"
]
}
}
what I need to do is to find a way to aggregate all "utm_medium" values from the "pageVisitsValues" field, so my expected result should be something like this:
Utm_term1": { -
"doc_count_error_upper_bound": 426,
"sum_other_doc_count": 1557591,
"buckets": [ -
{ -
"key": "email",
"doc_count": 31283
},
{ -
"key": "cpc",
"doc_count": 23615
Any idea how can I do that?

Answer 1

I'd suggest adding a new list to ES and aggregating on it since it would be less expensive, but if you don't want to do that, you can create a dynamic variable per record runtime_mappings and do a term aggregation on it, check the script section in the official docs for Elasticsearch.
To extract the keyword needed using painless script, you can use the .split function, check this ticket out.
If you need any further help please reply to this comment.