Using Cardinality but trying to find total length with it - elasticsearch

I have been using cardinality to find some unique fields, such as author
"aggs": {
"author_count" : {
"cardinality" : {
"field" : "author"
}
}
}
This works and counts all the author fields that have a unique author in it.
Now I want to find the total size of these unique authors. With other queries I have just done this by just adding
"aggs":{
"sum":{
"field" : "length" }}}
But when I have tried this it give me the total length of everything not just for the unique authors.
So for example if the field author contains only one "Kim" this should be returned.
I want every author who has written only single book and add all of their page lengths together too.
e.g
"author" : "kim",
"length": 100
"author" : "lolo",
"length": 100
The output should be author_count 2 and total_length 200.
But for
"author" : "kim",
"length": 100
"author" : "lolo",
"length": 100
"author" : "lolo",
"length": 100
The output should be author_count 1 and total_length 100. Because kim is the only unique author(author who has written only one book)
Any ideas?

After understanding the question, this can be achieved with bucket selector aggregation and sum bucket aggregation. First terms aggregation on author field will give all the unique authors, then value count aggregation will give books these unique authors have written.
total_sum sums the length of pages.
Now bucket selector will only retain buckets of those authors which have written only single book and finally sum_bucket sums all the length of those authors
{
"size": 0,
"aggs": {
"unique_author": {
"terms": {
"field": "author",
"size": 100
},
"aggs": {
"total_book_count": {
"value_count": {
"field": "author"
}
},
"total_sum": {
"sum": {
"field": "length"
}
},
"only_single_book_author": {
"bucket_selector": {
"buckets_path": {
"total_books": "total_book_count"
},
"script": "total_books==1"
}
}
}
},
"page_length": {
"sum_bucket": {
"buckets_path": "unique_author>total_sum"
}
}
}
}

Related

Elastic Search get top grouped sums with additional filters (Elasticsearch version5.3)

This is my Mapping :
{
"settings" : {
"number_of_shards" : 2,
"number_of_replicas" : 1
},
"mappings" :{
"cpt_logs_mapping" : {
"properties" : {
"channel_id" : {"type":"integer","store":"yes","index":"not_analyzed"},
"playing_date" : {"type":"string","store":"yes","index":"not_analyzed"},
"country_code" : {"type":"text","store":"yes","index":"analyzed"},
"playtime_in_sec" : {"type":"integer","store":"yes","index":"not_analyzed"},
"channel_name" : {"type":"text","store":"yes","index":"analyzed"},
"device_report_tag" : {"type":"text","store":"yes","index":"analyzed"}
}
}
}
}
I want to query the index similar to the way I do using the following MySQL query :
SELECT
channel_name,
SUM(`playtime_in_sec`) as playtime_in_sec
FROM
channel_play_times_bar_chart
WHERE
country_code = 'country' AND
device_report_tag = 'device' AND
channel_name = 'channel'
playing_date BETWEEN 'date_range_start' AND 'date_range_end'
GROUP BY channel_id
ORDER BY SUM(`playtime_in_sec`) DESC
LIMIT 30;
So far my QueryDSL looks like this
{
"size": 0,
"aggs": {
"ch_agg": {
"terms": {
"field": "channel_id",
"size": 30 ,
"order": {
"sum_agg": "desc"
}
},
"aggs": {
"sum_agg": {
"sum": {
"field": "playtime_in_sec"
}
}
}
}
}
}
QUESTION 1
Although the QueryDSL I have made does return me the top 30 channel_ids w.r.t playtimes but I am confused how to add other filters too within the search i.e country_code, device_report_tag & playing_date.
QUESTION 2
Another issue is that the result set contains only the channel_id and playtime fields unlike the MySQL result set which returns me channel_name and playtime_in_sec columns. This means I want to achieve aggregation using channel_id field but result set should instead return corresponding channel_name name of the group.
NOTE: Performance over here is a top priority as this is supposed to be running behind a graph generator querying millions or even more docs.
TEST DATA
hits: [
{
_index: "cpt_logs_index",
_type: "cpt_logs_mapping",
_id: "",
_score: 1,
_source: {
ChID: 1453,
playtime_in_sec: 35,
device_report_tag: "mydev",
channel_report_tag: "Sony Six",
country_code: "SE",
#timestamp: "2017-08-11",
}
},
{
_index: "cpt_logs_index",
_type: "cpt_logs_mapping",
_id: "",
_score: 1,
_source: {
ChID: 145,
playtime_in_sec: 25,
device_report_tag: "mydev",
channel_report_tag: "Star Movies",
country_code: "US",
#timestamp: "2017-08-11",
}
},
{
_index: "cpt_logs_index",
_type: "cpt_logs_mapping",
_id: "",
_score: 1,
_source: {
ChID: 12,
playtime_in_sec: 15,
device_report_tag: "mydev",
channel_report_tag: "HBO",
country_code: "PK",
#timestamp: "2017-08-12",
}
}
]
QUESTION1:
Are you looking to add a filter/query to the example above? If so you can simply add a "query" node to the query document:
{
"size": 0,
"query":{
"bool":{
"must":[
{"terms": { "country_code": ["pk","us","se"] } },
{"range": { "#timestamp": { "gt": "2017-01-01", "lte": "2017-08-11" } } }
]
}
},
"aggs": {
"ch_agg": {
"terms": {
"field": "ChID",
"size": 30
},
"aggs":{
"ch_report_tag_agg": {
"terms":{
"field" :"channel_report_tag.keyword"
},
"aggs":{
"sum_agg":{
"sum":{
"field":"playtime_in_sec"
}
}
}
}
}
}
}
}
You can use all normal queries/filters for elastic to pre-filter your search before you start aggregating (Regarding performance, elasticsearch will apply any filters / queries before starting to aggregate, so any filtering you can do here will help a lot)
Question2:
On the top of my head I would suggest one of two solutions (unless I'm not completely misunderstanding the question):
Add aggs levels for the fields you want in the output in the order you want to drill down. (you can nest aggs within aggs quite deeply without issues and get the bonus of count on each level)
Use the top_hits aggregation on the "lowest" level of aggs, and specify which fields you want in the output using "_source": { "include": [/fields/] }
Can you provide a few records of test data?
Also, it is useful to know which version of ElasticSearch you're running as the syntax and behaviour change a lot between major versions.

Return distinct values in Elasticsearch

I am trying to solve an issue where I have to get distinct result in the search.
{
"name" : "ABC",
"favorite_cars" : [ "ferrari","toyota" ]
}, {
"name" : "ABC",
"favorite_cars" : [ "ferrari","toyota" ]
}, {
"name" : "GEORGE",
"favorite_cars" : [ "honda","Hyundae" ]
}
When I perform a term query on favourite cars "ferrari". I get two results whose name is ABC. I simply want that the result returned should be one in this case. So my requirement will be if I can apply a distinct on name field to receive one 1 result.
Thanks
One way to achieve what you want is to use a terms aggregation on the name field and then a top_hits sub-aggregation with size 1, like this:
{
"size": 0,
"query": {
"term": {
"favorite_cars": "ferrari"
}
},
"aggs": {
"names": {
"terms": {
"field": "name"
},
"aggs": {
"single_result": {
"top_hits": {
"size": 1
}
}
}
}
}
}
That way, you'll get a single term ABC and then nested into it a single matching document

How to limit ElasticSearch results by a field value?

We've got a system that indexes resume documents in ElasticSearch using the mapper attachment plugin. Alongside the indexed document, I store some basic info, like if it's tied to an applicant or employee, their name, and the ID they're assigned in the system. A query that runs might look something like this when it hits ES:
{
"size" : 100,
"query" : {
"query_string" : {
"query" : "software AND (developer OR engineer)",
"default_field" : "fileData"
}
},
"_source" : {
"includes" : [ "applicant.*", "employee.*" ]
}
}
And gets me results like:
"hits": [100]
0: {
"_index": "careers"
"_type": "resume"
"_id": "AVEW8FJcqKzY6y-HB4tr"
"_score": 0.4530588
"_source": {
"applicant": {
"name": "John Doe"
"id": 338338
}
}
}...
What I'm trying to do is limit the results, so that if John Doe with id 338338 has three different resumes in the system that all match the query, I only get back one match, preferably the highest scoring one (though that's not as important, as long as I can find the person). I've been trying different options with filters and aggregates, but I haven't stumbled across a way to do this.
There are various approaches I can take in the app that calls ES to tackle this after I get results back, but if I can do it on the ES side, that would be preferable. Since I'm limiting the query to say, 100 results, I'd like to get back 100 individual people, rather than getting back 100 results and then finding out that 25% of them are docs tied to the same person.
What you want to do is an aggregation to get the top 100 unique records, and then a sub aggregation asking for the "top_hits". Here is an example from my system. In my example I'm:
setting the result size to 0 because I only care about the aggregations
setting the size of the aggregation to 100
for each aggregation, get the top 1 result
GET index1/type1/_search
{
"size": 0,
"aggs": {
"a1": {
"terms": {
"field": "input.user.name",
"size": 100
},
"aggs": {
"topHits": {
"top_hits": {
"size": 1
}
}
}
}
}
}
There's a simpler way to accomplish what #ckasek is looking to do by making use of Elasticsearch's collapse functionality.
Field Collapsing, as described in the Elasticsearch docs:
Allows to collapse search results based on field values. The collapsing is done by selecting only the top sorted document per collapse key.
Based on the original query example above, you would modify it like so:
{
"size" : 100,
"query" : {
"query_string" : {
"query" : "software AND (developer OR engineer)",
"default_field" : "fileData"
}
},
"collapse": {
"field": "id",
},
"_source" : {
"includes" : [ "applicant.*", "employee.*" ]
}
}
Using the answer above and the link from IanGabes, I was able to restructure my search like so:
{
"size": 0,
"query": {
"query_string": {
"query": "software AND (developer OR engineer)",
"default_field": "fileData"
}
},
"aggregations": {
"employee": {
"terms": {
"field": "employee.id",
"size": 100
},
"aggregations": {
"score": {
"max": {
"script": "scores"
}
}
}
},
"applicant": {
"terms": {
"field": "applicant.id",
"size": 100
},
"aggregations": {
"score": {
"max": {
"script": "scores"
}
}
}
}
}
}
This gets me back two buckets, one containing all the applicant Ids and the highest score from the matched docs, as well as the same for employees. The script is nothing more than a groovy script on the shard that contains '_score' as the content.

Having elasticsearch return a single result for each element in a list

I am trying to figure out a way in ES that I can return a single document for each element in a list of parameters. For instance, if I have a field called country and I pass in a list of a few countries, I want one result return for each of them.
So for instance if I provide a list such as:
{ "terms": { "country": ['France','Spain','Germany'] } }
I would receive one result for each but only one result.
You can use combination of query and aggregation.
GET /index/type/_search
{
"size": 0,
"query": {
"terms": {
"country": ["France","Spain","Germany"]
}
},
"aggs": {
"group_by_country": {
"terms": {
"field": "country",
"size" :0
},
"aggs":{
"top_hits_country" :{
"top_hits" :{
"size":1
}
}
}
}
}
}
In aggregated result you will get result for each country and since we have specified "size" :1 in "top_hits" you will get only one result.
Your question can be interpreted in a number of ways - as Paul Roub suggested, but I'm assuming you're looking for:
OR filter: https://www.elastic.co/guide/en/elasticsearch/reference/1.4/query-dsl-or-filter.html
SHOULD filter: https://www.elastic.co/guide/en/elasticsearch/guide/current/combining-filters.html

How to find top terms with occurrences in Elasticsearch

I have a fairly big dataset in Elasticsearch: 1 index, about 120 million records of one type. I am processing a large number of paragraphs on a given set of topics. The number of topics is limited and associated with a unique ID. Each paragraph has a couple of sentences identified by the sentence_id (unique across all topics). Each sentence has a number of words and each word can occur multiple times. So my mapping looks like the following:
{
"sentence_id": 1200,
"topic_id": 2,
"value": "ground",
"occurrences": 20
}
Now, I want to run a query which answers this:
"Find the top words for a given topic ID sorted by their occurrences."
So for each word in a topic, I have to sum up its occurrences across all the sentences, sort them and return.
I am not able to achieve this. I tried writing aggregation term query, but it does not sum occurrences and merely returns the unique count of records for each word.
{
"query": {
"term": {
"topic_id": {
"value": 3117
}
}
},
"aggs": {
"total_occurrences": {
"terms": {
"field": "occurrences",
"size": 1000
}
}
}
}
Can some one help me out?
I think first you need to aggregate on unique value, and then sum its occurrences, your query should look something like this assuming your occurrences field is numeric
{
"query": {
"term": {
"topic_id": {
"value": 3117
}
}
},
"aggs": {
"total_occurrences": {
"terms": {
"field": "value",
"size": 1000,
"order": {
"sum_occurrences": "desc" <--- to sort by top words
}
},
"aggs": {
"sum_occurrences": {
"sum": {
"field": "occurrences"
}
}
}
}
},
"size": 0
}
Hope this helps!

Resources