How to get distinct values of afeild in ES? - elasticsearch

Question

I am trying to calculate distinct values of a field in ES. For example if I have an index containing documents like:
{
"NAME": "XYZ",
"TITLE": "ABC"
}
{
"NAME": "RTY",
"TITLE": "BNM"
}
I want to have an output like this:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 10000,
"max_score": 1,
"hits": [
{
"_index": "record_new",
"_type": "record_new",
"_id": "AWChga1952qKS23vpN8J",
"_score": 1,
"_source": {
"TITLE":{
"ABC",
"BNM"
}
}
}]
}
}
How can I get the distinct values in "title" field in this format. I have tried using aggregation but the output is very weired. Please help.

Answer 1

Test Data:
PUT http://localhost:9200/stackoverflow/os/1
{
"NAME": "XYZ",
"TITLE": "LINUX OS"
}
PUT http://localhost:9200/stackoverflow/os/1
{
"NAME": "XYZ",
"TITLE": "WINDOWS SERVER"
}
First Query Attempt:
Note I have used POST here instead of GET since most REST clients do not support payload with GET.
POST http://localhost:9200/stackoverflow/_search
{
"size":0,
"aggs":{
"uniq_soft_tags":{
"terms":{
"field":"TITLE"
}
}
}
}
If you did not give a mapping for your data and run the above, most probably you'll end up with below error.
Fielddata is disabled on text fields by default. Set fielddata=true on [TITLE] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory
Read more about that here.
Adding the mapping to enable Fielddata:
PUT http://localhost:9200/stackoverflow/_mapping/os/
{
"properties": {
"TITLE": {
"type": "text",
"fielddata": true
}
}
}
Second Query Attempt:
POST http://localhost:9200/stackoverflow/_search
{
"size":0,
"aggs":{
"uniq_soft_tags":{
"terms":{
"field":"TITLE"
}
}
}
}
Results:
{
"took": 10,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0,
"hits": []
},
"aggregations": {
"uniq_soft_tags": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "linux",
"doc_count": 1
},
{
"key": "os",
"doc_count": 1
},
{
"key": "server",
"doc_count": 1
},
{
"key": "windows",
"doc_count": 1
}
]
}
}
}
Note that the doc_counts here are approximate.
Make sure to read the following section in the docs. https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#before-enabling-fielddata
That explains how you get that spilitting behaviour.
Before enabling fielddata
Before you enable fielddata, consider why you are using a text field
for aggregations, sorting, or in a script. It usually doesn’t make
sense to do so.
A text field is analyzed before indexing so that a value like New York
can be found by searching for new or for york. A terms aggregation on
this field will return a new bucket and a york bucket, when you
probably want a single bucket called New York.
UPDATE:
To prevent splitting behaviour you have to provide a mapping as follows. Note that with this you would not need the previous mapping where we set Fielddata to true.
PUT http://localhost:9200/stackoverflow/_mapping/os/
{
"properties": {
"TITLE": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
Now we can use,
TITLE field for searches.
TITLE.keyword field for aggregations, sorting, or in scripts.
Third Query Attempt:
POST http://localhost:9200/stackoverflow/_search
{
"size":0,
"aggs":{
"uniq_soft_tags":{
"terms":{
"field":"TITLE.keyword"
}
}
}
}
Results:
{
"took": 59,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 0,
"hits": []
},
"aggregations": {
"uniq_soft_tags": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "LINUX OS",
"doc_count": 1
},
{
"key": "WINDOWS SERVER",
"doc_count": 1
}
]
}
}
}