How to sort data in elastic search based on the filter data - elasticsearch

Question

I am relatively new to this elastic search. So I have data stored in the elastic search in a below-mentioned way:
[{
"name": "user1",
"city": [{
"name": "city1",
"count": 18
},{
"name": "city2",
"count": 15
},{
"name": "city3",
"count": 10
},{
"name": "city4",
"count": 5
}]
},{
"name": "user2",
"city": [{
"name": "city2",
"count": 2
},{
"name": "city5",
"count": 5
},{
"name": "city6",
"count": 8
},{
"name": "city8",
"count": 15
}]
},{
"name": "user3",
"city": [{
"name": "city1",
"count": 2
},{
"name": "city5",
"count": 5
},{
"name": "city7",
"count": 28
},{
"name": "city2",
"count": 1
}]
}]
So, what I am trying to do is, find out those users who have "city2" in their city list and order the data based on the "count" of "city2".
Here is my query what I have tried:
{
"sort": [{
"city.count": {
"order" : "desc"
}
}],
"query": {
"bool": {
"must": [
{"match": {"city.name": "city2"}}
]
}
}
}
So I am not able to figure out the sort part how to do it!
The sorting part is considering all the "count" value of all the cities based on the filter, but I just want the order to happen only based on the "count" of "city2".
Any kind of help would be appreciated. Thanks in advance.

Answer 1

Since the field city is object and not nested object, what you are trying to achieve won't be possible. The reason for this is when you define a field as object, elastics flattens each of the object field values as an array. So,
"city": [
{
"name": "city1",
"count": 18
},
{
"name": "city2",
"count": 15
},
{
"name": "city3",
"count": 10
},
{
"name": "city4",
"count": 5
}
]
is indexed as :
"city.name" : ["city1", "city2", "city3", "city4"]
"city.count": [18, 15, 10, 5]
As you can see, because of the way elastic index the object the relation between each city and its count is lost.
So, whenever you want to preserve the relation you should define the field as nested type.
{
"city": {
"type": "nested",
"properties": {
"name": {
"type": "text"
},
"count": {
"type": "long"
}
}
}
}
Sorting then can be achieved by using this nested field.
{
"sort": [
{
"city.count": {
"order": "desc",
"mode": "avg",
"nested": {
"path": "city",
"filter": {
"match": {
"city.name": "city2"
}
}
}
}
}
],
"query": {
"bool": {
"must": [
{
"match": {
"city.name": "city2"
}
}
]
}
}
}

Answer 2

Reaching your goal will be a little complex.
First, your query says that you want to get the docs with "city2" in them. Since at least one of the elements in the array "city" matches, the whole document will be returned.
The problem is that you only want to return the count for city2, not for all of them. This is where the complex part comes.
There are plenty of paths you can follow:
Change your index design. Instead of having an array of users, have one document per user with all their info, including the cities they have visited. However, the "I only want 1 element from the array" problem will still be there, but you will only will fight with one array at time, instead of n.
You can use Painless to only bring back the count of that particular city, but it would imply a lot of scripting. Don't trust the name. Painless is very Painful.
You can bring back all the elements and do the filtering within your code. For example, if you use the Python Elasticsearch Client, you can execute the query, return all the objects and only selec the wanted elements with Python.
Don't consider using the Terms aggregation. It would bring back the total counting of all the cities, without having the relationship with each user. And this is not what you want to do.
Hope this is helpful and sorry we can't get a straight-forward solution :(