So I have an issue here...
I'm using chewy ruby gem to communicate with Elasticsearch
=> #<Chewy::SnippetPagesIndex::Query:0x007f911c6b1610
#_collection=nil,
#_fully_qualified_named_aggs={"chewy::snippetpagesindex"=>{"chewy::snippetpagesindex::snippetpage"=>{}}},
#_indexes=[Chewy::SnippetPagesIndex],
#_named_aggs={},
#_request=nil,
#_response=nil,
#_results=nil,
#_types=[],
#criteria=
#<Chewy::Query::Criteria:0x007f911c6b1458
#aggregations=
{:group_by=>{:terms=>{:field=>"seo_area.suburb.id", :order=>{:_count=>"asc"}}, :aggs=>{:by_top_hit=>{:top_hits=>{:size=>10}}}}},
#facets={},
#fields=[],
#filters=
[{:geo_distance=>{:distance=>"100km", "seo_area.suburb.coordinates"=>"-27.9836052, 153.3977354"}},
{:bool=>
{:must_not=>[{:terms=>{:id=>[1]}}, {:terms=>{"seo_area.suburb.id"=>[5559]}}],
:must=>[{:term=>{:path_category=>"garden-services"}}, {:term=>{:status=>"active"}}, {:exists=>{:field=>"path_area"}}],
:should=>[]}}],
#options=
{:query_mode=>:must,
:filter_mode=>:and,
:post_filter_mode=>:and,
:preload=>
{:scope=>
#<Proc:0x007f911c6b1700#/Users/serviceseeking/Work/serviceseeking/engines/seo/app/concepts/seo/snippet_page/twins/search.rb:45 (lambda)>},
:loaded_objects=>true},
#post_filters=[],
#queries=[],
#request_options={},
#scores=[],
#script_fields={},
#search_options={},
#sort=[{:_geo_distance=>{"seo_area.suburb.coordinates"=>"-27.9836052, 153.3977354", :order=>"asc", :unit=>"km"}}],
#suggest={},
#types=[]>,
#options={}>
I'm using Elasticsearch aggregation so any sorting from the query/search phase will be gone upon accessing the aggregation.
What I've been passing is this...
aggs: {
by_seo_area_suburb_id: {
terms: {
field: "seo_area.suburb.id",
size: 10,
order: { by_distance: "desc" }
},
aggs: {
by_top_hit: {
top_hits: { size: 10 }
},
by_distance: {
geo_distance: {
field: "seo_area.suburb.coordinates",
origin: "52.3760, 4.894",
ranges: [
{ from: 0, to: 1 },
{ from: 1, to: 2 }
]
}
}
}
}
}
I'm getting this error though...
[500] {"error":{"root_cause":[{"type":"aggregation_execution_exception","reason":"Invalid terms aggregation order path [by_distance]. Terms buckets can only be sorted on a sub-aggregator path that is built out of zero or more single-bucket aggregations within the path and a final single-bucket or a metrics aggregation at the path end."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"snippet_pages","node":"srrlBssmSEGsqpZnPnOJmA","reason":{"type":"aggregation_execution_exception","reason":"Invalid terms aggregation order path [by_distance]. Terms buckets can only be sorted on a sub-aggregator path that is built out of zero or more single-bucket aggregations within the path and a final single-bucket or a metrics aggregation at the path end."}}]},"status":500}
Simply says...
Terms buckets can only be sorted on a sub-aggregator path that is built out of zero or more single-bucket aggregations within the path and a final single-bucket or a metrics aggregation at the path end.
Any ideas?
You have Buckets like this:
1-2
2-3
4-5
and so on. These are no single value buckets with a natural order. Thats what the exception is telling you. So you need something to melt it down to single values.
Even if you could order by that. Why would you? All with a distance between 1 and 2 would have the same value for comparison and their ordering would be undefined. If its enough for you to know which are 0-1 and 1-2 and so on just turn around the aggregation order. First take the distance and make a subaggregation for terms.
All in all I think you have a usecase in which aggregations are not what you want because consider the following two documents:
{ name: "peter", location: [0,0] }
{ name: "peter", location: [100,0] }
obviously both peters would melt to one in a terms aggregation. But they have two different locations and therefore the distance will (nearly) always be different. So how can you order peters by distance? As soon as you aggregate a field all other fields more or less become decoupled from it and you cannot use other fields for that.
So. If you want something like this you most likely have to go via the normal search. Have a look at this on how to sort a search by distance:
https://www.elastic.co/guide/en/elasticsearch/guide/current/sorting-by-distance.html
Related
I am trying to paginate over a specific field using the terms aggregation with partitions.
The problem is that the number of returned terms for each partition is not equal to the size parameter that I set.
These are the steps that I am doing:
Retrieve the number of different unique values for the field with "cardinality" aggregation.
In my data, the result is 21.
From the web page, the user wants to display a table with 10 items per page.
if unique_values % page_size != 0:
partitions_number = (unique_values // page_size) + 1
else:
partitions_number = (unique_values // page_size)
Than I am making this simple query:
POST my_index/_search?pretty
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"match": {
"field_to_paginate": "foo"
}
}
]
}
},
"aggs": {
"by_pchostname": {
"terms": {
"size": 10,
"field": "field_to_paginate",
"include": {
"partition": 0,
"num_partitions": 3
}
}
}
}
}
I am expecting to retrieve 10 results. But if I run the query I have only 7 results.
What am I missing here? Do I need to use a different solution here?
As a side note, I can't use composite aggregation because I need to sort results by doc_count over the whole dataset.
Partitons in terms aggregation divide the values in equal chunks.
In your case no of partition num_partitions is 3 so 21/3 == 7.
Partitons are meant for getting large values in the order of 1000 s.
You may be able to leverage shard_size parameter. My suggestion is to read this part of manual and work with the shard_size param
Terms aggregation does not allow pagination. Use composite aggregation instead (requires ES >= 6.1.0). Below is the quote from reference docs:
If you want to retrieve all terms or all combinations of terms in a
nested terms aggregation you should use the Composite aggregation
which allows to paginate over all possible terms rather than setting a
size greater than the cardinality of the field in the terms
aggregation. The terms aggregation is meant to return the top terms
and does not allow pagination.
Suppose I have 3 Documents:
A, B, C
All terms are very similar, and internal score is pretty identical
And people searching B more frequently than A and C, but C more frequently than A.
Can I get score to order like B, C, A ?
In the use case you described, you can not change the behaviour of scoring/relevancy computation. There is the possibility of boosting when using e.g. match query to affect scoring when searching for values. But that wouldn't be appropriate since you only want to sort the documents.
So the information about the search frequency has to be a part of the documents themselves, meaning it has to be an own field. Then you can simply add a sort clause like the following
{
"query": {
// your awesome query...
},
"sort": [
{
"search_frequency": {
"order": "desc"
}
},
"_score"
]
}
The challenge in this solution would be to keep the value of the field search_frequency up to date. You can do that via the Update API.
I have an ElasticSearch index with a list of "shops".
I'd like to allow customers to search these shops by both geo_distance (so, search for a point and get shops near that location), and textual match, like matches on shop name / address.
I'd like to get results that match either of these two criteria, and I'd like the order of these results to be a combination of both. The stronger the textual match, and the closer to the point searched, the higher the result. (Obviously, there's going to be a formula to combine these two, that'll need tweaking, not too worried about that part yet).
My issue / what I've tried:
geo_distance is a filter, not a query, so I can't combine both on the query part of the request.
I can use a bool => should filter (rather than query) that matches on either name or location. This gives me the results I want, but not in order.
I can also have _geo_distance as part of a sort clause so that documents closer to the point rank higher.
What I haven't figured out is how I would take the "regular" _score that ElasticSearch gives to documents when doing textual matches, and combine that with the geo_distance score.
By having the textual match in the filter, it doesn't seem to affect the score of documents (which makes sense). And I don't see how I could combine the textual match in the query part and a geo_distance filter so it's an OR rather than an AND.
I guess my best bet would be the equivalent of this:
{
function_score: {
query: { ... },
functions: [
{ geo_distance function },
{ multi_match_result score },
],
score_mode: 'multiply'
}
}
but I'm not sure you can do geo_distance as a score function, and I don't know how to have multi_match_result score as a score function, or if it's even possible.
Any pointers will be greatly appreciated.
I'm working with ElasticSearch v1.4, but I can upgrade if necessary.
but I'm not sure you can do geo_distance as a score function, and I don't know how to have multi_match_result score as a score function, or if it's even possible.
You can't really do it in the way that you're asking, but you can do what you want just as easily. For the simpler case, you get scoring just by using a normal query.
The problem with filters is that they're yes/no questions, so if you use them in a function_score, then it either boosts the score or it doesn't. What you probably want is degradation of the score as the distance from the origin grows. It's the yes/no nature that stops them from impacting the score at all. There's no improvement to relevancy implied by matching a filter -- it just means that it's part of the answer, but it doesn't make sense to say that it should be closer to the top/bottom as a result.
This is where the Decay function score helps. It works with numbers, dates, and -- most helpfully here -- geo_points. In addition to the types of data it accepts, it can decay using either gaussian, exponential, or linear decay functions. The one that you want to choose is honestly arbitrary and you should give the one that chooses the best "experience". I would suggest to start with gauss.
"function_score": {
"functions": [
"gauss": {
"my_geo_point_field": {
"origin": "0, 1",
"scale": "5km",
"offset": "500m",
"decay": 0.5
}
}
]
}
Note that origin is in x, y format (due to standard GeoJSON), which is longitude, latitude.
Each one of the values impacts how the score decays based on the graph (taken wholesale from the documentation). If you would use an offset of 0, then the score begins to drop once it's not exactly at the origin. With the offset, it allows it some buffer to be considered just as good.
The scale is directly associated with the decay in that the score will be chopped down by the decay value once it is scale-distance away from the origin (+/- the offset). In my above example, anything 5km from the origin would get half of the score as anything at the origin.
Again, just note that the different types of decay functions change the shape of scoring.
I'd like the order of these results to be a combination of both.
This is the purpose of the bool / should compound query. You get OR behavior with score improvement based on each match. Combining this with the above, you'd want something like:
{
"query": {
"bool": {
"should": [
{
"multi_match": { ... }
},
{
"function_score": {
"functions": [
"gauss": {
"my_geo_point_field": {
"origin": "0, 1",
"scale": "5km",
"offset": "500m",
"decay": 0.5
}
}
]
}
}
]
}
}
}
NOTE: If you add a must, then the should behavior changes from literal OR-like behavior (at least 1 must match) to completely optional behavior (none must match).
I'm working with ElasticSearch v1.4, but I can upgrade if necessary.
Starting with Elasticsearch 2.0, every filter is a query and every query is also a filter. The only difference is the context that it's used in. This doesn't change my answer here, but it's something that may help you in the future in addition to what I say next.
Geo-related performance increased dramatically in ES 2.2+. You should upgrade (and recreate your geo-related indices) to take advantage of those changes. ES 5.0 will have similar benefits!
Given an index of books that have a title, an author, and a description, I'd like the resulting search results to be sorted this way:
all books that match the title sorted by downloads (a numeric value)
all books that match on author sorted by downloads
all books that match on description sorted by downloads
I use the search query below, but the problem is that each entry has a different score thus making sorting by downloads irrelevant.
e.g. when the search term is 'sorting' - title: 'sorting in elastic search' will score higher than title: 'postgresql sorting is awesome' (because of the word position).
query = QueryBuilders.multiMatchQuery(queryString, "title^16", "author^8", "description^4")
elasticClient.prepareSearch(Index)
.setTypes(Book)
.setQuery(query)
.addSort(SortBuilders.scoreSort())
.addSort(SortBuilders.fieldSort("downloads").order(SortOrder.DESC))
How do I construct my query so that I could get the desired book sorting?
I use standard analysers and I need to the search query to be analysed, also I will have to handle multi-word search query strings.
Thx.
What you need here is a way to compute score based on three weighted field and a numeric field. Sort will sum the score obtained from both , due to which if either one of them is too large , it will supersede the other.
Hence a better approach would be to multiple downloads with the score obtained by the match.
So i would recommend function score query -
{
"query": {
"function_score": {
"query": {
"multi_match": {
"query": "sorting",
"fields": [
"title^16",
"author^8",
"description^4"
]
}
},
"function": [
{
"field_value_factor": {
"field": "downloads"
}
}
],
"boost_mode": "multiply"
}
}
}
This will compute the score based on all three fields. And then multiply that score with the value in download field to get the final score. The multiply boost_mode decides how the value computed by functions are clubbed together with the score computed by query.
I wonder how Elastic search is sorting (on what field) when no search query is specified (I just filter on documents) and no sort option specified. It looks like sorting is than random ... Default sort order is _score, but score is always 1 when you do not specify a search query ...
You got it right. Its then more or less random with score being 1. You still get consistent results as far as I remember. You have the "same" when you get results in SQL but don't specify ORDER BY.
Just in case someone may see this post even it posted over 6 yrs ago..
When you wanna know how elasticsearch calculate its own score known as _score, you can use the explain option.
I suppose that your query(with filter & without search) might like this more or less (but the point is making the explain option true) :
POST /goods/_search
{
"explain": true,
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"term": {
"maker_name": "nike"
}
}
}
}
}
As running this, you will notice that the _explaination of each hits describes as below :
"_explanation" : {
"value" : 1.0,
"description" : "ConstantScore(maker_name:nike)",
"details" : [ ]
}
which means ES gave constant score to all of the hits.
So to answer the question, "yes".
The results are sorted kinda randomly because all the filtered results have same (constant) score without any search query.
By the way, enabling an explain option is more helpful when you use search queries. You will see how ES calculates the score and will understand the reason why it returns in that order.
Score is mainly used for sorting, Score is calculated by lucene score calculating using several constraints,For more info refer here .