If I have set of documents as a response from elastic search,
how can I aggregate the results based on the score? the results should have two buckets , where the first bucket has documents whose score greater than 1 and the other less than 1.
I am new to elastic search, have seen that I can use script for this, but could not get that working.
Related
Is the Count of Search results from Elastic Search accurate? Or is this approximate? If accurate, will it always be accurate based on what is indexed? Assuming that all documents are indexed along with all the text in them and no documents are being ingested, what is the behavior of Count of Search results? Does it change base on the volume of the index and volume of the search results. Thanks for your help!!
I have data with the following format::
{
timestamp: Date,
x: number
}
I want to display these values simply in a line, without any aggregation over x, but in Kibana it always requires me to select some kind of aggregation, like average.
It is possible to create the line-chart that you request, but for Kibana to create an visualization, I'm afraid an aggregation would be necessary.
Kibana basis its visualization on buckets (Date, x-axis) and metrics (x, y-axis). Buckets are aggregations of documents over a specified search (almost 30 aggregation methods)
. Metrics are value(s) based on the documents contained in each bucket (almost 20 aggregation methods)
(https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html).
However, you could try to create buckets with 'date_histogram' for which the time interval is small enough so it contain one document. Then for the metric aggregation you could select min or max aggregation (Note: This assumes though that you timestamp is unique for each document).
I am using Elasticsearch6.8 and I get a list of document in the response. Some of the document have the same score but they appear on the same order in the response list consistently. I wonder what the algorithm ES uses to sort the document with the same score?
ES uses the index order when there is a tie when sorting on score.
The index order is defined by the _doc field.
I have small (max 50 char) keywords stored in text field in ElasticSearch index. I noticed that if I clear the index and add only 1 document, let's say "samsung galaxy", the score when I match the document is like 0.95.
But when I add 500k other docs and I make the same query, the score is like 20. I would like to set a min_score for this query because I need a certain level of relevancy.
But as the score is depending of the doc count. I can't set a min_score as the number of docs in the index will constantly evolve.
I already looked for solutions like constant_score but I need the power of Elastic to give me a score (and not 1 or 0).
1) Does this behavior come from the IDF method or not only from it?
2) Is there a way to keep the current search algorythm (or just without the term frequency) and have always the same score for a query without doc count dependency ? This would allow me to set a min_score
Suppose I have an index for cars on a dealer's car lot. Each document resembles the following:
{
color: 'red',
model_year: '2015',
date_added: '2015-07-20'
}
Suppose I have a million cars.
Suppose I want to present a view of the most recently added 1000 cars, along with facets over those 1000 cars.
I could just use from and size to paginate the results up to a fixed limit of 1000, but in doing so the totals and facets on model_year and color (i.e. aggregations) I get back from Elasticsearch aren't right--they're over the entire matched set.
How do I limit my search to the most recently added 1000 documents for pagination and aggregation?
As you probably saw in the documentation, the aggregations are performed on the scope of the query itself. If no query is given, the aggregations are performed on a match_all list of results. Even if you would use size at the query level, it will still not give you what you need because size is just a way of returning a set of documents from all the documents the query matched. Aggregations operate on what the query matches.
This feature request is not new and has been asked for before some time ago.
In 1.7 there is no straight forward solution. Maybe you can use the limit filter or terminate_after in-body request parameter, but this will not return the documents that were, also, sorted. This will give you the first terminate_after number of docs that matched the query and this number is per shard. This is not performed after the sorting has been applied.
In ES 2.0 there is, also, the sampler aggregation which works more or less the same way as the terminate_after is working, but this one takes into consideration the score of the documents to be considered from each shard. In case you just sort after date_added and the query is just a match_all all the documents will have the same score and it will be returning an irrelevant set of documents.
In conclusion:
there is no good solution for this, there are workarounds with number of docs per shard. So, if you want 1000 cars, then you need to take this number divide it by the number of primary shards, use it in sampler aggregation or with terminate_after and get a set of documents
my suggestion is to use a query to limit the number of documents (cars) by a different criteria instead. For example, show (and aggregate) the cars in the last 30 days or something similar. Meaning, the criteria should be included in the query itself, so that the resulting set of documents to be the one you want it aggregated. Applying aggregations to a certain number of documents, after they have been sorted, is not easy.