I'm trying to make a search that both limits and "offsets" (the keyword from in elasticsearch) the facet result set, so something like:
'{
"query" : {
"nested" : {
"_scope" : "my_scope",
"path" : "related_award_vendors",
"score_mode" : "avg",
"query" : {
"bool" : {
"must" : {
"text" : {"related_award_vendors.title" : "inc"}
}
}
}
}
},
"facets" : {
"facet1" : {
"terms_stats" : {
"key_field" : "related_award_vendors.django_id",
"value_field" : "related_award_vendors.award_amount",
"order":"term",
"size": 5,
"from":2
},
"scope" : "my_scope" }
}
}'
In the above, it returns id's 1,2,3,4,5 and if I remove "from" it still returns 1,2,3,5 in the result set.
The "size" is working correctly. In this case, it's returning five items in the result set.
My understanding is that solr can do this. Can this be done in elasticsearch?
The terms stats facet doesn't support the from parameter. The only way to achieve what you want is to set size to size + offset and ignore first offset entries on the client side. In your example it would mean to request 7 entries and ignore first 2.
Related
I have text document data (500k approximately) saved in elasticsearch where the document text is mapped with it's corresponding document number.
I am trying to fetch results in batches for "Sample Text" in particular set of document numbers (300k appoximately) with scoring and i am facing extreme slowness in the result.
Here is the the Mapping
PUT my_index
{
"mappings" : {
"doc_repo" : {
"properties" : {
"doc_number" : {
"type" : "integer"
},
"document" : {
"type" : "string",
"term_vector" : "with_positions_offsets_payloads"
}
}
}
}
}
Here is the request query
{
"query" : {
"bool" : {
"must" : [
{
"terms" : {
"document" : [
"sample text"
]
}
},
{
"terms" : {
"doc_number" : [1,2,3....,300K] //ArrayOf_300K_DocNumbers
}
}
]
}
},
"fields" : [
"doc_number"
],
"size" : 500,
"from" : 0
}
I Tried fetching result in two other ways
Result without scoring in particular set of document numbers(i used filtering for this)
Result with scoring but without any particular set of document numbers (in batches)
Both of these were pretty quick, but problem comes when i am trying achieve both.
Do i need to change mapping or search query or any other ways to achieve this.
Thanks in advance.
Issue was specifically with elasticsearch 2.X, Upgrading elasticsearch solves the issue.
I'm using a document query against a percolator that works ok. When I try to filter the percolator queries against which document percolate using queries ids, it doesn't return any result. For example:
{
"doc" : {
"text" : "This is the text within my document"
},
"highlight" : {
"order" : "score",
"pre_tags" : ["<example>"],
"post_tags" : ["</example>"],
"fields" : {
"text" : { "number_of_fragments" : 0 }
}
},
"filter":{"ids":{"values":[11,15]}}
,
"size" : 100
}
I know for sure that those ids are correct, but allways obtain "matches" : [ ]. When I don't use filter, ES retrieves correct matches.
Thanks for your help.
I think I've solved it. It seems that the filter only works on the "metadata" fields, meaning that you have to add customized fields to the queries indexed in the percolator in order to use them to filter when you need.
Using my previous example, I would have to index in percolator queries like:
{
"query" : {
"match_phrase" : {
"text" : "document"
}
},
"id" : 11
}
Adding "manually" a redundant id field in order to use it later as filter reference.
At percolation time, you have to use something like:
{
"doc" : {
"text" : "This is the text within my document"
},
"filter":{"match":{"id":11}},
"highlight" : {
"order" : "score",
"pre_tags" : ["<example>"],
"post_tags" : ["</example>"],
"fields" : {
"text" : { "number_of_fragments" : 0 }
}
},
"size" : 100
}
In order to use only that percolator query. Complementary information can be found here.
I have following match query string:
curl -XGET 'my-es.com/my_indice/_search?pretty' -d '{
"size" : 10,
"query" : {
"bool" : {
"must" : [ {
"match" : {
"state" : {
"query" : ["ACTIVE", "INACTIVE"],
"type" : "boolean"
}
}
}]
}
}
}'
I guess it means "state" = "ACTIVE" or "state" = "INACTIVE", but actually it performs "state" = "INACTIVE".
Then I tried term query string:
curl -XGET 'my-es.com/my_indice/_search?pretty' -d '{
"size" : 10,
"query" : {
"bool" : {
"must" : [{
"terms" : { "state" : ["ACTIVE", "INACTIVE"] }
}]
}
}
}'
It performs "state" = "ACTIVE" or "state" = "INACTIVE", showing term query supports multiple OR conditions via an array.
I'm curious about why match query does not support OR condition via an array? And it does not show any syntax error.
The match query only supports a single string value to be specified. It is not explicitly specified in the official match documentation, but if you're willing to read some source code for MatchQueryParser.java, you can see that while parsing the query field, the parser will skip the tokens delimiting the start and end of the array and always override the value with the latest one being parsed, hence why you're seeing what you see, i.e. that state will be matched with INACTIVE only.
What you can do, however, is to put both tokens inside the same string like below, and both tokens will be taken into account:
curl -XGET 'my-es.com/my_indice/_search?pretty' -d '{
"size" : 10,
"query" : {
"bool" : {
"must" : [ {
"match" : {
"state" : {
"query" : "ACTIVE INACTIVE",
"type" : "boolean"
}
}
}]
}
}
}'
In Elasticsearch I have an index containing documents with a timestamp and the number of observed requests to a webservice.
I would like to perform an aggregation to get, for each day, the hour where the maximum number of requests were observed (peak hour).
I succeed to get the result by performing the following request:
{
"aggregations" : {
"week_summary" : {
"filter" : {"range": {"#timestamp": {"gte": "2015-01-20||-7d","lte": "2015-01-20"}}},
"aggregations" : {
"oneday_interval" : {
"date_histogram" : {"field" : "#timestamp", "interval" : "1d","order" : { "_key" : "desc" }},
"aggregations" : {
"peak_hour_histogram" : {
"date_histogram" : {"field" : "#timestamp", "interval" : "1h","order" : { "peak_request_count.value" : "desc" }},
"aggregations" : {
"peak_request_count" : {
"sum" : { "field" : "request_count"}
}
}
}
}
}
}
}
},
size : 0
}
This is working ok in a sense: the first item in the peak_hour_histogram buckets array is indeed corresponding to the peak hour due to the ability to sort a date histogram on a sub-aggregation value.
Nevertheless, I don't need all the other buckets items (i.e. the other 23 hours of the day), and I'd like to receive only the first item. I tried to play with top_hits without any success.
Do you know a way to perform this filtering?
NB: In the real use case my aggregation is returning about 3MB of data. So filtering all those useless values becomes important.
Thanks for your answers.
I think this would be the feature that should answer your requirement: https://github.com/elasticsearch/elasticsearch/issues/6704. Started from this one: https://github.com/elasticsearch/elasticsearch/issues/7103
I'm new using Elastic Search, and i never used Lucene too.
I build this query:
{
"query" : {
"wildcard" : { "referer" : "*.domain.com*" }
},
"filter" : {
"query" : {
"term" : { "first" : "1" }
}
},
"facets" : {
"site_id" : {
"terms" : {
"field" : "site",
"size" : "70"
}
}
}
}
The wildcard is working great, but the term filter was ignored, what i did wrong?
I need to filter the results with both wildcard and term
Thanks!
Assuming what you are trying to do is applying the filter on the wildcard query results,
you can use a FilteredQuery. However, your case might fit better for a filter.
You use a query filter. Instead of that you may directly use a TermFilter in a FilteredQuery rather than making a filter out of a TermQuery. TermFilter should be faster as it directly uses the TermsEnum.
Note that results of Filters are cached in a FilterCache and Filters are faster because they do not do any scoring of documents. In your case, even though the filter part of the FilteredQuery will work fast, but the wildcard query will be unnecessarily do scoring. You may try to use an AND Filter to club both queryfilter(wildcard query) and term filter instead of a FilteredQuery.
To make just the filter work as required by you, try something like below. (Not tried myself)
{
"filtered" : {
"query" : {
"wildcard" : { "referer" : "*.domain.com*" }
},
"filter" : {
"term" : { "first" : "1" }
}
},
"facets" : {
"site_id" : {
"terms" : {
"field" : "site",
"size" : "70"
}
}
}
}