Can someone elaborate on these three filters in elastic search namely must, should, excludeFilters.
Are these parts of bool queries which come under compound category of queries in elastic search.
must is the same as logical AND operator and should is the same as logical OR operator
These clauses are used to combine multiple conditions (using bool query) when you are creating your DSL query
All these queries are included inside the bool query. For example
{
"query": {
"bool": {
"must": {},
"should": {},
"filter": {}
}
}
}
Update 1:
should (contribute to relevancy score) this means that when you use a should clause, then the search results are returned based on a number of factors like doc count, length of the field, frequency, total term frequency etc.
Whereas in the case of filter (doesn't contribute to relevancy score), this means that it just gives an answer of yes/no i.e whether a document matches or not. (It does not consider other factors as must and should clause considers)
Related
I am retrieving documents by filtering and using a term query to apply a score.
The query should match all animals having a specified color - the more colors are matched, the higher the score of a doc. Strange thing is, term and terms query result in a different scoring.
{
"query": {
"bool": {
"should": [
{"terms": {"color": ["brown","darkbrown"] } },
]
}
}
}
should be the same like using
{"term": {"color": {"value": "brown"} } },
{"term": {"color": {"value": "darkbrown"} } }
Query no. 1 gives me the exact same score for a document whether 1 or 2 terms are matched. The latter of course returns a higher score, if more colors are matched.
As stated by the coordination factor the returned score should be higher if more terms are matched. Therefore these two queries should result in the same score - or is because term queries do not analyze the search term?
My field is indexed as text. Strings are indexed as an "array" of strings, e.g. "brown","darkbrown"
Difference between term vs terms query:
Term query return documents that contain one or more exact term in a provided field.
The terms query is the same as the term query, except you can search for multiple values.
Warning: Avoid using the term query for text fields.
As far your this part is concerned
or is because term queries do not analyze the search term?
Yes, It is because the search term does not analyze the term searched. It just matches the exact search term.
I was following an ES tutorial, and at some point I wrote a query using term in the filter instead the recommended solution using match. My understanding is that match was used in the query part to get scoring, while term was used in the filter part to just remove hits before enter the query part. To my surprise match also works in the filter part.
What is the difference between:
GET blogs/_search
{
"query": {
"bool": {
"filter": {
"match": {
"category.keyword": "News"
}
}
}
}
}
and:
GET blogs/_search
{
"query": {
"bool": {
"filter": {
"term": {
"category.keyword": "News"
}
}
}
}
}
Both returns the same hits, and the score is 0 for all hits.
What is the behaviour or match in a filter clause? I would expect it to yield some score, but it does not.
What I thought:
term : does not analyze either the parameter or the field, and it is a yes/no scenario.
match : analyzes parameter and field and calculates a score of how good they match.
But when using match against a keyword in the filter part of the query, how does it behave?
The match query is a high-level query that resorts to using a term query if it needs to.
Scoring has nothing to do with using match instead of term. Scoring kicks in when you use bool/must/should instead of bool/filter.
Here is how the match query works:
First, it checks the type of the field.
If it's a text field then the value will be analyzed, either with the analyzer specified in the query (if any), or with the search- or index-time analyzer specified in the mapping.
If it's a keyword field (like in your case), then the input is not analyzed and taken "as is"
Since you're using the match query on a keyword field and your input is a single term, nothing is analyzed and the match query resorts to using a term query underneath. This is why you're seeing the same results.
In general, it's always best to use a match query as it is smart enough to know what to do given the field you're querying and the input data you're searching for.
You can read more about the difference between the two here.
I am using the below query :
GET customer/doc/_search?routing=123
{
"query": {
"bool": {
"filter": [
{
"term": {
"location": "Delhi"
}
}
],
"should": [
{
"match_phrase_prefix": {
"phone": {
"query": "650",
"max_expansions": 100
}
}
}
]
}
}
}
The problem is my search on phone isn't working anymore. It used to work fine when I had less data, now every shard has data for multiple locations. Search on phone now requires me to type in 6 or 7 characters at times. (There may be matching phone numbers that have different location but are on this shard)
This is due to max_expansions I am guessing. When I increase it to 500 it does return me search results (not all), but the query becomes slow.
Isn't there a way to force es to apply filter first (and restrict the dataset) and then apply the should clause, so that I get the matching results even with small value of max_expansions?
Any help is appreciated.
It is due to max_expansions. Restricting dataset is not exactly what you may want to do ( Thats also not very straight forward - you may have to use some script which will in turn slowdown query).
When you query for a wildcard expression, Lucene expands the wildcard expression into set of actual terms in your inverted index term dictionary. Now , when you restrict the term expansion to 500 - it might miss a few.
I would consider using prefixes during indexing phase. Prefixes helps to avoid the costly expansion in runtime phase.
I am new to elastic search and I am confused between must and filter. I want to perform an and operation between my terms, so I did this
POST /xyz/_search
{
"query": {
"bool": {
"must": [
{
"term": {
"city": "city1"
}
},
{
"term": {
"saleType": "sale_type1"
}
}
]
}
}
}
which gave me the required results matching both the terms, and on using filter like this
POST /xyz/_search
{
"query": {
"bool": {
"must": [
{
"term": {
"city": "city1"
}
}
],
"filter": {
"term": {
"saleType": "sale_type1"
}
}
}
}
}
I get the same result, so when should I use must and when should I use filter? What is the difference?
must contributes to the score. In filter, the score of the query is ignored.
In both must and filter, the clause(query) must appear in matching documents. This is the reason for getting same results.
You may check this link
Score
The relevance score of each document is represented by a positive floating-point number called the _score. The higher the _score, the more relevant the document.
A query clause generates a _score for each document.
To know how score is calculated, refer this link
must returns a score for every matching document. This score helps you rank the matching documents, and compare the relative relevance between documents (using the magnitude of the score of each document).
With this, one can say, Doc 1 is how many times more relevant than Doc 2. Or that Doc 1 to 7 are of much higher relevancy than Doc 8+.
For how the relative score is determined, you can refer to the references below.
Briefly, it is related to the number of term occurrences in the document, the document length, and the average number of term occurrences in your database index.
filter doesn't return a score. All one can say is, all matching documents are of relevance. But it won't help in evaluating if one is more relevant than the other. You can think of filter as a must with only 2 scores: zero or non-zero, and where all zero-scored documents are dropped.
filter is helpful if you just want to whitelist/blacklist for e.g., all documents belonging to the topic "pets".
In summary, there are 3 points that will help you in deciding when to use what:
must is your only choice when comparing/ranking documents by relevance
filter excludes all documents that don't match
filter is a lot faster because Elasticsearch doesn't need to compute the relative score
References:
Query vs Filter: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html
Computation of Relevance: https://www.infoq.com/articles/similarity-scoring-elasticsearch/
What is the difference between MUST and SHOULD bool query in ES?
If I ONLY want results that contain my terms should I then use must ?
I have a query that should only contain certain values, and also no results that has a lower date/timestamp than todays time/date - NOW
Also
Can i use multiple filters inside a must like the code bellow:
"filtered": {
"filter": {
"bool" : {
"must" : {
"term" : { "type" : 1 }
"term" : { "totals" : 14 }
"term" : { "groupId" : 3 }
"range" : {
"expires" : {
"gte": "now"
}
}
},
must means: The clause (query) must appear in matching documents. These clauses must match, like logical AND.
should means: At least one of these clauses must match, like logical OR.
Basically they are used like logical operators AND and OR. See this.
Now in a bool query:
must means: Clauses that must match for the document to be included.
should means: If these clauses match, they increase the _score; otherwise, they have no effect. They are simply used to refine the relevance score for each document.
Yes you can use multiple filters inside must.
Since this is a popular question, I would like to add that in Elasticsearch version 2 things changed a bit.
Instead of filtered query, one should use bool query in the top level.
If you don't care about the score of must parts, then put those parts into filter key. No scoring means faster search. Also, Elasticsearch will automatically figure out, whether to cache them, etc. must_not is equally valid for caching.
Reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html
Also, mind that "gte": "now" cannot be cached, because of millisecond granularity. Use two ranges in a must clause: one with now/1h and another with now so that the first can be cached for a while and the second for precise filtering accelerated on a smaller result set.
As said in the documentation:
Must: The clause (query) must appear in matching documents.
Should: The clause (query) should appear in the matching document. In a boolean query with no must clauses, one or more should clauses must match a document. The minimum number of should clauses to match can be set using the minimum_should_match parameter.
In other words, results will have to be matched by all the queries present in the must clause ( or match at least one of the should clauses if there is no must clause.
Since you want your results to satisfy all the queries, you should use must.
You can indeed use filters inside a boolean query.