I have an index in ElasticSearch 0.9 with some documents like this:
{"Id":1, "Title":"Hello World" , "Popularity":1},
{"Id":2, "Title":"Hello World" , "Popularity":3},
{"Id":3, "Title":"Hello" , "Popularity":10}
As you see the first two documents have the same title text but different popularity values. Now I do a Fuzzy search on the Title property with a simple scoring script in place:
Scoring script : "_score * doc['Popularity'].value"
My query is something like this:
{
"query": {
"custom_score": {
"lang": "mvel",
"script": "_score * doc['Popularity'].value",
"query": {
"fuzzy": {
"Title": {
"value": "Hello World",
"fuzziness": 3
}
}
}
}
}
}
Now what happens is that the third document (whose Id is equal to 3) comes to the top of the search result simply because it has a higher popularity. In the other words the scoring function completely overrides the relevancy of the search result. Whereas I expect to see the more relevant documents (Id = 1 and 2) on the top because they are more relevant to the search term (shorter distance) and then between the two top search results the scoring function boost the document with a higher popularity value. So the result would I expect would be like this:
{"Id":2, "Title":"Hello World" , "Popularity":3}
{"Id":1, "Title":"Hello World" , "Popularity":1}
{"Id":3, "Title":"Hello" , "Popularity":10}
As a real world example, we have a music store which has a search bar on the top. Users may enter a keyword such as "Blue" and then there will be tens of music tracks whose title is "Blue" and some other which are close to the search time (e.g. "BlueSky"). Each track has a popularity property as well however we want to see all the tracks whose title is "Blue" on the top even the track with title of "BlueSky" has a higher popularity simply because the users prefer to see the exact matches first. Then those whose title is exactly "Blue" must be ranked by the scoring script.
Can someone please guide me as to how can I update my query so that the relevant result (regardless of scoring) still get to the top of the result list and then among them scoring boost the more popular ones?
Related
Search after in elasticsearch must match its sorting parameters in count and order. So I was wondering how to get the score from previous result (example page 1) to use it as a search after for next page.
I faced an issue when using the score of the last document in previous search. The score was 1.0, and since all documents has 1.0 score, the result for next page turned out to be null (empty).
That's actually make sense, since I am asking elasticsearch for results that has lower rank (score) than 1.0 which are zero, so which score do I use to get the next page.
Note:
I am sorting by score then by TieBreakerID, so one possible solution is using high value (say 1000) for score.
What you're doing sounds like it should work, as explained by an Elastic team member. It works for me (in ES 7.7) even with tied scores when using the document ID (copied into another indexed field) as a tiebreaker. It's true that indexing additional documents while paginating will make your scores slightly unstable, but not likely enough to cause a significant problem for an end user. If you need it to be reliable for a batch job, the Scroll API is the better choice.
{
"query": {
...
},
"search_after": [
12.276552,
14173
],
"sort": [
{ "_score": "desc" },
{ "id": "asc" }
]
}
Imagine I have some documents, with the following values contained within a text field called name
Document1: abc xyz group
Document2: group x/group y
Document3: group 1, group 2, group 3, group 4
Now imagine I'm sending a simple match query to ES for the term 'group':
{
"query": {
"match": {
"name": "group"
}
}
}
My desired outcome would be that all 3 documents would return with the same score, no matter how often the term appears, where it appears, etc.
Now, I already know that I can do this by wrapping my match with a constant_score, like so:
{
"query": {
"constant_score": {
"filter": {
"match": {
"name": "group"
}
},
"boost": 1
}
}
}
BUT, say I now want to query using the search term abc group. In this case, what I want to happen is that Document2 and Document3 will return the same score (matches group), but Document1 to have a better score as it matches both abc and group.
With a constant_score wrapping my match query, documents that contain any of the terms return the same score (i.e Document1, 2 and 3 return the same score for abc group). If I remove the constant_score, then Document 3 has the best score presumably because it contains more matches with the search text (group appearing 4 times).
It seems as though I need a way of moving the constant_score query to after the match query has analyzed my search text. Effectively causing a query of abc group to be two constant_score queries - one for abc and one for group.
Does anyone know of a way to achieve this?
I've managed to solve this by utilising Elasticsearch's unique token filter: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-unique-tokenfilter.html
I've added that to my name field in the index mappings, and it looks to be retrieving the desired results without having to worry about constant_score.
Note however all this does is eliminate term frequencies from having any effect on the _score - other metrics (such as fieldLength) still have an effect on the results. This isn't, therefore, the equivalent of using a post-analyzed version of constant_score as I hypothesized in the question, however this will suffice for my current requirements.
I am in a scenario where I need to give more relevance to the document in Index if it has a unique keyword. Let me provide a scenario.
Let's say I need to search for a term znkdref unsuccessfull so the result will have contents which have znkdref or unsuccessfull or znkdref unsuccessfull but here I want that the contents which are having znkdref unsuccessfull should have highest relevance and then content having znkdref should have less relevance and then content having unsuccessfull should have least relevance.
Is there a way to achieve this ?? I would be glad to get any help
You want to use Query Time Boosting, in particular Prioritized Clauses.
In short you need to extract the keywords that you want boosted and build a query that boosts the parts that you want.
{
"query": {
"bool": {
"should": [{
"match": {
"content": {
"query": "znkdref",
"boost": 2
}
}
},
{
"match": {
"content": {
"query": "unsuccessfull"
}
}
}]
}
}
}
Update based on comment:
If you want to know why a document got the score that it did (maybe to identify "keywords") then you can pass in "explain" as a query parameter or set it in the root POST payload. The result will now have document frequency counts and sub scores.
Do you mean "znkdref" is a unique keyword? For example, "znkdref" is a special name of something. If so.
Of course, the documents match the whole query string "znkdref unsuccessfull" will have a highest relevance score in general.
The documents contain "znkdref" will usually have a higher relevance score than the documents contain "unsuccessfull". Because TF.IDF score of "znkdref" is bigger than TF.IDF score of "unsuccessfull".
The relevance score function is described at https://www.elastic.co/guide/en/elasticsearch/guide/current/practical-scoring-function.html
I hope that my answer is helpful for you.
I want to sort results, such that if one specific field (let's say 'first_name') is equal to an exact value (let's say 'Bob'), then those documents are returned first.
That would result in all documents where first_name is exactly 'Bob', would be returned first, and then all the other documents afterwards. Note that I don't intend to exclude documents where first_name is not 'Bob', merely sort them such that they're returned after all the Bobs.
I understand how numeric or alphabetical sorting works in Elasticsearch, but I can't find any part of the documentation covering this type of sorting.
Is this possible, and if so, how?
One solution is to manipulate the score of the results that contain the Bob in the first name field.
For example:
POST /test/users
{
"name": "Bob"
}
POST /test/users
{
"name": "Alice"
}
GET /test/users/_search
{
"query": {
"bool": {
"should": [
{
"match": {
"name": {
"query": "Bob",
"boost" : 2
}
}
},
{
"match_all": {}
}
]
}
}
}
Would return both Bob and Alice in that order (with approximate scores of 1 and 0.2 respectively).
From the book:
Query-time boosting is the main tool that you can use to tune
relevance. Any type of query accepts a boost parameter. Setting a
boost of 2 doesn’t simply double the final _score; the actual boost
value that is applied goes through normalization and some internal
optimization. However, it does imply that a clause with a boost of 2
is twice as important as a clause with a boost of 1.
Meaning that if you also wanted "Fred" to come ahead of Bob you could just boost it with a 3 factor in the example above.
Given an index of books that have a title, an author, and a description, I'd like the resulting search results to be sorted this way:
all books that match the title sorted by downloads (a numeric value)
all books that match on author sorted by downloads
all books that match on description sorted by downloads
I use the search query below, but the problem is that each entry has a different score thus making sorting by downloads irrelevant.
e.g. when the search term is 'sorting' - title: 'sorting in elastic search' will score higher than title: 'postgresql sorting is awesome' (because of the word position).
query = QueryBuilders.multiMatchQuery(queryString, "title^16", "author^8", "description^4")
elasticClient.prepareSearch(Index)
.setTypes(Book)
.setQuery(query)
.addSort(SortBuilders.scoreSort())
.addSort(SortBuilders.fieldSort("downloads").order(SortOrder.DESC))
How do I construct my query so that I could get the desired book sorting?
I use standard analysers and I need to the search query to be analysed, also I will have to handle multi-word search query strings.
Thx.
What you need here is a way to compute score based on three weighted field and a numeric field. Sort will sum the score obtained from both , due to which if either one of them is too large , it will supersede the other.
Hence a better approach would be to multiple downloads with the score obtained by the match.
So i would recommend function score query -
{
"query": {
"function_score": {
"query": {
"multi_match": {
"query": "sorting",
"fields": [
"title^16",
"author^8",
"description^4"
]
}
},
"function": [
{
"field_value_factor": {
"field": "downloads"
}
}
],
"boost_mode": "multiply"
}
}
}
This will compute the score based on all three fields. And then multiply that score with the value in download field to get the final score. The multiply boost_mode decides how the value computed by functions are clubbed together with the score computed by query.