Sort Elasticsearch results based on field value - elasticsearch

Assuming I have 3 documents (users), and they have knowledge of multiple programming languages - with scores associated, as described below, how can I search for multiple fields (multi-match for example), and if some search-keywords hits a language, sort by its score?
// user1
{
"name": "John Bayes",
"prog_langs": [
{
"name": "python",
"score": 10
},
{
"name": "java",
"score": 500
}
]
}
// user2
{
"name": "John Russel",
"prog_langs": [
{
"name": "python",
"score": 100
},
{
"name": "PHP",
"score": 200
}
]
}
// user3
{
"name": "Terry Guy",
"prog_langs": [
{
"name": "C++",
"score": 600
},
{
"name": "Javascript",
"score": 200
}
]
}
For example: searching "John python"
Should return user1 and user2, but user2 showing up first
**I've been trying to use sort and functions, but I think they always use lowest/highest/average values of score.
Thanks!
[Edit]
**In the meantime I got it working in a testing way to see if without full-text/multi-matched works, and I found out I had to make "prog_langs" nested, so I changed the mapping and it works as expected.
Now I'm only missing the part where a full-text search with multi-match merges with current query.
Thanks again!

I managed to fix the query and now it's working as expected.
Before posting my solution, just have to leave a few things to keep in mind:
I made a new mapping, and added some nested objects, so my original query had to suffer some changes (prog_langs are now of type nested)
I wanted at least two fields to match, being mandatory which should match at least once
{
"query": {
"bool": {
"must": [
{
"query": {
"match": {
"name": {
"query": "john python",
"boost": 5
}
}
}
},
{
"bool": {
"should": [
{
"nested": {
"path": "prog_langs",
"query": {
"match": {
"prog_langs.name": {
"query": "john python",
"boost": 5
}
}
}
}
}
]
}
}
],
"should": [
{
"function_score": {
"query": {
"match": {
"prog_langs.name": "john python"
}
},
"functions": [
{
"script_score": {
"script": "_score * (1 + doc['prog_langs.score'].value)"
}
}
]
}
}
]
}
},
"highlight": {
"fields": {
"name": {},
"prog_langs.name": {}
}
}
}

Related

Is there a way to build an Elastic query with changing search values?

I want to use Elastic in PHP to process a search request from my website. For example, I have the search parameter
name
age
height
weight.
But it should not be necessary to always search for all parameters.
So it could be that only (name AND age) have values and (height AND weight) have not.
Is there a way to build one query with flexible/changing input values?
The query below would not work when there are no search values for (height AND weight).
{
"query": {
"bool": {
"should": [
{ "match": { "name.keyword": "Anna" } },
{ "match": { "age": "30" } },
{ "match": { "height": "180" } },
{ "match": { "weight": "70" } }
]
}
}
}
Search templates to the rescue:
POST _scripts/my-search-template
{
"script": {
"lang": "mustache",
"source": """
{
"query": {
"bool": {
"should": [
{{#name}}
{ "match": { "name.keyword": "{{name}}" } },
{{/name}}
{{#age}}
{ "match": { "age": "{{age}}" } },
{{/age}}
{{#height}}
{ "match": { "height": "{{height}}" } },
{{/height}}
{{#weight}}
{ "match": { "weight": "{{weight}}" } },
{{/weight}}
{ "match_none": { } }
]
}
}
}
"""
}
}
Note that since you don't know how many criteria you have, the last condition is always false and is only there to make sure the JSON is valid (i.e. the last comma doesn't stay dangling)
You can then run your query like this:
POST my-index/_search/template
{
"id": "my-search-template",
"params": {
"name": "Anna",
"age": 30
}
}
You need to handle in your application that constructs your Elasticsearch query and its very easy to do it in the application as you know what all search parameter value you got from UI, if they are not null than only includes those fields in your Elasticsearch query.
Elasticsearch doesn't support if...else like condition in query.
Tldr;
They are multiple way to address your problem in Elasticsearch.
You could be playing with the parameter minimum_should_match
You could be using template queries with conditions.
You could also perform more complex bool queries, that enumerate the possibilities for a match.
You could also use scripts to program the logic you want to see.
Minimum should match
POST /_bulk
{"index":{"_index":"73121817"}}
{"name": "ana", "age": 1, "height": 180, "weight": 70}
{"index":{"_index":"73121817"}}
{"name": "jack", "height": 180, "weight": 70}
{"index":{"_index":"73121817"}}
{"name": "emma", "age": 1, "weight": 70}
{"index":{"_index":"73121817"}}
{"name": "william", "age": 1, "height": 180}
{"index":{"_index":"73121817"}}
{"name": "jenny", "weight": 70}
{"index":{"_index":"73121817"}}
{"name": "marco", "age": 1}
{"index":{"_index":"73121817"}}
{"name": "giulia", "height": 180}
{"index":{"_index":"73121817"}}
{"name": "paul"}
GET 73121817/_search
{
"query": {
"bool": {
"should": [
{ "match": { "name.keyword": "Anna" } },
{ "match": { "age": "30" } },
{ "match": { "height": "180" } },
{ "match": { "weight": "70" } }
],
"minimum_should_match": 2
}
}
}
with the minimum should match set to 2 only 2 documents are returned ana and jack
Template queries
Well Val's answer is quite complete
You could also refer to the doc
Complex queries
Refer to the so post behind the link
Scripted queries
GET 73121817/_search
{
"query": {
"bool": {
"filter": {
"script": {
"script": """
return (!doc["name.keyword"].empty && !doc["age"].empty);
"""
}
}
}
}
}

How to boost specific terms in elastic search?

If I have the following mapping:
PUT /book
{
"settings": {},
"mappings": {
"properties": {
"title": {
"type": "text"
},
"author": {
"type": "text"
}
}
}
}
How can i boost specific authors higher than others?
In case of the below example:
PUT /book/_doc/1
{
"title": "car parts",
"author": "john smith"
}
PUT /book/_doc/2
{
"title": "car",
"author": "bob bobby"
}
PUT /book/_doc/3
{
"title": "soap",
"author": "sam sammy"
}
PUT /book/_doc/4
{
"title": "car designs",
"author": "joe walker"
}
GET /book/_search
{
"query": {
"bool": {
"should": [
{ "match": { "title": "car" }},
{ "match": { "title": "parts" }}
]
}
}
}
How do I make it so my search will give me books by "joe walker" are at the top of the search results?
One solution is to make use of function_score.
The function_score allows you to modify the score of documents that are retrieved by a query.
From here
Base on your mappings try to run this query for example:
GET book/_search
{
"query": {
"function_score": {
"query": {
"bool": {
"should": [
{
"match": {
"title": "car"
}
},
{
"match": {
"title": "parts"
}
}
]
}
},
"functions": [
{
"filter": {
"match": {
"author": "joe walker"
}
},
"weight": 30
}
],
"max_boost": 30,
"score_mode": "max",
"boost_mode": "multiply"
}
}
}
The query inside function_score is the same should query that you used.
Now we want to take all the results from the query and give more weight (increase the score) to joe walker's books, meaning prioritize its books over the others.
To achieved that we created a function (inside functions) that compute a new score for each document returned by the query filtered by joe walker books.
You can play with the weight and other params.
Hope it helps

Multi match query with terms lookup searching multiple indices elasticsearch 6.x

All,
I am working on building a NEST 6.x query that takes a serach term and looks in different fields in different indices.
This is the one I got so far but is not returning any results that I am expecting.
Please see the details below
Indices used
dev-sample-search
user-agents-search
The way the search should work is as follows.
The value in the query field(27921093) is searched against the
fields agentNumber, customerName, fileNumber, documentid(These are all
analyzed fileds).
The search should limit the documents to the agentNumbers the user
sampleuser#gmail.com has access to( sample data for
user-agents-search) is added below.
agentNumber, customerName, fileNumber, documentid and status are
part of the index dev-sample-search.
status field is defined as a keyword.
The fields in the user-agents-search index are all keywords
Sample user-agents-search index data:
{
"id": "sampleuser#gmail.com"",
"user": "sampleuser#gmail.com"",
"agentNumber": [
"123.456.789",
"1011.12.13.14"
]
}
Sample dev-sample-search index data:
{
"agentNumber": "123.456.789",
"customerName": "Bank of america",
"fileNumber":"test_file_1123",
"documentid":"1234456789"
}
GET dev-sample-search/_search
{
"from": 0,
"size": 10,
"query": {
"bool": {
"must": [
{
"multi_match": {
"type": "best_fields",
"query": "27921093",
"operator": "and",
"fields": [
"agentNumber",
"customerName",
"fileNumber",
"documentid^10"
]
}
}
],
"filter": [
{
"bool": {
"must": [
{
"terms": {
"agentNumber": {
"index": "user-agents-search",
"type": "_doc",
"user": "sampleuser#gmail.com",
"path": "agentNumber"
}
}
},
{
"bool": {
"must_not": [
{
"terms": {
"status": {
"value": "pending"
}
}
},
{
"term": {
"status": {
"value": "cancelled"
}
}
},
{
"term": {
"status": {
"value": "app cancelled"
}
}
}
],
"should": [
{
"term": {
"status": {
"value": "active"
}
}
},
{
"term": {
"status": {
"value": "terminated"
}
}
}
]
}
}
]
}
}
]
}
}
}
I see a couple of things that you may want to look at:
In the terms lookup query, "user": "sampleuser#gmail.com", should be "id": "sampleuser#gmail.com",.
If at least one should clause in the filter clause should match, set "minimum_should_match" : 1 on the bool query containing the should clause

Search for documents matching all terms in a nested array Elasticsearch

I am learning to use Elasticsearch as a basic recommender engine.
My elasticsearch document contains records with nested entities as follows
PUT recs/user/1
{
"name" : "Brad Pitt",
"movies_liked": [
{
"name": "Forrest Gump",
"score": 1
},
{
"name": "Terminator",
"score": 4
},
{
"name": "Rambo",
"score": 4
},
{
"name": "Rocky",
"score": 4
},
{
"name": "Good Will Hunting",
"score": 2
}
]
}
PUT recs/user/2
{
"name" : "Tom Cruise",
"movies_liked": [
{
"name": "Forrest Gump",
"score": 2
},
{
"name": "Terminator",
"score": 1
},
{
"name": "Rocky IV",
"score": 1
},
{
"name": "Rocky",
"score": 1
},
{
"name": "Rocky II",
"score": 1
},
{
"name": "Predator",
"score": 4
}
]
}
I would like to search for users who specifically like "Forrest Gump","Terminator" and "Rambo".
I have used a nested query which currently looks like this
POST recs/user/_search
{
"query": {
"nested": {
"path": "movies_liked",
"query": {
"terms": {
"movies_liked.name": ["Forrest Gump","Terminator","Rambo"]
}
}
}
}
}
However when I execute this search, I expected to see only the first record which has all the required terms, but in the results I am getting both the records. In the second record the user clearly does not have "Rambo" in his liked list. I understand that this query is doing an "OR" operation with the given terms, How do I tweak this query to do an "AND" operation so that only the records having all the terms get matched?
How do I tweak this query to do an "AND" operation so that only the records having all the terms get matched?
By using a bool query:
POST recs/user/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "movies_liked",
"query": {
"bool": {
"must": [
{
"terms": {
"movies_liked.name": [
"Forrest Gump"
]
}
}
]
}
}
}
},
{
"nested": {
"path": "movies_liked",
"query": {
"bool": {
"must": [
{
"terms": {
"movies_liked.name": [
"Terminator"
]
}
}
]
}
}
}
},
{
"nested": {
"path": "movies_liked",
"query": {
"bool": {
"must": [
{
"terms": {
"movies_liked.name": [
"Rambo"
]
}
}
]
}
}
}
}
]
}
}
}
Note that bool wraps around several nested queries, not the other way around. It is important because the scope of a nested query is the nested document, because it basically a hidden separate object.
Hope that helps!

How to sort on elastic search query based on a matching property?

{
"users": [
{
"type": "admin",
"date": "2016-03-02T12:34:00+08:00",
"status": "2"
},
{
"type": "staff",
"date": "2016-03-02T12:34:00+08:00",
"status": "2"
}
],
"features": "1 2 3 4 5"
}
Lets say I have above Elasticsearch document structure. I want get all the documents on that index but those who has feature 3 should be at the top/begging of the search result. I went through Elasticsearch documentation but I couldn't figure out how to do it. Can anyone help me with this ?
Use this query:
{
"query": {
"bool": {
"should": [
{
"match": {
"features": "3"
}
},
{
"match_all": {}
}
]
}
}
}

Resources