access query value from function_score to compute new score - elasticsearch

I need to customize ES score. The score function I need to implement is:
score = len(document_term) - len(query_term)
For instance, one of my document in the ES index is :
{
"name": "foobar"
}
And the search query
{
"query": {
"function_score": {
"query": {
"match": {
"name": {
"query": "foo"
}
}
},
"functions": [
{
"script_score": {
"script": {
"source": "doc['name'].value.length() - ?LEN(query_tem)?"
}
}
}
],
"boost_mode": "replace"
}
}
}
The above search should provide a score of 6 - 3 = 3. But I didn't find a solution to get access the value of the query term.
Is it possible to access the value of the query term in a function_score context ?

There is no direct way to do this, however you can achieve that in the below way where you would need to add the query parameters in two different parts of the query.
Before that one important note, you cannot apply the doc['myfield'].value if the field is of type text, instead you would need to have its sibling field created as keyword and refer that in the script, which again I've mentioned below:
Mapping:
PUT myindex
{
"mappings" : {
"properties" : {
"myfield" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
Sample Document:
POST myquery/_doc/1
{
"myfield": "I've become comfortably numb"
}
Query:
POST <your_index_name>/_search
{
"query": {
"function_score": {
"query": {
"match": {
"myfield": "numb"
}
},
"functions": [
{
"script_score": {
"script": {
"source": "return doc['myfield.keyword'].value.length() - params.myquery.length()",
"params": {
"myquery": "numb" <---- Add the query string here as well
}
}
}
}
],
"boost_mode": "replace"
}
}
}
Response:
{
"took" : 558,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 24.0,
"hits" : [
{
"_index" : "myindex",
"_type" : "_doc",
"_id" : "1",
"_score" : 24.0,
"_source" : {
"myfield" : "I've become comfortably numb"
}
}
]
}
}
Hope this helps!

Related

ElasticSearch query fields in disabled object

I have an Elastic Search 6.8.7 cluster.
I have a column with this mapping:
"event_object": { "enabled": false, "type": "object" }
I want to search for records that match certain other criteria, and also have a particular value for a particular field field in this object.
So far, I have tried variations of doing a normal search for the indexed fields, and a filter script for the unindexed ones:
GET /my_index/_search
{
"query":{
"bool":{
"must":{
"query_string": {
"query": "foo:bar"
}
},
"filter": {
"script": {
"script": {
"source": "doc[\"event_object\"][\"state\"].value == \"R\""
}
}
}
}
},
"terminate_after":1000,
"from":0,
"size":1000
}
Which is a hodgepodge of testing myself forwards based on google searches. But I can't get things to even compile, let alone run and filter.
It is not possible to access the content of JSON objects that have enabled: false. From the official documentation:
Elasticsearch skips parsing of the contents of the field entirely. The JSON can still be retrieved from the _source field, but it is not searchable or stored in any other way
So even scripting will not help here.
However, there's one way to access this disabled data from scripting in a terms aggregation (using the include parameter and a top_hitssub-aggregation):
POST test/_search
{
"query": {
"match_all": {}
},
"aggs": {
"state": {
"terms": {
"script": "params._source.event_object.state",
"size": 100,
"include": "R"
},
"aggs": {
"hits": {
"top_hits": {
"size": 10
}
}
}
}
}
}
And you'd get a response like this one:
"aggregations" : {
"state" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "R",
"doc_count" : 1,
"hits" : {
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"event_object" : {
"state" : "R"
},
"test" : "hello"
}
}
]
}
}
}
]
}
}

Elasticsearch filter by multiple fields in an object which is in an array field

The goal is to filter products with multiple prices.
The data looks like this:
{
"name":"a",
"price":[
{
"membershipLevel":"Gold",
"price":"5"
},
{
"membershipLevel":"Silver",
"price":"50"
},
{
"membershipLevel":"Bronze",
"price":"100"
}
]
}
I would like to filter by membershipLevel and price. For example, if I am a silver member and query price range 0-10, the product should not appear, but if I am a gold member, the product "a" should appear. Is this kind of query supported by Elasticsearch?
You need to make use of nested datatype for price and make use of nested query for your use case.
Please see the below mapping, sample document, query and response:
Mapping:
PUT my_price_index
{
"mappings": {
"properties": {
"name":{
"type":"text"
},
"price":{
"type":"nested",
"properties": {
"membershipLevel":{
"type":"keyword"
},
"price":{
"type":"double"
}
}
}
}
}
}
Sample Document:
POST my_price_index/_doc/1
{
"name":"a",
"price":[
{
"membershipLevel":"Gold",
"price":"5"
},
{
"membershipLevel":"Silver",
"price":"50"
},
{
"membershipLevel":"Bronze",
"price":"100"
}
]
}
Query:
POST my_price_index/_search
{
"query": {
"nested": {
"path": "price",
"query": {
"bool": {
"must": [
{
"term": {
"price.membershipLevel": "Gold"
}
},
{
"range": {
"price.price": {
"gte": 0,
"lte": 10
}
}
}
]
}
},
"inner_hits": {} <---- Do note this.
}
}
}
The above query means, I want to return all the documents having price.price range from 0 to 10 and price.membershipLevel as Gold.
Notice that I've made use of inner_hits. The reason is despite being a nested document, ES as response would return the entire set of document instead of only the document specific to where the query clause is applicable.
In order to find the exact nested doc that has been matched, you would need to make use of inner_hits.
Below is how the response would return.
Response:
{
"took" : 128,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.9808291,
"hits" : [
{
"_index" : "my_price_index",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.9808291,
"_source" : {
"name" : "a",
"price" : [
{
"membershipLevel" : "Gold",
"price" : "5"
},
{
"membershipLevel" : "Silver",
"price" : "50"
},
{
"membershipLevel" : "Bronze",
"price" : "100"
}
]
},
"inner_hits" : {
"price" : {
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.9808291,
"hits" : [
{
"_index" : "my_price_index",
"_type" : "_doc",
"_id" : "1",
"_nested" : {
"field" : "price",
"offset" : 0
},
"_score" : 1.9808291,
"_source" : {
"membershipLevel" : "Gold",
"price" : "5"
}
}
]
}
}
}
}
]
}
}
Hope this helps!
Let me take show you how to do it, using the nested fields and query and filter context. I will take your example to show, you how to define index mapping, index sample documents, and search query.
It's important to note the include_in_parent param in Elasticsearch mapping, which allows us to use these nested fields without using the nested fields.
Please refer to Elasticsearch documentation about it.
If true, all fields in the nested object are also added to the parent
document as standard (flat) fields. Defaults to false.
Index Def
{
"mappings": {
"properties": {
"product": {
"type": "nested",
"include_in_parent": true
}
}
}
}
Index sample docs
{
"product": {
"price" : 5,
"membershipLevel" : "Gold"
}
}
{
"product": {
"price" : 50,
"membershipLevel" : "Silver"
}
}
{
"product": {
"price" : 100,
"membershipLevel" : "Bronze"
}
}
Search query to show Gold with price range 0-10
{
"query": {
"bool": {
"must": [
{
"match": {
"product.membershipLevel": "Gold"
}
}
],
"filter": [
{
"range": {
"product.price": {
"gte": 0,
"lte" : 10
}
}
}
]
}
}
}
Result
"hits": [
{
"_index": "so-60620921-nested",
"_type": "_doc",
"_id": "1",
"_score": 1.0296195,
"_source": {
"product": {
"price": 5,
"membershipLevel": "Gold"
}
}
}
]
Search query to exclude Silver, with same price range
{
"query": {
"bool": {
"must": [
{
"match": {
"product.membershipLevel": "Silver"
}
}
],
"filter": [
{
"range": {
"product.price": {
"gte": 0,
"lte" : 10
}
}
}
]
}
}
}
Above query doesn't return any result as there isn't any matching result.
P.S :- this SO answer might help you to understand nested fields and query on them in detail.
You have to use Nested fields and nested query to archive this: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html
Define you Price property with type "Nested" and then you will be able to filter by every property of nested object

Aggregation on .keyword to return only the keys that contain a specific string

New to aggregations in elasticsearch. Using 7.2. I am trying to write an aggregation on Tree.keyword to only return the count of documents that have a key that contains the word "Branch". I have tried sub aggregations, bucket_selector (which doesnt work for key strings) and scripts. Anyone have any ideas or suggestions on how to approach this?
Mapping:
{
"testindex" : {
"mappings" : {
"properties" : {
"Tree" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
}
}
}
}
}
Example Query that returns all the keys but what I need to do is limit to only return keys with "Branch" or better yet just the count of how many "Branch" keys there are:
GET testindex/_search
{
"aggs": {
"bucket": {
"terms": {
"field": "Tree.keyword"
}
}
}
}
Returns:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "testindex",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"Tree" : [
"Car:76",
"Branch:yellow",
"Car:one",
"Branch:blue"
]
}
}
]
},
"aggregations" : {
"bucket" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Car:76",
"doc_count" : 1
},
{
"key" : "Branch:yellow",
"doc_count" : 1
},
{
"key" : "Car:one",
"doc_count" : 1
},
{
"key" : "Branch:blue",
"doc_count" : 1
}
]
}
}
}
You have to add includes for limit result. Here's the code sample and hopefully this should help you.
GET testindex/_search
{
"_source": {
"includes": [
"Branch"
]
},
"aggs": {
"bucket": {
"terms": {
"field": "Tree.keyword"
}
}
}
}
It is possible to filter the values for which buckets will be created. This can be done using the include and exclude parameters which are based on regular expression strings or arrays of exact values. Additionally, include clauses that can filter using partition expressions.
For your case, it should be like this,
GET testindex/_search
{
"aggs": {
"bucket": {
"terms": {
"field": "Tree.keyword",
"include": "Branch:*"
}
}
}
}
Thanks for all the help! Unfortunately, none of those solutions worked for me. I ended up using a script to return all the branches and then setting everything else into a new key. Then used a bucket script to subtract 1 in Total_Buckets. Probably a better solution out there but hopefully it helps someone
GET testindex/_search
{
"aggs": {
"bucket": {
"cardinality": {
"field": "Tree.keyword",
"script": {
"lang": "painless",
"source": "if(_value.contains('Branches:')) { return _value} return 1;"
}
}
},
"Total_Branches": {
"bucket_script": {
"buckets_path": {
"my_var1": "bucket.value"
},
"script": "return params.my_var1-1"
}
}
}
}

Combining nested query get illegal_state_exception failed to find nested object under path

I'm creating a query on Elasticsearch, for find documents through all indices.
I need to combine should, must and nested query on Elasticsearch, i get the right result but i get an error inside the result.
This is the query I'm using
GET _all/_search
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{ "term": { "trimmed_final_url": "https://www.repubblica.it/t.../" } }
],
"must": [
{
"nested": {
"path": "entities",
"query": {
"bool": {
"must": [
{ "term": { "entities.id": "138511" } }
]
}
}
}
},
{
"term": {
"language": { "value": "it" }
}
}
]
}
}
And this is the result
{
"_shards" : {
"total" : 38,
"successful" : 14,
"skipped" : 0,
"failed" : 24,
"failures" : [
{
"shard" : 0,
"index" : ".kibana_1",
"node" : "7twsq85TSK60LkY0UiuWzA",
"reason" : {
"type" : "query_shard_exception",
"reason" : """
failed to create query: {
...
"index_uuid" : "HoHi97QFSaSCp09iSKY1DQ",
"index" : ".reporting-2019.06.02",
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "[nested] failed to find nested object under path [entities]"
}
}
},
...
"hits" : {
"total" : {
"value" : 50,
"relation" : "eq"
},
"max_score" : 16.90015,
"hits" : [
{
"_index" : "i_201906_v1",
"_type" : "_doc",
"_id" : "MugcbmsBAzi8a0oJt96Q",
"_score" : 16.90015,
"_source" : {
"language" : "it",
"entities" : [
{
"id" : 101580,
},
{
"id" : 156822,
},
...
I didn't write some fields because the code is too long
I am new to StackOverFlow (made this account to answer this question :D) so if this answer is out of line bear with me. I have been dabbling in nested fields in Elasticsearch recently so I have some ideas as to how this error could be appearing.
Have you defined a mapping for your document type? I don't believe Elasticsearch will recognize the field as nested if you do not tell it to do so in the mapping:
PUT INDEX_NAME
{
"mappings": {
"DOC_TYPE": {
"properties": {
"entities": {"type": "nested"}
}
}
}
}
You may have to specify this mapping for each index and document type. Not sure if there is a way to do that all with one request.
I also noticed you have a "should" clause with minimum matches set to 1. I believe this is exactly the same as a "must" clause so I am not sure what purpose this achieves (correct me if I'm wrong). If your mapping is specified, the query should look something like this:
GET /_all/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "entities",
"query": {
"term": {
"entities.id": {
"value": "138511"
}
}
}
}
},
{
"term": {
"language": {
"value": "it"
}
}
},
{
"term": {
"trimmed_final_url": {
"value": "https://www.repubblica.it/t.../"
}
}
}
]
}
}
}

Elastic search: exact match query on string array

Given this document:
{"name": "Perfect Sunny-Side Up Eggs","ingredientList": ["canola oil","eggs"]}
How can I build a query in elastic search to return exact matches on a string array given query term "oil eggs", so far this it what I have, but it returns other irrelevant documents:
POST /recipes/recipe/_search
{
"query": {
"match": {
"ingredientList": {
"query": [
"oil",
"eggs"
],
"operator": "and"
}
}
}
}
for instance, this document is returned but it doesn't contain "oil". Results should only contain "oil" and "eggs":
{"name": "Quick Baked French Toast","ingredientList": ["butter","cinnamon raisin bread","eggs"]}
Your query will look like this:
{
"query": {
"bool": {
"must": [
{
"term": {
"ingredientList": "oil"
}
},
{
"term": {
"ingredientList": "eggs"
}
}
]
}
}
}
Gives me the results:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "ingredients",
"_type" : "recipe",
"_id" : "AVeprXFrNutW6yNguPqp",
"_score" : 1.0,
"_source" : {
"name" : "Perfect Sunny-Side Up Eggs",
"ingredientList" : [ "canola oil", "eggs" ]
}
} ]
}
}
Elastic dont have API to exact match array. But same can be achieved using two methods:
Using multiple must blocks (not preferred)
Using terms set query and script
"query": {
"bool": {
"must": [
{
"terms_set": {
"ingredientList": {
"terms": ingredients,
"minimum_should_match_script": {
"source": "Math.min(params.num_terms, {})".format(len(ingredients))
}
}
}
},
{
"script": {
"script": {
"inline": "doc['ingredientList'].length == params.list_length",
"lang": "painless",
"params": {
"list_length": len(ingredients)
}
}
}
}
]
}
}

Resources