I'm trying to have a dynamic param based on a value in the document.
I tried so far this here
body: {
"script_fields": {
"potentialIncome": {
"script": {
"lang": "painless",
"source": "doc.rentPrice.value - params['doc.buyingPrice.value']",
"params": {
120000: 1200,
150000: 1500
}
}
}
}
}
This throws me following error:
type: 'script_exception',
reason: 'runtime error',
script_stack:
[ 'doc.rentPrice.value - params[\'doc.buyingPrice.value\']',
' ^---- HERE' ],
script: 'doc.rentPrice.value - params[\'doc.buyingPrice.value\']',
lang: 'painless'
I would like to have params dynamic in a way that the doc value buyingPrice decides which value to deduct.
Using ElasticSearch 7.2
A complicated and bad way is to use following script
if(doc['buyingPrice'].value==120000){return doc['rentPrice'].value-params['120000']}
else if(doc['buyingPrice'].value==150000){return doc['rentPrice'].value-params['150000']}
The Es object:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 2,
"relation": "eq"
},
"max_score": 1.0,
"hits": [{
"_index": "immo",
"_type": "objects",
"_id": "1",
"_score": 1.0,
"_source": {
"buyingPrice": 120000,
"rentPrice": 500
}
}, {
"_index": "immo",
"_type": "objects",
"_id": "2",
"_score": 1.0,
"_source": {
"buyingPrice": 150000,
"rentPrice": 500
}
}]
}
}
You need to try without the single quotes.
"source": "return (params[String.valueOf(doc.buyingPrice.value)] != null) ? doc.rentPrice.value - params[String.valueOf(doc.buyingPrice.value)] : 0",
^ ^
| |
Related
I have two nearly identical documents, one of which has the fields CONSTRUCTION: 1 and EDUCATION: 0.1, the other with CONSTRUCTION: 0.1 and EDUCATION: 1. I want to be able to sort results by the value of either the CONSTRUCTION or EDUCATION field
GET /objects/_search
{
"query": {
"function_score": {
"query": {
"match": {
"name": {
"query": "Monkeys"
}
}
},
"field_value_factor": {
"field" : "CONSTRUCTION",
"missing": 1
}
}
},
"_source": ["name", "CONSTRUCTION", "EDUCATION"]
}
Returns the incorrect results:
{
"took": 8,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1.7622693,
"hits": [
{
"_index": "objects__feed_id_key_pages__date_2019-12-10__timestamp_1575988952__batch_id_3gpnz7fc__",
"_type": "_doc",
"_id": "dit:greatDomesticUi:KeyPages:12",
"_score": 1.7622693,
"_source": {
"CONSTRUCTION": 0.1,
"name": "Space Monkeys - education",
"EDUCATION": 1
}
},
{
"_index": "objects__feed_id_key_pages__date_2019-12-10__timestamp_1575988952__batch_id_3gpnz7fc__",
"_type": "_doc",
"_id": "dit:greatDomesticUi:KeyPages:11",
"_score": 1.0226655,
"_source": {
"CONSTRUCTION": 1,
"name": "Space Monkeys - construction",
"EDUCATION": 0.1
}
}
]
}
}
This only always returns the same results. Indeed if you misspell the field_value_factor field, you get the same score "field_value_factor": { "field" : "WHATEVER",... }. This suggests the field simply isn't being read.
Dynamic mapping was turned off. The EDUCATION and CONSTRUCTION fields were not mapped. Mystery solved!
How do i get an exact sum aggregation in elasticsearch? Fore reference i am currently using elasticsearch 5.6 and the my index mapping looks like this:
{
"my-index":{
"mappings":{
"my-type":{
"properties":{
"id":{
"type":"keyword"
},
"fieldA":{
"type":"double"
},
"fieldB":{
"type":"double"
},
"fieldC":{
"type":"double"
},
"version":{
"type":"long"
}
}
}
}
}
}
The search query generated (using java client) is:
{
/// ... some filters here
"aggregations" : {
"fieldA" : {
"sum" : {
"field" : "fieldA"
}
},
"fieldB" : {
"sum" : {
"field" : "fieldB"
}
},
"fieldC" : {
"sum" : {
"field" : "fieldC"
}
}
}
}
However my result hits generate the following:
{
"took": 10,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 5,
"max_score": 3.8466966,
"hits": [
{
"_index": "my-index",
"_type": "my-type",
"_id": "25a203b63e264fd2be13db006684b06d",
"_score": 3.8466966,
"_source": {
"fieldC": 108,
"fieldA": 108,
"fieldB": 0
}
},
{
"_index": "my-index",
"_type": "my-type",
"_id": "25a203b63e264fd2be13db006684b06d",
"_score": 3.8466966,
"_source": {
"fieldC": -36,
"fieldA": 108,
"fieldB": 144
}
},
{
"_index": "my-index",
"_type": "my-type",
"_id": "25a203b63e264fd2be13db006684b06d",
"_score": 3.8466966,
"_source": {
"fieldC": -7.2,
"fieldA": 1.8,
"fieldB": 9
}
},
{
"_index": "my-index",
"_type": "my-type",
"_id": "25a203b63e264fd2be13db006684b06d",
"_score": 3.8466966,
"_source": {
"fieldC": 14.85,
"fieldA": 18.9,
"fieldB": 4.05
}
},
{
"_index": "my-index",
"_type": "my-type",
"_id": "25a203b63e264fd2be13db006684b06d",
"_score": 3.8466966,
"_source": {
"fieldC": 36,
"fieldA": 36,
"fieldB": 0
}
}
]
},
"aggregations": {
"fieldA": {
"value": 272.70000000000005
},
"fieldB": {
"value": 157.05
},
"fieldC": {
"value": 115.64999999999999
}
}
}
why do i get:
115.64999999999999 instead of 115.65 in fieldC
272.70000000000005 instead of 272.7 in fieldA
should i use float instead of double? or is there a way i can change the query without using painless script and using java's BigDecimal with specified precision and rounding mode?
It has to do with float number precision in JavaScript (similar to what can be seen here and explained here).
Here are two ways to check this:
A. If you node.js installed, just type node at the prompt and then enter the sum of all fieldA values:
$ node
108 - 36 - 7.2 + 14.85 + 36
115.64999999999999 <--- this is the answer
B. Open the Developer tools of your browser and pick the Console view. Then type the same sum as above:
> 108-36-7.2+14.85+36
< 115.64999999999999
As you can see, both results are consistent with what you're seeing in your ES response.
One way to circumvent this is to store your numbers either as normal integers (i.e. 1485 instead of 14.85, 3600 instead of 36, etc) or as scaled_float with a scaling factor of 100 (or bigger depending on the precision you need)
I'm trying to use elasticsearch MLT (More Like This) query.
Only one doc in store:
{
"_index": "monitors",
"_type": "monitor",
"_id": "AVTnvJ8SancUpEdFLMiq",
"_score": 1,
"_source": {
"ProcessGroup": "test",
"ProcessName": "test",
"OpName": "test",
"Domain": "test",
"LogLevel": "Info",
"StartDateTime": "2016-05-04 04:46:47",
"EndDateTime": "2016-05-04 04:47:47",
"MessageDateTime": "2016-05-04 04:46:47",
"ApplicationCode": "test",
"Status": "10",
}
}
Query:
POST /_search
{
"query": {
"more_like_this" : {
"fields" : ["ProcessName"],
"like" : "test",
"min_term_freq" : 1,
"max_query_terms" : 12
}
}
}
ProcessName is a not analyzed field.
I was expected to get this document as a response, but instead i got nada:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
Why is that ?
Another question:
Suppose I have search engines docs, and I search for "stph". I expect to get "Stephan Curry" suggestion because it's commonly searched. Fuzzy search doesn't fit because distance is greater than 2, so does using MLT query is a good option for this scenario ?
I was trying to come up with an elegant answer to this question and ran into an unexpected problem. The basic idea is to update a document based on its current timestamp. Seems straightforward enough, but I seem to be missing something. At the bottom of the Update API page, the ES docs say:
It also allows to update the ttl of a document using ctx._ttl and timestamp using ctx._timestamp. Note that if the timestamp is not updated and not extracted from the _source it will be set to the update date.
The ES documentation is often enigmatic at best, especially when it comes to scripting, but I took this to mean that I could use the _timestamp field in an update script.
So I set up a simple index with a timestamp:
PUT /test_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"doc": {
"_timestamp": {
"enabled": true,
"store": true,
"path": "doc_date",
"format" : "YYYY-MM-dd"
},
"properties": {
"doc_date": {
"type": "date",
"format" : "YYYY-MM-dd"
},
"doc_text": {
"type": "string"
}
}
}
}
}
and added some docs:
POST /test_index/_bulk
{"index":{"_index":"test_index","_type":"doc","_id":1}}
{"doc_text":"doc1", "doc_date":"2015-2-5"}
{"index":{"_index":"test_index","_type":"doc","_id":2}}
{"doc_text":"doc2", "doc_date":"2015-2-10"}
{"index":{"_index":"test_index","_type":"doc","_id":3}}
{"doc_text":"doc3", "doc_date":"2015-2-15"}
If I query for the first doc, I get back what I expect:
POST /test_index/_search
{
"query": {
"match": {
"doc_text": "doc1"
}
},
"fields": [
"_timestamp",
"_source"
]
}
...
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.4054651,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "1",
"_score": 1.4054651,
"_source": {
"doc_text": "doc1",
"doc_date": "2015-2-5"
},
"fields": {
"_timestamp": 1423094400000
}
}
]
}
}
So far so good. Now I want to conditionally update the first doc, based on its timestamp. First I tried this, and got an error:
POST /test_index/doc/1/_update
{
"script": "if(ctx._timestamp < new_ts){ctx._source.doc_date=new_date;ctx._source.doc_text=new_text}",
"params": {
"new_ts": 1423526400000,
"new_date": "2015-2-10",
"new_text": "doc1-updated"
}
}
...
{
"error": "ElasticsearchIllegalArgumentException[failed to execute script]; nested: PropertyAccessException[[Error: could not access: _timestamp; in class: java.util.HashMap]\n[Near : {... if(ctx._timestamp < new_ts){ctx._ ....}]\n ^\n[Line: 1, Column: 4]]; ",
"status": 400
}
Then I tried this:
POST /test_index/doc/1/_update
{
"script": "if(ctx[\"_timestamp\"] < new_ts){ctx._source.doc_date=new_date;ctx._source.doc_text=new_text}",
"params": {
"new_ts": 1423526400000,
"new_date": "2015-2-10",
"new_text": "doc1-updated"
}
}
...
{
"_index": "test_index",
"_type": "doc",
"_id": "1",
"_version": 2
}
I didn't get an error, but the update didn't happen:
POST /test_index/_search
{
"query": {
"match": {
"doc_text": "doc1"
}
},
"fields": [
"_timestamp",
"_source"
]
}
...
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.287682,
"hits": [
{
"_index": "test_index",
"_type": "doc",
"_id": "1",
"_score": 1.287682,
"_source": {
"doc_text": "doc1",
"doc_date": "2015-2-5"
},
"fields": {
"_timestamp": 1423094400000
}
}
]
}
}
Just out of curiosity, I inverted the conditional:
POST /test_index/doc/1/_update
{
"script": "if(ctx[\"_timestamp\"] > new_ts){ctx._source.doc_date=new_date;ctx._source.doc_text=new_text}",
"params": {
"new_ts": 1423526400000,
"new_date": "2015-2-10",
"new_text": "doc1-updated"
}
}
with the same result: no update.
Okay, so as a sanity check I tried to set the timestamp, and got an error:
POST /test_index/doc/1/_update
{
"script": "ctx._source.doc_date=new_date;ctx._source.doc_text=new_text;ctx._timestamp=new_ts",
"params": {
"new_ts": 1423526400000,
"new_date": "2015-2-10",
"new_text": "doc1-updated"
}
}
...
{
"error": "ClassCastException[java.lang.Long cannot be cast to java.lang.String]",
"status": 500
}
I also tried it with "ctx[\"_timestamp\"]=new_ts;", and got the same error.
So it seems that the _timestamp field is not available to the script, even though the documentation says it is. What am I doing wrong?
I also tried updating without the conditional or updating the timestamp, and it worked as expected.
I used Elasticsearch version 1.3.4 (with dynamic scripting enabled, obviously), running on an Ubuntu 12 VM.
Here is the code I used to set this up:
http://sense.qbox.io/gist/ca2b3c6b84572e5f87d57d22f8c38252fa4ee216
Let's say I have created a document like this:
PUT idx/type/1
{
"the_field": [1,2,3]
}
I can retrieve my document using GET /idx/type/1:
{
"_index": "idx",
"_type": "type",
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"the_field": [
1,
2,
3
]
}
}
Now, I want to check if the field "the_field" contains the value 2.
I know I can use a term clause, but I need to check this using a filter script, so I tried:
POST /idx/typ/_search
{
"query": {
"match_all": {}
},
"filter": {
"script": {
"script": "doc['the_field'].values.contains(2)"
}
}
}
and get no results:
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
To test if my mevl script syntax is right, I tried doing this:
POST /idx/type/_search
{
"query": {
"match_all": {}
},
"filter": {
"script": {
"script": "[1,2,3].contains(3)"
}
}
}
and get the right results:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "idx",
"_type": "type",
"_id": "1",
"_score": 1,
"_source": {
"the_field": [
1,
2,
3
]
}
}
]
}
}
What am I doing wrong?
I think doc['the_field'].values should return [1,2,3], is not it? If so, my code should work.
Does anybody can help me?
Thank you!
UPDATE
When I replace all the [1, 2, 3] in my code with ["a"," b", "c"], it works. Any idea?
It is working with "a", "b", "c" because the_field is being stored in Elasticsearch by default as a string and not an integer. You can validate by checking the mapping with:
$ curl -XGET 'http://localhost:9200/idx/type/_mapping'
The following should set the appropriate field type:
$ curl -XPUT 'http://localhost:9200/idx/type/_mapping' -d '
{
"type" : {
"properties" : {
"the_field" : {"type" : "integer" }
}
}
}
Update the mapping, re-index your data and see if that works. Please see the PUT Mapping API for additional guidance if needed.