Elasticsearch script query started to fail suddenly - elasticsearch

We have had this query for a long time
{
"size": 50,
"query": {
"match_all": {}
},
"version": false,
"seq_no_primary_term": false,
"sort": [
{
"_script": {
"script": {
"source": "(doc[params.f].size() != 0) ? (params.m['' + doc[params.f].value] ?: params.o): params.o",
"lang": "painless",
"params": {
"f": "scoreSerial",
"m": {
"0": "UNDEFINED",
"1": "FRUSTRATED",
"2": "TOLERATED",
"3": "SATISFIED"
},
"o": "ZZZZZ"
}
},
"type": "string",
"order": "asc"
}
}
]
}
This suddenly stopped working with the error
{
"error" : {
"root_cause" : [
{
"type" : "bootstrap_method_error",
"reason" : "bootstrap_method_error: CallSite bootstrap method initialization exception"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "visit-global-standard-w2022.31",
"node" : "olbLlZh7RYuwaf5S-B9v8g",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"(doc[params.f].size() != 0) ? (params.m['' + doc[params.f].value] ?: params.o): params.o",
" ^---- HERE"
],
"script" : "(doc[params.f].size() != 0) ? (params.m['' + doc[params.f].value] ?: params.o): params.o",
"lang" : "painless",
"position" : {
"offset" : 58,
"start" : 0,
"end" : 88
},
"caused_by" : {
"type" : "bootstrap_method_error",
"reason" : "bootstrap_method_error: CallSite bootstrap method initialization exception"
}
}
}
],
"caused_by" : {
"type" : "bootstrap_method_error",
"reason" : "bootstrap_method_error: CallSite bootstrap method initialization exception"
}
},
"status" : 400
}
The mapping of scoreSerial is
"scoreSerial" : {
"type" : "long"
}
It was working for more than 3 years. We are now on the 7.10 (for more than 6 months, when the above script was working) ES version. But suddenly it started to fail.
The strange thing is that it is not happening in all environments. Is it some java version related or something else? We can not also reproduce this locally.

It might have to do with type coercion, I've seen similar problems in the past (although not with painless but groovy), so simply replacing
'' + doc[params.f].value
by
doc[params.f].value.toString()
might do the trick
(i.e. explicitly getting the string version of the doc value instead of relying on type coercion). What you did should work according to the docs, though.

Related

cannot convert MethodHandle(Dates)JodaCompatibleZonedDateTime to (Object)double

I am trying to add conditions if field exist, then sort according to it otherwise use another field. Since one of either will exist.
Here is my query:
GET /my_index/_search
{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"type":"number",
"script": "if(doc['contextDates.event.date'].value != 0){ return doc['contextDates.event.date'].value} else { return doc['contextDates.start.date'].value}",
"order": "asc"
}
}
}
When I execute this query, I get following error:
"failed_shards" : [
{
"shard" : 0,
"index" : "my_inedx",
"node" : "UxKwS8SIR-uIbzo5_0IbcQ",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"return doc['contextDates.event.date'].value} else { ",
" ^---- HERE"
],
"script" : "if(doc['contextDates.event.date'].value != 0){ return doc['contextDates.event.date'].value} else { return doc['contextDates.start.date'].value}",
"lang" : "painless",
"position" : {
"offset" : 84,
"start" : 47,
"end" : 99
},
"caused_by" : {
"type" : "wrong_method_type_exception",
"reason" : "cannot convert MethodHandle(Dates)JodaCompatibleZonedDateTime to (Object)double"
}
}
}
]
I have tried Double.parseDouble method as well but it doesn't work. This is what I have inside document for contextDates
"contextDates" : {
"event" : {
"date" : "2020-06-26T00:00:00.000Z",
"resolution" : "day",
"score" : 0,
"type" : "event"
}
}
The doc value you're getting is of type JodaCompatibleZonedDateTime which you're trying to compare to a double value, so you need to modify your script like this
if(doc['contextDates.event.date'].value.getMillis() != 0){ return doc['contextDates.event.date'].value.getMillis()} else { return doc['contextDates.start.date'].value.getMillis()}

Elasticsearch - Missing Field Value For Nested Field - Function Score Query | v7.10.2

I have already posted this on the ES group but I got no response and so posted it on SO. Link https://discuss.elastic.co/t/missing-field-value-for-nested-field-function-score-query-v7-10-2/291365
I have been trying for a long time now but the nested field value is always saying missing field value while calculating the score.
Mapping:
{
"doctor_idx" : {
"mappings" : {
"properties" : {
"_class" : {
"type" : "keyword",
"index" : false,
"doc_values" : false
},
"service" : {
"type" : "nested",
"properties" : {
"_class" : {
"type" : "keyword",
"index" : false,
"doc_values" : false
},
"serviceTypeEarliestAvailability" : {
"type" : "nested",
"properties" : {
"_class" : {
"type" : "keyword",
"index" : false,
"doc_values" : false
},
"earliestAvailableDateTimeByType" : {
"type" : "date",
"format" : "date_hour_minute_second"
},
"serviceType" : {
"type" : "text"
},
"servicesMedium" : {
"type" : "keyword"
}
}
}
}
}
}
}
}
}
For simplicity, I have kept just one record. A glimpse of the record
"serviceTypeEarliestAvailability" : [
{
"serviceType" : "Service Type",
"earliestAvailableDateTimeByType" : "2021-12-09T19:39:16",
"servicesMedium" : [
"MED1",
"MED2",
"MED3",
"MED4"
]
}
],
The following query gives: "A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!"
I tried using field_value_factor instead of script_score but it's the same problem and it complains about the missing field value.
Query 1: with script_score
GET /doctor_idx/_search
{
"explain": true,
"query": {
"nested": {
"path": "service",
"query": {
"nested": {
"score_mode": "max",
"path": "service.serviceTypeEarliestAvailability",
"query": {
"function_score": {
"query": {
"match_all": {
"boost": 1
}
},
"functions": [
{
"filter": {
"match": {
"service.serviceTypeEarliestAvailability.serviceType": "type" // no complaints about this
}
},
"weight": 10
},
{
"script_score": {
"script": {
"source": "(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())"
}
}
}
]
}
}
}
}
}
}
}
Query 2 : With field_value_factor
GET /doctor_idx/_search
{
"explain": true,
"query": {
"nested": {
"path": "service",
"query": {
"nested": {
"score_mode": "max",
"path": "service.serviceTypeEarliestAvailability",
"query": {
"function_score": {
"query": {
"match_all": {
"boost": 1
}
},
"functions": [
{
"filter": {
"match": {
"service.serviceTypeEarliestAvailability.serviceType": "type"
}
},
"weight": 10
},
{
"field_value_factor": {
"field": "service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType"
}
}
]
}
}
}
}
}
}
}
Error From ES for Query 1:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.get(ScriptDocValues.java:160)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.getValue(ScriptDocValues.java:154)",
"(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
" ^---- HERE"
],
"script" : "(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
"lang" : "painless",
"position" : {
"offset" : 79,
"start" : 0,
"end" : 98
}
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "doctor_idx",
"node" : "mvh5k24dQPqM-d7JVeNomQ",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.get(ScriptDocValues.java:160)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.getValue(ScriptDocValues.java:154)",
"(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
" ^---- HERE"
],
"script" : "(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
"lang" : "painless",
"position" : {
"offset" : 79,
"start" : 0,
"end" : 98
},
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
}
}
}
]
},
"status" : 400
}
Error From ES for Query 2:
{
"error" : {
"root_cause" : [
{
"type" : "exception",
"reason" : "Missing value for field [service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType]"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "doctor_idx",
"node" : "mvh5k24dQPqM-d7JVeNomQ",
"reason" : {
"type" : "exception",
"reason" : "Missing value for field [service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType]"
}
}
]
},
"status" : 500
}
The only record I have in the ES
{
"took" : 32,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "doctor_idx",
"_type" : "_doc",
"_id" : "xM20oH0Bmp1zsT0w8pQe",
"_score" : 1.0,
"_source" : {
"_class" : "com.insig.clinic_apps.services_server.booking.dao.DoctorSchema",
"id" : "xM20oH0Bmp1zsT0w8pQe",
"metadata" : {
"doctorId" : "xhnz2lGvXxelWyVekz82c2v6Srb2",
"fullName" : "Dave Insig Email Test",
"credentials" : [
"MD"
],
"languagesSpoken" : [
"EN",
"NOT_DEFINED"
],
"specialitiesServed" : [ ],
"city" : "VAUGHAN",
"provincesPermitted" : [ ],
"province" : "ON",
"country" : "CANADA"
},
"service" : {
"servicesMedium" : [
"IN_CLINIC",
"VIDEO",
"MESSAGING",
"PHONE"
],
"servicesTypeDuration" : [
{
"serviceType" : "Acne Symptoms",
"duration" : 5,
"servicesMedium" : [
"IN_CLINIC",
"MESSAGING",
"PHONE",
"VIDEO"
]
}
],
"serviceTypeEarliestAvailability" : [
{
"serviceType" : "Acne Symptoms",
"earliestAvailableDateTimeByType" : "2021-12-09T19:39:16",
"servicesMedium" : [
"IN_CLINIC",
"MESSAGING",
"PHONE",
"VIDEO"
]
}
],
"bufferTimeForNextAvailability" : 0
},
"earliestAvailableDateTime" : "2021-12-09T19:39:16",
"patientRating" : 4.384481,
"onTimeRating" : 3.171053
}
}
]
}
}
/!\ Partial solution /!\
Query 1
When I was trying to execute your query I got this error:
{
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "dynamic method [org.elasticsearch.script.JodaCompatibleZonedDateTime, toEpochMilli/0] not found"
}
}
So I did a slight change to your query .toInstant().toEpochMilli()
GET /so_custom_score/_search
{
"explain": true,
"query": {
"nested": {
"path": "service",
"query": {
"nested": {
"score_mode": "max",
"path": "service.serviceTypeEarliestAvailability",
"query": {
"function_score": {
"query": {
"match_all": {
"boost": 1
}
},
"functions": [
{
"filter": {
"match": {
"service.serviceTypeEarliestAvailability.serviceType": "type"
}
},
"weight": 10
},
{
"script_score": {
"script": {
"source": """
def availability = doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value;
return availability.toInstant().toEpochMilli();
"""
}
}
}
]
}
}
}
}
}
}
}
Well, wasted a lot of time here and figured that there was no problem with the queries. Elasticsearch v7.10.2 has an issue when I try to get the explanation of the query.
It works perfectly fine without the explain parameter.
See the at org.elasticsearch.common.lucene.search.function.FieldValueFactorFunction$1.explainScore(FieldValueFactorFunction.java:103) ~[elasticsearch-7.10.2.jar:7.10.2]
PS: One of the comments in the OG question mentioned that the latest Elasticsearch v7.15 works fine even with explain parameter.

Cannot apply [*] operation to types [org.elasticsearch.index.fielddata.ScriptDocValues.Doubles] and [java.lang.Double]

Here is the mappeing of the attribute for the index-
"hoi" : {
"type" : "float"
}
This is the query that I want to execute. I want to change my attribute hoi with dynamic value and sort the doc on top of that, don't want to use runtime mapping as I have lower version of ES-
GET test/_search
{
"sort" : {
"_script" : {
"type" : "number",
"script" : {
"lang": "painless",
"source": "doc['hoi'] * params.factor",
"params" : {
"factor" : 1.1
}
},
"order" : "asc"
}
}
}
Error-
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"doc['hoi'] * params.factor",
" ^---- HERE"
],
"script" : "doc['hoi'] * params.factor",
"lang" : "painless",
"position" : {
"offset" : 40,
"start" : 0,
"end" : 47
}
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "test",
"node" : "5sfrbDiLQDOg82_sneaU1g",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"doc['hoi'] * params.factor",
" ^---- HERE"
],
"script" : "doc['hoi'] * params.factor",
"lang" : "painless",
"position" : {
"offset" : 40,
"start" : 0,
"end" : 47
},
"caused_by" : {
"type" : "class_cast_exception",
"reason" : "Cannot apply [*] operation to types [org.elasticsearch.index.fielddata.ScriptDocValues.Doubles] and [java.lang.Double]."
}
}
}
]
},
"status" : 400
}
How to make it work?
doc['hoi'] itself is of type org.elasticsearch.index.fielddata.ScriptDocValues.Doubles and params.factor is of type java.lang.Double.
So the error you get means that some document has multiple values in your hoi field. So you need to reference it with doc['hoi'].value * params.factor instead, which will take the first hoi value from the array to compute the sort.
You can also check how many values there are with doc['hoi'].size().
UPDATE
If some docs don't have any values for hoi then you can adapt the script like this:
doc['hoi'].size() > 0 ? doc['hoi'].value * params.factor : 0

Elasticsearch Query DSL: Length of field, if field exists

Say I have a field, data.url. Some our logs contain this field, some do not. I want to return only results where data.url is more than, say, 50 characters long. Really I just need a list of URLs.
I'm trying:
GET _search
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['data.url'].value.length() > 50",
"lang": "painless"
}
}
}
}
}
}
But get mixed errors:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"doc['data.url'].value.length() > 50",
" ^---- HERE"
],
"script" : "doc['data.url'].value.length() > 50",
"lang" : "painless",
"position" : {
"offset" : 4,
"start" : 0,
"end" : 35
}
},
or
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:496)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:503)",
"doc['data.url'].value.length() > 50",
" ^---- HERE"
],
"script" : "doc['data.url'].value.length() > 50",
"lang" : "painless",
"position" : {
"offset" : 15,
"start" : 0,
"end" : 35
}
With
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "No field found for [data.url] in mapping with types []"
}
and sometimes
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
}
This field definitely exists; I can see it in the logs, search in the search field, and using term works:
GET _search
{
"query": {
"bool": {
"filter": {
"term": {
"data.url": "www.google.com"
}
}
}
}
}
What am I missing?
I'm using Elasticsearch 7.8.
Since you are using version 7.*, you need to use this below script query
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['data.url.keyword'].length > 50",
"lang": "painless"
}
}
}
}
}
}
If data.url field is of keyword type, then ignore the ".keyword" at the end of the field

Add date field and boolean with ? in name to existing Elasticsearch documents

We need to add two new fields to an existing ElasticSearch (7.9 oss) instance.
Field 1: Date Field
We want to add an optional date field. It shouldn't have a value upon creation.
How to do this with update_by_query?
Tried this:
POST orders/_update_by_query
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.new_d3_field",
"lang": "painless",
"type": "date",
"format": "yyyy/MM/dd HH:mm:ss"
}
}
Field 2: Boolean field with ? in name
We want to keep the ? so that it matches the other fields that we already have in ES.
Also worth noting that even removing the ? and doing the below the field doesn't appear to be a boolean.
Tried this:
POST orders/_update_by_query
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.new_b_field? = false",
"lang": "painless"
}
}
Which gave the error:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"ctx._source.new_b_field? = false",
" ^---- HERE"
],
"script" : "ctx._source.new_b_field? = false",
"lang" : "painless",
"position" : {
"offset" : 25,
"start" : 0,
"end" : 32
}
}
],
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"ctx._source.new_b_field? = false",
" ^---- HERE"
],
"script" : "ctx._source.new_b_field? = false",
"lang" : "painless",
"position" : {
"offset" : 25,
"start" : 0,
"end" : 32
},
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "invalid sequence of tokens near ['='].",
"caused_by" : {
"type" : "no_viable_alt_exception",
"reason" : null
}
}
},
"status" : 400
}
Also tried:
POST orders/_update_by_query?new_b_field%3F=false
Which gave:
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "request [/orders/_update_by_query] contains unrecognized parameter: [new_b_field?]"
}
],
"type" : "illegal_argument_exception",
"reason" : "request [/orders/_update_by_query] contains unrecognized parameter: [new_b_field?]"
},
"status" : 400
}
If you want to add two new fields to an existing ElasticSearch index that don't have value upon creation you should update its mapping using Put mapping API
PUT /orders/_mapping
{
"properties": {
"new_d3_field": {
"type": "date",
"format": "yyyy/MM/dd HH:mm:ss"
},
"new_b_field?": {
"type": "boolean"
}
}
}
If you still want to use _update_by_query you should set an initial value, then the field will be added.
POST orders/_update_by_query?wait_for_completion=false&conflicts=proceed
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.new_d3_field=params.date;ctx._source.new_b_field = params.val",
"lang": "painless",
"params": {
"date": "1980/01/01",
"val": false
}
}
}
Update By Query API is used to update documents so I guess you can't add a field to your schema without updating at list one doc. what you can do is to set a dummy doc and update only this certain doc. Something like that:
POST orders/_update_by_query
{
"query": {
"match": {
"my-field":"my-value"
}
},
"script": {
"source": "ctx._source.new_d3_field=params.date;ctx._source.new_b_field = params.val",
"lang": "painless",
"params": {
"date": "1980/01/01",
"val": false
}
}
}

Resources