Say I have a field, data.url. Some our logs contain this field, some do not. I want to return only results where data.url is more than, say, 50 characters long. Really I just need a list of URLs.
I'm trying:
GET _search
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['data.url'].value.length() > 50",
"lang": "painless"
}
}
}
}
}
}
But get mixed errors:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"doc['data.url'].value.length() > 50",
" ^---- HERE"
],
"script" : "doc['data.url'].value.length() > 50",
"lang" : "painless",
"position" : {
"offset" : 4,
"start" : 0,
"end" : 35
}
},
or
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:496)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:503)",
"doc['data.url'].value.length() > 50",
" ^---- HERE"
],
"script" : "doc['data.url'].value.length() > 50",
"lang" : "painless",
"position" : {
"offset" : 15,
"start" : 0,
"end" : 35
}
With
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "No field found for [data.url] in mapping with types []"
}
and sometimes
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
}
This field definitely exists; I can see it in the logs, search in the search field, and using term works:
GET _search
{
"query": {
"bool": {
"filter": {
"term": {
"data.url": "www.google.com"
}
}
}
}
}
What am I missing?
I'm using Elasticsearch 7.8.
Since you are using version 7.*, you need to use this below script query
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['data.url.keyword'].length > 50",
"lang": "painless"
}
}
}
}
}
}
If data.url field is of keyword type, then ignore the ".keyword" at the end of the field
Related
Unable to do Elastic Search update by query script for a string field. This is my query
curl -X POST "http://localhost:9200/z5-purchase-orders/_update_by_query?pretty" -H 'Content-Type: application/json' -d'
{
"script": {
"source": "ctx._source.facility_name = VCU",
"lang": "painless"
},
"query": {
"bool" : {
"must" : [
{ "match" : {"facility_id" : 0} },
{ "match" : {"customer_id" : 1002} }
]
}
}
}
'
I did the same query for integer field. It worked fine, with string its causing error
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"... ._source.facility_name = VCU",
" ^---- HERE"
],
"script" : "ctx._source.facility_name = VCU",
"lang" : "painless"
}
],
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"... ._source.facility_name = VCU",
" ^---- HERE"
],
"script" : "ctx._source.facility_name = VCU",
"lang" : "painless",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Variable [VCU] is not defined."
}
},
"status" : 400
}
Please help with this string update by query. I use elastic search version 7.
To fix that error, you simply need to stringify the value VCU
"ctx._source.facility_name = 'VCU'"
If you still get an error after that, it's going to be a different one, feel free to share it.
After escaping the characters it worked fine.
"script": {
"source": "ctx._source.facility_name = \"VCU\"",
"lang": "painless"
},
I am trying to add conditions if field exist, then sort according to it otherwise use another field. Since one of either will exist.
Here is my query:
GET /my_index/_search
{
"query": {
"match_all": {}
},
"sort": {
"_script": {
"type":"number",
"script": "if(doc['contextDates.event.date'].value != 0){ return doc['contextDates.event.date'].value} else { return doc['contextDates.start.date'].value}",
"order": "asc"
}
}
}
When I execute this query, I get following error:
"failed_shards" : [
{
"shard" : 0,
"index" : "my_inedx",
"node" : "UxKwS8SIR-uIbzo5_0IbcQ",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"return doc['contextDates.event.date'].value} else { ",
" ^---- HERE"
],
"script" : "if(doc['contextDates.event.date'].value != 0){ return doc['contextDates.event.date'].value} else { return doc['contextDates.start.date'].value}",
"lang" : "painless",
"position" : {
"offset" : 84,
"start" : 47,
"end" : 99
},
"caused_by" : {
"type" : "wrong_method_type_exception",
"reason" : "cannot convert MethodHandle(Dates)JodaCompatibleZonedDateTime to (Object)double"
}
}
}
]
I have tried Double.parseDouble method as well but it doesn't work. This is what I have inside document for contextDates
"contextDates" : {
"event" : {
"date" : "2020-06-26T00:00:00.000Z",
"resolution" : "day",
"score" : 0,
"type" : "event"
}
}
The doc value you're getting is of type JodaCompatibleZonedDateTime which you're trying to compare to a double value, so you need to modify your script like this
if(doc['contextDates.event.date'].value.getMillis() != 0){ return doc['contextDates.event.date'].value.getMillis()} else { return doc['contextDates.start.date'].value.getMillis()}
We need to add two new fields to an existing ElasticSearch (7.9 oss) instance.
Field 1: Date Field
We want to add an optional date field. It shouldn't have a value upon creation.
How to do this with update_by_query?
Tried this:
POST orders/_update_by_query
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.new_d3_field",
"lang": "painless",
"type": "date",
"format": "yyyy/MM/dd HH:mm:ss"
}
}
Field 2: Boolean field with ? in name
We want to keep the ? so that it matches the other fields that we already have in ES.
Also worth noting that even removing the ? and doing the below the field doesn't appear to be a boolean.
Tried this:
POST orders/_update_by_query
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.new_b_field? = false",
"lang": "painless"
}
}
Which gave the error:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"ctx._source.new_b_field? = false",
" ^---- HERE"
],
"script" : "ctx._source.new_b_field? = false",
"lang" : "painless",
"position" : {
"offset" : 25,
"start" : 0,
"end" : 32
}
}
],
"type" : "script_exception",
"reason" : "compile error",
"script_stack" : [
"ctx._source.new_b_field? = false",
" ^---- HERE"
],
"script" : "ctx._source.new_b_field? = false",
"lang" : "painless",
"position" : {
"offset" : 25,
"start" : 0,
"end" : 32
},
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "invalid sequence of tokens near ['='].",
"caused_by" : {
"type" : "no_viable_alt_exception",
"reason" : null
}
}
},
"status" : 400
}
Also tried:
POST orders/_update_by_query?new_b_field%3F=false
Which gave:
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "request [/orders/_update_by_query] contains unrecognized parameter: [new_b_field?]"
}
],
"type" : "illegal_argument_exception",
"reason" : "request [/orders/_update_by_query] contains unrecognized parameter: [new_b_field?]"
},
"status" : 400
}
If you want to add two new fields to an existing ElasticSearch index that don't have value upon creation you should update its mapping using Put mapping API
PUT /orders/_mapping
{
"properties": {
"new_d3_field": {
"type": "date",
"format": "yyyy/MM/dd HH:mm:ss"
},
"new_b_field?": {
"type": "boolean"
}
}
}
If you still want to use _update_by_query you should set an initial value, then the field will be added.
POST orders/_update_by_query?wait_for_completion=false&conflicts=proceed
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.new_d3_field=params.date;ctx._source.new_b_field = params.val",
"lang": "painless",
"params": {
"date": "1980/01/01",
"val": false
}
}
}
Update By Query API is used to update documents so I guess you can't add a field to your schema without updating at list one doc. what you can do is to set a dummy doc and update only this certain doc. Something like that:
POST orders/_update_by_query
{
"query": {
"match": {
"my-field":"my-value"
}
},
"script": {
"source": "ctx._source.new_d3_field=params.date;ctx._source.new_b_field = params.val",
"lang": "painless",
"params": {
"date": "1980/01/01",
"val": false
}
}
}
In elastic search index, how to get the documents where array field has length greater than 0?
I tried following multiple syntaxes but didn't get any breakthrough. I got same error in all of the syntaxes.
GET http://{{host}}:{{elasticSearchPort}}/student_details/_search
Syntax 1:
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['enrolledCourses'].values.length > 0",
"lang": "painless"
}
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Syntax 2:
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['enrolledCourses'].values.size() > 0",
"lang": "painless"
}
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Syntax 3:
{
"query": {
"bool": {
"filter" : {
"script" : {
"script" : "doc['enrolledCourses'].values.size() > 0"
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Syntax 4:
{
"query": {
"bool": {
"filter" : {
"script" : {
"script" : "doc['enrolledCourses'].values.length > 0"
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Please help me in solving this.
I don't know what version of elastic you run, then all my test I'd running on latest 7.9.0 version of Elasticsearch.
I will use painless script for scripting.
I put to documents to index test:
PUT test/_doc/1
{
"name": "Vasia",
"enrolledCourses" : ["test1", "test2"]
}
PUT test/_doc/2
{
"name": "Petya"
}
How you can see one document contains enrolledCourses field and second not.
In painless you don't need use values field and you can take length directly, this is according to painless documentation. Then I skip using values operator in my script:
GET test/_search
{
"query": {
"bool": {
"filter": [
{
"script": {
"script": {
"source": "doc['enrolledCourses'].length > 0",
"lang": "painless"
}
}
}
]
}
}
}
After running I'd received 2 different errors:
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:757)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:331)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
"java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"doc['enrolledCourses'].length > 0",
" ^---- HERE"
]
}
and
{
"type" : "illegal_argument_exception",
"reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [enrolledCourses] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
Both of errors is pretty clear. First for document where field doesn't exists and second because Elasticsearch indexed string array field as default mapping type text.
Both of cases is very easy to fix by mapping enrolledCourses field as keyword.
In first case mapping will always provide empty field and in second keyword word be allow to run fielddata property.
PUT test
{
"settings": {
"number_of_replicas": 0
},
"mappings": {
"properties": {
"name": {
"type": "keyword"
},
"enrolledCourses": {
"type": "keyword"
}
}
}
}
Now I will receive right answer for query:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : 0.0,
"_source" : {
"name" : "Vasia",
"enrolledCourses" : [
"test1",
"test2"
]
}
}
]
}
}
I want to search multiple indices in Elasticsearch (news items in search_news and documents in search_documents) and whenever an index has a publicationDate field (news items only), I want to 'sort' this, so I boost newer news items. I am using Elasticsearch 6.8.
I found the script_scoring example in https://dzone.com/articles/23-useful-elasticsearch-example-queries (last one). But this throws errors and based on the documentation I came up to
GET /search_*/_search
{
"query": {
"function_score": {
"query": {
"bool": {
"must": {
"query_string": {
"query": "Lorem Ipsum"
}
},
"must_not": {
"exists": {
"field": "some_exlusion_field"
}
}
}
},
"script_score": {
"script": {
"params" : {
"threshold": "2019-04-04"
},
"source": "publishDate = doc['publishDate'].value; if (publishDate > Date.parse('yyyy-MM-dd', threshold).getTime()) { return log(2.5) } return log(1);"
}
}
}
}
}
This results in the error:
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"publishDate = doc['publis ...",
"^---- HERE"
],
"script": "publishDate = doc['publishDate'].value; if (publishDate > Date.parse('yyyy-MM-dd', threshold).getTime()) { return log(2.5) } return log(1);",
"lang": "painless"
}
}
I managed to minify the source to:
"source": "if (doc['publishDate'] > '2019-04-04') { return 5 } return 1;"
But no success:
"failures" : [
{
"shard" : 0,
"index" : "search_document_page",
"node" : "c0iLpxiJRqmgwS0KY8OybA",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:81)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
"if (doc['publishDate'] > '2019-04-04') { ",
" ^---- HERE"
],
"script" : "if (doc['publishDate'] > '2019-04-04') { return 5 } return 1;",
"lang" : "painless",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "No field found for [publishDate] in mapping with types []"
}
}
},
{
"shard" : 0,
"index" : "search_news",
"node" : "c0iLpxiJRqmgwS0KY8OybA",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"if (doc['publishDate'] > '2019-04-04') { ",
" ^---- HERE"
],
"script" : "if (doc['publishDate'] > '2019-04-04') { return 5 } return 1;",
"lang" : "painless",
"caused_by" : {
"type" : "class_cast_exception",
"reason" : "Cannot apply [>] operation to types [org.elasticsearch.index.fielddata.ScriptDocValues.Dates] and [java.lang.String]."
}
}
}
]
}
}
Any suggestion for checking the existence of the field in doc and how to check the date properly?
For the existence check ( doc here ) :
if (!doc.containsKey('publishDate')) {
return 1;
}
And for the date comparison, you can try this way
if (Date.parse('yyyy-MM-dd', params.threshold).getMillis() > doc['publishDate'].getMillis()) {
return 5;
} else {
return 1;
}