elasticsearch kibana 5.6 too many dynamic script compilations painless - elasticsearch

I'm trying to extract a term using a pattern from a text field using a painless script on elasticsearch 5.6.
GET _search
{
"size": 5,
"script_fields": {
"ipaddress": {
"script": {
"inline": "Pattern pattern=Pattern.compile(\\\"([A-Z])\\w+\\\"); Matcher matcher =pattern.matcher(doc['Message'].value); if(matcher.find()){return matcher.group();}return null;",
"lang": "painless"
}
}
},
"query": {
"bool": {
"should": [
{
"match_phrase": {
"Message": "Failed password"
}
},
{
"regexp": {
"Message": {
"value": "([A-Z])\\w+"
}
}
}
]
}
}
}
Instead I receive the error
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"... pattern=Pattern.compile(\\\"([A-Z])\\w+\\\"); Matcher ...",
" ^---- HERE"
],
"script": "Pattern pattern=Pattern.compile(\\\"([A-Z])\\w+\\\"); Matcher matcher =pattern.matcher(doc['Message'].value); if(matcher.find()){return matcher.group();}return null;",
"lang": "painless"
},
{
"type": "circuit_breaking_exception",
"reason": "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting",
"bytes_wanted": 0,
"bytes_limit": 0
}
],
It seems quoting the regex pattern is the issue as shown in the error ^-------HERE. How can I quote the pattern?

Related

Mustache double quotes problem in search templates

What is the best way to use mustache False values feature in Elasticsearch template?
At the moment I am trying to select function based on boolean value.
Rendering seems to be working according to the logic, but it prints empty double quotes and I cannot get rid of those.
Code example mustache template snippet:
"must": {
"function_score": {
"functions": [
"{{^isLocationFunctionNeeded}}",
{
"exp": {
"location": {
"origin": {
"lat": "0.0",
"lon": "0.0"
},
"offset": "1km",
"scale": "50km"
}
}
},
"{{/isLocationFunctionNeeded}}",
{
"random_score": {},
"weight": 0.00001
}
],
"score_mode": "sum"
}
}
Render snippet:
"must": {
"function_score": {
"functions": [
"",
{
"random_score": {},
"weight": 1.0E-5
}
],
"score_mode": "sum"
}
}
Error I get trying to run the template on ELK:
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "failed to parse [START_OBJECT]. malformed query, expected a [VALUE_STRING] while parsing functions but got a [function_score] instead",
"line": x (where "" is visible in Render snippet),
"col": x (where "" is visible in Render snippet)
}
],
"type": "x_content_parse_exception",
"reason": " x (where "" is visible in Render snippet),[bool] failed to parse field [must]",
"caused_by": {
"type": "parsing_exception",
"reason": "failed to parse [START_OBJECT]. malformed query, expected a [VALUE_STRING] while parsing functions but got a [function_score] instead",
"line": x (where "" is visible in Render snippet),,
"col": x (where "" is visible in Render snippet),
}
Without mustache values it's working fine. Also I noticed in some cases if you surround empty double quotes with random functions it tends to work sometimes. Seems Elastic don't like must cases starting with empty double quotes.
I also asked the same question in ELK community with no luck so far:
https://discuss.elastic.co/t/mustache-double-quotes-problem-in-search-templates/318736
As an example for rendering template we can try using the following:
{
"script": {
"lang": "mustache",
"source": {
"must": {
"function_score": {
"functions": [
"{{^isLocationFunctionNeeded}}",
{
"exp": {
"location": {
"lat": "0.0",
"lon": "0.0"
},
"offset": "1km",
"scale": "50km"
}
},
"{{/isLocationFunctionNeeded}}",
{
"random_score": {},
"weight": 0.00001
}
],
"score_mode": "sum"
}
}
}
}
}
Calling template with params:
{
"id": "example_template",
"params": {
"isLocationFunctionNeeded" : true
}
}
The query inside your template must be a full fledge query, not just must. Also you need to enclose it in triple quotes """, like this, and it will work
POST _scripts/example_template
{
"script": {
"lang": "mustache",
"source": """
{
"query": {
"bool": {
"must": {
"function_score": {
"functions": [
{{^isLocationFunctionNeeded}}
{
"exp": {
"location": {
"lat": "0.0",
"lon": "0.0"
},
"offset": "1km",
"scale": "50km"
}
},
{{/isLocationFunctionNeeded}}
{
"random_score": {},
"weight": 0.00001
}
],
"score_mode": "sum"
}
}
}
}
}
"""
}
}
If you cannot use the triple quotes (e.g. when using Postman), you need to send it as a one-line string and escape all quote characters:
POST _scripts/example_template
{
"script": {
"lang": "mustache",
"source": " { \"query\": { \"bool\": { \"must\": { \"function_score\": { \"functions\": [ {{^isLocationFunctionNeeded}} { \"exp\": { \"location\": { \"lat\": \"0.0\", \"lon\": \"0.0\" }, \"offset\": \"1km\", \"scale\": \"50km\" } }, {{/isLocationFunctionNeeded}} { \"random_score\": {}, \"weight\": 0.00001 } ], \"score_mode\": \"sum\" }}}}"
}
}

ElasticSearch [script] unknown field [file], parser not found

I used script_score to customize the scoring:
GET /customer/_search
{
"query": {
"function_score": {
"query": {
"match": {
"name": "Mark"
}
},
"script_score": {
"script": {
"lang": "painless",
"file": "test"
}
}
}
}
}
I set "file": "test", and put test.groovy file in config/scripts directory, but I got these error:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "[script] unknown field [file], parser not found"
}
],
"type": "illegal_argument_exception",
"reason": "[script] unknown field [file], parser not found"
},
"status": 400
}
[script] unknown field [file], parser not found! Why? Should I need to install some plugins?
Elasticsearch version : 6.2.3
Plugins installed: None
JVM version : 1.8.0_181
OS version: Ubuntu Linux 4.4.0-124-generic
File scripts have been removed in ES 6.0, you should now use stored scripts instead.
You can easily migrate your Groovy script to Painless.
First, store your script:
POST _scripts/test
{
"script": {
"lang": "painless",
"source": "Math.log(_score * 2)"
}
}
Then use it in your query:
GET /customer/_search
{
"query": {
"function_score": {
"query": {
"match": {
"name": "Mark"
}
},
"script_score": {
"script": {
"id": "test"
}
}
}
}
}

script_score query does not support [source]

I'm using the official Docker image for Elasticsearch OSS (docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4) and can't seem to get script_score working at all. It seems like scripting isn't enabled.
For example, this:
POST http://localhost:9200/address/address/_search
{
"query": {
"function_score": {
"query": {
"match": {
"fullAddress": {
"query": "13 fake",
"operator": "and"
}
}
},
"script_score": {
"lang": "expression",
"source": "doc['flatNumber'].length"
}
}
}
}
gives me this:
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "script_score query does not support [source]",
"line": 13,
"col": 15
}
],
"type": "parsing_exception",
"reason": "script_score query does not support [source]",
"line": 13,
"col": 15
},
"status": 400
}
I tried enabling it:
PUT http://localhost:9200/_cluster/settings
{
"persistent": {
"script.engine.groovy.inline.aggs": "on"
}
}
but to no avail:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "persistent setting [script.engine.groovy.inline.aggs], not recognized"
}
],
"type": "illegal_argument_exception",
"reason": "persistent setting [script.engine.groovy.inline.aggs], not recognized"
},
"status": 400
}
How do I get script_score working?
You're simply missing a script section in your script_score. Modify it like this and it will work:
"script_score": {
"script": {
"lang": "expression",
"source": "doc['flatNumber'].length"
}
}

Update nested string field

I am trying to update a field image.uri by _update_by_query:
POST user/_update_by_query
{
"script": {
"source": "ctx._source.image.uri = 'https://example.com/default/image/profile.jpg'",
"lang": "painless"
},
"query": {
"bool": {
"must_not": [
{
"exists": {
"field": "image.id"
}
}
]
}
}
}
But it throws error:
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"ctx._source.image.uri = 'https://example.com/default/image/profile.jpg'",
" ^---- HERE"
],
"script": "ctx._source.image.uri = 'https://example.com/default/image/profile.jpg'",
"lang": "painless"
}
],
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"ctx._source.image.uri = 'https://example.com/default/image/profile.jpg'",
" ^---- HERE"
],
"script": "ctx._source.image.uri = 'https://example.com/default/image/profile.jpg'",
"lang": "painless",
"caused_by": {
"type": "null_pointer_exception",
"reason": null
}
},
"status": 500
}
A sample document:
{
"image": {
"uri": "https://example.com/resources/uploads/default_files/profile/thumb/large/default_profile.jpg"
},
"created": "2018-06-06T21:49:26Z",
"uid": 1,
"name": "Jason Cameron",
"username": "jason"
}
UPDATED RESPONE
The problem could be coming from a document without image object in it.
Try to add strict mapping if possible, to avoid indexing documents without image object.
OLD RESPONSE/"\' are correct for use inside painless script as string
Your problem comes as use of ' to encapsulate your uri, strings must be encapsulated by ".
Try to modify your script as:
"script": {
"source": "ctx._source.image.uri = \"https://example.com/default/image/profile.jpg\"",
"lang": "painless"
}

Error in executing groovy script in elasticsearch

below is my query, I want to change score calculation using function_score feature:
{
"size": 1,
"query":{
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {
"messageText": "car"
}
}
]
}
},
"script_score" : {
"script" : "doc['time_views'].values[doc['time_views'].values.length-1]"
}
,
"boost_mode": "replace"
}
},
"from": 0
}
but I got this error response
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "failed to run inline script [doc['time_views'].values[doc['time_views'].values.length-1]] using lang [groovy]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "datacollection",
"node": "TWeZV3R6Rq-WYQ2YIHjILQ",
"reason": {
"type": "script_exception",
"reason": "failed to run inline script [doc['time_views'].values[doc['time_views'].values.length-1]] using lang [groovy]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [time_views] in mapping with types [message]"
}
}
}
]
},
"status": 500
}
some solutions says using quotation in "doc['time_views']" causes the problem when query has been send from command prompt tools. I don't know why!
I don't use any command prompt tools. I create the query in java code directly
EDIT
this is my index mapping:
"mappings": {
"message": {
"properties": {
"text": {
"type": "string"
},
"time_views": {
"type": "nested",
"properties": {
"backupTimestamp": {
"type": "long"
},
"views": {
"type": "integer"
}
}
}
}
}
}
}
I want to use "views" of last item of "time_views". so I try below scripts too, but each of them throw different error:
"doc['time_views.views'].values[doc['time_views.views'].values.length-1]"
error: java.util.ArrayList cannot be cast to java.lang.Number
"doc['time_views.views'].values[doc['time_views.views'].values.size()-1]"
error: failed to run inline script [doc['time_views.views'].values[doc['time_views.views'].values.size()-1]] using lang [groovy]
"doc['time_views'].values[doc['time_views'].values.size()-1].views"
error: failed to run inline script [doc['time_views'].values[doc['time_views'].values.size()-1].views] using lang [groovy]"
I'm really new in elasticsearch and groovy language. I didn't care about that "time_views" is nested Object, also I don't know syntax of groovy exactly, after some affort I found my mistakes and the solution:
{
"size": 1,
"query":{
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {
"messageText": "car"
}
}
]
}
},
"script_score" : {
"script" : "doc['time_views.views'].values.get(doc['time_views.views'].values.size()-1)"
}
,
"boost_mode": "replace"
}
},
"from": 0
}
It's work as I expected

Resources