I am trying to perform terms aggregation over a field of type 'ip' using inline script like this :
{
"aggs": {
"by_ipaddress": {
"terms": {
"script": {
"inline": "doc['ipAddressFrom'].value",
"lang": "painless"
}
}
}
}
}
It throws following exception :
"reason": {
"type": "script_exception",
"reason": "runtime error",
"caused_by": {
"type": "array_index_out_of_bounds_exception",
"reason": "16"
},
"script_stack": [
"org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:602)",
"org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:152)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:83)",
"doc['ipAddressFrom'].value",
" ^---- HERE"
],
"script": "doc['ipAddressFrom'].value",
"lang": "painless"
}
But when I aggregate over the same field :
{
"aggs": {
"by_ipaddress": {
"terms": {
"field": "ipAddressFrom"
}
}
}
}
It works.
Mapping for the field "ipAddressFrom" is :
"ipAddressFrom" : {
"type" : "ip"
}
Please let me know how to use ip fields in script.
For elasticsearch 6.x, there is nothing wrong with using ip type in painless scripts.
Your aggregation with inline doesn't work because for some documents field ipAddressFrom does not exist.
You can fix the aggregation with something like:
"script": {
"inline": "if (doc.containsKey('ipAddressFrom') && !doc['ipAddressFrom'].empty){ return doc['ipAddressFrom'].value} else {return '0'}",
"lang": "painless"
}
Related
I'm trying to extract a term using a pattern from a text field using a painless script on elasticsearch 5.6.
GET _search
{
"size": 5,
"script_fields": {
"ipaddress": {
"script": {
"inline": "Pattern pattern=Pattern.compile(\\\"([A-Z])\\w+\\\"); Matcher matcher =pattern.matcher(doc['Message'].value); if(matcher.find()){return matcher.group();}return null;",
"lang": "painless"
}
}
},
"query": {
"bool": {
"should": [
{
"match_phrase": {
"Message": "Failed password"
}
},
{
"regexp": {
"Message": {
"value": "([A-Z])\\w+"
}
}
}
]
}
}
}
Instead I receive the error
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"... pattern=Pattern.compile(\\\"([A-Z])\\w+\\\"); Matcher ...",
" ^---- HERE"
],
"script": "Pattern pattern=Pattern.compile(\\\"([A-Z])\\w+\\\"); Matcher matcher =pattern.matcher(doc['Message'].value); if(matcher.find()){return matcher.group();}return null;",
"lang": "painless"
},
{
"type": "circuit_breaking_exception",
"reason": "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting",
"bytes_wanted": 0,
"bytes_limit": 0
}
],
It seems quoting the regex pattern is the issue as shown in the error ^-------HERE. How can I quote the pattern?
in ES version 5.x, I used this query and works well.
"bool": {
"must": [
{
"script": {
"script": {
"inline": "params['_source']['my_field'] != null",
"lang": "painless"
},
BUT in ES version 6.2, It's error with below error message.
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"params['_source']['my_field'] != null"
,
" ^---- HERE"
],
"script": "params['_source']['my_field'] != null",
"lang": "painless"
}
],
There is no '_source' data in params. because there is no return data with below query.
"inline": "params['_source'] != null",
How should I do with this error?
Please, try something like this:
{
"query" : {
"bool": {
"must": [
{
"script": {
"script": {
"source": "(doc.containsKey('my_field') ? doc['my_field'].value : null) != null",
"lang": "painless"
}
}
}
]
}
}
}
It works on my 6.2.2 version.
Is it possible to make a comparison with respect to the length of an array field in Elasticsearch?
For instance the following works (sourceId is a text type field)
GET /entity_active/_count
{
"query": {
"bool" : {
"must" : {
"script" : {
"script" : {
"inline": "doc['sourceId'].values.size() > 0",
"lang": "painless"
}
}
}
}
}
}
However the following does not work (users is an array field)
GET /entity_active/_count
{
"query": {
"bool" : {
"must" : {
"script" : {
"script" : {
"inline": "doc['users'].values.size() > 0",
"lang": "painless"
}
}
}
}
}
}
The latter returns such a response:
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:80)",
"doc['users'].values.size() > 0",
" ^---- HERE"
],
"script": "doc['users'].values.size() > 0",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
Do I need to use an alternative way to make use of such array fields?
Arrays are not indexed in ES, hence you cannot access an array via doc_values (i.e. through doc.). You can access the _source document in which the array is present, though, using ctx._source. Try this script instead:
_source.users.size() > 0
or
_source.users.length > 0
I'm following the example over here for updating a set of tags using the partial update in Elasticsearch.
Following is my script:
{
"script": {
"lang": "painless",
"inline": "ctx._source.deviceTags.add(params.tags)",
"params": {
"tags": "search"
}
}
}
Request URL is:
https://aws-es-service-url/devices/device/123/_update
But I'm getting the following response:
{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[fBaExM8][x.x.x.x:9300][indices:data/write/update[s]]"
}
],
"type": "illegal_argument_exception",
"reason": "failed to execute script",
"caused_by": {
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"ctx._source.deviceTags.add(params.tags)",
" ^---- HERE"
],
"script": "ctx._source.deviceTags.add(params.tags)",
"lang": "painless",
"caused_by": {
"type": "null_pointer_exception",
"reason": null
}
}
},
"status": 400
}
Any idea of what I have done wrong?
Since your deviceTags array is initially null, you have two ways to solve this
A. Use upsert to make sure that deviceTags gets added to your document initially
{
"script": {
"lang": "painless",
"inline": "ctx._source.deviceTags.add(params.tags)",
"params": {
"tags": "search"
}
},
"upsert": {
"deviceTags": ["search"]
}
}
B. Protect your code against NPE
{
"script": {
"lang": "painless",
"inline": "(ctx._source.deviceTags = ctx._source.deviceTags ?: []).add(params.tags)",
"params": {
"tags": "search"
}
}
}
I have inserted a new field wiki_collection and set its value to true in my document by using:
POST /pibtest1/_update_by_query
{
"script": {
"inline": "ctx._source.wiki_collection=true"
},
"query": {
"match": {
"url":"http%3A%2F%2Fwww.searchtechnologies.com%2Fbundles%2Fmodernizr%3Fv%3D7-yR01kdRVQ7W1RQzMBVKYLDhCt0itEATWHixfzE8Os1"
}
}
}
But now I want to delete this field. I am trying to do this:
POST /pibtest1/_update
{
"script" : "ctx._source.remove(\"wiki_collection\")"
}
But I am getting an error which says:
{
"error": {
"root_cause": [
{
"type": "invalid_type_name_exception",
"reason": "Document mapping type name can't start with '_'"
}
],
"type": "invalid_type_name_exception",
"reason": "Document mapping type name can't start with '_'"
},
"status": 400
}
Is there any other way to delete a field in elasticsearch?
Edit: I have updated by query:
POST /pibtest1/_update_by_query
{
"script": {
"inline": "ctx._source.remove(\"wiki_collection\")"
},
"query": {
"match": {
"url": "http://www.searchtechnologies.com/bundles/modernizr?v=7-yR01kdRVQ7W1RQzMBVKYLDhCt0itEATWHixfzE8Os1"
}
}
}
But now I am getting a new filed called "remove='wiki_collection'" in my documents which looks like this: http://i.stack.imgur.com/QvxIa.png
I want to remove/delete this wiki_collection field from my documents.
Your update is wrong. It should either be (you specify the complete path - index/type/ID):
POST /pibtest1/test/234/_update
{
"script": {
"inline": "ctx._source.remove(\"wiki_collection\")"
}
}
Or you use the same _update_by_query:
POST /pibtest1/_update_by_query
{
"script": {
"inline": "ctx._source.remove(\"wiki_collection\")"
},
"query": {
"match": {
"url": "whatever"
}
}
}