I am facing an issue while trying to execute a script within an ES JSON request
The request:
POST _search
{
"query": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
},
"aggs": {
"bucket_histogram": {
"histogram": {
"field": "dayTime",
"interval": 10
},
"aggs": {
"get_average": {
"avg": {
"field": "value"
}
},
"check-threshold": {
"bucket_script": {
"buckets_path": {
"averageValue": "get_average"
},
"script": "averageValue - doc[\"thresholdValue\"].value"
}
}
}
}
}
}
But I get this error instead of returning values
{
"error": {
"root_cause": [],
"type": "reduce_search_phase_exception",
"reason": "[reduce] ",
"phase": "fetch",
"grouped": true,
"failed_shards": [],
"caused_by": {
"type": "groovy_script_execution_exception",
"reason": "failed to run inline script [averageValue - doc[\"thresholdValue\"].value] using lang [groovy]",
"caused_by": {
"type": "missing_property_exception",
"reason": "No such property: doc for class: 7dcca7d142ac809a7192625d43d95bde9883c434"
}
}
},
"status": 503
}
Yet if I remove doc[\"thresholdValue\"] and enter a number everything works fine.
You are using a bucket_script, which is a part of the pipeline aggregations released with Elasticsearch 2.0. Pipeline aggregations work against other aggregations and not documents, which is why the doc context is not supplied to the aggregation.
If you want to process aggregations against specific documents, then perhaps you want the scripted metric aggregation instead.
Related
I'm trying to make range aggregation on the following data set:
{
"ProductType": 1,
"ProductDefinition": "fc588f8e-14f2-4871-891f-c73a4e3d17ca",
"ParentProduct": null,
"Sku": "074617",
"VariantSku": null,
"Name": "Paraboot Avoriaz/Jannu Marron Brut Marron Brown Hiking Boot Shoes",
"AllowOrdering": true,
"Rating": null,
"ThumbnailImageUrl": "/media/1106/074617.jpg",
"PrimaryImageUrl": "/media/1106/074617.jpg",
"Categories": [
"399d7b20-18cc-46c0-b63e-79eadb9390c7"
],
"RelatedProducts": [],
"Variants": [
"84a7ff9f-edf0-4aab-87f9-ba4efd44db74",
"e2eb2c50-6abc-4fbe-8fc8-89e6644b23ef",
"a7e16ccc-c14f-42f5-afb2-9b7d9aefbc5c"
],
"PriceGroups": [
"86182755-519f-4e05-96ef-5f93a59bbaec"
],
"DisplayName": "Paraboot Avoriaz/Jannu Marron Brut Marron Brown Hiking Boot Shoes",
"ShortDescription": "",
"LongDescription": "<ul><li>Paraboot Avoriaz Mountaineering Boots</li><li>Marron Brut Marron (Brown)</li><li>Full leather inners and uppers</li><li>Norwegien Welted Commando Sole</li><li>Hand made in France</li><li>Style number : 074617</li></ul><p>As featured on Pritchards.co.uk</p>",
"UnitPrices": {
"EUR 15 pct": 343.85
},
"Taxes": {
"EUR 15 pct": 51.5775
},
"PricesInclTax": {
"EUR 15 pct": 395.4275
},
"Slug": "paraboot-avoriazjannu-marron-brut-marron-brown-hiking-boot-shoes",
"VariantsProperties": [
{
"Key": "ShoeSize",
"Value": "8"
},
{
"Key": "ShoeSize",
"Value": "10"
},
{
"Key": "ShoeSize",
"Value": "6"
}
],
"Guid": "0d4f6899-c66a-4416-8f5d-26822c3b57ae",
"Id": 178,
"ShowOnHomepage": true
}
I'm aggregating on VariantsProperties which have the following mapping
"VariantsProperties": {
"type": "nested",
"properties": {
"Key": {
"type": "keyword"
},
"Value": {
"type": "keyword"
}
}
}
Terms aggregations are working fine with following code:
{
"aggs": {
"Nest": {
"nested": {
"path": "VariantsProperties"
},
"aggs": {
"fieldIds": {
"terms": {
"field": "VariantsProperties.Key"
},
"aggs": {
"values": {
"terms": {
"field": "VariantsProperties.Value"
}
}
}
}
}
}
}
}
However when I try to do a range aggregation to get shoes in size between 8 - 12 such as:
{
"aggs": {
"Nest": {
"nested": {
"path": "VariantsProperties"
},
"aggs": {
"fieldIds": {
"range": {
"field": "VariantsProperties.Value",
"ranges": [ { "from": 8, "to": 12 }]
}
}
}
}
}
}
I get the following error:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Field [VariantsProperties.Value] of type [keyword] is not supported for aggregation [range]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "product-avenueproductindexdefinition-24476f82-en-us",
"node": "ejgN4XecT1SUfgrhzP8uZg",
"reason": {
"type": "illegal_argument_exception",
"reason": "Field [VariantsProperties.Value] of type [keyword] is not supported for aggregation [range]"
}
}
],
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Field [VariantsProperties.Value] of type [keyword] is not supported for aggregation [range]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Field [VariantsProperties.Value] of type [keyword] is not supported for aggregation [range]"
}
}
},
"status": 400
}
Is there a way to "transform" the terms aggregation into a range aggregation, without the need of changing the schema? I know I could build the ranges myself by extracting the data from the terms aggregation and building the ranges out of it, however, I would prefer a solution within the elastic itself.
There are two ways to solve this:
Option A: Use a script instead of a field. This option will work without having to reindex your data, but depending on your volume of data, the performance might suffer.
POST test/_search
{
"aggs": {
"Nest": {
"nested": {
"path": "VariantsProperties"
},
"aggs": {
"fieldIds": {
"range": {
"script": "Integer.parseInt(doc['VariantsProperties.Value'].value)",
"ranges": [
{
"from": 8,
"to": 12
}
]
}
}
}
}
}
}
Option B: Add an integer sub-field in your mapping.
PUT my-index/_mapping
{
"properties": {
"VariantsProperties": {
"type": "nested",
"properties": {
"Key": {
"type": "keyword"
},
"Value": {
"type": "keyword",
"fields": {
"numeric": {
"type": "integer",
"ignore_malformed": true
}
}
}
}
}
}
}
Once your mapping is modified, you can run _update_by_query on your index in order to reindex the VariantsProperties.Value data
PUT my-index/_update_by_query
Finally, when this last command is done, you can run the range aggregation on the VariantsProperties.Value.numeric field.
Also note that this second but will be more performant on the long term.
I have a handful of indexes, some of which have a particular date property indicating when it was published (date_publish), and others do not. I am trying to apply a gauss function to decay the score of documents which were published a long time ago. The relevant indexes are correctly configured to recognise the date_publish property as a date.
I have set up my query as follows, specifically filtering documents which do not have the property:
{
"index": "index_contains_prop,index_does_not_contains_prop",
"body": {
"query": {
"function_score": {
"score_mode": "avg",
"query": {
"match_all": {}
},
"functions": [
{
"script_score": {
"script": {
"source": "0"
}
}
},
{
"filter": {
"exists": {
"field": "date_publish"
}
},
"gauss": {
"date_publish": {
"origin": "now",
"scale": "728d",
"offset": "7d",
"decay": 0.5
}
}
}
]
}
},
"from": 0,
"size": 1000
}
}
However, the query errors with the following:
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "unknown field [date_publish]",
"line": 1,
"col": 0
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "index_does_not_contains_prop",
"node": "1hfXZK4TT3-K288nIr0UWA",
"reason": {
"type": "parsing_exception",
"reason": "unknown field [date_publish]",
"line": 1,
"col": 0
}
}
]
},
"status": 400
}
I have RTFM'd many times, and i can't see any discrepancy - I ahve also tried wrapping the exists condition in a bool:must object, to no avail.
Have I misunderstood the purpose of the filter argument?
The exists query will only work on fields that are part of the index mapping. It will return only documents that have a value for this field, but the field itself still needs to be defined in the mapping. This is why you're getting an error - index_does_not_contains_prop does not have date_publish mapped. You can use the put mapping API to add this field to the indexes who don't have it (it won't change any document), and then your query should work.
Hi I am trying to update documents a elasticsearch which meets specific criteria. I am using google sense(chrome extension) for making request. The request that I am making is as shown below:
GET styling_rules2/product_line_filters/_update
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{"term":{"product_line_attribute": "brand"}}
],
"minimum_should_match": 1
}
},
"filter": {
"term": {
"product_line_name": "women_skirts"
}
}
}
},
"script" : "ctx._source.brand=brands"
}
sample document is as shown below:
{
"product_line_attribute_db_path": "product_filter.brand",
"product_line_attribute": "brand",
"product_line_name": "women_skirts",
"product_line_attribute_value_list": [
"vero moda",
"faballey",
"only",
"rider republic",
"dorothy perkins"
]
}
desired result: update all the document which has product_line_attribute="brand" and product_line_name="women_skirts" to product_line_attribute="brands".
problem: I am getting the error as follows:
{
"error": {
"root_cause": [
{
"type": "search_parse_exception",
"reason": "failed to parse search source. unknown search element [script]",
"line": 18,
"col": 4
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "styling_rules2",
"node": "2ijp1pXwT46FN4on4-JPlg",
"reason": {
"type": "search_parse_exception",
"reason": "failed to parse search source. unknown search element [script]",
"line": 18,
"col": 4
}
}
]
},
"status": 400
}
thanks in advance!
You should use the _update_by_query endpoint and not _update. Also the script section is not correct, which is probably why you're getting a class_cast_exception.
Try this instead:
POST styling_rules2/product_line_filters/_update_by_query
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"term": {
"product_line_attribute": "brand"
}
}
],
"minimum_should_match": 1
}
},
"filter": {
"term": {
"product_line_name": "women_skirts"
}
}
}
},
"script": {
"inline": "ctx._source.brand=brands"
}
}
below is my query, I want to change score calculation using function_score feature:
{
"size": 1,
"query":{
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {
"messageText": "car"
}
}
]
}
},
"script_score" : {
"script" : "doc['time_views'].values[doc['time_views'].values.length-1]"
}
,
"boost_mode": "replace"
}
},
"from": 0
}
but I got this error response
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "failed to run inline script [doc['time_views'].values[doc['time_views'].values.length-1]] using lang [groovy]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "datacollection",
"node": "TWeZV3R6Rq-WYQ2YIHjILQ",
"reason": {
"type": "script_exception",
"reason": "failed to run inline script [doc['time_views'].values[doc['time_views'].values.length-1]] using lang [groovy]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [time_views] in mapping with types [message]"
}
}
}
]
},
"status": 500
}
some solutions says using quotation in "doc['time_views']" causes the problem when query has been send from command prompt tools. I don't know why!
I don't use any command prompt tools. I create the query in java code directly
EDIT
this is my index mapping:
"mappings": {
"message": {
"properties": {
"text": {
"type": "string"
},
"time_views": {
"type": "nested",
"properties": {
"backupTimestamp": {
"type": "long"
},
"views": {
"type": "integer"
}
}
}
}
}
}
}
I want to use "views" of last item of "time_views". so I try below scripts too, but each of them throw different error:
"doc['time_views.views'].values[doc['time_views.views'].values.length-1]"
error: java.util.ArrayList cannot be cast to java.lang.Number
"doc['time_views.views'].values[doc['time_views.views'].values.size()-1]"
error: failed to run inline script [doc['time_views.views'].values[doc['time_views.views'].values.size()-1]] using lang [groovy]
"doc['time_views'].values[doc['time_views'].values.size()-1].views"
error: failed to run inline script [doc['time_views'].values[doc['time_views'].values.size()-1].views] using lang [groovy]"
I'm really new in elasticsearch and groovy language. I didn't care about that "time_views" is nested Object, also I don't know syntax of groovy exactly, after some affort I found my mistakes and the solution:
{
"size": 1,
"query":{
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {
"messageText": "car"
}
}
]
}
},
"script_score" : {
"script" : "doc['time_views.views'].values.get(doc['time_views.views'].values.size()-1)"
}
,
"boost_mode": "replace"
}
},
"from": 0
}
It's work as I expected
I can't figure out what's wrong in my ES query.
I want to filter on a specific field, and also sort by other field.
Request:
GET /_search
{
"query" : {
"term": {
"_type" : "monitor"
},
"filtered" : {
"filter" : { "term" : { "ProcessName" : "myProc" }}
}
},
"sort": { "TraceDateTime": { "order": "desc", "ignore_unmapped": "true" }}
}
Response:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "failed to parse search source. expected field name but got [START_OBJECT]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": ".kibana",
"node": "94RPDCjhQh6eoTe6XoRmSg",
"reason": {
"type": "parse_exception",
"reason": "failed to parse search source. expected field name but got [START_OBJECT]"
}
}
]
},
"status": 400
}
You have a syntax error in your query, you need to enclose both of your term queries inside a bool/must compound query, it needs to be like this:
POST /_search
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"ProcessName": "myProc"
}
},
{
"term": {
"_type": "monitor"
}
}
]
}
}
}
},
"sort": {
"TraceDateTime": {
"order": "desc",
"ignore_unmapped": "true"
}
}
}
PS: Always use POST when sending a payload in your query.