Is it possible to use range aggregations in ElasticSearch with alphabetical ranges? - elasticsearch

In ElasticSearch range queries can be used with text ("from" : "Bread").
Is it possible to do the same with "range aggregations"?
I am trying :
"aggs" : "slice" {
"range" : { "ranges" : [{"from" : "Bread"}],"field" : "content.keyword"}
}
and I am getting :
"root_cause" : [
{
"type" : "number_format_exception",
"reason" : "For input string: \"Bread\""
}
],
"reason" : "all shards failed",
"phase" : "query",
"caused_by" : {
"reason" : "For input string: \"Bread\"",
"type" : "number_format_exception",
"caused_by" : {
"reason" : "For input string: \"Bread\"",
"type" : "number_format_exception"
}
},
"type" : "search_phase_execution_exception"
If so, would there be a workaround?


You can apply that range query to a terms agg if that's what you're after:
{
"size": 0,
"aggs": {
"filtered_content_terms": {
"filter": {
"range": {
"content.keyword": {
"gte": "Bread"
}
}
},
"aggs": {
"content_terms": {
"terms": {
"field": "content.keyword",
"size": 10
}
}
}
}
}
}

Related

match_only_text fields do not support sorting and aggregations elasticsearch

I would like to count and sort the number of occurred message on a field of type match_only_text. Using a DSL query the output needed to have to look like this:
{" Text message 1":615
" Text message 2":568
....}
So i tried this on kibana:
GET my_index_name/_search?size=0
{
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message"
}
}
}
}
However i get this error:
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "match_only_text fields do not support sorting and aggregations"
}
I am interested in the field "message" this is its mapping:
"message" : {
"type" : "match_only_text"
}
This is a part of the index mapping:
"mappings" : {
"_meta" : {
"package" : {
"name" : "system"
},
"managed_by" : "ingest-manager",
"managed" : true
},
"_data_stream_timestamp" : {
"enabled" : true
},
"dynamic_templates" : [
{
"strings_as_keyword" : {
"match_mapping_type" : "string",
"mapping" : {
"ignore_above" : 1024,
"type" : "keyword"
}
}
}
],
"date_detection" : false,
"properties" : {
"#timestamp" : {
"type" : "date"
}
.
.
.
"message" : {
"type" : "match_only_text"
},
"process" : {
"properties" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
},
"pid" : {
"type" : "long"
}
}
},
"system" : {
"properties" : {
"syslog" : {
"type" : "object"
}
}
}
}
}
}
}
Please Help

Yes, by design, match_only_text is of the text field type family, hence you cannot aggregate on it.
You need to:
A. create a message.keyword sub-field in your mapping of type keyword:
PUT my_index_name/_mapping
{
"properties": {
"message" : {
"type" : "match_only_text",
"fields": {
"keyword": {
"type" : "keyword"
}
}
}
}
}
B. update the whole index (using _update_by_query) so the sub-field gets populated and
POST my_index_name/_update_by_query?wait_for_completion=false
Then, depending on the size of your index, call GET _tasks?actions=*byquery&detailed regularly to check the progress of the task.
C. run the aggregation on that sub-field.
POST my_index_name/_search
{
"size": 0,
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message.keyword"
}
}
}
}

Elasticsearch - Missing Field Value For Nested Field - Function Score Query | v7.10.2

I have already posted this on the ES group but I got no response and so posted it on SO. Link https://discuss.elastic.co/t/missing-field-value-for-nested-field-function-score-query-v7-10-2/291365
I have been trying for a long time now but the nested field value is always saying missing field value while calculating the score.
Mapping:
{
"doctor_idx" : {
"mappings" : {
"properties" : {
"_class" : {
"type" : "keyword",
"index" : false,
"doc_values" : false
},
"service" : {
"type" : "nested",
"properties" : {
"_class" : {
"type" : "keyword",
"index" : false,
"doc_values" : false
},
"serviceTypeEarliestAvailability" : {
"type" : "nested",
"properties" : {
"_class" : {
"type" : "keyword",
"index" : false,
"doc_values" : false
},
"earliestAvailableDateTimeByType" : {
"type" : "date",
"format" : "date_hour_minute_second"
},
"serviceType" : {
"type" : "text"
},
"servicesMedium" : {
"type" : "keyword"
}
}
}
}
}
}
}
}
}
For simplicity, I have kept just one record. A glimpse of the record
"serviceTypeEarliestAvailability" : [
{
"serviceType" : "Service Type",
"earliestAvailableDateTimeByType" : "2021-12-09T19:39:16",
"servicesMedium" : [
"MED1",
"MED2",
"MED3",
"MED4"
]
}
],
The following query gives: "A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!"
I tried using field_value_factor instead of script_score but it's the same problem and it complains about the missing field value.
Query 1: with script_score
GET /doctor_idx/_search
{
"explain": true,
"query": {
"nested": {
"path": "service",
"query": {
"nested": {
"score_mode": "max",
"path": "service.serviceTypeEarliestAvailability",
"query": {
"function_score": {
"query": {
"match_all": {
"boost": 1
}
},
"functions": [
{
"filter": {
"match": {
"service.serviceTypeEarliestAvailability.serviceType": "type" // no complaints about this
}
},
"weight": 10
},
{
"script_score": {
"script": {
"source": "(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())"
}
}
}
]
}
}
}
}
}
}
}
Query 2 : With field_value_factor
GET /doctor_idx/_search
{
"explain": true,
"query": {
"nested": {
"path": "service",
"query": {
"nested": {
"score_mode": "max",
"path": "service.serviceTypeEarliestAvailability",
"query": {
"function_score": {
"query": {
"match_all": {
"boost": 1
}
},
"functions": [
{
"filter": {
"match": {
"service.serviceTypeEarliestAvailability.serviceType": "type"
}
},
"weight": 10
},
{
"field_value_factor": {
"field": "service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType"
}
}
]
}
}
}
}
}
}
}
Error From ES for Query 1:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.get(ScriptDocValues.java:160)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.getValue(ScriptDocValues.java:154)",
"(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
" ^---- HERE"
],
"script" : "(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
"lang" : "painless",
"position" : {
"offset" : 79,
"start" : 0,
"end" : 98
}
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "doctor_idx",
"node" : "mvh5k24dQPqM-d7JVeNomQ",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.get(ScriptDocValues.java:160)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Dates.getValue(ScriptDocValues.java:154)",
"(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
" ^---- HERE"
],
"script" : "(doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value.getMillis())",
"lang" : "painless",
"position" : {
"offset" : 79,
"start" : 0,
"end" : 98
},
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
}
}
}
]
},
"status" : 400
}
Error From ES for Query 2:
{
"error" : {
"root_cause" : [
{
"type" : "exception",
"reason" : "Missing value for field [service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType]"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "doctor_idx",
"node" : "mvh5k24dQPqM-d7JVeNomQ",
"reason" : {
"type" : "exception",
"reason" : "Missing value for field [service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType]"
}
}
]
},
"status" : 500
}
The only record I have in the ES
{
"took" : 32,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "doctor_idx",
"_type" : "_doc",
"_id" : "xM20oH0Bmp1zsT0w8pQe",
"_score" : 1.0,
"_source" : {
"_class" : "com.insig.clinic_apps.services_server.booking.dao.DoctorSchema",
"id" : "xM20oH0Bmp1zsT0w8pQe",
"metadata" : {
"doctorId" : "xhnz2lGvXxelWyVekz82c2v6Srb2",
"fullName" : "Dave Insig Email Test",
"credentials" : [
"MD"
],
"languagesSpoken" : [
"EN",
"NOT_DEFINED"
],
"specialitiesServed" : [ ],
"city" : "VAUGHAN",
"provincesPermitted" : [ ],
"province" : "ON",
"country" : "CANADA"
},
"service" : {
"servicesMedium" : [
"IN_CLINIC",
"VIDEO",
"MESSAGING",
"PHONE"
],
"servicesTypeDuration" : [
{
"serviceType" : "Acne Symptoms",
"duration" : 5,
"servicesMedium" : [
"IN_CLINIC",
"MESSAGING",
"PHONE",
"VIDEO"
]
}
],
"serviceTypeEarliestAvailability" : [
{
"serviceType" : "Acne Symptoms",
"earliestAvailableDateTimeByType" : "2021-12-09T19:39:16",
"servicesMedium" : [
"IN_CLINIC",
"MESSAGING",
"PHONE",
"VIDEO"
]
}
],
"bufferTimeForNextAvailability" : 0
},
"earliestAvailableDateTime" : "2021-12-09T19:39:16",
"patientRating" : 4.384481,
"onTimeRating" : 3.171053
}
}
]
}
}

/!\ Partial solution /!\
Query 1
When I was trying to execute your query I got this error:
{
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "dynamic method [org.elasticsearch.script.JodaCompatibleZonedDateTime, toEpochMilli/0] not found"
}
}
So I did a slight change to your query .toInstant().toEpochMilli()
GET /so_custom_score/_search
{
"explain": true,
"query": {
"nested": {
"path": "service",
"query": {
"nested": {
"score_mode": "max",
"path": "service.serviceTypeEarliestAvailability",
"query": {
"function_score": {
"query": {
"match_all": {
"boost": 1
}
},
"functions": [
{
"filter": {
"match": {
"service.serviceTypeEarliestAvailability.serviceType": "type"
}
},
"weight": 10
},
{
"script_score": {
"script": {
"source": """
def availability = doc['service.serviceTypeEarliestAvailability.earliestAvailableDateTimeByType'].value;
return availability.toInstant().toEpochMilli();
"""
}
}
}
]
}
}
}
}
}
}
}

Well, wasted a lot of time here and figured that there was no problem with the queries. Elasticsearch v7.10.2 has an issue when I try to get the explanation of the query.
It works perfectly fine without the explain parameter.
See the at org.elasticsearch.common.lucene.search.function.FieldValueFactorFunction$1.explainScore(FieldValueFactorFunction.java:103) ~[elasticsearch-7.10.2.jar:7.10.2]
PS: One of the comments in the OG question mentioned that the latest Elasticsearch v7.15 works fine even with explain parameter.

aggregation_execution_exception : Invalid aggregation order path,Sub-path points to non single-bucket aggregation

when i run es aggration :
"aggregations": {
"author": {
"terms": {
"field": "author",
"size": 100,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": {
"interactions-c>interactions-sum": "desc"
}
},
"aggregations": {
"interactions-c": {
"children": {
"type": "interactions"
},
"aggregations": {
"interactions-sum": {
"sum": {
"field": "interactions.likes"
}
}
}
}
}
}
}
exception:
{
"error" : {
"root_cause" : [
{
"type" : "aggregation_execution_exception",
"reason" : "Invalid aggregation order path [interactions-c>interactions-sum]. Buckets can only be sorted on a sub-aggregator path that is built out of zero or more single-bucket aggregations within the path and a final single-bucket or a metrics aggregation at the path end. Sub-path [interactions-c] points to non single-bucket aggregation"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "article_20200910",
"node" : "fLYvCQjfTEKG0QIivtn3Hg",
"reason" : {
"type" : "aggregation_execution_exception",
"reason" : "Invalid aggregation order path [interactions-c>interactions-sum]. Buckets can only be sorted on a sub-aggregator path that is built out of zero or more single-bucket aggregations within the path and a final single-bucket or a metrics aggregation at the path end. Sub-path [interactions-c] points to non single-bucket aggregation"
}
}
]
},
"status" : 500
}
this is my index mapping:
{
"article" : {
"aliases" : { },
"mappings" : {
"properties" : {
"author" : {
"type" : "keyword"
},
"interactions" : {
"properties" : {
"comments" : {
"type" : "long"
},
"dislikes" : {
"type" : "long"
},
"forwards" : {
"type" : "long"
},
"likes" : {
"type" : "long"
},
"views" : {
"type" : "long"
}
}
},
"joinField" : {
"type" : "join",
"eager_global_ordinals" : false,
"relations" : {
"article" : [
"interactions"
]
}
}
}
}
}
}
i create a index which use join field (parent :article ;children :interations )
what i want:
aggreate interations number by author (author is a parent index field,interations is a children field)
then order by interations sum value desc
but es says childern aggs is not a single bucket aggs!So is there any way to this?

Elasticsearch Aggregation sorting

My Elasticsearch mapping is
{
"mappings" : {
"loc" : {
"dynamic": "true",
"properties" : {
"geoip" : {
"properties" : {
"location" : { "type": "geo_point"}
}
},
"lon" : { "type" : "double" },
"lat" : { "type" : "double" },
"altitude" : { "type" : "double" },
"id" : { "type" : "long" },
"date" : { "type" : "date", "format" : "epoch_millis" },
"ip" : { "type" : "string" },
"port" : { "type" : "string" }
}
}
}
}
And I want to sort by time.
So i made query.
{
"query": {
"bool" : {
"must" : {
"match_all" : {}
},
"filter" : {
"geo_distance" : {
"distance" : "0.2km",
"geoip.location" : {
"lat" : 36.773353,
"lon" : 126.933847
}
}
}
}
},
"size" : 0,
"sort" : { "date" : { "order" : "desc" } },
"aggs" : {
"ids" : {
"terms" : {
"field" : "id"
},
"aggs" : {
"dedup_docs" : {
"top_hits" : {"size" : 1}
}
}
}
}
}
I want to return the latest time by grouping the results of applying the gps filter by id and sorting in chronological order.
However, the date value of the result is an unordered result.
I do not know how to modify the query.

How to index percolator queries containing filters on inner objects?

Using Elasticsearch 2.1.1
I have documents with inner objects:
{
"level1": {
"level2": 42
}
}
I want to register percolator queries applying filters on the inner property:
$ curl -XPUT http://localhost:9200/myindex/.percolator/myquery?pretty -d '{
"query": {
"filtered": {
"filter": {
"range": {
"level1.level2": {
"gt": 10
}
}
}
}
}
}'
It fails because I don't have a mapping:
{
"error" : {
"root_cause" : [ {
"type" : "query_parsing_exception",
"reason" : "Strict field resolution and no field mapping can be found for the field with name [level1.level2]",
"index" : "myindex",
"line" : 1,
"col" : 58
} ],
"type" : "percolator_exception",
"reason" : "failed to parse query [myquery]",
"index" : "myindex",
"caused_by" : {
"type" : "query_parsing_exception",
"reason" : "Strict field resolution and no field mapping can be found for the field with name [level1.level2]",
"index" : "myindex",
"line" : 1,
"col" : 58
}
},
"status" : 500
}
So I start again, but this time I add a mapping template before:
curl -XDELETE http://localhost:9200/_template/myindex
curl -XDELETE http://localhost:9200/myindex
curl -XPUT http://localhost:9200/_template/myindex?pretty -d 'x
{
"template": "myindex",
"mappings" : {
"mytype" : {
"properties" : {
"level1" : {
"properties" : {
"level2" : {
"type" : "long"
}
}
}
}
}
}
}
'
I try to register my percolator query again:
curl -XPUT http://localhost:9200/myindex/.percolator/myquery?pretty -d '{
"query": {
"filtered": {
"filter": {
"range": {
"level1.level2": {
"gt": 10
}
}
}
}
}
}'
And now it succeeds:
{
"_index" : "myindex",
"_type" : ".percolator",
"_id" : "myquery",
"_version" : 1,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"created" : true
}
And I can see the mapping that has been created:
curl http://localhost:9200/myindex/_mapping?pretty
{
"myindex" : {
"mappings" : {
".percolator" : {
"properties" : {
"query" : {
"type" : "object",
"enabled" : false
}
}
},
"mytype" : {
"properties" : {
"level1" : {
"properties" : {
"level2" : {
"type" : "long"
}
}
}
}
}
}
}
}
Now my problem is that I also need to perform searches on my percolator queries and the default percolate mapping doesn’t index the query field.
So I start again, this time specifying in my mapping template that I want percolator queries to be indexed (note "enabled": true):
curl -XPUT http://localhost:9200/_template/myindex?pretty -d '
{
"template": "myindex",
"mappings" : {
".percolator" : {
"properties" : {
"query" : {
"type" : "object",
"enabled" : true
}
}
},
"mytype" : {
"properties" : {
"level1" : {
"properties" : {
"level2" : {
"type" : "long"
}
}
}
}
}
}
}
'
I try to register my percolator query again:
curl -XPUT http://localhost:9200/myindex/.percolator/myquery?pretty -d '{
"query": {
"filtered": {
"filter": {
"range": {
"level1.level2": {
"gt": 10
}
}
}
}
}
}'
But now I get an error:
{
"error" : {
"root_cause" : [ {
"type" : "mapper_parsing_exception",
"reason" : "Field name [level1.level2] cannot contain '.'"
} ],
"type" : "mapper_parsing_exception",
"reason" : "Field name [level1.level2] cannot contain '.'"
},
"status" : 400
}
How can I create and index a percolator query matching an inner property?

Resources