I have documents in the following style in my index:
{
"docType": {
"valuesOverTime": [
{
"begin": 1488442858570,
"end": 1488442860570,
"values": [
{
"name": "level",
"segCount": 4
}
]
},
{
"begin": 1488442860571,
"end": 1488442890592,
"mcdn": [
{
"name": "level",
"segCount": 10
}
]
},
{
"begin": 1488442890593,
"end": 1488442890600,
"mcdn": [
{
"name": "level",
"segCount": 7
}
]
}
]
}
}
and want to query the sum of the docType.valuesOverTime.values.segCount in a certain time range, like the following range
{
"range": {
"docType.valuesOverTime.begin": {
"gte": 1488442858570,
"lte": 1488442860571
}
}
},
{
"range": {
"docType.valuesOverTime.end": {
"gte": 1488442860570,
"lte": 1488442890592
}
}
}
should get me the sum of the first two entries: 14.
However I am absolutely stuck getting the query right! I always get the sum of all the entries in docType.valueOverTime.values.segCount being 21 in this case.
I tried the following queries and some variations on them which where of course all wrong:
{
"size": 0,
"aggs": {
"myfilter": {
"filter": {
"bool": {
"must": [
{
"range": {
"docType.valuesOverTime.begin": {
"gte": 1488442858570,
"lte": 1488442860571
}
}
},
{
"range": {
"docType.valuesOverTime.end": {
"gte": 1488442860570,
"lte": 1488442890592
}
}
}
]
}
},
"aggs": {
"summe": {
"sum": {
"field": "docType.valuesOverTime.values.segCount"
}
}
}
}
}
}
and
{
"_source": "docType.valuesOverTime.values",
"query": {
"constant_score" : {
"filter" : {
"bool": {
"must": [
{
"range": {
"docType.valuesOverTime.begin": {
"gte": 1488442858570,
"lte": 1488442860571
}
}
},
{
"range": {
"docType.valuesOverTime.end": {
"gte": 1488442860570,
"lte": 1488442890592
}
}
}
]
}
}
}
},
"aggs": {
"summe": {
"sum": {
"field": "docType.valuesOverTime.values.segCount"
}
}
}
}
Can someone please tell me, what I got wrong? And how to do it right!
Working on es 5.2.2 with the following mapping
"valuesOverTime": {
"properties": {
"begin": {
"type": "long"
},
"end": {
"type": "long"
},
"values": {
"properties": {
"name": {
"type": "keyword"
},
"segCount": {
"type": "long"
}
}
}
}
}
I solved it myself:
first, everything in valuesOverTime needs to be nested, so that the mapping looks like
"valuesOverTime": {
"type": nested"
"properties": {
"begin": {
"type": "long"
},
"end": {
"type": "long"
},
"values": {
"properties": {
"name": {
"type": "keyword"
},
"segCount": {
"type": "long"
}
}
}
}
}
Then I can query as follows:
{
"size": 0,
"aggs": {
"nestedAcq": {
"nested": {"path": "docType.valuesOverTime"},
"aggs": {
"rangeAcq": {
"range": {
"field": "ocType.valuesOverTime.begin",
"ranges": [
{
"from": 1488442858570,
"to": 1488442860572
}
]
},
"aggs": {
"theSum": {
"sum": {
"field": "docType.valuesOverTime.values.segCount"
}
}
}
}
}
}
}
}
and get
"theSum": {
"value": 14
}
Related
I can use aggregate to make some stats between two timestamps as following:
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"status": "ok"
}
},
{
"term": {
"deviceId": "123456789"
}
},
{
"range": {
"time": {
"gte": 1669852800,
"lt": 1671062400
}
}
}
]
}
},
"aggs": {
"results": {
"date_histogram": {
"field": "time",
"fixed_interval": "60",
}
}
}
}
Is it possible to query the results contain specific time range daily only? For example, 7am - 9am daily between Dec.1 to Dec.15. How to achieve it?
I found the solution on elasticsearch v7.15.2 as following:
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"status": "ok"
}
},
{
"term": {
"deviceId": "123456789"
}
},
{
"range": {
"time": {
"gte": 1669852800,
"lt": 1671062400
}
}
},
{
"script": {
"script": {
"source": "doc.time.value.getHourOfDay() >= params.min && doc.time.value.getHourOfDay() < params.max",
"params": {
"min": 8,
"max": 10
}
}
}
}
]
}
},
"aggs": {
"results": {
"date_histogram": {
"field": "time",
"fixed_interval": "60"
}
}
}
}
The syntax is slightly different from the comment above, but it works.
Need help to understand sorting in a query where _score is either 0 or 1 based on some conditions
but there is also an additional_filed type integer, sort should be in the order of _score * additional_field
sample index
PUT /product_t
{
"mappings": {
"properties": {
"name": {
"type": "text"
},
"placed" :{
"type": "integer"
},
"store" : {
"type": "nested"
}
}
}
}
sample document
PUT /product_t/_doc/10
{
"name": "awesome",
"count_sold": 199,
"stock": [
{
"id": 1,
"count": 10
},
{
"id": 2,
"count": 5
},
{
"id": 3,
"count": 0
}
]
}
Query Used :
POST /product_t/_search
{
"from": 0,
"size": 100,
"timeout": "300ms",
"query": {
"bool": {
"filter": [
{
"match": {
"name": {
"value": "awesome"
}
}
},
{
"nested": {
"path": "stock",
"query": {
"bool": {
"must": [
{
"match": {
"stock.id": 3
}
}
]
}
}
}
}
],
"should": [
{
"constant_score": {
"filter": {
"nested": {
"path": "stock",
"query": {
"bool": {
"must": [
{
"match": {
"stock.id": 3
}
},
{
"range": {
"stock.count": {
"gt": 0
}
}
}
]
}
}
}
},
"boost": 1
}
},
{
"constant_score": {
"filter": {
"nested": {
"path": "stock",
"query": {
"bool": {
"must": [
{
"match": {
"stock.id": 3
}
},
{
"range": {
"stock.count": {
"lte": 0
}
}
}
]
}
}
}
},
"boost": 0
}
}
]
}
}
}
count_sold is additional field
You can use script sort
"sort": {
"_script": {
"type": "number",
"script": {
"lang": "painless",
"source": "doc['count_sold'].value * _score"
},
"order": "asc"
}
}
Suppose i want this structure for a document:
{
"hours": {
"open": [
{
"start": 10,
"end": 19
},
{
"start": 21,
"end": 29
}
...
],
"closed": [
{
"start": 100,
"end": 199
},
{
"start": 201,
"end": 299
}
...
]
}
}
whose index has this mapping:
{
"mappings": {
"_doc": {
"properties": {
"hours": {
"properties": {
"open": {
"type": "nested",
"properties": {
"start": { "type": "integer" },
"end": { "type": "integer" }
}
},
"closed": {
"type": "nested",
"properties": {
"start": { "type": "integer" },
"end": { "type": "integer" }
}
}
}
}
}
}
}
}
In the Elasticsearch Query DSL, how do i find all documents where 20 lies inside an open segment and not inside a closed segment. The query I tried was incorrect.
failed query
{
"query": {
"bool": {
"must": [
{
"bool": {
"must": [
{
"nested": {
"path": "hours.open",
"query": {
"range": {
"hours.open.start": { "lte": 20 }
}
}
}
},
{
"nested": {
"path": "hours.open",
"query": {
"range": {
"hours.open.end": { "gte": 20 }
}
}
}
}
]
}
},
{
"bool": {
"must_not": [
{
"bool": {
"must": [
{
"nested": {
"path": "hours.closed",
"query": {
"range": {
"hours.closed.start": { "lte": 20 }
}
}
}
},
{
"nested": {
"path": "hours.closed",
"query": {
"range": {
"hours.closed.end": { "gte": 20 }
}
}
}
}
]
}
}
]
}
}
]
}
}
}
whats wrong with my query? it is returning this document which is not what i intended. 20 does not lie inside an open segment.
I finally got it working. The following is the correct query:
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "hours.open",
"query": {
"bool": {
"must": [
{ "range": { "hours.open.start": { "lte": 20 } } },
{ "range": { "hours.open.end": { "gte": 20 } } }
]
}
}
}
}
],
"must_not": [
{
"nested": {
"path": "hours.closed",
"query": {
"bool": {
"must": [
{ "range": { "hours.closed.start": { "lte": 20 } } },
{ "range": { "hours.closed.end": { "gte": 20 } } }
]
}
}
}
}
]
}
}
}
With that said, it looks like my original attempt was wrong because there were two different hours.open nested path queries and likewise two different hours.closed nested path queries. The parser must only take one of them for a single path.
Seems like you need to swap lte and gte:
"hours.open.start": { "gte": 20 }
"hours.open.end": { "lte": 20 }
and same for the closing times:
"hours.closed.start": { "gte": 20 }
"hours.closed.end": { "lte": 20 }
Edit:
The must and must_not need to be part of the same bool query:
{
"query": {
"bool": {
"must": [{
"nested": {
"path": "hours.open",
"query": {
"range": {
"hours.open.start": {
"gte": 20
}
}
}
}
},
{
"nested": {
"path": "hours.open",
"query": {
"range": {
"hours.open.end": {
"lte": 20
}
}
}
}
}
],
"must_not": [{
"bool": {
"must": [{
"nested": {
"path": "hours.closed",
"query": {
"range": {
"hours.closed.start": {
"gte": 20
}
}
}
}
},
{
"nested": {
"path": "hours.closed",
"query": {
"range": {
"hours.closed.end": {
"lte": 20
}
}
}
}
}
]
}
}]
}
}
}
I am not able to perform nested aggregation in a specific date range.
Ideally, I would want to get % values of two fields for the previous day and the current day. Here's the JSON (this is a part of the elastic watcher and not the whole config):
{
"metadata":{
"threshold":5,
"interval":"2m",
"window":"2d"
},
"trigger":{
"schedule":{
"interval":"2m"
}
},
"input":{
"search":{
"request":{
"indices":[
"filebeat-*"
],
"types":[
"doc"
],
"body": {
"aggs": {
"aggs1": {
"range": {
"date_range": {
"ranges": [
{
"from": "now-2d/d"
},
{
"to": "now-2d/d"
}
]
},
"aggs": {
"max": {
"script": {
"source": "(doc['upstream'].value\/100)"
}
}
}
}
},
"aggs2": {
"range": {
"date_range": {
"ranges": [
{
"from": "now-2d/d"
},
{
"to": "now-2d/d"
}
]
}
},
"aggs": {
"max": {
"script": {
"source": "(doc['downstream'].value\/100)"
}
}
}
},
"aggs3": {
"range": {
"date_range": {
"ranges": [
{
"from": "now-1d/d"
},
{
"to": "now/d"
}
]
}
},
"aggs": {
"max": {
"script": {
"source": "(doc['upstream'].value\/100)"
}
}
}
},
"aggs4": {
"range": {
"date_range": {
"ranges": [
{
"from": "now-1d/d"
},
{
"to": "now/d"
}
]
}
},
"aggs": {
"max": {
"script": {
"source": "(doc['downstream'].value\/100)"
}
}
}
}
},
"query": {
"bool": {
"filter": {
"range": {
"#timestamp": {
"lte": "now",
"gte": "now-{{ctx.metadata.window}}"
}
}
}
}
}
}
}
}
}
}
All I want is to have 4 values for two fields for current day and previous day so that I can get the difference between the values for further procesing.
Thanks.
PUT _xpack/watcher/watch/log_error_watch
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"search": {
"request": {
"indices": [
"filebeat-2017.01.02"
],
"body": {
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
],
"query": {
"range": {
"offset": {
"gte": 1000,
"lte": 2000
}
},
"match": {
"source": "/var/log/apache2/access.log"
}
},
"size": 5
}
}
}
}
}
[o.e.m.j.JvmGcMonitorService] [hj-test156] [gc][11042] overhead, spent [701ms] collecting in the last [1s]
[2017-01-02T15:32:04,311][ERROR][o.e.x.w.i.s.ExecutableSimpleInput] [hj-test156] failed to execute [search] input for watch [log_error_watch], reason [[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]]
Your query is malformed, you need to write it like this
...
"query": {
"bool": {
"must": [
{
"range": {
"offset": {
"gte": 1000,
"lte": 2000
}
}
},
{
"match": {
"source": "/var/log/apache2/access.log"
}
}
]
}
}
},
...
UPDATE
For a range with a date field you can do it like this:
{
"range": {
"#timestamp": {
"gte": "2017-01-02T05:23:34.731Z",
"lte": "2017-01-03T05:23:34.731Z"
}
}
},