I'm trying to create a script field that will calculate a time difference between two timestamps and then aggregate an avg on that script field.
I first tried:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"and": [
{
"exists": {
"field": "time.new_time"
}
},
{
"exists": {
"field": "time.first_alert_time"
}
}
]
}
}
},
"script_fields": {
"timedifference": {
"script": "doc['time.new_time'].value - doc['time.first_alert_time'].value"
}
},
"aggs": {
"avg_timedifference": {
"avg": {
"field" : "timedifference"
}
}
}
}
Which resulted in null value under the aggregated avg avg_timedifference.
Then I tried:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"and": [
{
"exists": {
"field": "time.new_time"
}
},
{
"exists": {
"field": "time.first_alert_time"
}
}
]
}
}
},
"script_fields": {
"timedifference": {
"script": "doc['time.new_time'].value - doc['time.first_alert_time'].value"
}
},
"aggs": {
"avg_timedifference": {
"avg": {
"script" : "doc['timedifference'].value"
}
}
}
}
Which generated an error message saying: "No field found for [timedifference] in mapping"
How about simply moving the script to the aggregation?
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"and": [
{
"exists": {
"field": "time.new_time"
}
},
{
"exists": {
"field": "time.first_alert_time"
}
}
]
}
}
},
"aggs": {
"avg_timedifference": {
"avg": {
"script" : "Math.ceil(doc['time.new_time'].value - doc['time.first_alert_time'].value)"
}
}
}
}
Related
recently I am working with the ELK stack
where I have an index with docs the has the following properties "name, value, date"
and I have performed some aggregations on the data using elasticsearch query
like bellow:
GET abcd/_search
{
"aggs": {
"per_date": {
"date_histogram": {
"field": "DATE",
"calendar_interval": "month"
},
"aggs": {
"succ": {
"filter": {
"bool": {
"must": [
{
"term": {
"Name": "some name"
}
}
]
}
},
"aggs": {
"sum_init": {
"sum": {
"field": "value"
}
}
}
},
"init": {
"filter": {
"bool": {
"must": [
{
"term": {
"Name.keyword": "some other name "
}
}
]
}
},
"aggs": {
"sum_init": {
"sum": {
"field": "value"
}
}
}
},
"ccn_kpi": {
"bucket_script": {
"buckets_path": {
"succ_req": "succ>sum_init",
"total_req": "init>sum_init"
},
"script": "params.succ_req / params.total_req * 100 "
}
}
}
}
}
}
what I need is a way to store the result of the query in a new index and want this operation keep going as a new data coming in... any advice would help
I need to get the last document of each interface, I have played around with different queries but I can get the desired result, below is my las attempt.
Can you help me to get the last document of each interface where the field throughput exist?
Thanks
GET /interface-2021.11/_search
{
"query": {
"bool": {
"should": [
{
"term": {
"interface_name.keyword": {
"value": "Gi0/0/2 on (EXT-01)"
}
}
},
{
"term": {
"interface_name.keyword": {
"value": "Gi0/0/1 on (EXT-02)"
}
}
},
{
"term": {
"interface_name.keyword": {
"value": "Ethernet1/61 on (DC-01)"
}
}
},
{
"term": {
"interface_name.keyword": {
"value": "Ethernet1/17 on (DC-02)"
}
}
}
],
"minimum_should_match": 1,
"filter": [
{
"exists": {
"field": "throughput"
}
}
]
}
},
"aggs": {
"top_date": {
"top_hits": {
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
]
}
}
}
}
Good job, you're on the right path! You just need to aggregate by interface_name.keyword and get the top hit for each interface.
Here is the query that will work as you expect:
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"terms": {
"interface_name.keyword": [
"Gi0/0/2 on (EXT-01)",
"Gi0/0/1 on (EXT-02)",
"Ethernet1/61 on (DC-01)",
"Ethernet1/17 on (DC-02)"
]
}
},
{
"exists": {
"field": "throughput"
}
}
]
}
},
"aggs": {
"interfaces": {
"terms": {
"field": "interface_name.keyword"
},
"aggs": {
"top_date": {
"top_hits": {
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
]
}
}
}
}
}
}
I have query like below. It returns 320 results for the below condition-
{
"size": "5000",
"sort": [
{
"errorDateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"range": {
"errorDateTime": {
"gte": "2021-04-07T20:08:20.516",
"lte": "2021-04-08T00:08:20.516"
}
}
},
{
"bool": {
"should": [
{
"match": {
"businessFunction": "PriceUpdate"
}
},
{
"match": {
"businessFunction": "PriceFeedIntegration"
}
},
{
"match": {
"businessFunction": "StoreConnectivity"
}
},
{
"match": {
"businessFunction": "Transaction"
}
},
{
"match": {
"businessFunction": "SalesSummary"
}
}
]
}
}
]
}
},
"aggs": {
"genres_and_store": {
"terms": {
"field": "storeId"
},
"aggs": {
"genres_and_error": {
"terms": {
"field": "errorCode"
},
"aggs": {
"genres_and_business": {
"terms": {
"field": "businessFunction"
}
}
}
}
}
}
}
}
However the aggregation results are not matching. I have so many stores which are not returned in aggregation but I can see them in query result. What am I missing? My schema looks like -
{
"errorDescription": "FTP Service unable to connect to Store to list the files for Store 12345",
"errorDateTime": "2021-04-07T21:01:15.040546",
"readBy": [],
"errorCode": "e004",
"businessFunction": "TransactionError",
"storeId": "12345"
}
Please let me know if I am writing the query wrong. I want to aggregare per store, per errorcode and per businessFunction.
If no size param is set in the terms aggregation, then by default it returns the top 10 terms, which are ordered by their doc_count. You need to add the size param in the terms aggregation, to get all the matching total hits.
Try out the below query
{
"size": "5000",
"sort": [
{
"errorDateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"range": {
"errorDateTime": {
"gte": "2021-04-07T20:08:20.516",
"lte": "2021-04-08T00:08:20.516"
}
}
},
{
"bool": {
"should": [
{
"match": {
"businessFunction": "PriceUpdate"
}
},
{
"match": {
"businessFunction": "PriceFeedIntegration"
}
},
{
"match": {
"businessFunction": "StoreConnectivity"
}
},
{
"match": {
"businessFunction": "Transaction"
}
},
{
"match": {
"businessFunction": "SalesSummary"
}
}
]
}
}
]
}
},
"aggs": {
"genres_and_store": {
"terms": {
"field": "storeId",
"size": 100 // note this
},
"aggs": {
"genres_and_error": {
"terms": {
"field": "errorCode"
},
"aggs": {
"genres_and_business": {
"terms": {
"field": "businessFunction"
}
}
}
}
}
}
}
}
I think I was missing size parameter inside aggs and was getting default 10 aggregations only:
"aggs": {
"genres_and_store": {
"terms": {
"field": "storeId",
"size": 1000
},
I am getting a reason": "[query] query malformed, no start_object after query name" error, not sure why.
The query is meant to grab the difference between two date fields and calculate average of all the results, I believe this should work but it may not work.
Any help would be greatly appreciated.
I am on elastic version 5.6.12
query below:
POST index_my.test/_search
{
"size":10,
"query": {
"bool": {
"must": [
{
"query":
"match_all": {}
}
}
]
}
"filter": {
"and": [
{
"exists": {
"field": "activity.timeline.found"
}
}
{
"exists": {
"field": "activity.timeline.sent"
}
}
]
},
"aggs": {
"avg_timedifference": {
"avg": {
"script" : "Math.ceil(doc['activity.timeline.found'].value - doc['activity.timeline.sent'].value)"
}
}
}
}
You forgot a comma before "filter". Try this:
POST index_my.test/_search
{
"size":10,
"query": {
"bool": {
"must": [
{
"query":
"match_all": {}
}
}
]
},
"filter": {
"and": [
{
"exists": {
"field": "activity.timeline.found"
}
}
{
"exists": {
"field": "activity.timeline.sent"
}
}
]
},
"aggs": {
"avg_timedifference": {
"avg": {
"script" : "Math.ceil(doc['activity.timeline.found'].value - doc['activity.timeline.sent'].value)"
}
}
}
}
I have the following query in Elasticsearch:
{
"script_fields": {
"travel_time": {
"script": {
"inline": "doc['DateTo'].value - doc['DateFrom'].value"
}
}
},
"stored_fields": [
"_source"
],
"query": {
"bool": {
"filter": {
"exists": {
"field": "DateTo"
}
}
}
}
}
How can I add DateFrom into exists filter?
You can add multiple exists criteria:
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "DateFrom"
}
},
{
"exists": {
"field": "DateTo"
}
},
{
"script": {
"script": {
"inline": "doc['DateTo'].value - doc['DateFrom'].value > 0"
}
}
}
]
}
}