I'm facing some errors with DSL query builder and aggregations.
Tried several approaches and none of them seem to work.
If I remove aggs clause, the query works seamlessly.
Queries below return error: [bool] malformed query, expected [END_OBJECT] but found [FIELD_NAME]
{
"query": {
"bool": {
"filter": [
{
"range": {
"json.#timestamp": {
"gt": "2021-08-22T00:00:00.000Z",
"lt": "2022-10-22T13:41:09.000Z"
}
}
},
{
"term": {
"json.path": "/api/v1/discover"
}
},
{
"wildcard": {
"container.image.name": {
"value": "*prod*"
}
}
}
]
}
},
"aggs": {
"totalCount": {
"sum": {
"field": "count"
}
}
}
}
Using aggs inside body also does not work.
{
"query": {
"bool": {
"filter": [
{
"range": {
"json.#timestamp": {
"gt": "2021-08-22T00:00:00.000Z",
"lt": "2022-10-22T13:41:09.000Z"
}
}
},
{
"term": {
"json.path": "/api/v1/discover"
}
},
{
"wildcard": {
"container.image.name": {
"value": "*prod*"
}
}
}
]
}
},
"body": {
"aggs": {
"group_by_id": {
"terms": {
"field": "cloud.image.id"
}
}
}
}
}
Not even a basic aggs example will succeed.
{
"query": {
"match_all": {}
},
"aggs": {
"objects": {
"terms": {
"field": "json.path"
}
}
}
}
This one returns error: [1:16806] unknown field [aggs]
{
"query": {
"aggs": {
"my-agg-name": {
"terms": {
"field": "json.path"
}
}
}
}
}
What am I doing wrong?
I'm on Elastic Cloud v7.16.2
Just found out what the problem is... Aggregations will only work on Dev Tools page. It will not work on Discover page Seach box.
Related
recently I am working with the ELK stack
where I have an index with docs the has the following properties "name, value, date"
and I have performed some aggregations on the data using elasticsearch query
like bellow:
GET abcd/_search
{
"aggs": {
"per_date": {
"date_histogram": {
"field": "DATE",
"calendar_interval": "month"
},
"aggs": {
"succ": {
"filter": {
"bool": {
"must": [
{
"term": {
"Name": "some name"
}
}
]
}
},
"aggs": {
"sum_init": {
"sum": {
"field": "value"
}
}
}
},
"init": {
"filter": {
"bool": {
"must": [
{
"term": {
"Name.keyword": "some other name "
}
}
]
}
},
"aggs": {
"sum_init": {
"sum": {
"field": "value"
}
}
}
},
"ccn_kpi": {
"bucket_script": {
"buckets_path": {
"succ_req": "succ>sum_init",
"total_req": "init>sum_init"
},
"script": "params.succ_req / params.total_req * 100 "
}
}
}
}
}
}
what I need is a way to store the result of the query in a new index and want this operation keep going as a new data coming in... any advice would help
I need to get the last document of each interface, I have played around with different queries but I can get the desired result, below is my las attempt.
Can you help me to get the last document of each interface where the field throughput exist?
Thanks
GET /interface-2021.11/_search
{
"query": {
"bool": {
"should": [
{
"term": {
"interface_name.keyword": {
"value": "Gi0/0/2 on (EXT-01)"
}
}
},
{
"term": {
"interface_name.keyword": {
"value": "Gi0/0/1 on (EXT-02)"
}
}
},
{
"term": {
"interface_name.keyword": {
"value": "Ethernet1/61 on (DC-01)"
}
}
},
{
"term": {
"interface_name.keyword": {
"value": "Ethernet1/17 on (DC-02)"
}
}
}
],
"minimum_should_match": 1,
"filter": [
{
"exists": {
"field": "throughput"
}
}
]
}
},
"aggs": {
"top_date": {
"top_hits": {
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
]
}
}
}
}
Good job, you're on the right path! You just need to aggregate by interface_name.keyword and get the top hit for each interface.
Here is the query that will work as you expect:
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"terms": {
"interface_name.keyword": [
"Gi0/0/2 on (EXT-01)",
"Gi0/0/1 on (EXT-02)",
"Ethernet1/61 on (DC-01)",
"Ethernet1/17 on (DC-02)"
]
}
},
{
"exists": {
"field": "throughput"
}
}
]
}
},
"aggs": {
"interfaces": {
"terms": {
"field": "interface_name.keyword"
},
"aggs": {
"top_date": {
"top_hits": {
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
]
}
}
}
}
}
}
I have query like below. It returns 320 results for the below condition-
{
"size": "5000",
"sort": [
{
"errorDateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"range": {
"errorDateTime": {
"gte": "2021-04-07T20:08:20.516",
"lte": "2021-04-08T00:08:20.516"
}
}
},
{
"bool": {
"should": [
{
"match": {
"businessFunction": "PriceUpdate"
}
},
{
"match": {
"businessFunction": "PriceFeedIntegration"
}
},
{
"match": {
"businessFunction": "StoreConnectivity"
}
},
{
"match": {
"businessFunction": "Transaction"
}
},
{
"match": {
"businessFunction": "SalesSummary"
}
}
]
}
}
]
}
},
"aggs": {
"genres_and_store": {
"terms": {
"field": "storeId"
},
"aggs": {
"genres_and_error": {
"terms": {
"field": "errorCode"
},
"aggs": {
"genres_and_business": {
"terms": {
"field": "businessFunction"
}
}
}
}
}
}
}
}
However the aggregation results are not matching. I have so many stores which are not returned in aggregation but I can see them in query result. What am I missing? My schema looks like -
{
"errorDescription": "FTP Service unable to connect to Store to list the files for Store 12345",
"errorDateTime": "2021-04-07T21:01:15.040546",
"readBy": [],
"errorCode": "e004",
"businessFunction": "TransactionError",
"storeId": "12345"
}
Please let me know if I am writing the query wrong. I want to aggregare per store, per errorcode and per businessFunction.
If no size param is set in the terms aggregation, then by default it returns the top 10 terms, which are ordered by their doc_count. You need to add the size param in the terms aggregation, to get all the matching total hits.
Try out the below query
{
"size": "5000",
"sort": [
{
"errorDateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"range": {
"errorDateTime": {
"gte": "2021-04-07T20:08:20.516",
"lte": "2021-04-08T00:08:20.516"
}
}
},
{
"bool": {
"should": [
{
"match": {
"businessFunction": "PriceUpdate"
}
},
{
"match": {
"businessFunction": "PriceFeedIntegration"
}
},
{
"match": {
"businessFunction": "StoreConnectivity"
}
},
{
"match": {
"businessFunction": "Transaction"
}
},
{
"match": {
"businessFunction": "SalesSummary"
}
}
]
}
}
]
}
},
"aggs": {
"genres_and_store": {
"terms": {
"field": "storeId",
"size": 100 // note this
},
"aggs": {
"genres_and_error": {
"terms": {
"field": "errorCode"
},
"aggs": {
"genres_and_business": {
"terms": {
"field": "businessFunction"
}
}
}
}
}
}
}
}
I think I was missing size parameter inside aggs and was getting default 10 aggregations only:
"aggs": {
"genres_and_store": {
"terms": {
"field": "storeId",
"size": 1000
},
I`m building product list page with filters. There a lot of filters, and data for them are counting in ES with aggregation functions.
Simplest example if min/max price:
{
"size": 0,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"shop_id": 44
}
},
{
"term": {
"CategoryId": 36898
}
},
{
"term": {
"products_status": 1
}
},
{
"term": {
"availability": 3
}
}
]
}
}
}
},
"aggs": {
"min_price": {
"min": {
"field": "products_price"
}
},
"max_price": {
"max": {
"field": "products_price"
}
}
}
}
So, this request in ES return me minimal and maximal price according rules installed in filter (category_id 36898, shop_id 44 etc).
It is working perfect.
The question is: is it possible to update this request and get aggregations without filters? Or is it maybe possible to return aggregation data with another filter in one request?
So I want:
min_price and max_price for filtered data (query1)
and mix_price and max_price for unfiltered data (or filtered data with query 2)?
You can use global option for the aggregations to not applying any filters provided in query block.
For example, for your query use the following json input.
{
"size": 0,
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"shop_id": 44
}
},
{
"term": {
"CategoryId": 36898
}
},
{
"term": {
"products_status": 1
}
},
{
"term": {
"availability": 3
}
}
]
}
}
}
},
"aggs": {
"min_price": {
"min": {
"field": "products_price"
}
},
"max_price": {
"max": {
"field": "products_price"
}
},
"without_filter_min": {
"global": {},
"aggs": {
"price_value": {
"min": {
"field": "products_price"
}
}
}
},
"without_filter_max": {
"global": {},
"aggs": {
"price_value": {
"max": {
"field": "products_price"
}
}
}
}
}
}
I have an elastic search running with documents like this one:
{
id: 1,
price: 620000,
propertyType: "HO",
location: {
lat: 51.41999,
lon: -0.14426
},
active: true,
rentOrSale: "S",
}
I'm trying to use aggregates to get statistics about a certain area using aggregations and the query I'm using is the following:
{
"sort": [
{
"id": "desc"
}
],
"query": {
"bool": {
"must": [
{
"term": {
"rentOrSale": "s"
}
},
{
"term": {
"active": true
}
}
]
},
"filtered": {
"filter": {
"and": [
{
"geo_distance": {
"distance": "15.0mi",
"location": {
"lat": 51.50735,
"lon": -0.12776
}
}
}
]
}
}
},
"aggs": {
"propertytype_agg": {
"terms": {
"field": "propertyType"
},
"aggs": {
"avg_price": {
"avg": {
"field": "price"
}
}
}
},
"bed_agg": {
"terms": {
"field": "numberOfBedrooms"
},
"aggs": {
"avg_price": {
"avg": {
"field": "price"
}
}
}
}
}
}
But in the result I can't see the aggregations. As soon as I remove either the bool or filtered part of the query I can see the aggregations. I can't figure out why this is happening, nor how do I get the aggregations for these filters. I've tried using the answer to this question but I've not been able to solve it. Any ideas?
I think your query need to be slightly re-arranged - move the "filtered" further up and repeat the "query" command:
"query": {
"filtered": {
"query" : {
"bool": {
...
}
},
"filter": {
...
}
}
}