We are sending the IIS Logs from Log Stash to Elastic Search. But the date histogram search is not returning the correct result. Below is the Query
{
"size":0,
"query":{
"filtered":{
"filter":{
"bool":{
"must":[
{
"query":{
"terms":{
"request":[
"/hotel/getrates.aspx",
"/planner/travelplan.aspx"
]
}
}
},
{
"query":{
"match":{
"method":"GET"
}
}
},
{
"range":{
"EventTime":{
"gte":"now-22h",
"lte":"now-16h"
}
}
}
]
}
}
}
},
"aggs":{
"timevalue":{
"date_histogram":{
"field":"EventTime",
"interval":"hour",
"format":"yyyy-MM-dd hh:mm:ss"
},
"aggs":{
"request":{
"terms":{
"field":"request"
},
"aggs":{
"status":{
"terms":{
"field":"status"
}
}
}
}
}
}
}
}
Results are only for hour window 00-12, not for the 13-23.
Can someone please explain the reason.
Related
I'm using an Elasticsearch filter aggregation with a nested top_hits aggregation to retrieve top matching documents based on different filters, but I can't seem to change the scores of results in each bucket via boosting or a nested function_score query. Is this just not possible? I haven't found any explicit documentation saying it won't work, and the query executes just fine, however the resulting scores aren't impacted.
Example query (note the huge boost in the first aggregation):
GET _search
{
"size":0,
"query":{
"bool":{
"should":[
{
"multi_match":{
"type":"phrase",
"query":"TV",
"fields":[
"categories^4"
]
}
}
]
}
},
"aggs":{
"1":{
"filter":{
"bool":{
"must":[
{
"multi_match":{
"type":"phrase",
"query":"Music",
"fields":[
"categories^10"
]
}
}
]
}
},
"aggs":{
"1_hits":{
"top_hits":{
"size":10,
"sort":[
{
"_score":{
"order":"desc"
}
}
]
}
}
}
},
"2":{
"filter":{
"bool":{
"must":[
{
"multi_match":{
"type":"phrase",
"query":"Music",
"fields":[
"categories"
]
}
}
]
}
},
"aggs":{
"2_hits":{
"top_hits":{
"size":10,
"sort":[
{
"_score":{
"order":"desc"
}
}
]
}
}
}
}
}
}
I'm trying to create a query with elasticsearch to filter the records of the same city and price.
But the city filter is not working.
POST diadeturista/services/_search
{
"query":{
"bool":{
"must":[
],
"filter":{
"bool":{
"must":{
"terms":{
"city":[
"Contagem"
]
},
"range":{
"price_adult":{
"lte":"300",
"gte":"150"
}
}
}
}
}
}
}
}
SHow me this error:
[terms] malformed query, expected [END_OBJECT] but found [FIELD_NAME]
I think what you want todo is
{
"query":{
"bool":{
"must": [
{
"terms":{
"city":[
"Contagem"
]
}
},
{
"range":{
"price_adult":{
"lte":"300",
"gte":"150"
}
}
}
]
}
}
}
I have orders documents like this:
{
"customer":{
"id":1,
"Name":"Foobar"
},
"products":[
{
"id":1,
"name":"Television",
"category": 11
},
{
"id":2,
"name":"Smartphone",
"category": 12
}
]
}
And I am performing a top_terms_aggregation in order to know the products best sellers. To do it globally, I use:
{
"size":0,
"aggs":{
"products":{
"nested":{
"path":"products"
},
"aggs":{
"top_terms_aggregation":{
"terms":{
"field":"products.id",
"size":10
}
}
}
}
}
}
But, how would I filter the products given a category_id? I tried adding this filter:
"query":{
"bool":{
"must":[
{
"match":{
"products.category":11
}
}
]
}
}
But this filters the orders itself that has some product with the given category, and the aggregation gets corrupted.
I want to get the best sellers products that belongs to a given category.
Solved this way:
{
"size":0,
"aggs":{
"products":{
"nested":{
"path":"products"
},
"aggs":{
"first_filter":{
"filter":{
"term":{
"products.category":11
}
},
"aggs":{
"top_terms_aggregation":{
"terms":{
"field":"products.id",
"size":10
}
}
}
}
}
}
}
}
Must be this exact sequence of aggs or "stranger things" happens.
You may use filter aggregation
GET _search
{
"size":0,
"aggs":{
"products":{
"nested":{
"path":"products"
},
"filter": {
"term": {
"category": 11
}
},
"aggs":{
"top_terms_aggregation":{
"terms":{
"field":"products.id",
"size":10
}
}
}
}
}
}
I want to perform a query equivalent to the following MYSQL query
SELECT http_user, http_req_method, dst dst_port count(*) as total
FROM my_table
WHERE http_req_method='GET' OR http_req_method="POST"
GROUP BY http_user, http_req_method, dst dst_port
I built the following query:
{
"query":{
"bool":{
"should":[
{
"term":{"http_req_method":"GET"}
},
{
"term":{"http_req_method":"POST"}
}
],
}
},
"aggs":{
suser":{
"terms":{
"field":"http_user"
},
"aggs":{
"dst":{
"terms":{
"field":"dst"
},
"aggs":{
"dst_port":{
"terms":{
"field":"dst_port"
},
"aggs":{
"http_req_method":{
"terms":{
"field":"http_req_method"
}
}
}
}
}
}
}
}
}
}
( I might be missing some branches there but it's correct in my code). The problem is that results also include other methods too like CONNECT, although I only ask for GET or POST. I thought aggregations are applied on the results after the query. Am I doing something wrong here?
I would leverage "minimum_should_match", like this:
"query":{
"bool":{
"minimum_should_match": 1,
"should":[
{
"term":{"http_req_method":"GET"}
},
{
"term":{"http_req_method":"POST"}
}
],
}
},
Another way that works better would be to leverage the terms query in a bool/filter clause instead
"query":{
"bool":{
"filter":[
{
"terms": {"http_req_method": ["GET", "POST"] }
}
]
}
},
According to the latest Elasticsearch documentation, you should move the filter part inside the aggregation. Something like this:
{
"aggs":{
get_post_requests":{
"filter" : {
"bool": [
{ "term":{"http_req_method":"GET"} },
{ "term":{"http_req_method":"POST"} },
]
},
"aggs": {
"suser"{
"terms":{
"field":"http_user"
}
},
"aggs":{
"dst":{
"terms":{
"field":"dst"
},
"aggs":{
"dst_port":{
"terms":{
"field":"dst_port"
},
"aggs":{
"http_req_method":{
"terms":{
"field":"http_req_method"
}
}
}
}
}
}
}
}
}
}
}
Hope the parentheses are ok. Let me know if this gets you closer to the result :)
What I want to achieve: I want my "age" aggregation to not be filtered by the query filter and I want to be able to apply filters to it.
So if I start with this query:
{
"query":{
"filtered":{
"filter":{ "terms":{ "family_name":"Brown" } } //filter_1
}
},
"aggs":{
"young_age":{
"filter":{
"range":{ "lt":40, "gt":18 } //filter_2
},
"aggs":{
"age":{
"terms":{
"field":"age"
}
}
}
}
}
}
My aggregation "young_age" will be filtered by both filter_1 and filter_2. I don't want my aggregation to be filtered by filter_1.
As I was looking into the documentation, I thought global aggregation would solve my problem, and I wrote that query:
{
"query":{
"filtered":{
"filter":{ "terms":{ "family_name":"Brown" } } //filter_1
}
},
"aggs":{
"young_age":{
"global":{}, //<----------- add global
"filter":{
"range":{ "lt":40, "gt":18 } //filter_2
},
"aggs":{
"age":{
"terms":{
"field":"age"
}
}
}
}
}
}
But then elastic search complains about my filter_2:
"""
Found two aggregation type definitions [age] in [global] and [filter]
"""
And of course if I remove the filter_2:
{
"query":{
"filtered":{
"filter":{
"terms":{
"family_name":"Brown"
}
}
}
},
"aggs":{
"young_age":{
"global":{},
"aggs":{
"age":{
"terms":{
"field":"age"
}
}
}
}
}
}
Then my aggregation won't be filtered by filter_1 (as expected).
So how am I suppose to apply filter_2 to my global aggregation? Or how am I supposed to achieved that? I remember writing something similar with the facet filters...
In my opinion this is the typical use case of a post_filter. As the doc says:
The post_filter is applied to the search hits at the very end of a search request, after aggregations have already been calculated
Your query will look like:
{
"post_filter":{
"terms":{
"family_name":"Brown" //filter_1
}
},
"aggs":{
"young_age":{
"filter":{
"range":{ "lt":40, "gt":18 } //filter_2
},
"aggs":{
"age":{
"terms":{
"field":"age"
}
}
}
}
}
}
In this case the search hits are all the documents in the index. Then the aggregation is calculated (before filter_1). And after that the post_filter with the filter_1 will be executed.
Edit: As you said in your commend you have many aggregations and only one that shouldn't be affected by filter_1 I fixed your query using global aggregation
{
"query": {
"filtered": {
"filter": {
"term": {
"family_name": "Brown"
}
}
}
},
"aggs": {
"young_age": {
"global": {},
"aggs": {
"filter2": {
"filter": {
"range": {
"lt": 40,
"gt": 18
}
},
"aggs": {
"age": {
"terms": {
"field": "age"
}
}
}
}
}
}
}
}
globals and filters are not allowed at same level. you have to put filter at one level inside to global aggregation.
something like this should do for you.
{
"query":{
"filtered":{
"filter":{
"terms":{
"family_name":"Brown"
}
}
}
},
"aggs":{
"young_age":{
"global":{},
"aggs":{
"filter": {"term": {"family_name": "Brown"}}, #or {"bool": {"filter": {"term": {"family_name": "Brown"}}}}
"aggs": {
"age":{
"terms":{
"field":"age"
}
}
}
}
}
}
}