I have a requirement of find sum one fields in a single query. I have managed to sum specific field in specific datetime range .
I try to use aggregation query to get the field total in specific datetime range,but get total value is 0
My document json look like the following way :
{
"_index": "flow-2018.02.01",
"_type": "doc",
"_source": {
"#timestamp": "2018-02-01T01:02:40.701Z",
"dest": {
"ip": "120.119.37.237",
"mac": "d4:6d:50:21:f8:44",
"port": 3280
},
"final": true,
"flow_id": "EQQA////DP//////FP8BAAFw5CIXZxTUbVAh+ERn/7FMeHcl7S6z0Aw",
"last_time": "2018-02-01T01:01:48.349Z",
"source": {
"ip": "100.255.177.76",
"mac": "70:e4:30:15:67:14",
"port": 45870,
"stats": {
"bytes_total": 60,
"packets_total": 1
}
},
"start_time": "2018-02-01T01:01:48.349Z",
"transport": "tcp",
"type": "flow"
},
"fields": {
"start_time": [
1517446908349
],
"#timestamp": [
1517446960701
],
"last_time": [
1517446908349
]
},
"sort": [
1517446960701
]
}
My search query :
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"_source.#timestamp":{
"gte": "2018-02-01T01:00:00.000Z",
"lte": "2018-02-01T01:05:00.000Z"
}
}
}
]
}
},
"aggs":{
"total":{
"sum":{
"field":"stats.packets_total "
}
}
}
}
Please help me to solve this issue. Thank you
You're almost there, stats.packets_total should be source.stats.packets_total (and make sure to remove the space at the end of the field name), like this:
{
"size":0,
"query":{
"bool":{
"must":[
{
"range":{
"#timestamp":{
"gte": "2018-02-01T01:00:00.000Z",
"lte": "2018-02-01T01:05:00.000Z"
}
}
}
]
}
},
"aggs":{
"total":{
"sum":{
"field":"source.stats.packets_total"
}
}
}
}
Related
Here is a document in ElasticSearch
"CompanyId": 5733,
"PartNumber": "W8S038",
"Name_en": "#8 Washer, M4 Compatible, Stainless Steel, Pack of 100",
"ProductId": 90023,
"CompanyName": "Washers Ltd",
"Prices": [
{
"BuyerId": 308,
"Price": 2.42
}
,
{
"BuyerId": 406,
"Price": 2.22
}
]
}
Obviously we can't let on to buyer 308 that buyer 406 is getting a better price. Therefore when buyer 308 is searching I need to remove all of the prices for other buyers.
I'd like to do this by using source filtering. But how?!
(I could exclude Prices and add back in the required price by using a script_field. However, that means that the price is not part of the source document and therefore ReactiveSearch can't see it and therefore can't sort on it.)
Update: here is the query generated by ReactiveSearch to which I need to append the limit on prices:
"query":{
"bool":{
"must":[
{
"bool":{
"must":[
{
"bool":{
"must":[
{
"bool":{
"should":[
{
"multi_match":{
"query":"m4 washer",
"fields":[
"Name_en"
],
"type":"cross_fields",
"operator":"and"
}
},
{
"multi_match":{
"query":"m4 washer",
"fields":[
"Name_en"
],
"type":"phrase_prefix",
"operator":"and"
}
}
],
"minimum_should_match":"1"
}
}
]
}
}
]
}
}
],
"filter": [
{
"nested": {
"path": "Prices",
"query": {
"term": {
"Prices.CompanyId": 1474
}
},
"inner_hits": {}
}
}
]
}
},
"size":10,
"aggs":{
"CompanyName.raw":{
"terms":{
"field":"CompanyName.raw",
"size":1000,
"order":{
"_count":"desc"
}
}
}
},
"_source":{
"excludes":[
"PurchasingViews",
"ContractFilters",
"SearchField*",
"Keywords*",
"Menus*"
]
},
"from":0,
"sort":[
{
"Name_en.raw":{
"order":"asc"
}
}
],
"script_fields":{
"price":{
"script":{
"lang":"painless",
"inline":"if(params['_source']['Prices'] != null){for(p in params['_source']['Prices']){ if(p.CompanyId == 1474) return p.Price; }} return null;"
}
}
}
}
(That bool, must, bool, must, bool, must, bool, should seems rather stupid?)
You need to use the nested inner_hits feature like below.
{
"_source": [
"CompanyId", "PartNumber", "Name_en", "ProductId", "CompanyName"
],
"query": {
"bool": {
"filter": [
{
"nested": {
"path": "Prices",
"query": {
"term": {
"Prices.BuyerId": 308
}
},
"inner_hits": {}
}
}
]
}
}
}
In the output you'll get exactly what you expect, namely all the root-level fields and the matching prices for the given buyer.
UPDATE:
Here is how I would rewrite your query:
{
"query": {
"bool": {
"minimum_should_match": "1",
"should": [
{
"multi_match": {
"query": "m4 washer",
"fields": [
"Name_en"
],
"type": "cross_fields",
"operator": "and"
}
},
{
"multi_match": {
"query": "m4 washer",
"fields": [
"Name_en"
],
"type": "phrase_prefix",
"operator": "and"
}
}
],
"filter": [
{
"nested": {
"path": "Prices",
"query": {
"term": {
"Prices.CompanyId": 1474
}
},
"inner_hits": {}
}
}
]
}
},
"size": 10,
"aggs": {
"CompanyName.raw": {
"terms": {
"field": "CompanyName.raw",
"size": 1000,
"order": {
"_count": "desc"
}
}
}
},
"_source": {
"excludes": [
"PurchasingViews",
"ContractFilters",
"SearchField*",
"Keywords*",
"Menus*",
"Prices"
]
},
"from": 0,
"sort": [
{
"Name_en.raw": {
"order": "asc"
}
}
],
"script_fields": {
"price": {
"script": {
"lang": "painless",
"inline": "if(params['_source']['Prices'] != null){for(p in params['_source']['Prices']){ if(p.CompanyId == 1474) return p.Price; }} return null;"
}
}
}
}
I am trying to filter hotel rooms by price range in Elastic Search. The rooms have a default nightly price and also custom prices can be set for specific days.
I'm storing the nightlyPrice and a nested object for custom prices together with the dates. The mapping is smt. like:
{
"adverts": {
"mappings": {
"advert": {
"properties": {
"nightlyPrice": {"type": "float"},
"customPrices": {
"type": "nested",
"properties": {
"date": {"type": "date"},
"price": {"type": "float"}
}
}
}
}
}
}
}
For example I want to get the rooms within the price range of 100 and 200$ between the dates 1st and 7th of July.
So I came up with this logic:
Either customPrices.date must be between 2019-07-01 and 2019-07-07 and customPrices.price between 100 and 200.
or the nightlyPrice must be between 100 and 200 and no customPrices.date is set between 05 and 07 July.
However I couldn't be able to apply this logic to Elastic Search, nested objects / queries are kinda tricky I guess.
This is the final query I came up with:
{
"query": {
"bool": {
"filter": [
{
"term": {
"status": "active"
}
}
],
"must": [
{
"bool": {
"should": [
{
"nested": {
"path": "customPrices",
"query": {
"bool": {
"must": [
{
"range": {
"date": {
"from": "2019-07-01",
"to": "2019-07-07"
}
}
},
{
"range": {
"price": {
"from": 100,
"to": 200
}
}
}
]
}
}
}
},
{
"bool": {
"must": [
{
"range": {
"nightlyPrice": {
"from": 100,
"to": 200
}
}
}
],
"must_not": [
{
"nested": {
"path": "customPrices",
"query": {
"range": {
"customPrices.date": {
"from": "2019-07-01",
"to": "2019-07-07"
}
}
}
}
}
]
}
}
]
}
}
]
}
}
}
The problem with this query is if customPrices.date matches the date range it never matches the document no matter the price range is. I experimented with 1 - 100000$ price range and it still doesn't match.
Tried to use the explain API to understand why a specific document didn't match but I don't understand it, it says user requested match_none query but there's this should query so it should match the nested query (first one):
{
"_index": "adverts",
"_type": "advert",
"_id": "13867",
"matched": false,
"explanation": {
"value": 0.0,
"description": "Failure to meet condition(s) of required/prohibited clause(s)",
"details": [
{
"value": 0.0,
"description": "no match on required clause (+(ToParentBlockJoinQuery (MatchNoDocsQuery(\"User requested \"match_none\" query.\")) (+nightlyPrice:[100.0 TO 200.0] -ToParentBlockJoinQuery (customListingPrices.date:[1561939200000 TO 1562543999999]))) #status:active",
"details": [
{
"value": 0.0,
"description": "Failure to meet condition(s) of required/prohibited clause(s)",
"details": [
{
"value": 0.0,
"description": "no match on required clause (ToParentBlockJoinQuery (MatchNoDocsQuery(\"User requested \"match_none\" query.\")) (+nightlyPrice:[100.0 TO 200.0] -ToParentBlockJoinQuery (customListingPrices.date:[1561939200000 TO 1562543999999])))",
"details": [
{
"value": 0.0,
"description": "No matching clauses",
"details": []
}
]
},
{
"value": 0.0,
"description": "match on required clause, product of:",
"details": [
{
"value": 0.0,
"description": "# clause",
"details": []
},
{
"value": 1.0,
"description": "status:active",
"details": []
}
]
}
]
}
]
},
{
"value": 0.0,
"description": "match on required clause, product of:",
"details": [
{
"value": 0.0,
"description": "# clause",
"details": []
},
{
"value": 1.0,
"description": "DocValuesFieldExistsQuery [field=_primary_term]",
"details": []
}
]
}
]
}
}
Any help or idea is greatly appreciated...
If you closely look at the first must clause, it appears that you haven't mentioned the entire path of the field.
{
"range":{
"date":{ <-- must be "customPrices.date"
"from":"2019-07-01",
"to":"2019-07-07"
}
}
},
{
"range":{
"price":{ <-- must be "customPrices.price"
"from":100,
"to":200
}
}
}
Below is how the query should be and should work fine for your use case.
Query
POST <your_index_name>/_search
{
"query":{
"bool":{
"filter":{
"term":{
"status":"active"
}
},
"must":[
{
"bool":{
"should":[
{
"bool":{
"must":[
{
"nested":{
"path":"customPrices",
"query":{
"bool":{
"must":[
{
"range":{
"customPrices.date":{
"gte":"2019-07-01",
"lte":"2019-07-09"
}
}
},
{
"range":{
"customPrices.price":{
"gte":100,
"lte":200
}
}
}
]
}
}
}
}
]
}
},
{
"bool":{
"must":[
{
"range":{
"nightlyPrice":{
"gte":100,
"lte":200
}
}
}
],
"must_not":[
{
"nested":{
"path":"customPrices",
"query":{
"range":{
"customPrices.date":{
"gte":"2019-07-05",
"lte":"2019-07-07"
}
}
}
}
}
]
}
}
]
}
}
]
}
}
}
Hope it helps!
I'm trying to acheve this goals:
Filter out results by bool query, like "status=1"
Filter out results by bool range query, like "discance: gte 10 AND lte 60"
Filter out results by match at least one int value from int array
Search words in many fields with calculating document score. Some fields needs wildcard, some boosting, like importantfield^2, somefield*, someotherfield^0.75
All above points join by AND operator. All terms in one point join by OR operator.
Now I wrote something like this, but wildcards not working. Searching "abc" don't finds "abcd" in "name" field.
How to solve this?
{
"filtered": {
"query": {
"multi_match": {
"query": "John Doe",
"fields": [
"*name*^1.75",
"someObject.name",
"tagsArray",
"*description*",
"ownerName"
]
}
},
"filter": {
"bool": {
"must": [
{
"term": {
"status": 2
}
},
{
"bool": {
"should": [
{
"term": {
"someIntsArray": 1
}
},
{
"term": {
"someIntsArray": 5
}
}
]
}
},
{
"range": {
"distanceA": {
"lte": 100
}
}
},
{
"range": {
"distanceB": {
"gte": 50,
"lte": 100
}
}
}
]
}
}
}
}
Mappings:
{
"documentId": {
"type": "integer"
},
"ownerName": {
"type": "string",
"index": "not_analyzed"
},
"description": {
"type": "string"
},
"status": {
"type": "byte"
},
"distanceA": {
"type": "short"
},
"createdAt": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"distanceB": {
"type": "short"
},
"someObject": {
"properties": {
"someObject_id": {
"type": "integer"
},
"name": {
"type": "string",
"index": "not_analyzed"
}
}
},
"someIntsArray": {
"type": "integer"
},
"tags": {
"type": "string",
"index": "not_analyzed"
}
}
You can make use of Query String if you would want to apply wildcard for multiple fields and at the same time apply various boosting values for individual fields:
Below is how your query would be:
POST <your_index_name>/_search
{
"query":{
"bool":{
"must":[
{
"query_string":{
"query":"abc*",
"fields":[
"*name*^1.75",
"someObject.name",
"tagsArray",
"*description*",
"ownerName"
]
}
}
],
"filter":{
"bool":{
"must":[
{
"term":{
"status":"2"
}
},
{
"bool":{
"minimum_should_match":1,
"should":[
{
"term":{
"someIntsArray":1
}
},
{
"term":{
"someIntsArray":5
}
}
]
}
},
{
"range":{
"distanceA":{
"lte":100
}
}
},
{
"range":{
"distanceB":{
"gte": 50,
"lte":100
}
}
}
]
}
}
}
}
}
Note that for the field someIntsArray, I've made use of "minimum_should_match":1 so that you won't end up with documents that'd have neither of those values.
Updated Answer:
Going by the updated comment, you can have the fields with wildcard search used by query_string and you can make use of simple match query with boosting as shown in below. Include both these queries (can even add more match queries depending on your requirement) in a combine should clause. That way you can control where wildcard query can be used and where not.
{
"query":{
"bool":{
"should":[
{
"query_string":{
"query":"joh*",
"fields":[
"name^2"
]
}
},
{
"match":{
"description":{
"query":"john",
"boost":15
}
}
}
],
"filter":{
"bool":{
"must":[
{
"term":{
"status":"2"
}
},
{
"bool":{
"minimum_should_match":1,
"should":[
{
"term":{
"someIntsArray":1
}
},
{
"term":{
"someIntsArray":5
}
}
]
}
},
{
"range":{
"distanceA":{
"lte":100
}
}
},
{
"range":{
"distanceB":{
"lte":100
}
}
}
]
}
}
}
}
}
Let me know if this helps
I'm using Elasticsearch 6.4. I need to perform date range query for nested type. And my Mapping is :
"items": {
"type": "nested",
"properties": {
"date": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss"
},
"value": {
"type": "long"
}
}
}
And the sample data for items is:
"items": [{"value": 53.79585560271698,"date": "2018-10-30 07:30:00"
},
{"value": 53.15659716175469,"date": "2018-10-30 07:45:00"
},
{"value": 52.13847544211876,"date": "2018-10-30 08:00:00"
},
{"value": 51.30296218052354,"date": "2018-10-30 08:15:00"
},
{"value": 50.9705640874663,"date": "2018-10-30 08:30:00"
},
{"value": 51.37812337892956,"date": "2018-10-30 08:45:00"
},
{"value": 51.162125032933545,"date": "2018-10-30 09:00:00"
}................etc]
I used the query :
{
"query":{
"bool":{
"must":[
{
"match":{
"objectId":2078
}
},
{
"nested":{
"path":"items",
"query":{
"bool":{
"must":[
{
"range":{
"items.date":{
"gte":"2018-10-30 07:30:00",
"lte":"2018-10-30 08:30:00"
}
}
}
]
}
}
}
}
]
}
}
}
But the result not filtering with date range that i given.i searched for this query , but solutions not working for me.
I'm new to ES. please give me solution.
Hello I have the following query that I am running:
{
"_source": [
"source1",
"source2",
"source3",
"source4",
],
"query": {
"bool": {
"minimum_should_match": 1,
"must": {
"filter": [
{
"term": {
"_type": {
"value": "someval1"
}
}
},
{
"term": {
"_type": {
"value": "someval2"
}
}
}
],
"query_string": {
"analyze_wildcard": "true",
"query": "tesla*",
"rewrite": "scoring_boolean"
}
}
}
},
"size": 50,
"sort": [
"_score"
]
}
That is currently returning:
'"reason":"[bool] malformed query, expected [END_OBJECT] but found [FIELD_NAME]","line":1,"col":343},"status":400}'
Any idea how to use multiple filters on a query? I was able to do it just fine on elastic 2.4 but since OR is now deprecated as well as filtered, I am a bit lost.
Thanks!
The syntax of the query is wrong. filter should not be wrapped into the must statement. It should be in the same level with must. Also bool queries must statement should be an array, not an object. So your query should look like this
{
"_source":[
"source1",
"source2",
"source3",
"source4"
],
"query":{
"bool":{
"minimum_should_match":1,
"must":[
{
"query_string":{
"analyze_wildcard":"true",
"query":"tesla*",
"rewrite":"scoring_boolean"
}
}
],
"filter":{
"bool":{
"should":[
{
"term":{
"_type":{
"value":"someval1"
}
}
},
{
"term":{
"_type":{
"value":"someval2"
}
}
}
]
}
}
}
},
"size":50,
"sort":[
"_score"
]
}
I think your filter is OR, that's why I wrap it inside should