I have a elasticsearch range query like this
curl 'localhost:9200/myindex/_search?pretty' -d '
{
"query": {
"range" : {
"total" : {
"gte" :174,
"lte" :180
}
}
}
}'
I need to use this query in grafana for my graph. i am trying to add this as a part of the Lucene query. but i am not able to find the desired result. can anyone help.
If "total" is a field, you can do something like this in Lucene:
total:[174 TO 180]
reference: https://lucene.apache.org/core/2_9_4/queryparsersyntax.html
First off I think you may be missing the document type from the request URL, should look like so:
http://localhost:9200/[INDEX]/[TYPE]/_search?pretty
Second, I've looked at previous answers providing detailed examples of range filtering and the query should work just fine like so
{
"query":
{
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"range": {
"total": {
"gte": 174,
"lte": 180
}
}
}
}
}
}
Related
I need to add a parameter to my search that filters results containing a specific word in a value. The query is searching for user history records and contains a url key. I need to filter out /history and any other url containing that string.
Here's my current query:
GET /user_log/_search
{
"size" : 50,
"query": {
"match": {
"user_id": 56678
}
}
}
Here's an example of a record, boiled down to just the value we're looking at:
"_source": {
"url": "/history?page=2&direction=desc",
},
How can the parameters of the search be changed to filter out this result.
You can use the filter param of boolean query in Elasticsearch.
if your url field is of type keyword, you can use the below query
{
"query": {
"bool": {
"must": {
"match": {
"user_id": 56678
}
},
"filter": { --> note filter
"term": {
"url": "/history"
}
}
}
}
}
I found a way to solve my specific issue. Instead of filtering on the url I'm filtering on a different value. Here's what I'm using now:
{
"size" : 50,
"query": {
"bool" : {
"must" : {
"match" : { "user_id" : 56678 }
},
"must_not": {
"match" : { "controller": "History" }
}
}
}
}
I'm still going to leave this question open for a while to see if anyone has other ways of solving the original problem.
recently I use Kibana for get data from ElasticSearch.
There are that node:
{ "_index" : "impasti",
"_type" : "impasti",
"_id" : "2019-01-02T15:25:20",
"_score" : 1.9806902,
"_source" : {
"sensor" : "Temperature",
"mac_address" : "",
"time" : "2019-01-02T14:25:19.728709Z",
"unit" : "'C",
"value" : 20.937
}},
I try get the data by the time field, into datetime range or exactly datetime.
But when I run this query
POST /impasti/impasti/_search
{'query':{
"query_string": {
"default_field": "time",
"query": "2019-01-02T14:25:19.728709Z"
}
}
}
the response is an error like this:
"type": "query_shard_exception" "reason": "Failed to parse query [2019-01-02T14:25:19.728709Z]",
Where is the mistake?
Thanks guys
Error states that you would need to do something more in order for the ES query parser to understand that the value is a date and not a string/text/keyword type:
Moreover date fields are usually used for a particular range. Below is how it can be done via adding Range feature
Using query_string:
POST your_index_name/_search
{
"query": {
"query_string": {
"default_field": "time",
"query": "time:[2019-01-02T14:25:19.728709Z TO 2019-01-02T14:25:19.728709Z]"
}
}
}
Generally it is [min To max] for finding docs in specified time range, but in case if you want to find documents for that date, mention same date for both min and max.
Using Range Bool Query:
POST you_index_name/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"mydate": {
"gte": "2019-01-02T14:25:19.728709Z",
"lte": "2019-01-02T14:25:19.728709Z"
}
}
}
]
}
}
}
The above is an example of Range Query using query DSL
Alternatively you can also make use of a simple match query to get what you want whilst using query DSL via Term Query, that is because internally it is stored as long value.
POST your_index_name/_search
{
"query": {
"term": {
"mydate": "2019-01-02T14:25:19.728709Z"
}
}
}
Note: Elasticsearch internally stores the date values in the form of long datatype in its inverted index as mentioned in this link
Could you update that query which gives you this error in the question. I can quickly check and let you know.
Sure!
This is the query
`POST /impasti/impasti/_search{
"query":{
"filtered": {
"query": {
"query_string": {
"default_field": "time",
"query": "time:[2019-01-02T14:20:19.728709Z TO 2019-02-02T14:25:19.728709Z]"
}
},
"filter": {
"term":{ "sensor": "temperature" }
}
}
}
}`
It get this error:
no [query] registered for [filtered]
I am working on ES recently and I found that I could achieve the almost same result but I have no clear idea as to the DIFFERENCE between these two.
"Filter then Aggregation"
POST kibana_sample_data_flights/_search
{
"size": 0,
"query": {
"constant_score": {
"filter": {
"term": {
"DestCountry": "CA"
}
}
}
},
"aggs": {
"ca_weathers": {
"terms": { "field": "DestWeather" }
}
}
}
"Filter Aggregation"
POST kibana_sample_data_flights/_search
{
"size": 0,
"aggs": {
"ca": {
"filter": {
"term": {
"DestCountry": "CA"
}
},
"aggs": {
"_weathers": {
"terms": { "field": "DestWeather" }
}
}
}
}
}
My Questions
Why there are two similar functions? I believe I am wrong about it but what's the difference then?
(please do ignore the result format, it's not the question I am asking ;p)
Which is better if I want to filter out the unrelated/unmatched and start the aggregation on lots of documents?
When you use it in "query", you're creating a context on ALL the docs in your index. In this case, it acts like a normal filter like: SELECT * FROM index WHERE (my_filter_condition1 AND my_filter_condition2 OR my_filter_condition3...).
When you use it in "aggs", you're creating a context on ALL the docs that might have (or haven't) been previously filtered. Let's say that if you have an structure like:
#OPTION A
{
"aggs":{
t_shirts" : {
"filter" : { "term": { "type": "t-shirt" } }
}
}
}
Without a "query", is exactly the same as having
#OPTION B
{
"query":{
"filter" : { "term": { "type": "t-shirt" } }
}
}
BUT the results will be returned in different fields.
In the Option A, the results will be returned in the aggregations field.
In the Option B, the results will be returned in the hits field.
I would recommend to apply your filters always on the query part, so you can work with subsecuent aggregations of the already filtered docs. Also because Aggrgegations cost more performance than queries.
Hope this is helpful! :D
Both filters, used in isolation, are equivalent. If you load no results (hits), then there is no difference. But you can combine listing and aggregations. You can query or filter your docs for listing, and calculate aggregations on bucket further limited by the aggs filter. Like this:
POST kibana_sample_data_flights/_search
{
"size": 100,
"query": {
"bool": {
"filter": {
"term": {
... some other filter
}
}
}
},
"aggs": {
"ca_filter": {
"term": {
"TestCountry": "CA"
}
},
"aggs": {
"ca_weathers": {
"terms": { "field": "DestWeather" }
}
}
}
}
But more likely you will need the other way, ie. make aggregations on all docs, to display summary informations, while you display docs from specific query. In this case you need to combine aggragations with post_filter.
Answer from #Val's comment, I may just quote here for reference:
In option A, the aggregation will be run on ALL documents. In option B, the documents are first filtered and the aggregation will be run only on the selected documents. Say you have 10M documents and the filter select only a 100, it's pretty evident that option B will always be faster.
I have two fields,let us name them "fieldA" and "fieldB" in my documents and i need to find the difference between them and check if that value falls under a specific range say "rangeA" or " rangeB" and then return the documents that matches my criteria.
The schema for data is as shown below:
{
"fieldA": 45
"fieldB":13
}
I need to find all the document which have the difference between "fieldA" and "fieldB" in between 30 and 35. How can i do this using scripting in elasticsearch?
This can also be done using aggregations and scripts like below:
{
"aggregations": {
"age_diff": {
"range": {
"script": "doc[\"fieldA\"].value - doc[\"fieldB\"].value",
"ranges": [
{
"from": 30,
"to": 35
}
]
}
}
}
}
This way you can just check how many documents falls under the specified range.But if you want to get the documents under the aggregations you can use "top_hits" aggregations.
More detailed discussion on aggregations can be found here and more about "top_hits" can be found in detail here
{
"query": {
"filtered": {
"filter": {
"script": {
"script": "difference=doc['fieldA'].value-doc['fieldB'].value;return (difference>param1 && difference<param2);",
"params": {
"param1":30,
"param2":35
}
}
}
}
}
}
I'm attempting to do a count query such that I return the number of unsuccessful attempts to log into my system within the last 10 minutes. I created this query:
{
"term": {
"success":false
},
"range": {
"_timestamp": {
"gt": "now-10m"
}
}
}
However, this returns all of the unsuccessful attempts for any time, disregarding the range filter in my query. Am I structuring this query correctly? The query works when I do a search with terms and ranges.
In other words, the output of the above query and curl -XGET localhost:9200/application/_count is the same (I have only tested unsuccessful attempts).
Try using the search_type parameter instead of using the countAPI. This is actually preferred:
curl -XGET localhost:9200/application/_search&search_type=count -d'{
query:....
}'
Documentation:
http://www.elasticsearch.org/guide/reference/api/search/search-type/
The range is a filter, so I think you have to create a filtered query to take it correctly into account :
{
"filtered": {
"query": {
"term": {
"success":false
},
},
"filter: {
"range": {
"_timestamp": {
"gt": "now-10m"
}
}
}
}
}