I know for ES now means today
Is there any way to get last Sunday using some combination of now?
Context: We use Kibana and want to create a dashboard with a filter such that everyday it shows data for date ranges from last Sunday to today.
You can simply use below range query:
Here, now/w will give result from Monday to today and now/w-1d will give result from Sunday to today.
POST timeindex/_search
{
"query": {
"range": {
"timestamp": {
"gte": "now/w-1d"
}
}
}
}
Related
I would like to find all documents within my index which have a timestamp between time 12:00:00 and 13:00:00 (24hr-format), regardless the date.
I can not find the KQL expression for this. Also the graphical filter method requires a specific date (wildcards does not work).
In the console I tried something like this:
# Does not work:
GET /_search
{
"query": {
"range": {
"#timestamp": {
"gt": "12:00:00",
"lt": "13:00:00",
"format": "HH:mm:ss"
}
}
}
}
Filter methods with date (e.g. "2020-11-16T12:00:00") work properly, so there is no issue with the timestamp. I am aware of the timezone of UTC, so at the end I have also to specify the timezone.
My ELK-stack has versions Kibana v7.10.0 and Elasticsearch v7.10.0.
My documents in ES contain a field that stores date value in epoch format like this:
"received": 1521055535062
I am trying to build a query where I can filter documents based on a certain date with or without UTC time difference taken into account.
Since I have over a million records in ES, I would like a way to figure out how many of those documents belonged to a specific date.
What you are looking for is the date range query with date format specification that you want. The following should return the docs received on first day of 2018
GET _search
{
"query": {
"range" : {
"received" : {
"gte": "01/01/2018",
"lt": "01/02/2018",
"format": "dd/MM/yyyy"
}
}
}
}
See here for the details https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html
Just adding a snippet of Java code (using the range query). From the question, it's not clear in which language the query should be written but if it's in Java, following snippet should be helpful:
BoolQueryBuilder boolQuery = new BoolQueryBuilder();
Date asOfDate = new Date(System.currentTimeMillis() - historySeconds * 1000);
boolQuery.must(QueryBuilders.rangeQuery("created").from(asOfDate).includeLower(true));
Can I make a range query on default timestamp field ignoring date values i.e. using only time in timestamp - say 2 hours of each day?
My intentions are to search for all the documents but exclude the documents indexed between 9 PM and 12 AM (I have seen example with date ranges in filtering).
timestamp example stands following:
"#timestamp": [
"2015-12-21T15:18:17.120Z"
]
Elasticsearch version: 1.5.2
My first idea would be to use the date math in Elasticsearch query, e.g. if you run your query at 1PM, this would work:
{
"query": {
"range" : {
"#timestamp" : {
"gte": "now-16h/h",
"lte": "now-1h/h"
}
}
}
}
(watch out for the timezone though).
As far as I know, the only other possibility would be to use scripting.
Please note also that you are running a very old version of Elasticsearch.
Edit If you need simply absolute date, then check how your #timestamp field look, and use the same format, for instance on my Elasticsearch, it would be:
{
"query": {
"range" : {
"#timestamp" : {
"gte": "2015-03-20T01:21:00.01Z",
"lte": "2015-03-21T01:12:00.04Z"
}
}
}
}
I have a query that properly parses date ranges. However, my database has a default value that all dates have a timestamp of 00:00:00. This means that items that are still valid today are shown as expired even if they should still be valid. How can I adjust the following to look at just the date and not the time of the item (expirationDate).
{
"range": {
"expirationDate": {
"gte": "now"
}
}
}
An example of the data is:
"expirationDate": "2014-06-24T00:00:00.000Z",
Did you look into the different format options for dates stored in ElasticSearch? If this does not work for you or you don't want to store dates without the time you can try this query, which will work for your exact use case I guess:
{
"range": {
"expirationDate": {
"gt": "now-1d"
}
}
}
You can also round down the time so that your query returns anything that occurred since the beginning of the day:
Assuming that
now is 2017-03-07T07:00:00.000,
now/d is 2017-03-07T00:00:00.000
Your query would be:
{
"range": {
"expirationDate": {
"gte": "now/d"
}
}
}
elastic search documentation on rounding times
QUESTION Maybe anyone has solution how to filter/query ElasticSearch data by month or day ? Let's say I need to get all users who celebrating birthdays today.
mapping
mappings:
dob: { type: date, format: "dd-MM-yyyy HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ss+SSSS"}
and stored in this way:
dob: 1950-06-03T00:00:00Z
main problem is how to search users by month and day only. Ignore the year, because birthday is annually as we know.
SOLUTION
I found solution to query birthdays with wildcards. As we know if we want use wildcards, the mapping of field must be a string, so I used multi field mapping.
mappings:
dob:
type: multi_field
fields:
dob: { type: date, format: "yyyy-MM-dd'T'HH:mm:ss'Z'}
string: { type: string, index: not_analyzed }
and query to get users by only month and day is:
{
"query": {
"wildcard": {
"dob.string": "*-06-03*"
}
}
}
NOTE
This query can be slow, as it needs to iterate over many terms.
CONCLUSION
It's not pretty nice way, but it's the only one I've found and it works!.
You should store the value-to-be-searched in Elasticsearch. The string/wildcard solution is half the way, but storing the numbers would be even better (and faster):
mappings:
dob:
type: date, format: "yyyy-MM-dd'T'HH:mm:ss'Z'
dob_day:
type: byte
dob_month:
type: byte
Example:
dob: 1950-03-06
dob_day: 06
dob_month: 03
Filtering (or querying) for plain numbers is easy: Match on both fields.
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"dob_day": 06,
}
},
{
"term": {
"dob_month": 03,
}
},
]
}
}
}
}
PS: While thinking about the solution: Storing the date as self-merged number like "06-03" -> "603" or "6.03" would be less obvious, but allow range queries to be used. But remember that 531 (05-31) plus one day would be 601 (06-01).
A manually-computed julian date might also be handy, but the calculation must always assume 29 days for February and the range query would have a chance of being off-by-1 if the range includes the 29st of Feb.
Based on your question I am assuming that you want a query, and not a filter (they are different), you can use the date math/format combined with a range query.
See: range query for usage
For explanation of date math see the following link
curl -XPOST http://localhost:9200/twitter/tweet/_search -d
{
"query": {
"range": {
"birthday": {
"gte" : "2014-01-01",
"lte" : "2014-01-01"
}
}
}
}
I have tested this with the latest elastic search.
If you don't want to parse strings, you can use a simple script. This filter will match when dateField has a specific month in any year:
"filter": {
"script": {
"lang": "expression",
"script": "doc['dateField'].getMonth() == month",
"params": {
"month": 05,
}
}
}
Note: the month parameter is 0-indexed.
The same method will work for day of month or any other date component.