I'm looking for a solution to be able to store and query dates before Epoch (1st January 1970).
For example, I'm trying to store into Elasticsearch's index the date December, 25 1969 00:00:00 +0100.
I suppose it possible to store into into string or integer, but is there any solution to keep the field type date into Elasticsearch's mapping ?
You can definitely store dates before the epoch, simply by storing the date string or negative numbers.
If your index mapping looks like this:
PUT epoch
{
"mappings": {
"properties": {
"my_date": {
"type": "date"
}
}
}
}
Then you can index documents before the epoch, like this:
PUT epoch/_doc/1
{
"my_date": -608400
}
PUT epoch/_doc/2
{
"my_date": "1969-12-25T00:00:00"
}
When searching for document with a date before the epoch, both would be returned:
POST epoch/_search
{
"query": {
"range": {
"before": {
"lt": "1970-01-01"
}
}
}
}
Related
Given an index in Elasticsearch (opensearch) with a field, "etl_at", whose entries are unix timestamp integers, how to I find all records that are newer than yesterday?
"request_id" : "20230208201622189E3A8AFF810DB804F3",
"etl_at" : 1675888029122
I'm trying to use a CAST, but am met with error;
GET users/_search
{
"query": {
"range": {
"etl_at": {
"gt": "CAST('now-1d/d' as int)"
}
}
}
}
Can I make a range query on default timestamp field ignoring date values i.e. using only time in timestamp - say 2 hours of each day?
My intentions are to search for all the documents but exclude the documents indexed between 9 PM and 12 AM (I have seen example with date ranges in filtering).
timestamp example stands following:
"#timestamp": [
"2015-12-21T15:18:17.120Z"
]
Elasticsearch version: 1.5.2
My first idea would be to use the date math in Elasticsearch query, e.g. if you run your query at 1PM, this would work:
{
"query": {
"range" : {
"#timestamp" : {
"gte": "now-16h/h",
"lte": "now-1h/h"
}
}
}
}
(watch out for the timezone though).
As far as I know, the only other possibility would be to use scripting.
Please note also that you are running a very old version of Elasticsearch.
Edit If you need simply absolute date, then check how your #timestamp field look, and use the same format, for instance on my Elasticsearch, it would be:
{
"query": {
"range" : {
"#timestamp" : {
"gte": "2015-03-20T01:21:00.01Z",
"lte": "2015-03-21T01:12:00.04Z"
}
}
}
}
Hi i am kinda new to elasticsearch. I need to get an aggregation with date comparison and a dynamic range filter.
Like i need to get documents count where created_at document is 1 week earlier than their identification_date.
So i tried something like this but my date param seems unused, actually changing it never changes my results.
"aggs": {
"identified": {
"terms": {
"script": "doc['created_at'].value > (doc['identification_date'].value - diff_date)
&& doc['created_at'].value < doc['identification_date'].value",
"params": {
"diff_date": 604800
}
}
}
}
Thank you for taking time helping.
since on the abstracted level you just want count of documents created 7 days before on some date field.you don't need aggregation for this if you don't want to group the results sets further on some fields.Simply you can a)use range filter query on your date field
{
"range" : {
"date" : {
"gte" : "now-7d/d",
"lt" : "now"
}
}
}
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html
I have a query that properly parses date ranges. However, my database has a default value that all dates have a timestamp of 00:00:00. This means that items that are still valid today are shown as expired even if they should still be valid. How can I adjust the following to look at just the date and not the time of the item (expirationDate).
{
"range": {
"expirationDate": {
"gte": "now"
}
}
}
An example of the data is:
"expirationDate": "2014-06-24T00:00:00.000Z",
Did you look into the different format options for dates stored in ElasticSearch? If this does not work for you or you don't want to store dates without the time you can try this query, which will work for your exact use case I guess:
{
"range": {
"expirationDate": {
"gt": "now-1d"
}
}
}
You can also round down the time so that your query returns anything that occurred since the beginning of the day:
Assuming that
now is 2017-03-07T07:00:00.000,
now/d is 2017-03-07T00:00:00.000
Your query would be:
{
"range": {
"expirationDate": {
"gte": "now/d"
}
}
}
elastic search documentation on rounding times
QUESTION Maybe anyone has solution how to filter/query ElasticSearch data by month or day ? Let's say I need to get all users who celebrating birthdays today.
mapping
mappings:
dob: { type: date, format: "dd-MM-yyyy HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss'Z'||yyyy-MM-dd'T'HH:mm:ss+SSSS"}
and stored in this way:
dob: 1950-06-03T00:00:00Z
main problem is how to search users by month and day only. Ignore the year, because birthday is annually as we know.
SOLUTION
I found solution to query birthdays with wildcards. As we know if we want use wildcards, the mapping of field must be a string, so I used multi field mapping.
mappings:
dob:
type: multi_field
fields:
dob: { type: date, format: "yyyy-MM-dd'T'HH:mm:ss'Z'}
string: { type: string, index: not_analyzed }
and query to get users by only month and day is:
{
"query": {
"wildcard": {
"dob.string": "*-06-03*"
}
}
}
NOTE
This query can be slow, as it needs to iterate over many terms.
CONCLUSION
It's not pretty nice way, but it's the only one I've found and it works!.
You should store the value-to-be-searched in Elasticsearch. The string/wildcard solution is half the way, but storing the numbers would be even better (and faster):
mappings:
dob:
type: date, format: "yyyy-MM-dd'T'HH:mm:ss'Z'
dob_day:
type: byte
dob_month:
type: byte
Example:
dob: 1950-03-06
dob_day: 06
dob_month: 03
Filtering (or querying) for plain numbers is easy: Match on both fields.
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"dob_day": 06,
}
},
{
"term": {
"dob_month": 03,
}
},
]
}
}
}
}
PS: While thinking about the solution: Storing the date as self-merged number like "06-03" -> "603" or "6.03" would be less obvious, but allow range queries to be used. But remember that 531 (05-31) plus one day would be 601 (06-01).
A manually-computed julian date might also be handy, but the calculation must always assume 29 days for February and the range query would have a chance of being off-by-1 if the range includes the 29st of Feb.
Based on your question I am assuming that you want a query, and not a filter (they are different), you can use the date math/format combined with a range query.
See: range query for usage
For explanation of date math see the following link
curl -XPOST http://localhost:9200/twitter/tweet/_search -d
{
"query": {
"range": {
"birthday": {
"gte" : "2014-01-01",
"lte" : "2014-01-01"
}
}
}
}
I have tested this with the latest elastic search.
If you don't want to parse strings, you can use a simple script. This filter will match when dateField has a specific month in any year:
"filter": {
"script": {
"lang": "expression",
"script": "doc['dateField'].getMonth() == month",
"params": {
"month": 05,
}
}
}
Note: the month parameter is 0-indexed.
The same method will work for day of month or any other date component.