Kibana not showing nano seconds, it is showing zeros
Actually timestamp is available in nano seconds
How to sort the data in kibana using nano seconds precision
date data type stores dates in millisecond resolution. The date_nanos data type stores dates in nanosecond resolution, which limits its range of dates from roughly 1970 to 2262, as dates are still stored as a long representing nanoseconds since the epoch.
Queries on nanoseconds are internally converted to range queries on this long representation, and the result of aggregations and stored fields is converted back to a string depending on the date format that is associated with the field.
Date formats can be customized, but if no format is specified then it uses the default. As an example, You can customise the date field like this:
PUT my-index-000001
{
"mappings": {
"properties": {
"date": {
"type": "date_nanos"
}
}
}
}
Related
I have a date field defined in index as
"_reportDate": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}
and I have a query to query from _source field which gives _reportDate field in string of 2015-12-05 01:05:00.
I can't seems to find a way to get date in different date format during query retrieval apart from using script field (which is not preferable). From what I understand a date field will be parse to long value to be indexed in elastic search, can we retrieve the long value as well during elasticsearch query?
You need to store the field and at search time ask for this stored field.
If it does not work you can always apply the script at index time with ingest feature and a script processor.
Is it possible to set a fixed timespan for a saved visualization or a saved search in Kibana 4?
Scenario:
I want to create one dashboard with 2 visualizations with different time spans.
A metric counting unique users within 10 min (last 10 minutes)
A metric counting todays unique users (from 00.00am until now)
Note that changing the time span on the dashboard does not affect the visualizations. Possible?
You could add a date range query to the saved search you base each visualisation on. Eg, if your timestamp field is called timestamp:
timestamp:[now-6M/M TO now]
where the time range is from 'now' to '6 months ago, rounding to the start of the month.
Because Kibana also now supports JSON-based query DSL, you could also achieve the same thing by entering this into the search box instead:
{
"range" : {
"timestamp" : {
"gte": "now-6M/M",
"lte": "now"
}
}
}
For more on date range queries see https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html#ranges-on-dates
However changing the dashboard timescale will override this if it's a subset. So if you use the above 6 month range in the saved search, but a 3 month range in the dashboard, you'll filter to 3 months of data.
I'm using such command to import data to RethinkDB
rethinkdb import --force -f ${folder}/json/data.json --table test.data -c localhost:28015
It imports data perfectly. But I have some of fields in my json as time:
{
"id": "1",
"date": "2015-09-19",
"time": {
"begin": "09:00",
"end": "10:30"
}
}
When I'm trying to query these fields like data or time.begin, time.end treating them as time - RethinkDB doesn't understand it and throw exception
r.db('test').table('data').filter(function(t) {
return t("date").date()
})
RqlRuntimeError: Not a TIME pseudotype: `"2015-09-19"` in:
r.db("test").table("data").filter(function(var_43) { return var_43("date").date(); })
^^^^^^^^^^^^^^
Is any way to specify for RethinkDB which field in the JSON are with time type?
JSON doesn't provide a standard way of specifying a time field, but there are a couple ways you can do this with RethinkDB: either modify the data before or after inserting it. RethinkDB time objects are more than just the strings you have shown here, and contain millisecond time resolution along with timezone data.
Time objects can be constructed using r.now(), r.time(), r.epoch_time(), and r.ISO8601(). Because of the format of your time strings, I would use r.ISO8601(). It is important to note that your data doesn't appear to contain timezone information, so you should be sure that your data won't return incorrect results if they are all put in the same timezone.
Another thing to keep in mind when using times in RethinkDB is that the data will be converted into an appropriate time object in your client. Since it appears that you are using Javascript, you will get back a Date object. For Python, you would get a datetime.datetime object, etc. If you would rather get the raw time pseudotype format (see below), you can specify timeFormat: "raw" as a global optarg to your query (see the documentation for run() for details).
Post-process the data inside RethinkDB
This is probably the easiest option, and what I would recommend. After importing your data, you can run a query to modify each row to convert the strings into time objects. Based on the format of your data, this should work:
r.db('test').table('data').replace(function(row) {
return row.merge({
'begin_time': r.ISO8601(row('date').add('T').add(row('time')('begin')), { defaultTimezone: '+00:00' }),
'end_time': r.ISO8601(row('date').add('T').add(row('time')('end')), { defaultTimezone: '+00:00' })
}).without('date', 'time');
}).run(conn, callback)
This replaces the date and time fields from all the rows in your test.data table with begin_time and end_time time objects that can be used as you expect. The defaultTimezone field is required because the time string doesn't contain timezone information, but you should change these values to whatever is appropriate.
Modify the JSON data
This is a bit lower-level and can be tricky, but if you don't mind getting your hands dirty, this could be more suited to your needs.
RethinkDB time objects are communicated in JSON using a particular format to represent a 'pseudotype'. These are types not standardized in JSON that still exist in RethinkDB. The format for a time pseudotype looks like this:
{
"$reql_type$": "TIME",
"epoch_time": 1413843783.195,
"timezone": "+00:00"
}
Where epoch_time is the number of seconds since the UNIX epoch (Jan 1, 1970). If the data you are importing follows this format, you can insert this directly and it will be interpreted by the database as a valid time object. It would be up to you to modify the data you are importing, but your example row would look something like this:
{
"id": "1",
"begin_time": {
"$reql_type$": "TIME",
"epoch_time": 1442653200,
"timezone": "+00:00"
},
"end_time': {
"$reql_type$": "TIME",
"epoch_time": 1442658600,
"timezone": "+00:00"
}
}
My same caveat for timezones applies here as well.
I'm aggregating documents that each have a timestamp. The timestamp is UTC, but the documents each also have a local time zone ("timezone": "America/Los_Angeles") that can be different across documents.
I'm trying to do a date_histogram aggregation based on local time, not UTC or a fixed time zone (e.g., using the option "time_zone": "America/Los_Angeles").
How can I convert the timezone for each document to its local time before the aggregation?
Here's the simple aggregation:
{
"aggs": {
"date": {
"date_histogram": {
"field": "created_timestamp",
"interval": "day"
}
}
}
}
I'm not sure if I fully understand it, but it seems like the time_zone property would be for that:
The zone value accepts either a numeric value for the hours offset, for example: "time_zone" : -2. It also accepts a format of hours and minutes, like "time_zone" : "-02:30". Another option is to provide a time zone accepted as one of the values listed here.
If you store another field that's the local time without timezone information it should work.
Take every timestamp you have (which is in UTC), convert it to a date in the local timezone (this will contain the timezone information). Now simply drop the timezone information from this datetime. Now you can perform actions on this new field.
Suppose you start with this time in UTC:
'2016-07-17T01:33:52.412Z'
Now, suppose you're in PDT you can convert it to:
'2016-07-16T18:33:52.412-07:00'
Now, hack off the end so you end up with:
'2016-07-16T18:33:52.412Z'
Now you can operate on this field.
I am currently working on a project in which I am storing user activity logs in elasticsearch. the user field in the log is like {"user":"abc#yahoo.com"}. I have a timestamp field for each activity, that describes when this activity was recorded. Can i generate date histogram on the basis of number of users in a particular time period. eg the histogram entry must show the number of users on that time. I can have this implemented by obtaining facet counts, but i need to get counts on various intervals and various ranges with minimum queries. Please guide me in this regard. Thanks.
Add a facet to your query something like the following:
{"facets": {
"daily_volume": {
"date_histogram": {
"size": 100,
"field": "created_at",
"interval": "day"
"order": "time"
}
}
}
This returns a nice set of ordered data for the number of items per day.
I then feed this to a Google Chart (the ColumnChart works nicely for histograms), doing a conversion on the returned timestamp integer to convert it to a Date type understood correctly by the Javascript charts API.