I want to create a date histogram with opensearch dashboards. The time format of my data is YYYY-MM-DD HH:mm:ss.SSS, which I have set under Stack Management > Advanced Settings > Date Format. I get an error like this:
Under Discover, I can sort by "date", as it is of type "float". My field "timestamp", by which I would like to sort, is of type "string", and I cannot change this via the API:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [timestamp] cannot be changed from type [text] to [date]"}],"type":"illegal_argument_exception","reason":"mapper [timestamp] cannot be changed from type [text] to [date]"},"status":400}
I'm stuck, can someone please help?
To use a field for date histogram aggregation, the field type should be a date.
Unfortunately, it's not possible to change the field type from Kibana => Stack management.
Here is some solution for your case:
Use Histogram aggregation
Set the field type and re-index the data
Here are the steps for the second option.
#Check the mapping old_index = your existing index name
GET old_index
#Put the new mapping before reindexing
PUT new_index
{
"mappings": {
"properties": {
"timestamp": {
"type": "date",
"format": ["YYYY-MM-DD HH:mm:ss.SSS"]
}
}
}
}
#reindex the data
POST _reindex?wait_for_completion=false
{
"source": {
"index": "old_index"
},
"dest": {
"index": "new_index"
}
}
Related
I'm trying to implement partial search on a date field in elastic search. For example if startDate is stored as "2019-08-28" i should be able to retrieve the same while querying just "2019" or "2019-08" or "2019-0".
For other fields i'm doing this:
{
"simple_query_string": {
"fields": [
"customer"
],
"query": "* Andrew *",
"analyze_wildcard": "true",
"default_operator": "AND"
}}
which perfectly works on text fields, but the same doesn't work on date fields.
This is the mapping :
{"mappings":{"properties":{"startDate":{"type":"date"}}}}
Any way this can be achieved, be it change in mapping or other query method? Also i found this discussion related to partial dates in elastic, not sure if it's much relevant but here it is:
https://github.com/elastic/elasticsearch/issues/45284
Excerpt from ES-Docs
Internally, dates are converted to UTC (if the time-zone is specified)
and stored as a long number representing milliseconds-since-the-epoch.
It is not possible to do searching as we can do on a text field. However, we can tell ES to index date field as both date & text.e.g
Index date field as multi-type:
PUT sample
{
"mappings": {
"properties": {
"my_date": {
"type": "date",
"format": "year_month_day",//<======= yyyy-MM-dd
"fields": {
"formatted": {
"type": "text", //<========= another representation of type TEXT, can be accessed using my_date.formatted
"analyzer": "whitespace" //<======= whitespace analyzer (standard will tokenized 2020-01-01 into 2020,01 & 01)
}
}
}
}
}
}
POST dates/_doc
{
"date":"2020-01-01"
}
POST dates/_doc
{
"date":"2019-01-01"
}
Use wildcard query to search: You can even use n-grams at indexing time for faster search if required.
GET dates/_search
{
"query": {
"wildcard": {
"date.formatted": {
"value": "2020-0*"
}
}
}
}
I have an external ES db (i.e. I can't change its structure) with the following mapping
"failure_url": {
"properties": {
"lastAccessTime": {
"type": "long"
},
"url": {
"type": "keyword"
}
}
}
lastAccessTime represents a date, but is mapped as a long. A standard "range" filter fails with
"caused_by": {
"type": "number_format_exception",
"reason": "For input string: "now""
}
is the error in my filter expression or is it due to the field not being a "date"? If the latter, how can I still query this date?
If you want to query your field with now and other date match expressions, then your field must be defined as a date type.
One thing you can do is to create another field of type date and then update your index to index that new field.
So first change your mapping like this:
PUT my-index/_mappings
{
"properties": {
"lastAccessDate": {
"type": "date"
}
}
}
And then you can leverage the update by query API in order to index your dates from the long field:
POST my-index/_update_by_query
{
"script": {
"source": "ctx._source.lastAccessDate = ctx._source.lastAccessTime"
}
}
When that's done, you'll be able to run your range query on the new lastAccessDate field of type date.
I have json data that has a "product_ref" field that can take these values as an example:
"product_ref": "N/A"
"product_ref": "90323"
"product_ref": "SN3005"
"product_ref": "2015-05-23"
When pushing the data to the index i get a mapping error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [root.product_ref] of different type, current_type [date], merged_type [text]"}],"type":"illegal_argument_exception","reason":"mapper [root.product_ref] of different type, current_type [date], merged_type [text]"},"status":400}
Any idea?
There is something called date detection, and by default, it is enabled.
If date_detection is enabled (default), then new string fields are checked to see whether their contents match any of the date patterns specified in dynamic_date_formats. If a match is found, a new date field is added with the corresponding format.
You just need to disable it by modifying your mappings:
PUT /products
{
"mappings": {
"doc": {
"date_detection": false,
"properties": {
"product_ref": { "type": "keyword" },
}
}
}
}
This is happening because ElasticSearch assumed you're indexing dates of a particular format, and a value which doesn't match that was attempted to be indexed. i.e. after indexing date, you index wrong format.
Make sure all the values are dates and none are empty,perhaps remove these in your ingestion layer.
EDIT: If you don't care to lose the date value you can use the dynamic mapping.
{
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "date",
"mapping": {
"type": "text"
}
}
}
]
}
I want to index one field that contains only time in HH:mm:ss.SSS format. Which datatype I can use to store this field in elasticsearch.
You can use the date mapping type with a specific date format to accomodate your data:
{
"my_field": {
"type": "date",
"format": "hour_minute_second_fraction"
}
}
I have memory problems with aggregation queries.
my elastic version is 1.3.2
I tired to define _timestamp as doc value ,
but when I checked the mapping I can see it didn't work
It didn't happen in other fields.
Is there any known issue with timestamp field and doc values?
Lib
Have you tried this mapping?
{
"tweet" : {
"_timestamp" : {
"enabled" : true,
"format" : "YYYY-MM-dd"
}
}
I'm using specified version (13.2). I set up custom date field in my project like this and it's worked for me:
PUT 'http://127.0.0.1:9200/a252e39969665bb4d065/' -d
'{
"a252e39969665bb4d065": {
"mappings": {
"_default_": {
"properties": {
"createdDate": {
"type": "date",
"format": "dateOptionalTime"
}
}
}
}
}
}'
Please, note that i'm using default mapping here (default mapping for all types in index). You can use specified type in an index by replacing "default" in mapping.