I have an issue similar to this Difference between #timestamp and timestamp field in elasticsearch. But I will need solution for it.
We use Kibana which by default use #timestamp as time filter. Yes, I can change it to whatever field manually EVERYTIME people create the time filter, but it is impossible for EVERYBODY in our big team to know it. So we need #timestamp.
#timestamp won't show up even I use the mapping here,
"_timestamp" : {
"enabled" : true,
"store" : true
}
So I workaround by adding a field name called #timestamp. I can use curl to add documents to it and The time filer start working.
However, when I move to use NEST api which cannot create #timestamp field. Even I define the field name as #timestamp, NEST api automatically change it to timestamp.
So Kibana time filter broken again.
Any suggestion?
Just figured it out. Nest API does have a way to explicitly set the field name.
[ElasticProperty(Name = "#timestamp", Type = FieldType.Date, DateFormat = "yyyy-MM-dd'T'HH:mm:ss", Store = true)]
So this is resolved.
Related
I am new to apache NiFi.
I am trying to put data into elasticsearch using nifi.
I want to specify an index name by combining a specific string and the value converted from a timestamp field into date format.
I was able to create the desired shape with the expression below, but failed to create the index name with the value of the timestamp field of the content.
${now():format('yyyy-MM-dd')}
example json data
{
"timestamp" :1625579799000,
"data1": "abcd",
"date2": 12345
}
I would like to get the following result:
index : "myindex-2021.07.06"
What should I do? please tell me how
I know that if you use the PutElasticSearch Processor, you can provide it with a specific index name to use. And as long as the index name meets the proper ElasticSearch format for naming a new index, if the enable auto index creation in ElasticSearch is turned on, then when sent, Elastic will create the new index with the desired name. This has worked for me. Double check the Elastic Naming Rules that can be found here or https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-indexing.html
I'm using Elastic Search 5.6.2 with Kibana and I'm currently facing a problem
My documents are indexed on the field timestamp which is normally an integer, however recently somebody has logged a document with a timestamp that is not an integer, and Kibana complains of conflicting type.
The discover panels display nothing and the following errors pop:
Saved "field" parameter is now invalid. Please select a new field.
Discover: "field" is a required parameter
How can I look for the document(s) causing these conflicts so that to find the service creating bad logs ?
The field type (either integer or text/keyword) is not defined on per document basis but rather on per index basis (in the mappings). I guess you are manipulating timeseries data, and you probably have un index per day (or per month or ...).
In Kibana Dev Tools:
List the created indices with GET _cat/indices
For each index (logstash-2017.09.28 in my example) do a GET logstash-2017.09.28/_mapping and check the type of the field in #timestamp
The field type is probably different between indices.
You won't be able to change the field type on created indices. Deleting document won't solve you're problem. The only solution is to drop the index or reindex the whole index with a new field type (in a specific mapping).
To avoid this problem on future indices, the solution is to create an index template with a mapping telling that the field #timestamp is of type date or whatever.
What is the most appropriate name for the timestamp when utilizing Logstash to parse logs into Elasticsearch, then visualizing with Kibana?
I am defining the timestamp using date in a filter:
date {
match => [ "logtime", "yy-MM-dd HH:mm:ss" ]
}
Logstash automatically puts this into the #timestamp field. Kibana can be configured to use any correctly formatted field as the timestamp, but it seems to be correct to use _timestamp in Elasticsearch. To do that, you have to mutate and rename the datestamp field.
mutate {
rename => { "#timestamp" => "_timestamp" }
}
Which is slightly annoying.
This question could be entirely semantic - but is it most correct to use _timestamp, or is it just fine to use #timestamp? Are there any other considerations which should influence the naming of the timestamp field?
Elasticsearch allows you to define fields starting with an underscore, however, Kibana (since v4) will only show the ones declared outside of the _source document.
You should definitely keep with #timestamp which is the standard way to name the timestamp field in Logstash. Kibana will not allow you to use _timestamp.
Please note that _timestamp is reserved and deprecated special field name. Actually any field names starting with underscore are reserved for elasticsearch future internal usage. AFAIK logstash documentation examples use #timestamp as field name
without any renaming.
I am building a project using elastic search for indexing and query large data.
The automatic mapper created on of my fields as double and i would like to change it to int, i can of course do it manually but i have been trying to do it with put command on the index\type
What i tried is :
PUT myindex/_mapping/model
{
"model" : {
"properties" : {
"programnumber" : {"type" : "integer", "store" : "yes"}
}
}
}
}
Thank you
Once created, fields cannot be changed anymore (with a few notable exceptions)
If your programnumber field was created as a double, it's most probably because the value of that field in the first document you indexed was a floating-point value.
Nowadays ElasticSearch allows you to go around this.
Just change your mapping and then reindex.
As you can read from the answer to a similar question: Change field type from string to integer? and how to re-index?
... you have to reindex. You can do that with Logstash (example configs have been posted in the past) or third-party tools like es-reindex. After reindexing to a new name (e.g. the original name with an underscore appended) you can delete the original index and create an alias named like the original index that points to the new index. Thereby everything will work as before.
Interesting link: Reindex API
Our data model doesn't have version field separately. One of the ways we versioned the data model is by the id and the last updated timestamp, the version will be incremented when a new record with same id but latest last updated timestamp is received.
However in elastic search, there is no way to derive the value of _id field. Multi fields cannot applied to _id field.
Our system is reactive and message driven, so can't rely on the order in which we receive the message.
is there anyways we can solve versioning in a performant way?
The _version field in elasticsearch is not for versioning. It's to ensure you are working on the expected document (e.g. you read a doc and decide to delete it, than it would be wise to add the version-number of the read doc to the delete command).
You can set the _id field to "[your_id]_[timestamp]" and add two additional fields "my_id" and "timestamp".
How to set the _id to "[your_id]_[timestamp]"? If you use logstash than you can use the mutate filter:
mutate { add_field => ["id", "%{your_id}", "_", "%{timestamp}"] }
should work. If you don't use logstash, you have to create the id-field similar.