I fail at indexing timestamp fields with ElasticSearch (version 7.10.2) and I do not understand why.
So I create an index with the following mapping. You can copy & paste it directly to Kibana:
PUT /my-dokumente
{
"mappings" : {
"properties" : {
"aufenthalt" : {
"properties" : {
"aufnahme" : {
"properties" : {
"zeitpunkt" : {
"type" : "date",
"format": "yyyy-MM-dd HH:mm:ss",
"ignore_malformed": true
}
}
},
"entlassung" : {
"properties" : {
"zeitpunkt" : {
"type" : "date",
"format": "yyyy-MM-dd HH:mm:ss",
"ignore_malformed": true
}
}
}
}
}
}
}
}
Then I try to index a document:
PUT /my-dokumente/dokumente/1165963
{
"aufenthalt" :
{
"aufnahme" :
{
"zeitpunkt" : "2019-08-18 15:02:13"
},
"entlassung" :
{
"zeitpunkt" : "2019-08-20 10:29:22"
}
}
}
Now, i get this error:
"mapper [aufenthalt.entlassung.zeitpunkt] cannot be changed from type [date] to [text]
Why is elastic search not parsing my date?
I also tried with many different mapping settings like strict_date_hour_minute_second or to send the timestamp as "2019-08-18T15:02:13" or "2019-08-18T15:02:13Z" also, I converted it to epoch millis, but I always get some different error message, for example Cannot update parameter [format] from [strict_date_hour_minute_second] to [strict_date_optional_time||epoch_millis].
So the basic question is just: How can I send a timestamp value to ElasicSearch? (with Kibana/CURL).
PS: I am not using a Client SDK like Java High Level Rest Client. Why are talking about basic Kibana/CURL.
It can't be that complicated. What am I missing?
Thank you!
Mapping types are removed in 7.x. Refer to this official documentation
You need to add _doc in URL when indexing a document to Elasticsearch
Modify the URL as PUT /my-dokumente/_doc
Related
I'm trying to get Elasticsearch to recognise strings in the format yyyy-MM-dd HH:mm:ss as date fields. I've created a dynamic date format and applied that to the default mapping. It works nicely when I index documents of my first type - any new fields where the data is in this format get initialised as date fields.
The problem comes when I try to create documents of a new type, but with date format fields with the same name as in my first type. These fail with a malformed date error.
Here's an example set of Kibana commands to demonstrate:
DELETE /datetest
PUT /datetest
PUT /datetest/_mapping/_default_
{
"dynamic_date_formats" : ["yyyy-MM-dd HH:mm:ss"]
}
PUT /datetest/doc/1
{
"date" : "2015-01-01 12:00:00"
}
# This one works fine
PUT /datetest/otherdoc/1
{
"otherdate" : "2015-01-01 12:00:00"
}
# This one does not
PUT /datetest/otherdoc/2
{
"date" : "2015-01-01 12:00:00"
}
The last command gives this error:
"Invalid format: \"2015-01-01 12:00:00\" is malformed at \" 12:00:00\""
I know that fields with the same name in different types must have the same data type, but in this case, I want them to have the same data type - date. I could manually create the mappings for each new type, but I want it to automatically support new types added to my source data. It seems to be what the dynamic date format is supposed to do. Am I doing something wrong here?
I would create a custom dynamic template. Something like this:
PUT /datetest/_mapping/_default_
{
"date_detection" : true,
"dynamic_templates" : [
{
"dates" : {
"match" : ".*date.*",
"mapping" : {
"type" : "date",
"format" : 'yyyy-MM-dd HH:mm:ss'
}
}
}
]
}
Just tried it. It seems to work. I hope this helps :)
All I can think of, would be adding dynamic template:
PUT /datetest
{
"mappings": {
"_default_": {
"date_detection": false,
"dynamic_templates": [
{
"dates": {
"match": ".*Date|date",
"match_pattern": "regex",
"mapping": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||dateOptionalTime"
}
}
}
]
}
}
}
All three statements will run just fine:
PUT /datetest/doc/1
{ "date" : "2015-01-01 12:00:00" }
PUT /datetest/otherdoc/1
{ "otherdate" : "2015-01-01 12:00:00" }
PUT /datetest/otherdoc/2
{ "date" : "2015-01-01 12:00:00" }
Dynamic template will try to match your field name on based pattern. They need to end with date or Date.
I know you need just yyyy-MM-dd HH:mm:ss format, but I've also added default one, so that Elasticsearch can pick one from multiple ones.
I have multiple mappings which come from the same datasource but have small differences, like the example below.
{
"type_A" : {
"properties" : {
"name" : {
"type" : "string"
}
"meta_A" : {
"type" : "string"
}
}
}
}
{
"type_B" : {
"properties" : {
"name" : {
"type" : "string"
}
"meta_B" : {
"type" : "string"
}
}
}
}
What I want to be able to is:
Directly query specific fields (like meta_A)
Directly query all documents from the datsource
Query all documents from a specific mapping
What I was looking into is the type filter, so preferably I could write a query like this:
{
"query": {
"filtered" : {
"filter" : {
"type" : { "value" : "unified_type" }
}
}
// other query clauses
}
}
So instead of typing "type_A","type_B" in an or clause in the type filter I would like to have this "unified_type", but without giving up the possibility to directly query "type_A".
How could I achive this?
I don't think that it's possible. However, you could use copy_to functionality, so you would have your fields as they are now and their values copied into unified name.
The copy_to parameter allows you to create custom _all fields. In
other words, the values of multiple fields can be copied into a group
field, which can then be queried as a single field. For instance, the
first_name and last_name fields can be copied to the full_name field
as follows:
So you'd be copying both "meta_A" and "meta_B" into some "unified_meta" field and query this one.
Let me first explain my scenario.
I am fetching data from RDBMS and pushing it into ElasticSearch.
Fetched Results are in the form of List and i am preparing bulk index request like this:
BulkRequestBuilder bulkRequest = client.prepareBulk();
for (Map<String,Object> singleDataRow : ResultSet)
{
IndexRequest indexRequest = new IndexRequest("testindex","testtype",singleDataRow.getObject("NAME"));
bulkRequest.add(indexRequest);
}
bulkRequest.execute().actionGet();
My Map = includes Map of string to string, string to big decimal, string to big integer etc.
eg.
{ BIRTHDATE : 2015-03-05 , NAME : deepankar , AGE : 22 , AMOUNT : 15.5 }
But when i see the mapping of my testtype in testindex, all mapping of fields are of "type" : "string"
Why the fields does not maps to "type": "string" , or "type" : "long" , and even "type" : "date" as elasticsearch does it by default?
Elasticsearch will attempt to 'guess' the field type by the first insert, unless you create and map fields beforehand.
There are two possible reasons why your fields are being indexed as string instead of long or any other type:
You're not really sending these fields as int, so you're sending '10' instead of 10
You've already inserted at least 1 document that had a string value for that field, so if you've inserted your first document with AGE: '22' Elasticsearch will set that field to type: string and any future inserts will have a string value.
If you want to make sure, you can delete the current index, re-create it and manually set up mapping before inserting the first document, like so:
curl -XPUT 'http://localhost:9200/testindex/_mapping/testmapping' -d '
{
"testmapping" : {
"properties" : {
"birthdate" : { "type" : "date", "format": "dateOptionalTime" },
"name" : { "type" : "string" },
"age" : { "type" : "long" },
"amount" : { "type" : "double" }
}
}
}
'
I am trying to measure the latency involved with using the ELK stack. I am logging in my test application and I want to find out how long it takes before it appears in ElasticSearch. I understand this will only be a rough estimate and is very specific to my environment.
How can I measure the latency between my app/logstash/elasticsearch?
EDIT:
I am following suggestion and enabled _timestamp but I don't see the field in my records.
{
logaggr : {
order : 0,
template : "logaggr-*",
settings : {},
mappings : {
logaggr : {
date_detection : false,
_timestamp : {
enabled : true,
store: true
},
properties : {
level : {
type : "string"
},
details : {
type : "string"
},
logts : {
format : "yyyy-MM-dd HH:mm:ss,SSS",
type : "date"
},
classname : {
type : "string"
},
thread : {
type : "string"
}
}
}
},
aliases : {}
}
}
Thanks in advance!
There are three timestamps that will answer your question:
the log file timestamp, e.g. when the application wrote the information. Make sure your server's clock is correct.
#timestamp, which is set by logstash to the time when it receives the log.
_timestamp, which elasticsearch can set to the time when it receives the log. This setting must be enabled in elasticsearch.
Between these three, you can track the progress of your logs through ELK.
Seems I'm way out of my league here and hope that someone frome the Elasticsearch community can tune in and pinpoint me to my error.
I wanted to try out Kibana + Elastic search for fiddling with some COUNTER3-compliant stats we have and for this needed to import CSV files.
I choose the CSV River plugin to import the csv files.
The river-csv was configured with following PUT request to http://localhost:9200/_river/test_csv_river/_meta
{
"type": "csv",
"csv_file": {
"folder": "C:\\localdata\\UsageStats\\COUNTER3",
"first_line_is_header": "true"
}
}
This works fine but it turns out that the number fields are imported as strings, not numbers. Confirmed by checking on the indexes type mapping : http://localhost:9200/test_csv_river/csv_type/_mapping
{
test_csv_river: {
mappings: {
csv_type: {
properties: {
ft_total: {
type: "string"
},
.....
}
}
}
}
}
This related SO question made me believe that I can change the types field properties AFTER I have created the index: How to update a field type in elasticsearch?
But when I made the following PUT Request I get an error
{"error":"MergeMappingException[Merge failed with failures {[mapper [ft_total] of different type, current_type [string], merged_type [integer]]}]","status":400}
http://localhost:9200/test_csv_river/csv_type/_mapping
{
"csv_type" : {
"properties" : {
"ft_total" : {"type" : "integer", "store" : "yes"}
}
}
}
Is there no way to update that type from string > integer?
Alternatively, is there any way I can make sure the index is created with a specific field predefined as "type" : "integer" when using the CSV river plugin ?