failed to parse timestamp elasticsearch - elasticsearch

I'm new to elasticsearch and am trying to create my first index but am having issues with a timestamp field that was working before...
I created my index like this:
PUT /kafkasdp
{
"mappings": {
"kafka_logs": {
"properties": {
"timestamp": {
"type": "date"
},
"log_level": {
"type": "string"
},
"message1": {
"type": "string"
},
"message2": {
"type": "string"
}
}
}
}
}
and then I'm trying to send data like this:
post /kafkasdp/kafka_logs
{
"timestamp": "2017-02-03 19:27:20,606",
"log_level": "INFO",
"message2": "Deleting segment 1 from log omega-replica-sync-dev-8. (kafka.log.Log)"
}
but keep getting this error:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "failed to parse [timestamp]"
}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse [timestamp]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Invalid format: \"2017-02-03 19:27:20,606\" is malformed at \" 19:27:20,606\""
}
},
"status": 400
}
I thought my timestamp is a valid date type?

Read about date type on Elasticsearch reference: you should specify format of date you are expecting in your documents:
PUT your_index_name
{
"mappings": {
"your_index_type": {
"properties": {
"date": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss,SSS"
}
}
}
}
}
As you did not specify it, Elasticsearch will expect date value in ISO format:
yyyyMMdd'T'HHmmss.SSS'Z' (e.g., 2017-02-03T19:27:20.606Z)

Related

Update a string parameter in Elasticsearch _mapping

I have such a _mapping in Elasticsearch 6.8:
{
"grch38_test__wes__grch38__variants__20210222" : {
"mappings" : {
"variant" : {
"_meta" : {
"gencodeVersion" : "25",
"hail_version" : "0.2.20",
"genomeVersion" : "38",
"sampleType" : "WES",
"sourceFilePath" : "s3://my_folder/my_vcf.vcf"
},
...
My goal is to issue a query in Kibana to modify variant._meta.sourceFilePath. Following thread:
Elastic search mapping for nested json objects
I was able to come up with the query:
PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
"properties": {
"variant": {
"type": "nested",
"properties": {
"_meta": {
"type": "nested",
"properties": {
"type": "text",
"sourceFilePath": "s3://my_folder/my_vcf.vcf"
}
}
}
}
}
}
But its giving me an error:
elasticsearch mapping Expected map for property [fields] on field [name] but got a class java.lang.String
Full error message:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Expected map for property [fields] on field [type] but got a class java.lang.String"
}
],
"type": "mapper_parsing_exception",
"reason": "Expected map for property [fields] on field [type] but got a class java.lang.String"
},
"status": 400
}
I have also tried:
PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
"properties": {
"variant": {
"type": "nested",
"properties": {
"_meta": {
"type": "nested",
"properties": {
"sourceFilePath": {
"type": "text",
"value":"s3://my_folder/my_vcf.vcf"
}
}
}
}
}
}
}
But its telling me that value is unsupported:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [sourceFilePath] has unsupported parameters: [value : s3://seqr-dp-data--prod/vcf/dev/grch38_test_contracted.vcf]"
}
],
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [sourceFilePath] has unsupported parameters: [value : s3://seqr-dp-data--prod/vcf/dev/grch38_test_contracted.vcf]"
},
"status": 400
}
What am I doing wrong? How to modify the field?
_meta is a reserved field for storing application-specific metadata. It's not meant to be searchable and can be only retrieved through the GET Mapping API.
This means that if your _meta content was intended to be consistent with what the _meta field is designed for, you cannot apply any mappings to it. It's a "final" hashmap of concrete values and would need to be defined at the top level of your update-mapping payload:
PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
"_meta": {
"variant": { <-- shared index-level metadata
"gencodeVersion": "25",
"hail_version": "0.2.20",
"genomeVersion": "38",
"sampleType": "WES",
"sourceFilePath": "s3://my_folder/my_vcf.vcf"
}
},
"properties": {
"some_text_field": { <-- actual document properties
"type": "text"
}
}
}
If, on the other hand, your _meta field is an unfortunate naming coincidence, you can declare the mappings for it like so:
PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
"properties": {
"_meta": {
"properties": {
"variant": {
"properties": {
"gencodeVersion": {
"type": "text"
},
"genomeVersion": {
"type": "text"
},
"hail_version": {
"type": "text"
},
"sampleType": {
"type": "text"
},
"sourceFilePath": {
"type": "text"
}
}
}
}
}
}
}
and ingest documents of the form:
POST grch38_test__wes__grch38__variants__20210222/variant/_doc
{
"_meta": {
"variant": {
"gencodeVersion": "25",
"hail_version": "0.2.20",
"genomeVersion": "38",
"sampleType": "WES",
"sourceFilePath": "s3://my_folder/my_vcf.vcf"
}
}
}
But again, the _meta content would be document-specific, not index-wide!
And BTW, the nested mapping only makes sense if you're dealing with arrays of objects, not objects of objects.
But if you insist on wanting it, here's how you'd do it:
PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant?include_type_name
{
"properties": {
"_meta": {
"type": "nested", <---
"properties": {
"variant": {
"type": "nested", <---
"properties": {
"gencodeVersion": {
"type": "text"
},
"genomeVersion": {
"type": "text"
},
"hail_version": {
"type": "text"
},
"sampleType": {
"type": "text"
},
"sourceFilePath": {
"type": "text"
}
}
}
}
}
}
}

Trying to parital update a doc but getting error regarding date field with epoch_second format

I'm trying to partially update an existing document which is already in the index and is indexed well, meaning I can view it in Kibana which uses timestamp field to display the documents. I'm trying to update only the doc's name of id test_id
my request is:
POST shirkan_test/_update/test_id
{
"doc": {
"name": "shirkan"
},
"doc_as_upsert": true
}
Getting the following error:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "failed to parse field [timestamp] of type [date] in document with id 'test_id'. Preview of field's value: '1.602505857664299E9'"
}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse field [timestamp] of type [date] in document with id 'test_id'. Preview of field's value: '1.602505857664299E9'",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "failed to parse date field [1.602505857664299E9] with format [epoch_second]",
"caused_by": {
"type": "date_time_parse_exception",
"reason": "Failed to parse with all enclosed parsers"
}
}
},
"status": 400
}
Much appreciate any help with this.
Thanks.
EDIT: adding index mapping
{
"mapping": {
"properties": {
"condition": {
"type": "text",
"index": false
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"rank": {
"type": "double"
},
"timestamp": {
"type": "date",
"format": "epoch_second"
}
}
}
}
EDIT 2: Changing timestamp format to strict_date_optional_time_nanos doesn't yield such an error and the upsert works well.
Looks like for now, the solution which worked for me to this problem is to change the timestamp field format from epoch_second to strict_date_optional_time_nanos. Other formats may work as well. I had to completely delete the index and recreate it since I came across the same error message when trying to re-index.
As mentioned in one of my comments, I filed a bug report here:
https://github.com/elastic/elasticsearch/issues/64050

elasticsearch mapping issue: failed to parse field

I have this mapping
PUT /mytest
{
"mappings":{
"properties": {
"value": { "type": "object" }
}
}
}
When I insert this document
POST /mytest/_doc/4
{
"value": { "value": "test"}
}
I got the following error:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "failed to parse field [value.value] of type [long] in document with id '4'. Preview of field's value: 'test'"
}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse field [value.value] of type [long] in document with id '4'. Preview of field's value: 'test'",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "For input string: \"test\""
}
},
"status": 400
}
I know the naming convention is bad, still, this is a valid JSON request, not sure why it doesn't allow it.
This error is telling you that you don't have a mapping for the property value within your value object property. The below example would property set the value.value property within your mytest index:
PUT mytest
{
"mappings": {
"properties": {
"value": {
"type": "object",
"properties": {
"value": {
"type": "text"
}
}
}
}
}
}
However, I don't think that's what your intention was. As a best practice, try following the Elastic Common Schema (ECS) for naming your index properties.

ElasticSearch action_request_validation_exception

I'm creating mapping for multiple type
here my query
PUT opl_consultation/_mapping
my json mapping file
{
"mappings": {
"article": {
"properties": {
"numero_noeud": { "type": "text" },
"intitule_fr": { "type": "text" },
"path_audio": { "type": "text" }
}
},
"hierarchie": {
"properties": {
"id_type_noeud_hie": { "type": "integer" },
"noeud_numero_hie": { "type": "text" },
"intitule_hie_fr": { "type": "text" }
}
},
"law_type": {
"properties": {
"id_type_loi": { "type": "integer" },
"Desc_law_type": { "type": "text" }
}
}
}
}
below the error a got
"type": "action_request_validation_exception",
"reason": "Validation Failed: 1: mapping type is missing;"
},
"status": 400
}
the version is Elasticsearch\6.4.2
In Elasticsearch 6.4.2 you cannot have more than one mapping type. See https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
If you run your query instead as PUT opl_consultation with your mapping definition you will get the below error
"type": "illegal_argument_exception",
"reason": "Rejecting mapping update to [opl_consultation] as the final mapping would have more than 1 type: [law_type, article, hierarchie]"
Instead, use a custom type field as described here

Elasticsearch percolate mapping error

I want to use percolate query in elasticsearch. But I couldn't setup mapping. I have received the following error. where is my fault?
PUT /my-index
{
"mappings": {
"doctype": {
"properties": {
"message": {
"type": "string"
}
}
},
"queries": {
"properties": {
"query": {
"type": "percolator"
}
}
}
}
}
Error Message:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "No handler for type [percolator] declared on field [query]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [queries]: No handler for type [percolator] declared on field [query]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "No handler for type [percolator] declared on field [query]"
}
},
"status": 400
}
Thanks

Resources