In elasticsearch, each document can specify a _parent document, creating a one-to-many relationship. Can the same relationship be created with nested documents?
It does not appear so. Below are commands that create an index, with a mapping that has a _parent defined for a nested property. After executing these commands, the mapping is successfully created, but the _parent relationship is not. Am I doing something wrong or is this not possible?
curl -XPUT localhost:9200/my_index
curl -XPUT localhost:9200/my_index/_mapping/new_obj -d '{
"new_obj" : {
"properties" : {
"my_prop" : { "type" : "string"},
"my_nested_prop" : {
"type" : "nested",
"_parent" : { "type" : "new_obj"},
"properties" : {
"amount" : { "type" : "integer" },
"description" : { "type" : "string" }
}
}
}
}
}'
Related
I have an index in elasticsearch 6.2.4, for which i created a new index in new elasticsearch cluster 7.6.1.
I have copied the mapping for this index from 6.2.4 to 7.6.1, but when i tried to _reindex from 6.2.4 to 7.6.1.
I get the below error.
"failures" : [
{
"index" : "newindex",
"type" : "_doc",
"id" : "someid",
"cause" : {
"type" : "mapper_parsing_exception",
"reason" : "failed to parse field [UPDATES.when] of type [date] in document with id 'someid'. Preview of field's value: '1.528501444E9'",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "failed to parse date field [1.528501444E9] with format [epoch_second]",
"caused_by" : {
"type" : "date_time_parse_exception",
"reason" : "Failed to parse with all enclosed parsers"
}
}
},
"status" : 400
}
_reindex call done at 7.6.1's kibana
POST _reindex/?pretty
{
"source": {
"remote": {
"host": "http://oldserver:9200"
},
"index": "oldindex",
"query": {
"match_all": {}
}
},
"dest": {
"index": "newindex"
}
}
Mapping of updates field is same in both places
"UPDATES" : {
"properties" : {
"key" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"when" : {
"type" : "date",
"format" : "epoch_second"
}
What am I missing here ?
I guess the timestamp which is appearing in one of your date feild 1.528501444E9 is UNIX timestamp in scientific notation.
But Elasticsearch fails because it can't parse 1.528501444E9 since I suppose as per you exception the format that you have given for this field is epoch_second which does not take this format.
You can read futher related to this format from here https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html
I'm trying an example from https://www.elastic.co/blog/data-visualization-elasticsearch-aggregations
When I try to create indecies and upload data, I get the folllowing error:
rolf#PE:~/nfl/scripts/Elasticsearch-datasets-master/mappings$ curl -XPUT localhost:9200/nfl?pretty
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "nfl"
}
rolf#PE~/nfl/scripts/Elasticsearch-datasets-master/mappings$ curl -XPUT localhost:9200/nfl/2013/_mapping?pretty -d #nfl_mapping.json
{
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "_index is not configurable"
}
],
"type" : "mapper_parsing_exception",
"reason" : "_index is not configurable"
},
"status" : 400
}
The start of the mapping file is as follows:
{
"2013" : {
"_index" : {
"enabled" : true
},
"_id" : {
"index" : "not_analyzed",
"store" : "yes"
},
"properties" : {
"gameid" : {
"type" : "string",
"index" : "not_analyzed",
"store" : "yes"
}, ...
Appreciate some hints. Thanks.
You're probably using a recent version of ES and the nfl_mapping.json mapping is for an older version. In recent versions, it is not possible anymore to specify _index and _id in your mapping. Change it to this and it will work
{
"2013" : {
"properties" : {
"gameid" : {
"type" : "keyword"
}, ...
Also change all occurrences of string with text and string+not_analyzedto keyword.
After that you should be good to go.
Also note that "index" : "not_analyzed" is no longer supported.
I am trying to stream logs from logstash to elasticsearch (5.5.0). I am using filebeat to send logs to logstash.
I have not defined any index; it is defined automatically (say "test1") when data is pushed for the first time.
Now, I want to create another index ("test2") so that I can manage field data types. For that, I got the mappings for test1. Updated the index name. And did PUT call for test2 with this data. However, it fails with following result:
`ubuntu#elasticsearch:~$ curl -XPUT 'localhost:9200/test2?pretty' -H 'Content-Type: application/json' -d'#/tmp/mappings_test.json'
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "unknown setting [index.test2.mappings.log.properties.#timestamp.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
}
],
"type" : "illegal_argument_exception",
"reason" : "unknown setting [index.test2.mappings.log.properties.#timestamp.type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
},
"status" : 400
}
`
Following is the excerpt of the json which I am using.
`
{
"test2" : {
"mappings" : {
"log" : {
"properties" : {
"#timestamp" : {
"type" : "date"
},
"#version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"accept_date" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
....
`
I modified index name only. Rest of the content is same as mapping of test1 index.
Any help is appreciated on how to create this new index by updating types?
You need to remove test2 on the second line and have only mappings:
PUT test2
{
"mappings" : { <---- this needs to be at the top level
"log" : {
"properties" : {
"#timestamp" : {
"type" : "date"
},
"#version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"accept_date" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
....
I would like to disable all the "raw" fields that are created in Elasticsearch by logstash-forwarder. So if I have a field as "host" logstash-forwarder won't create a "host.raw" field. But I need a general solution for all the string fields.
I have my string fields as "not_analyzed" so having raw fields has no point and just a duplicate of the data.
I tried to remove "fields" part of the mapping below but it's added back after the first log message. The closest thing I could achieve was to add the following mapping but that still creates empty raw fields:
curl -XPUT 'localhost:9200/myindex/' -d '{
"mappings": {
"_default_": {
"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string",
"fields" : {
"raw" : {
"ignore_above" : 0,
"index" : "not_analyzed",
"type" : "string"
}
}
},
"match" : "*",
"match_mapping_type" : "string"
}
} ],
"_all": { "enabled": false }
}
}
}'
So how can I disable these fields?
I create an index with a mapping which contains a field not_analyzed with command below and index a document with next command.
curl -XPUT localhost:9200/twitter -d '{
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1
},
"mappings": {
"tweet" : {
"properties" : {
"message" : { "type" : "string",
"index": "not_analyzed"}
}
}
}
}'
curl -XPOST 'http://localhost:9200/twitter/tweet?' -d '{
"user" : "kimchy",
"postDate" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}
'
I checked to mappings with http://localhost:9200/twitter/_mapping?pretty=true and it outputs:
{
"twitter" : {
"mappings" : {
"tweet" : {
"properties" : {
"message" : {
"type" : "string",
"index" : "not_analyzed"
},
"post_date" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"user" : {
"type" : "string"
}
}
}
}
}
}
Finally when I search with this http://localhost:9200/twitter/tweet/_search?pretty=1&q=trying it finds the indexed document. Is it normal? I thought it should not find it unless I search the complete text "trying out Elasticsearch".
not_analyzed means that it's not doing tokenizing/other analysis to index the values, but it does still store the full value in Elasticsearch and it can be used as an exact match in a terms query. The field value is still getting included/analyzed into the _all field and indexed there so that it's searchable.
You need to set "include_in_all": false or "index": "no" to disable that.
See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html for more information.