Ok, in my elastisearch I am using the following mapping for an index:
{
"mappings": {
"mytype": {
"type":"object",
"dynamic" : "false",
"properties": {
"name": {
"type": "string"
},
"address": {
"type": "string"
},
"published": {
"type": "date"
}
}
}
}
}
it works. In fact if I put a malformed date in the field "published" it complains and fails.
Also I've the following configuration:
...
node.name : node1
index.mapper.dynamic : false
index.mapper.dynamic.strict : true
...
And without the mapping, I can't really use the type. The problem is that if I insert something like:
{
"name":"boh58585",
"address": "hiohio",
"published": "2014-4-4",
"test": "hophiophop"
}
it will happily accept it. Which is not the behaviour that I expect, because the field test is not in the mapping. How can I restrict the fields of the document to only those that are in the mapping???
The use of "dynamic": false tells Elasticsearch to never allow the mapping of an index to be changed. If you want an error thrown when you try to index new documents with fields outside of the defined mapping, use "dynamic": "strict" instead.
From the docs:
"The dynamic parameter can also be set to strict, meaning that not only new fields will not be introduced into the mapping, parsing (indexing) docs with such new fields will fail."
Since you've defined this in the settings, I would guess that leaving out the dynamic from the mapping definition completely will default to "dynamic": "strict".
Is your problem with the malformed date field?
I would fix the date issue and continue to use dynamic: false.
You can read about the ways to set up the date field mapping for a custom format here.
Stick the date format string in a {type: date, format: ?} mapping.
Related
I am new to elastic search. I have an index named users, which has a lot of fields I know. But a few more fields can be added in the future.
So when defining my mapping, I want to include the fields that I know currently with dynamic "strict", but in the future, if I want to add the new field, how will update the new mapping and if I do it, will I have to reindex everything.
I found in the ES documents that mappings are applied only during index creation time. So I am a little confused here, what's the right way to approach this.
You can always update the mapping in future, even after keeping it strict using the put mapping api. You'll not require existing data to be re-indexed unless you want the newly added field have some value for the older documents which were added before updating the mapping with the new field.
Lets assume you already have an index test with one field say field1 of type keyword. Now in future you have a requirement to add new field say field2 of integer type. You can do so by the put mapping api as below,
PUT test/_mapping
{
"properties": {
"field2": {
"type": "integer"
}
}
}
After executing the above if you check the mapping using
GET test/_mapping
You can see the new field as well in the response,
{
"test" : {
"mappings" : {
"dynamic" : "strict",
"properties" : {
"field1" : {
"type" : "keyword"
},
"field2" : {
"type" : "integer"
}
}
}
}
}
Inner objects inherit the dynamic setting from their parent object or from the mapping type. In the following example, dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
However, the user.social_networks object enables dynamic mapping, so you can add fields to this inner object.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
PUT my-index-000001
{
"mappings": {
"dynamic": false,
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}
I have a big size field in MySQL and do not want to save the original value to ElasticSearch. Is there a method just like Lucene Field.Store.NO?
Thanks.
You just need to define the "store" mapping accordingly, eg. :
PUT your-index
{
"mappings": {
"properties": {
"some_field": {
"type": "text",
"index": true,
"store": false
}
}
}
}
You may also want to disable the _source field :
#disable-source-field
The _source field contains the original JSON document body that was passed at index time [...] Though very handy to have around, the source field does incur storage overhead within the index.
For this reason, it can be disabled as follows:
PUT your-index
{
"mappings": {
"_source": {
"enabled": false
}
}
}
I have json data that has a "product_ref" field that can take these values as an example:
"product_ref": "N/A"
"product_ref": "90323"
"product_ref": "SN3005"
"product_ref": "2015-05-23"
When pushing the data to the index i get a mapping error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [root.product_ref] of different type, current_type [date], merged_type [text]"}],"type":"illegal_argument_exception","reason":"mapper [root.product_ref] of different type, current_type [date], merged_type [text]"},"status":400}
Any idea?
There is something called date detection, and by default, it is enabled.
If date_detection is enabled (default), then new string fields are checked to see whether their contents match any of the date patterns specified in dynamic_date_formats. If a match is found, a new date field is added with the corresponding format.
You just need to disable it by modifying your mappings:
PUT /products
{
"mappings": {
"doc": {
"date_detection": false,
"properties": {
"product_ref": { "type": "keyword" },
}
}
}
}
This is happening because ElasticSearch assumed you're indexing dates of a particular format, and a value which doesn't match that was attempted to be indexed. i.e. after indexing date, you index wrong format.
Make sure all the values are dates and none are empty,perhaps remove these in your ingestion layer.
EDIT: If you don't care to lose the date value you can use the dynamic mapping.
{
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "date",
"mapping": {
"type": "text"
}
}
}
]
}
as mentioned in the title, I want to disable index a specified field in elasticsearch, for example, I have a fields named #fileds which contains three sub-fields like name、age、salary, now I do not want to index the field #fields.age in elasticsearch, how can I achieve that? I have tried to use include_in_all parameters, but it doesn't work. mapping configuration like:
"mappings": {
"fluentd": {
"properties": {
"#fields": {
"properties": {
"age": {
"type": "text",
"include_in_all": false,
"index": "no"
}
}
}
}
}
}
when I use this mapping configuration above, I can only see #fields.age in the index's mapping, #fields.name and #fields.salary should appear on the index's mapping not the #fields.age, how can this happen? any answers will be appreciated.
After updating to ElasticSearch 2 I am no more able to map the ContextSuggester for different types:
PUT /test/foo/_mapping
{
"properties": {
"suggest": {
"type": "completion",
"context": {
"type": {
"type": "category",
"path": "_type",
"default": [
"foo"
]
}
}
}
}
}
PUT /test/bar/_mapping
{
"properties": {
"suggest": {
"type": "completion",
"context": {
"type": {
"type": "category",
"path": "_type",
"default": [
"bar"
]
}
}
}
}
}
Putting the map for the second type ends in the following exception:
Mapper for [suggest] conflicts with existing mapping in other types: [mapper [suggest] has different [context_mapping] values]
The problem is that the default value differs for the different types. From my point of view, this should be the expected approach. How can I solve this problem?
Tested version of ES: 2.2.1
You have a field conflict.
Mapping - field conflicts
Mapping types are used to group fields, but the fields in each
mapping type are not independent of each other. Fields with:
the same name
in the same index
in different mapping types
map to the same field internally, and must have the same mapping. If a
title field exists in both the user and blogpost mapping types, the
title fields must have exactly the same mapping in each type. The only
exceptions to this rule are the copy_to, dynamic, enabled,
ignore_above, include_in_all, and properties parameters, which may
have different settings per field.
Either create a separate index or rename the field for the other type.