I have an EFK pipeline set up. Everyday a new index is created using the logstash-* prefix. Every time a new field is sent by Fluentd, the field is added to the index pattern logstash-*. I'm trying to create an index template that will disable indexing on a specific field when an index is created. I got this to work in ES 7.1 using the PUT below:
PUT _template/logstash-test
{
"index_patterns": ["logstash-*"],
"mappings": {
"dynamic_templates" : [
{
"params" : {
"path_match" : "params",
"mapping" : {
"enabled": false
}
}
}
]
}
}
However when I try this on Elasticsearch 6.8 I get the following error:
"type": "illegal_argument_exception",
"reason": "Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping"
It is a little different in Elasticsearch 6.X as it had mapping types, which is not used anymore.
Try something like this:
PUT _template/logstash-test
{
"index_patterns": ["logstash-*"],
"mappings": {
"_doc": {
"dynamic_templates" : [
{
"params" : {
"path_match" : "params",
"mapping" : {
"enabled": false
}
}
}
]
}
}
}
If your index has a different custom type and is not using the _doc type, you should use that in the mapping.
Related
I have the mapping
{
"test" : {
"mappings" : {
"properties" : {
"description" : {
"type" : "text"
},
"location" : {
"type" : "keyword",
"index" : false
},
"title" : {
"type" : "text"
}
}
}
}
}
and I want to update the index parameter of the location field to true
I am trying
PUT /test/_mapping
{
"properties": {
"location": {
"type": "keyword",
"index": true
}
}
}
and I am getting
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Mapper for [location] conflicts with existing mapping:\n[mapper [location] has different [index] values]"}],"type":"illegal_argument_exception","reason":"Mapper for [location] conflicts with existing mapping:\n[mapper [location] has different [index] values]"},"status":400}
How to update the index parameter?
What you are trying to achieve is called breaking changes or conflicting changes and is not possible and same is mentioned in the error message.
Think of what index param does and why its breaking changes, from index docs
The index option controls whether field values are indexed. It accepts
true or false and defaults to true. Fields that are not indexed are
not queryable.
Earlier index value was false so your existing documents didn't have value indexed and wasn't queryable and now you changing to true which doesn't make sense as your earlier documents will not have the indexed value and that's the reason its called breaking changes.
You have to create a new index with new index value and you can use the reindex API for that.
hi I want to set custom routing on specific field "userId" on my Es v2.0.
But it giving me error.I don't know how to set custom routing on ES v2.0
Please guys help me out.Thanks in advance.Below is error message, while creating custom routing with existing index.
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
}
],
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
},
"status": 400
}
In ES 2.0, the _routing.path meta-field has been removed. So now you need to do it like this instead:
In your mapping, you can only specify that routing is required (but you cannot specify path anymore):
PUT my_index
{
"mappings": {
"my_type": {
"_routing": {
"required": true
},
"properties": {
"name": {
"type": "string"
}
}
}
}
}
And then when you index a document, you can specify the routing value in the query string like this:
PUT my_index/my_type/1?routing=bar
{
"name": "foo"
}
You can still use custom routing based on a field from the data being indexed. You can setup a simple pipeline and then use the pipeline every time you index a document, or you can also change the index settings to use the pipeline whenever the index receives a document indexing request.
Read about the pipeline here
Do read above and below the docs for more clarity. It's not meant for setting custom routing, but can be used for the purpose. Custom routing was disabled for a good reason that the field to be used can turn out to have null values leading to unexpected behavior. Hence take care of that issue by yourself.
For routing, here is a sample pipeline PUT :
PUT localhost:9200/_ingest/pipeline/nameRouting
Content-Type: application/json
{
"description" : "Set's the routing value from name field in document",
"processors" : [
{
"set" : {
"field": "_routing",
"value": "{{_source.name}}"
}
}
]
}
The index settings will be :
{
"settings" : {
"index" : {
"default_pipeline" : "nameRouting"
}
}
}
I'm trying to disable dynamic mapping creation for only specific indexes, not for all. For some reason I can't put default mapping with 'dynamic' : 'false'.
So, here left two options as I can see:
specify property 'index.mapper.dynamic' in file elasticsearch.yml.
put 'index.mapper.dynamic' at index creation time, as described here https://www.elastic.co/guide/en/kibana/current/setup.html#kibana-dynamic-mapping
First option may only accept values: true, false and strict. So there is no way to specify subset of specific indexes (like we do by pattern with property 'action.auto_create_index' https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation).
Second option just not works.
I've created index
POST http://localhost:9200/test_idx/
{
"settings" : {
"mapper" : {
"dynamic" : false
}
},
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
}
}
}
}
}
Then checked index settings:
GET http://localhost:9200/test_idx/_settings
{
"test_idx" : {
"settings" : {
"index" : {
"mapper" : {
"dynamic" : "false"
},
"creation_date" : "1445440252221",
"number_of_shards" : "1",
"number_of_replicas" : "0",
"version" : {
"created" : "1050299"
},
"uuid" : "5QSYSYoORNqCXtdYn51XfA"
}
}
}
}
and mapping:
GET http://localhost:9200/test_idx/_mapping
{
"test_idx" : {
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
}
}
}
}
}
}
so far so good, let's index document with undeclared field:
POST http://localhost:9200/test_idx/test_type/1
{
"field1" : "it's ok, field must be in mapping and in source",
"somefield" : "but this field must be in source only, not in mapping"
}
Then I've checked mapping again:
GET http://localhost:9200/test_idx/_mapping
{
"test_idx" : {
"mappings" : {
"test_type" : {
"properties" : {
"field1" : {
"type" : "string"
},
"somefield" : {
"type" : "string"
}
}
}
}
}
}
As you can see, mapping is extended regardless of index setting "dynamic" : false.
I've also tried to create index exactly as described in doc
PUT http://localhost:9200/test_idx
{
"index.mapper.dynamic": false
}
but got the same behavior.
Maybe I've missed something?
Thanks a lot in advance!
You're almost there: the value needs to be set to strict.
And the correct usage is the following:
PUT /test_idx
{
"mappings": {
"test_type": {
"dynamic":"strict",
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
And pushing this a bit further, if you want to forbid the creation even of new types, not only fields in that index, use this:
PUT /test_idx
{
"mappings": {
"_default_": {
"dynamic": "strict"
},
"test_type": {
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
Without _default_ template:
PUT /test_idx
{
"settings": {
"index.mapper.dynamic": false
},
"mappings": {
"test_type": {
"dynamic": "strict",
"properties": {
"field1": {
"type": "string"
}
}
}
}
}
You must know about that the below part just mean that ES could'nt create a type dynamically.
"mapper" : {
"dynamic" : false
}
You should configure ES like this:
PUT http://localhost:9200/test_idx/_mapping/test_type
{
"dynamic":"strict"
}
Then you cant't index other field that without mapping any more ,and get an error as follow:
mapping set to strict, dynamic introduction of [hatae] within [data] is not allowed
If you wanna store the data,but make the field can't be index,you could take the setting like this:
PUT http://localhost:9200/test_idx/_mapping/test_type
{
"dynamic":false
}
Hope these can help the people with the same issue :).
The answer is in the doc (7x.): https://www.elastic.co/guide/en/elasticsearch/reference/7.x/dynamic.html
The dynamic setting controls whether new fields can be added
dynamically or not. It accepts three settings:
true
Newly detected fields are added to the mapping. (default)
false
Newly detected fields are ignored. These fields will not be indexed so
will not be searchable but will still appear in the _source field of
returned hits. These fields will not be added to the mapping, new
fields must be added explicitly.
strict
If new fields are detected, an exception is thrown and the document is
rejected. New fields must be explicitly added to the mapping.
PUT my_index
{
"mappings": {
"dynamic": "strict",
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}
You cannot disable dynamic mapping in ES 7 anymore, what you can do if you have completely unstructured data is to disable completely the mapping for the index like this:
curl -X PUT "localhost:9200/my_index?pretty" -H 'Content-Type: application/json' -d'
{
"mappings": {
"enabled": false
}
}
'
if you are using python you can do this:
from elasticsearch import Elasticsearch
# Connect to the elastic cluster
es=Elasticsearch([{'host':'localhost','port':9200}])
request_body = {
"mappings": {
"enabled": False
}
}
es.indices.create(index = 'my_index', body = request_body)
For ES 7 if you want to update an existing index:
PUT customers/_mapping
{
"dynamic": "strict"
}
first, please be concern aboout value false or strict,they work in a different way.
using "dynamic": "false" and create documents with fields not covered by the mapping, those fields will be ignored (so they won't be stored) and wouldn't show up in _source when you GET the document.
where value strict will not allow you to create the document rather it will throw an exception
Inner objects inherit the dynamic setting from their parent object or from the mapping type. In the following example, dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
However, the user.social_networks object enables dynamic mapping, so you can add fields to this inner object.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
PUT my-index-000001
{
"mappings": {
"dynamic": false,
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}
if you are using node.js client
await this.client.indices.putMapping({
index: ElasticIndex.UserDataFactory,
body: {
dynamic: 'strict',
properties: {
...this.schema,
},
},
});
I have a custom object that I wish to store in ElasticSearch as its own type in an index, but I don't want any field in the object to be analyzed. How should I go about doing this?
I have been using the ElasticSearch NEST client but can also manually create the mapping if needed.
You have a few options that will all work. Personally, I would go with either of the first two. If it's a daily index, then the second one is the better option.
Define the mapping upfront and disable dynamic fields. This is by far the safest approach and it will help you to avoid mistakes, and it will prevent fields from being added afterward.
{
"mappings": {
"_default_": {
"_all": {
"enabled": false
}
},
"mytype" : {
"dynamic" : "strict",
"properties" : {
...
}
}
}
}
Create an index template that also disables dynamic fields, but allows you to continuously roll new indices with the same mapping(s).
You can create tiered index templates so that more than one applies to any given index.
{
"template": "mytimedindex-*",
"settings": {
"number_of_shards": 2
},
"mappings": {
"_default_": {
"_all": {
"enabled": false
}
},
"mytype" : {
"dynamic" : "strict",
"properties" : {
...
}
}
}
}
Create a dynamic mapping that allows new fields, but defaults all strings to not_analyzed:
"dynamic_templates" : [ {
"strings" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string"
},
"match" : "*",
"match_mapping_type" : "string"
}
} ]
This will allow you to dynamically add fields to the mapping.
I have memory problems with aggregation queries.
my elastic version is 1.3.2
I tired to define _timestamp as doc value ,
but when I checked the mapping I can see it didn't work
It didn't happen in other fields.
Is there any known issue with timestamp field and doc values?
Lib
Have you tried this mapping?
{
"tweet" : {
"_timestamp" : {
"enabled" : true,
"format" : "YYYY-MM-dd"
}
}
I'm using specified version (13.2). I set up custom date field in my project like this and it's worked for me:
PUT 'http://127.0.0.1:9200/a252e39969665bb4d065/' -d
'{
"a252e39969665bb4d065": {
"mappings": {
"_default_": {
"properties": {
"createdDate": {
"type": "date",
"format": "dateOptionalTime"
}
}
}
}
}
}'
Please, note that i'm using default mapping here (default mapping for all types in index). You can use specified type in an index by replacing "default" in mapping.