How to set elasticsearch index mapping as not_analysed for all the fields - elasticsearch

I want my elasticsearch index to match the exact value for all the fields. How do I map my index to "not_analysed" for all the fields.

I'd suggest making use of multi-fields in your mapping (which would be default behavior if you aren't creating mapping (dynamic mapping)).
That way you can switch to traditional search and exact match searches when required.
Note that for exact matches, you would need to have keyword datatype + Term Query. Sample examples are provided in the links I've specified.
Hope it helps!

You can use dynamic_templates mapping for this. As a default, Elasticsearch is making the fields type as text and index: true like below:
{
"products2": {
"mappings": {
"product": {
"properties": {
"color": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
As you see, also it creates a keyword field as multi-field. This keyword fields indexed but not analyzed like text. if you want to drop this default behaviour. You can use below configuration for the index while creating it :
PUT products
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"product": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"index": false
}
}
}
]
}
}
}
After doing this the index will be like below :
{
"products": {
"mappings": {
"product": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"index": false
}
}
}
],
"properties": {
"color": {
"type": "keyword",
"index": false
},
"type": {
"type": "keyword",
"index": false
}
}
}
}
}
}
Note: I don't know the case but you can use the multi-field feature as mentioned by #Kamal. Otherwise, you can not search on the not analyzed fields. Also, you can use the dynamic_templates mapping set some fields are analyzed.
Please check the documentation for more information :
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html
Also, I was explained the behaviour in this article. Sorry about that but it is Turkish. You can check the example code samples with google translate if you want.

Related

How to detect whether elasticsearch has enabled dynamic field

I don't know whether my index has enabled/disabled dynamic field. When I use get index mapping command it just responses these informations:
GET /my_index1/_mapping
{
"my_index1": {
"mappings": {
"properties": {
"goodsName": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"auditTime": {
"type": "long"
},
"createUserId": {
"type": "long"
}
}
}
}
}
If you don't explicitly set the dynamic to false or strict, it will be true by default. If you explicitly set that, you will see that in your mappings:
{
"mappings": {
"dynamic": false,
"properties": {
"name": {
"type": "text"
}
}
}
}
And when you index the following document:
{"name":"products", "clickCount":1, "bookingCount":2, "isPromoted":1}
Only the field name will be indexed, the rest won't. If you call the _mapping endpoint again, it will give you the exact mappings above.

How to declare mapping for nested fields in Elasticsearch to allow for storing different types?

In essence, I want my mapping to be as schemaless as possible, but allow for nested types and being able to store data that may have different types:
When I try to add a document where some fields have different types of values, I get an error like this:
"type": "illegal_argument_exception",
"reason": "mapper [data.customData.value] of different type, current_type [long], merged_type [text]"
This can easily be solved by mapping the field value to text (or create it dynamically by first inserting a document with only text). However, I would like to avoid having a schema. Perhaps having all of the fields nested in customData to be set to text? How do I do that?
I had the problem earlier, but then it started working after accidentally managing to get a dynamical mapping that worked (since everything was regarded as text. I was later made aware of this problem since I needed to change the mapping to allow for nested types.
Documents with this kind of data are troublesome to store successfully:
"customData": [
{
"value": "some_text",
"key": "some_text"
},
{
"value": 0,
"key": "some_text"
}
]
A part of the mapping that works:
{
"my_index": {
"aliases": {},
"mappings": {
"_doc": {
"properties": {
"data": {
"properties": {
"customData": {
"properties": {
"key": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"value": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
},
"some_list": {
"type": "nested",
"properties": {
"some_field": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
}
In essence, I want the mapping to be as schemaless as possible, but allow for nested types and being able to store data that may have different types:
{
"mappings": {
"_doc": {
"properties": {
"data": {
"type": "object"
},
"somee_list": {
"type": "nested"
}
}
}
}
}
So what would be the best approach to go about this problem?

Avoid creating dual mappings from logstash

I notice that logstash creates an extra "keyword" field in the index mapping for every string field that it extracts from the log files and sends to elastic search.
There are many fields that I've removed completely with the prune plugin, but there are other fields that I don't want to remove completely, but I also don't need to have a *.keyword for them.
Is there a way to have logstash only create *.keyword fields for some fields and not others? Specifically, is there a way for logstash to have a whitelist of fields that it is OK to create *.keywords for, and not do it for anything else?
(using elasticsearch 6.x)
I think you need to change the mapping of the desired fields. The mapping page shows the default text type mapping:
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/breaking_50_mapping_changes.html
I tried to set a field without a keyword field and it worked except you couldn't agregate on that field (I tried terms aggregation) even if you set index: true in the mapping. I might have missed something but I think this is where you should start.
The solution I'm working with for now is a dynamic templates.
I can map some fields to just text and others to text and a keyword. For example:
{
"mappings": {
"doc": {
"dynamic_templates": [
{
"match_my_custom_fields": {
"match_mapping_type": "string",
"match": "custom_prefix_*",
"mapping": {
"type": "text",
"fields": {
"raw": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
{
"strings_as_keywords": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"ignore_above": 256
}
}
}
],
"properties": {
"geoip": {
"dynamic": true,
"properties": {
"ip": {
"type": "ip"
},
"location": {
"type": "geo_point"
},
"latitude": {
"type": "half_float"
},
"longitude": {
"type": "half_float"
}
}
}
}
}
}
This way, everything beginning with custom_prefix_ will have a text and keyword field, and everything else will just have a keyword.
Of course, I somehow broke the geoip.geo_point that was being emitted from the geoip logstash plugin, and now my map visualizations won't work, so I need to figure out how to restore that.
EDIT: Got geo_point working again, see the "geoip" prop

ElasticSearch Reindex API not analyzing the new field

I have an existing index named "Docs" which has documents in it.
I am creating a new Index named "Docs1" exactly same like "Docs" with only one extra field with analyzer in one property, which I want to use for autocomplete purpose.
Property in "Docs" index
"name": {
"type": "text",
"analyzer": "text_standard_analyzer",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
Property in the "Docs1" index going to be
{
"name": {
"type": "text",
"analyzer": "text_standard_analyzer",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
},
"pmatch": {
"type": "text",
"analyzer": "text_partialmatching_analyzer"
}
}
}
}
I am using Reindex API to copy records from "Docs" to "Docs1"
POST _reindex
{
"source": {
"index": "Docs"
},
"dest": {
"index": "Docs1"
}
}
when I reindex, I expect for the older documents to contain the new field with the information in that field.
I am noticing the new field in my destination index "Docs1" is not analyzed for existing data. But it is analyzed for any new documents I am adding.
Please suggest
Reindex by adding "type" worked
POST _reindex
{
"source":
{ "index": "sourceindex" },
"dest":
{ "index": "destindex",
"type":"desttype"
}
}

Set fields to not_analysed in all (future) types under an index in elasticsearch 1.7

I have an index with quite a few types, and new types' creation is not controlled by me. I know that the data structure is pretty solid, but I don't know the type's name in advance.
I would like to set some fields as not_analysed, while some should be analysed. Is there a way to achieve this?
I would also add to Val's excellent answer that you are probably wanting to add these dynamic templates to the _default_ mapping for your index, since you mentioned you do not know the types in advance. For example:
PUT /my_index/_mapping/_default_
{
"dynamic_templates": [
{
"analyzed": {
"match_mapping_type": "string",
"match": "*_text",
"mapping": {
"type": "string"
}
}
},
{
"not_analyzed": {
"match_mapping_type": "string",
"match": "*_key",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
With that in place, you can add any arbitrary type to the index and any fields in the documents added to the new type that end with "_text" will be analyzed. Any fields ending with "_key" will not be analyzed. You can read more about the default mapping in the docs.
Dynamic mappings are the way to go. Since you're mentioning analyzed vs not_analyzed I reckon you're talking about string fields.
The idea is to update your index and mapping in order to include a dynamic template for your string fields:
PUT my_index
{
"mappings": {
"my_type": {
"dynamic_templates": [ <--- include this section in your existing mapping
{
"analyzed": {
"match_mapping_type": "string",
"match": "field1",
"mapping": {
"type": "string"
}
}
},
{
"not_analyzed": {
"match_mapping_type": "string",
"match": "field2",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
}
]
}
}
}
Another way would be to make each new string field both analyzed and not_analyzed so you don't have to enumerate all your fields, simply using:
PUT my_index
{
"mappings": {
"my_type": {
"dynamic_templates": [ <--- include this section in your existing mapping
{
"strings": {
"match_mapping_type": "string", <-- match all string fields
"mapping": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}
}
}
}
}
]
}
}
}

Resources