This is my index template(simplified):
{
"template": "user-*",
"mappings" : {
"ESHOP_USER" : {
"properties" : {
"id" : {
"type" : "long"
},
"nickname" : {
"type" : "text"
},
"createTime" : {
"type" : "keyword"
},
"updateTime" : {
"type" : "keyword"
}
}
}
},
"aliases" : {
"user-test-alias" : {
"index" : "user-test*"
},
"user-prod-alias" : {
"index" : "user-prod*"
}
}
}
What I want to do:
Indices with name pattern user-* share the same template, and I want to add user-test-alias to all indices with name pattern user-test*, and user-prod-alias to all indices with name pattern user-prod*.
What I got:
With above template, I got all indices with name pattern user-* getting two aliases: user-test-alias and user-prod-alias.
I knew if I split this template to test and prod templates, or use POST /_aliases after indices creation, I can solve the problem. But is there some way to achieve my goal with only one index template?
What I would do is to use 3 templates:
one that is common to both environments
one for test (just for the alias)
one for prod (just for the alias as well)
First template (common) applied first:
{
"template": "user-*",
"order": 0,
"mappings" : {
"ESHOP_USER" : {
"properties" : {
"id" : {
"type" : "long"
},
"nickname" : {
"type" : "text"
},
"createTime" : {
"type" : "keyword"
},
"updateTime" : {
"type" : "keyword"
}
}
}
}
}
Second template (test alias) applied next:
{
"template": "user-test-*",
"order": 1,
"aliases" : {
"user-test-alias" : {
"index" : "user-test*"
}
}
}
Third template (prod alias) applied next:
{
"template": "user-prod-*",
"order": 1,
"aliases" : {
"user-prod-alias" : {
"index" : "user-prod*"
}
}
}
Related
I would like to count and sort the number of occurred message on a field of type match_only_text. Using a DSL query the output needed to have to look like this:
{" Text message 1":615
" Text message 2":568
....}
So i tried this on kibana:
GET my_index_name/_search?size=0
{
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message"
}
}
}
}
However i get this error:
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "match_only_text fields do not support sorting and aggregations"
}
I am interested in the field "message" this is its mapping:
"message" : {
"type" : "match_only_text"
}
This is a part of the index mapping:
"mappings" : {
"_meta" : {
"package" : {
"name" : "system"
},
"managed_by" : "ingest-manager",
"managed" : true
},
"_data_stream_timestamp" : {
"enabled" : true
},
"dynamic_templates" : [
{
"strings_as_keyword" : {
"match_mapping_type" : "string",
"mapping" : {
"ignore_above" : 1024,
"type" : "keyword"
}
}
}
],
"date_detection" : false,
"properties" : {
"#timestamp" : {
"type" : "date"
}
.
.
.
"message" : {
"type" : "match_only_text"
},
"process" : {
"properties" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
},
"pid" : {
"type" : "long"
}
}
},
"system" : {
"properties" : {
"syslog" : {
"type" : "object"
}
}
}
}
}
}
}
Please Help
Yes, by design, match_only_text is of the text field type family, hence you cannot aggregate on it.
You need to:
A. create a message.keyword sub-field in your mapping of type keyword:
PUT my_index_name/_mapping
{
"properties": {
"message" : {
"type" : "match_only_text",
"fields": {
"keyword": {
"type" : "keyword"
}
}
}
}
}
B. update the whole index (using _update_by_query) so the sub-field gets populated and
POST my_index_name/_update_by_query?wait_for_completion=false
Then, depending on the size of your index, call GET _tasks?actions=*byquery&detailed regularly to check the progress of the task.
C. run the aggregation on that sub-field.
POST my_index_name/_search
{
"size": 0,
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message.keyword"
}
}
}
}
No matter what I do, when the index is created with a heartbeat process (7.10.2)
Elasticsearch maps all fields and the monitor.id will be like:
GET /heartbeat-7.10.2-2021.05.25
[...]
"monitor" : {
"properties" : {
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
[...]
even if i delete the index, template, and update the template as:
{
"order" : 1,
"index_patterns" : [
"heartbeat-7.10.2-*"
],
"settings" : {
},
"mappings" : {
"dynamic": false,
"properties" : {
"monitor" : {
"properties" : {
"id" : {
"ignore_above" : 1024,
"type" : "keyword"
}
}
}
}
},
"aliases" : { }
}
It appears to be that template configuration is ignored.
There is no other heartbeat template.
This is problematic, because in this way I cannot use e.g. monitor.id for aggregation. This is a problem with multiple fields.
I'm relatively new to templates, so maybe I'm missing something.
So apparently I had both _template and _index_template and the _index_template had the priority
After
delete _index_template/heartbeat*
it works just fine.
I have been trying to update my mapping but not able to do that. Majorly this question is related to updating the nested part. Suppose there is a field "Anand" which contains a field "hello"
{
"properties": {
"anand": {
"hello": {
"type": "short"
}
}
}
}
But I am getting the error
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "No type specified for field [anand]"
}
],
"type" : "mapper_parsing_exception",
"reason" : "No type specified for field [anand]"
},
"status" : 400
}
Current Mapping is
{
"anandschool" : {
"mappings" : {
"properties" : {
"anand" : {
"type" : "nested"
},
"doc" : {
"properties" : {
"properties" : {
"properties" : {
"shop_tier" : {
"type" : "long"
}
}
}
}
},
"message" : {
"type" : "byte"
},
"properties" : {
"properties" : {
"shop_tier" : {
"type" : "long"
},
"shop_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"shop" : {
"type" : "long"
}
}
}
}
}
I even created a nested type anand so that it can work
{
"properties": {
"anand": {
"type": "nested"
}
}
}
Self Answer
When updating mapping for nested things need to update properties of the nested field.
For above example update by
"properties": {
"anand": {
"properties":{
"hello": {
"type": "short"
}
}
}
}
}
THough This will not work inside a field which is nested.Ex if anand type was "nested", it would not work. If anyone knows solution for that let me know.
I have a simple doc structure as follows.
{
"did" : "1",
"uid" : "user1",
"mid" : "pc-linux1",
"path" : "/tmp/path1"
}
I need to query elastic ,that matches all fields exactly
GET index2/_search
{
"query": {
"bool":{
"must": [
{
"term" : { "uid" : "user1"}
},
{
"term" : { "mid" : "pc-linux1"}
},
{
"term" : { "did" : "1"}
},
{
"term" : { "path" : "/tmp/path1"}
}
]
}
}
}
The matching should happen without any kind of elastic 'analysis' on keywords, so that "/tmp/path1" is matched as a full term.
I tried to use a custom mapping: with
"index" : false
which does not work.
PUT /index2?include_type_name=true
{
"mappings" : {
"_doc": {
"properties" : {
"did" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"mid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
}
}
}
}
}
I am using elastic7.0 and few posts suggesting a custom mapping with
"index" : "not_analysed"
does not get accepted as a valid mapping in elastic 7.0
Any suggestions?
If you want to match exact terms, try this query:
GET index2/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"uid": "user1"
}
},
{
"match": {
"mid": "pc-linux1"
}
},
{
"match": {
"did": "1"
}
},
{
"match": {
"path": "/tmp/path1"
}
}
]
}
}
}
Say I have a document like this:
{
"events" : [
{
"event_id" : 123,
"props" : {
"version": "33"
},
{
"event_id" : 124,
"props" : {
"version": "44a"
}
]
}
Is it possible to specify that the events.props.version be mapped to some type?
I've tried:
{
"template" : "logstash-*",
...
"mappings" : {
"_default_" : {
"properties" : {
"events.props.version" : { "type" : "string" }
}
}
}
}
But that doesn't seem to work.
Please have a look at mapping API in elasticsearch Mapping API.
To set any analyzer in the inner element we need to consider each and every inner field as a separate properties set. try the following
{
"mappings": {
"properties": {
"events": {
"properties": {
"event_id": {
"type": "string",
"analyzer": "keyword"
},
"props": {
"properties": {
"version": {
"type": "string"
}
}
}
}
}
}
}
}
if this not works please provide me you mapping.
Sure, but you need to use the "object" type:
From the doc ( https://www.elastic.co/guide/en/elasticsearch/reference/1.5/mapping-object-type.html ) if you want to map
{
"tweet" : {
"person" : {
"name" : {
"first_name" : "Shay",
"last_name" : "Banon"
},
"sid" : "12345"
},
"message" : "This is a tweet!"
}
}
you can write:
{
"tweet" : {
"properties" : {
"person" : {
"type" : "object",
"properties" : {
"name" : {
"type" : "object",
"properties" : {
"first_name" : {"type" : "string"},
"last_name" : {"type" : "string"}
}
},
"sid" : {"type" : "string", "index" : "not_analyzed"}
}
},
"message" : {"type" : "string"}
}
}
}