Elasticsearch mapping select all fields via template to change their data type Elasticsearch - elasticsearch

Hi All I am using elasticsearch-template.json to set data type of all of my fields to string. Below is the snippet of the template:
{
"template": "logstash-*",
"settings": {
"index.refresh_interval": "5s",
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"logs": {
"_all": {
"enabled": true
},
"properties": {
"level1": {
"properties": {
"level2": {
"properties": {
"_all": {"type": "string"}
}
}
}
}
}
}
}
}
Here under level2 i have got lots of fields which get created i want to set all of them to string how can i set it. I have tried "*" character as well as "%" character to select all the fields. but unfortunately it only gets added as a new field to the mapping. How to specify in template to select all the fields under a certain level?

I believe what you are looking for is a dynamic_templates and using path_match instead of match. This demonstrates how that might work:
curl -DELETE localhost:9200/test-*
curl -XDELETE http://localhost:9200/_template/test
curl -XPOST http://localhost:9200/_template/test -d '
{
"template": "test-*",
"mappings": {
"_default_": {
"dynamic_templates": [
{
"level1_level2_all": {
"path_match": "level1.level2.*",
"match_mapping_type": "*",
"mapping": {
"index": "not_analyzed",
"type": "string"
}
}
}
]
}
}
}
'
curl -XPOST http://localhost:9200/test-1/a -d '
{
"level1": {
"level2": {
"x":1
}
}
}'
curl -XPOST http://localhost:9200/test-1/a -d '
{
"level1": {
"level2": {
"y":1
}
}
}'
curl http://localhost:9200/test-1/_mapping?pretty
The output of which is:
"test-1" : {
"mappings" : {
"_default_" : {
"dynamic_templates" : [ {
"level1_level2_all" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string"
},
"match_mapping_type" : "*",
"path_match" : "level1.level2.*"
}
} ],
"properties" : { }
},
"a" : {
"dynamic_templates" : [ {
"level1_level2_all" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string"
},
"match_mapping_type" : "*",
"path_match" : "level1.level2.*"
}
} ],
"properties" : {
"level1" : {
"properties" : {
"level2" : {
"properties" : {
"x" : {
"type" : "string",
"index" : "not_analyzed"
},
"y" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}
}
}
}
}
}
}

Related

How to change mapping type on a very large ElasticSearch index?

I tried to this solution from here, but it didn't worked for me.
I needed to convert the nested text fields like one.second.third to ip type. So I added mapping like
PUT .the-index-2022.12.20-0000025/_mapping
{
"properties": {
"one": {
"type": "object",
"properties": {
"second": {
"type":"object",
"properties": {
"ip_address1": {
"type": "ip"
},
"ip_address2": {
"type": "ip"
}
}
}
}
}
}
}
Then I used
POST .the-index-2022.12.20-0000025/_update_by_query?wait_for_completion=false&slices=auto
It executed successfully but still didnt change the fields type to ip.
What can I do else or what is the problem here?
Thanks in advance
Also tried like:
PUT .the-index-2022.12.20-0000025/_mapping
{
"properties": {
"one": {
"properties": {
"second": {
"properties": {
"ip_address1": {
"type": "ip"
},
"ip_address2": {
"type": "ip"
}
}
}
}
}
}
}
Could you please help?
I've added 3 new fields in ip type:
{
".the-index-2022.12.20-0000025" : {
"mappings" : {
"one.sec.ip1" : {
"full_name" : "one.sec.ip1",
"mapping" : {
"ip1" : {
"type" : "ip"
}
}
},
"one.sec.ip2" : {
"full_name" : "one.sec.ip2",
"mapping" : {
"ip2" : {
"type" : "ip"
}
}
},
"one.sec.ip3" : {
"full_name" : "one.sec.ip3",
"mapping" : {
"ip3" : {
"type" : "ip"
}
}
}
}
}
}

Mapping array of long values from hive to elastic search index

I have data in hive in following format
user_ids name city owner_ids
[1, 324, 456] some_name some_city [4567, 12345678]
I want to be able to search by user_ids = 324 as filter criteria or owner_ids = 12345678 and be able to get back above document as response. (Exact match on ids)
Currently I am using dynamic template for mapping which maps user_ids field to long and I am unable to get any results, what type should I force field mapping of user_ids and owner_ids to get this response?
Mapping configuration
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1
},
"mappings": {
"doc": {
"dynamic_templates": [
{
"strings_as_keywords": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
}
}
}
Result mapping
{
"user_search" : {
"mappings" : {
"doc" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"city" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"ds" : {
"type" : "date"
},
"user_ids" : {
"type" : "long"
},
"owner_ids" : {
"type" : "long"
}
}
}
}
}
}

Elastic Search error - variable [relevancy] is not defined

I am trying to query my products ElasticSearch index and create a script_score but I keep receiving the error Variable [relevancy] is not defined.
I tried replacing the script with just a number, then with Math.log(_score) to make sure the script_score was working properly and the math function is ok, and both queries executed as expected. I also tried doc['relevancy'].value and received the same error.
My query is:
curl -X GET "localhost:9200/products/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"function_score": {
"query": {
"multi_match" : {
"query": "KQ",
"fields": [ "item_id", "extended_desc", "mfg_part_no" ]
}
},
"script_score" : {
"script": "Math.log(_score) + Math.log(doc['relevancy'])"
},
"boost_mode": "replace"
}
}
}
'
And the mapping for this index is:
{
"products" : {
"mappings" : {
"properties" : {
"#timestamp" : {
"type" : "date"
},
"#version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"extended_desc" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"frecno" : {
"type" : "long"
},
"item_id" : {
"type" : "text",
"analyzer" : "my_analyzer"
},
"mfg_part_no" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"relevancy" : {
"type" : "long"
}
}
}
}
}
Replaced ' with \u0027 because this is curl.
curl -X GET "localhost:9200/products/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": {
"function_score": {
"query": {
"multi_match" : {
"query": "KQ",
"fields": [ "item_id", "extended_desc", "mfg_part_no" ]
}
},
"script_score" : {
"script": "Math.log(_score) + Math.log(doc[\u0027relevancy\u0027].value)"
},
"boost_mode": "replace"
}
}
}
'

How to find out what is my index sorted by in elasticsearch?

I created new index in elasticsearch (v6) using command:
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/sorttest -d '
{
"settings" : {
"index" : {
"sort.field" : ["username", "date"],
"sort.order" : ["asc", "desc"]
}
},
"mappings": {
"_doc": {
"properties": {
"username": {
"type": "keyword",
"doc_values": true
},
"date": {
"type": "date"
}
}
}
}
}
'
The response was
{"acknowledged":true,"shards_acknowledged":true,"index":"sorttest"}
Next I checked out generated mapping
curl -XGET localhost:9200/sorttest/_mapping?pretty
And the result was
{
"sorttest" : {
"mappings" : {
"_doc" : {
"properties" : {
"date" : {
"type" : "date"
},
"username" : {
"type" : "keyword"
}
}
}
}
}
}
The question is: how can I find out what kind of sorting is set for my index?
Just
curl -XGET localhost:9200/sorttest?pretty
and you will see:
"settings" : {
"index" : {
...
"sort" : {
"field" : [
"username",
"date"
],
"order" : [
"asc",
"desc"
]
},
...
}
}

Is it possible to define default mapping for an inner object in ElasticSearch?

Say I have a document like this:
{
"events" : [
{
"event_id" : 123,
"props" : {
"version": "33"
},
{
"event_id" : 124,
"props" : {
"version": "44a"
}
]
}
Is it possible to specify that the events.props.version be mapped to some type?
I've tried:
{
"template" : "logstash-*",
...
"mappings" : {
"_default_" : {
"properties" : {
"events.props.version" : { "type" : "string" }
}
}
}
}
But that doesn't seem to work.
Please have a look at mapping API in elasticsearch Mapping API.
To set any analyzer in the inner element we need to consider each and every inner field as a separate properties set. try the following
{
"mappings": {
"properties": {
"events": {
"properties": {
"event_id": {
"type": "string",
"analyzer": "keyword"
},
"props": {
"properties": {
"version": {
"type": "string"
}
}
}
}
}
}
}
}
if this not works please provide me you mapping.
Sure, but you need to use the "object" type:
From the doc ( https://www.elastic.co/guide/en/elasticsearch/reference/1.5/mapping-object-type.html ) if you want to map
{
"tweet" : {
"person" : {
"name" : {
"first_name" : "Shay",
"last_name" : "Banon"
},
"sid" : "12345"
},
"message" : "This is a tweet!"
}
}
you can write:
{
"tweet" : {
"properties" : {
"person" : {
"type" : "object",
"properties" : {
"name" : {
"type" : "object",
"properties" : {
"first_name" : {"type" : "string"},
"last_name" : {"type" : "string"}
}
},
"sid" : {"type" : "string", "index" : "not_analyzed"}
}
},
"message" : {"type" : "string"}
}
}
}

Resources