Mapping definition for has unsupported parameters: [dynamic : true] - elasticsearch

I have been trying to make one of the fields being indexed to be dynamic and also changed the elasticsearch.yml for the same by adding
index.mapper.dynamic: false
in the end and also restarted elasticsearch and kibana sense. Also tried with different fields and index names but I am still getting the same error :
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [package]: Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]"
}
},
"status": 400
}
the code for adding index is below :
PUT /worldtest221
{
"settings" : {
"index" : {
"creation_date" : "1474989052008",
"uuid" : "Ae-7aFrLT466ZJL4U9QGLQ",
"number_of_replicas" : "0",
"analysis" : {
"char_filter" : {
"quotes" : {
"type" : "mapping",
"mappings" : [ "=>'", "=>'", "‘=>'", "’=>'", "‛=>'" ]
}
},
"filter" : {
"nGram_filter" : {
"max_gram" : "400",
"type" : "nGram",
"min_gram" : "1",
"token_chars" : [ "letter", "digit", "punctuation", "symbol" ]
}
},
"analyzer" : {
"quotes_analyzer" : {
"char_filter" : [ "quotes" ],
"tokenizer" : "standard"
},
"nGram_analyzer" : {
"type" : "custom",
"filter" : [ "lowercase", "asciifolding", "nGram_filter" ],
"tokenizer" : "whitespace"
},
"whitespace_analyzer" : {
"type" : "custom",
"filter" : [ "lowercase", "asciifolding" ],
"tokenizer" : "whitespace"
}
}
},
"cache" : {
"query_result" : {
"enable" : "true"
}
},
"number_of_shards" : "1",
"version" : {
"created" : "2030099"
}
}
},
"mappings" : {
"package" : {
"properties" : {
"autosuggestionpackagedetail" : {
"type" : "string",
"index" : "not_analyzed"
},
"availability" : {
"type" : "nested",
"include_in_all" : false,
"properties" : {
"displaysequence" : {
"type" : "long",
"include_in_all" : false
},
"isquantityavailable" : {
"type" : "boolean"
},
.....
"metatags" : {
"type" : "string",
"include_in_all" : false
},
"minadultage" : {
"type" : "long",
"include_in_all" : false
},
"newmemberrewardpoints" : {
"type" : "long",
"include_in_all" : false
},
"packagealbum" : {
"include_in_all" : false,
"properties" : {
"albumdetailid" : {
"type" : "string",
"include_in_all" : false,
"dynamic": true
},
....
Look at the second last line where I mention "dynamic" : true

This happens because you are trying to set "dynamic: true" on a field ("albumdetailid") of type string! This make no sense! no new fields can be created under a "string" field. The "dynamic" parameter should be set under an "object" field. so Either define "albumdetailid" as "object" or put the "dynamic: true" one level higher - under "packagealbum" like this:
"packagealbum" : {
"include_in_all" : false,
"dynamic": true,
"properties" : {
"albumdetailid" : {
"type" : "string",
"include_in_all" : false
},
....

Related

Elasticsearch suggest from multiple indices

I am working on Elasticsearch. I want to use search suggestor for multiple indices at a time. I have two indices, tags and pool_tags which has name field in each index. How to use suggestor on this two indices having a similarly named field name.
I tried naming the suggestor (pool_tag_suggest in pool_tags) differently and I tried. Here are the mappings
tags:
{
"tags" : {
"mappings" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
},
"suggest" : {
"type" : "completion",
"analyzer" : "simple",
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
}
}
}
}
}
}
}
pool_tags:
{
"pool_tags" : {
"mappings" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword"
},
"pool_tag_suggest" : {
"type" : "completion",
"analyzer" : "simple",
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
}
}
}
}
}
}
}
WHAT I TRIED
POST pool_tags,tags/_search
{
"suggest": {
"tags_suggestor": {
"text": "ww",
"term": {
"field": "name.suggest"
}
},
"pooltags_suggestor": {
"text": "ww",
"term": {
"field": "name.pool_tag_suggest"
}
}
}
}
ERROR
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]"
},
{
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.pool_tag_suggest]"
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "pool_tags",
"node" : "g2rCnS4PQMWyldWABVJawQ",
"reason" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]"
}
},
{
"shard" : 0,
"index" : "tags",
"node" : "g2rCnS4PQMWyldWABVJawQ",
"reason" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.pool_tag_suggest]"
}
}
],
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "no mapping found for field [name.suggest]"
}
}
},
"status" : 400
}

Elasticsearch 'failed to find filter under name '

I'am just started with ES 5.2.2
Trying ad analyzer with support russian morhology.
Run ES using docker, i create image with installed elasticsearch-analysis-morphology.
then i:
Create index,
then put settings
after that get settings, and all sems right
curl http://localhost:9200/news/_settings?pretty
{
"news" : {
"settings" : {
"index" : {
"number_of_shards" : "5",
"provided_name" : "news",
"creation_date" : "1489343955314",
"analysis" : {
"analyzer" : {
"russian_analyzer" : {
"filter" : [
"stop",
"custom_stop",
"russian_stop",
"custom_word_delimiter",
"lowercase",
"russian_morphology",
"english_morphology"
],
"char_filter" : [
"html_strip",
"ru"
],
"type" : "custom",
"tokenizer" : "standard"
}
},
"char_filter" : {
"ru" : {
"type" : "mapping",
"mappings" : [
"Ё=>Е",
"ё=>е"
]
}
},
"filter:" : {
"custom_stop" : {
"type" : "stop",
"stopwords" : [
"n",
"r"
]
},
"russian_stop" : {
"ignore_case" : "true",
"type" : "stop",
"stopwords" : [
"а",
"без",
]
},
"custom_word_delimiter" : {
"split_on_numerics" : "false",
"generate_word_parts" : "false",
"preserve_original" : "true",
"catenate_words" : "true",
"generate_number_parts" : "true",
"catenate_all" : "true",
"split_on_case_change" : "false",
"type" : "word_delimiter",
"catenate_numbers" : "false"
}
}
},
"number_of_replicas" : "1",
"uuid" : "IUkHHwWrStqDMG6fYOqyqQ",
"version" : {
"created" : "5020299"
}
}
}
}
}
then i try open index but ES give me this:
{
"error" : {
"root_cause" : [
{
"type" : "exception",
"reason" : "Failed to verify index [news/IUkHHwWrStqDMG6fYOqyqQ]"
}
],
"type" : "exception",
"reason" : "Failed to verify index [news/IUkHHwWrStqDMG6fYOqyqQ]",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Custom Analyzer [russian_analyzer] failed to find filter under name [custom_stop]"
}
},
"status" : 500
}
Can't understand where i'm wrong.
Can anyone see what the problem is?
There was mistake in "filter" section
was:
look here this This colon was a mistake
|
v
"filter:" : {
"custom_stop" : {
"type" : "stop",
"stopwords" : [
"n",
"r"
]
}...
Thanks #asettou and #andrey-morozov

mapping social relation elasticsearch

we have a social application where user can make group under some specific categories.User will have user Education , Certification ,location etc.
i want to search user on the basis of location,education etc. similarly search group based on categories . i want to use Elasticsearch
this is user mapping
"userData" : {
"dynamic" : "true",
"properties" : {
"allSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"email" : {
"type" : "string",
"index" : "not_analyzed"
},
"firstName" : {
"type" : "string"
},
"gender" : {
"type" : "object",
"enabled" : false
},
"id" : {
"type" : "string"
},
"isActive" : {
"type" : "string",
"index" : "not_analyzed"
},
"lastName" : {
"type" : "string"
},
"location" : {
"properties" : {
"country" : {
"type" : "string",
"index" : "not_analyzed"
},
"region" : {
"type" : "string"
}
}
},
"mId" : {
"type" : "object",
"enabled" : false
},
"profilePic" : {
"type" : "object",
"enabled" : false
},
"profileStatus" : {
"type" : "object",
"enabled" : false
},
"status" : {
"type" : "object",
"enabled" : false
},
"userId" : {
"type" : "object",
"enabled" : false
},
"userSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
}
}
}
​group model
"groupData" : {
"dynamic" : "true",
"properties" : {
"allSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"cDate" : {
"type" : "object",
"enabled" : false
},
"categoryId" : {
"type" : "integer"
},
"groupId" : {
"type" : "object",
"enabled" : false
},
"groupName" : {
"type" : "string"
},
"groupPic" : {
"type" : "object",
"enabled" : false
},
"groupStatus" : {
"type" : "object",
"enabled" : false
},
"groupSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"isActive" : {
"type" : "string",
"index" : "not_analyzed"
},
"mId" : {
"type" : "object",
"enabled" : false
}
}
}
​the problem is that how can i say user is a member of the group. search members in a particular group
should i add the education details with the userData itself as nested or parent child.
Or it is difficult to handle social relations in Elasticsearch?
the problem is that how can i say user is a member of the group
For this you can have an array of group ids in user mapping itself and then use term filter to filter the group id.
search members in a particular group
This can be done using the above method itself.
should i add the education details with the userData itself as nested or parent child.
Education details should be a part of the user mapping itself. This can be made using nested array. Using parent child relationships for this will be a overkill.

How to do ES Moving Avearge Prediction with Logstash?

I am using Elasticsearch 2.3.2, and Logstash 2.3.3. I have found from https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-movavg-aggregation.html which states that moving average can do predictions. I know it is possible to only make query in ES, but I am not sure how should I do that with logstash.
I have a logstash file which reads a csv log file storing CPU usage for every 15 seconds. Should I just include the following into the logstash output json file for the related index as an output mapping?
{
"the_movavg":{
"moving_avg":{
"buckets_path": "the_sum",
"window" : 30,
"model" : "holt_winters",
"settings" : {
"type" : "mult",
"alpha" : 0.5,
"beta" : 0.5,
"gamma" : 0.5,
"period" : 7,
"pad" : true
}
}
}
This is my json file for logstash
{
"template" : "linux_cpu-*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" }
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"#timestamp": { "type": "date" },
"#version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
}
}
}
}
}
And is it possible to have it as a graph as to be shown in Kibana?

Elasticsearch 1.7.3: doc_values treated as fielddata

I'm new to ElasticSearch, started working with ElasticSearch 1.7.3 as part of a Logstash-ElasticSearch-Kibana deployment.
I've defined a mapping template for my log messages, this is the interesting part:
{
"template" : "logstash-*",
"settings" : { "index.refresh_interval" : "5s" },
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}],
"properties" : {
"#version" : { "type" : "string", "index" : "not_analyzed" },
"#timestamp" : { "type" : "date", "format" : "dateOptionalTime" },
"message" : { "type" : "string" }
}
} ,
"my_log" : {
"_all" : { "enabled" : true, "omit_norms" : true },
"dynamic_templates" : [ {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}],
"properties" : {
"#timestamp" : { "type" : "date", "format" : "dateOptionalTime" },
"file" : { "type" : "string" },
"message" : { "type" : "string" }
"geolocation" : { "type" : "string" },
}
}
}
}
Although the #timestamp field is defined as doc_value:true I have an error of MemoryException because it is a fielddata:
[FIELDDATA] Data too large, data for [#timestamp] would be larger than
limit of [633785548/604.4 mb]
NOTE:
I know I can change the memory or add more nodes to the cluster, but in my point of view this is a design problem where this field should not be indexed in memory.

Resources