we have a social application where user can make group under some specific categories.User will have user Education , Certification ,location etc.
i want to search user on the basis of location,education etc. similarly search group based on categories . i want to use Elasticsearch
this is user mapping
"userData" : {
"dynamic" : "true",
"properties" : {
"allSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"email" : {
"type" : "string",
"index" : "not_analyzed"
},
"firstName" : {
"type" : "string"
},
"gender" : {
"type" : "object",
"enabled" : false
},
"id" : {
"type" : "string"
},
"isActive" : {
"type" : "string",
"index" : "not_analyzed"
},
"lastName" : {
"type" : "string"
},
"location" : {
"properties" : {
"country" : {
"type" : "string",
"index" : "not_analyzed"
},
"region" : {
"type" : "string"
}
}
},
"mId" : {
"type" : "object",
"enabled" : false
},
"profilePic" : {
"type" : "object",
"enabled" : false
},
"profileStatus" : {
"type" : "object",
"enabled" : false
},
"status" : {
"type" : "object",
"enabled" : false
},
"userId" : {
"type" : "object",
"enabled" : false
},
"userSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
}
}
}
group model
"groupData" : {
"dynamic" : "true",
"properties" : {
"allSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"cDate" : {
"type" : "object",
"enabled" : false
},
"categoryId" : {
"type" : "integer"
},
"groupId" : {
"type" : "object",
"enabled" : false
},
"groupName" : {
"type" : "string"
},
"groupPic" : {
"type" : "object",
"enabled" : false
},
"groupStatus" : {
"type" : "object",
"enabled" : false
},
"groupSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"isActive" : {
"type" : "string",
"index" : "not_analyzed"
},
"mId" : {
"type" : "object",
"enabled" : false
}
}
}
the problem is that how can i say user is a member of the group. search members in a particular group
should i add the education details with the userData itself as nested or parent child.
Or it is difficult to handle social relations in Elasticsearch?
the problem is that how can i say user is a member of the group
For this you can have an array of group ids in user mapping itself and then use term filter to filter the group id.
search members in a particular group
This can be done using the above method itself.
should i add the education details with the userData itself as nested or parent child.
Education details should be a part of the user mapping itself. This can be made using nested array. Using parent child relationships for this will be a overkill.
Related
Im looking for a clean solution for this. Basically I have arrays which are numbered by an integer. This number may be from 1-50. Rather than repeating my index 50 times, is there a work around for this?
Here is an example of how I would do it for level 1.
Thanks
"test" : {
"properties" : {
"1" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
"2" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
"3" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
repeat 47 times more until
"50" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
I have been trying to make one of the fields being indexed to be dynamic and also changed the elasticsearch.yml for the same by adding
index.mapper.dynamic: false
in the end and also restarted elasticsearch and kibana sense. Also tried with different fields and index names but I am still getting the same error :
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [package]: Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [albumdetailid] has unsupported parameters: [dynamic : true]"
}
},
"status": 400
}
the code for adding index is below :
PUT /worldtest221
{
"settings" : {
"index" : {
"creation_date" : "1474989052008",
"uuid" : "Ae-7aFrLT466ZJL4U9QGLQ",
"number_of_replicas" : "0",
"analysis" : {
"char_filter" : {
"quotes" : {
"type" : "mapping",
"mappings" : [ "=>'", "=>'", "‘=>'", "’=>'", "‛=>'" ]
}
},
"filter" : {
"nGram_filter" : {
"max_gram" : "400",
"type" : "nGram",
"min_gram" : "1",
"token_chars" : [ "letter", "digit", "punctuation", "symbol" ]
}
},
"analyzer" : {
"quotes_analyzer" : {
"char_filter" : [ "quotes" ],
"tokenizer" : "standard"
},
"nGram_analyzer" : {
"type" : "custom",
"filter" : [ "lowercase", "asciifolding", "nGram_filter" ],
"tokenizer" : "whitespace"
},
"whitespace_analyzer" : {
"type" : "custom",
"filter" : [ "lowercase", "asciifolding" ],
"tokenizer" : "whitespace"
}
}
},
"cache" : {
"query_result" : {
"enable" : "true"
}
},
"number_of_shards" : "1",
"version" : {
"created" : "2030099"
}
}
},
"mappings" : {
"package" : {
"properties" : {
"autosuggestionpackagedetail" : {
"type" : "string",
"index" : "not_analyzed"
},
"availability" : {
"type" : "nested",
"include_in_all" : false,
"properties" : {
"displaysequence" : {
"type" : "long",
"include_in_all" : false
},
"isquantityavailable" : {
"type" : "boolean"
},
.....
"metatags" : {
"type" : "string",
"include_in_all" : false
},
"minadultage" : {
"type" : "long",
"include_in_all" : false
},
"newmemberrewardpoints" : {
"type" : "long",
"include_in_all" : false
},
"packagealbum" : {
"include_in_all" : false,
"properties" : {
"albumdetailid" : {
"type" : "string",
"include_in_all" : false,
"dynamic": true
},
....
Look at the second last line where I mention "dynamic" : true
This happens because you are trying to set "dynamic: true" on a field ("albumdetailid") of type string! This make no sense! no new fields can be created under a "string" field. The "dynamic" parameter should be set under an "object" field. so Either define "albumdetailid" as "object" or put the "dynamic: true" one level higher - under "packagealbum" like this:
"packagealbum" : {
"include_in_all" : false,
"dynamic": true,
"properties" : {
"albumdetailid" : {
"type" : "string",
"include_in_all" : false
},
....
I am using Elasticsearch 2.3.2, and Logstash 2.3.3. I have found from https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-movavg-aggregation.html which states that moving average can do predictions. I know it is possible to only make query in ES, but I am not sure how should I do that with logstash.
I have a logstash file which reads a csv log file storing CPU usage for every 15 seconds. Should I just include the following into the logstash output json file for the related index as an output mapping?
{
"the_movavg":{
"moving_avg":{
"buckets_path": "the_sum",
"window" : 30,
"model" : "holt_winters",
"settings" : {
"type" : "mult",
"alpha" : 0.5,
"beta" : 0.5,
"gamma" : 0.5,
"period" : 7,
"pad" : true
}
}
}
This is my json file for logstash
{
"template" : "linux_cpu-*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" }
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"#timestamp": { "type": "date" },
"#version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
}
}
}
}
}
And is it possible to have it as a graph as to be shown in Kibana?
I'm new to ElasticSearch, started working with ElasticSearch 1.7.3 as part of a Logstash-ElasticSearch-Kibana deployment.
I've defined a mapping template for my log messages, this is the interesting part:
{
"template" : "logstash-*",
"settings" : { "index.refresh_interval" : "5s" },
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}],
"properties" : {
"#version" : { "type" : "string", "index" : "not_analyzed" },
"#timestamp" : { "type" : "date", "format" : "dateOptionalTime" },
"message" : { "type" : "string" }
}
} ,
"my_log" : {
"_all" : { "enabled" : true, "omit_norms" : true },
"dynamic_templates" : [ {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}],
"properties" : {
"#timestamp" : { "type" : "date", "format" : "dateOptionalTime" },
"file" : { "type" : "string" },
"message" : { "type" : "string" }
"geolocation" : { "type" : "string" },
}
}
}
}
Although the #timestamp field is defined as doc_value:true I have an error of MemoryException because it is a fielddata:
[FIELDDATA] Data too large, data for [#timestamp] would be larger than
limit of [633785548/604.4 mb]
NOTE:
I know I can change the memory or add more nodes to the cluster, but in my point of view this is a design problem where this field should not be indexed in memory.
I tried to display a Kibana dashboard and it works well. Unfortunately, when I want to add a pie chart (or another representation) containing the countries of the companies locations, I have an empty panel.
I'm able to use the kibana queries to filter on a specific country but I'm not able to display a panel with nested documents.
My mapping (I have to use nested fields because a company can have several locations):
{
"settings" : {
"number_of_shards" : 1
},
"mappings": {
"company" : {
"properties" : {
"name" : { "type" : "string", "store" : "yes" },
"website" : { "type" : "string", "store" : "yes" },
"employees" : { "type" : "string", "store" : "yes" },
"type": { "type" : "string", "store" : "yes" },
"locations" : {
"type" : "nested",
"properties" : {
"city" : { "type" : "string", "store" : "yes" },
"country" : { "type" : "string", "store" : "yes" },
"coordinates" : { "type" : "geo_point", "store" : "yes" }
}
}
}
}
}
}
Do you know how could I display panel with nested objects? Is it implemented?
Thanks,
Kevin
you are missing one parameter ("include_in_parent": true) in your mapping.
The correct mapping should be:
{
"settings" : {
"number_of_shards" : 1
},
"mappings": {
"company" : {
"properties" : {
"name" : { "type" : "string", "store" : "yes" },
"website" : { "type" : "string", "store" : "yes" },
"employees" : { "type" : "string", "store" : "yes" },
"type": { "type" : "string", "store" : "yes" },
"locations" : {
"type" : "nested",
"include_in_parent": true,
"properties" : {
"city" : { "type" : "string", "store" : "yes" },
"country" : { "type" : "string", "store" : "yes" },
"coordinates" : { "type" : "geo_point", "store" : "yes" }
}
}
}
}
}
}
It's clearly a Kibana bug. The facet query generated by Kibana is missing the "nested" field to indicate this.