I tried to display a Kibana dashboard and it works well. Unfortunately, when I want to add a pie chart (or another representation) containing the countries of the companies locations, I have an empty panel.
I'm able to use the kibana queries to filter on a specific country but I'm not able to display a panel with nested documents.
My mapping (I have to use nested fields because a company can have several locations):
{
"settings" : {
"number_of_shards" : 1
},
"mappings": {
"company" : {
"properties" : {
"name" : { "type" : "string", "store" : "yes" },
"website" : { "type" : "string", "store" : "yes" },
"employees" : { "type" : "string", "store" : "yes" },
"type": { "type" : "string", "store" : "yes" },
"locations" : {
"type" : "nested",
"properties" : {
"city" : { "type" : "string", "store" : "yes" },
"country" : { "type" : "string", "store" : "yes" },
"coordinates" : { "type" : "geo_point", "store" : "yes" }
}
}
}
}
}
}
Do you know how could I display panel with nested objects? Is it implemented?
Thanks,
Kevin
you are missing one parameter ("include_in_parent": true) in your mapping.
The correct mapping should be:
{
"settings" : {
"number_of_shards" : 1
},
"mappings": {
"company" : {
"properties" : {
"name" : { "type" : "string", "store" : "yes" },
"website" : { "type" : "string", "store" : "yes" },
"employees" : { "type" : "string", "store" : "yes" },
"type": { "type" : "string", "store" : "yes" },
"locations" : {
"type" : "nested",
"include_in_parent": true,
"properties" : {
"city" : { "type" : "string", "store" : "yes" },
"country" : { "type" : "string", "store" : "yes" },
"coordinates" : { "type" : "geo_point", "store" : "yes" }
}
}
}
}
}
}
It's clearly a Kibana bug. The facet query generated by Kibana is missing the "nested" field to indicate this.
Related
Im looking for a clean solution for this. Basically I have arrays which are numbered by an integer. This number may be from 1-50. Rather than repeating my index 50 times, is there a work around for this?
Here is an example of how I would do it for level 1.
Thanks
"test" : {
"properties" : {
"1" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
"2" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
"3" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
repeat 47 times more until
"50" : {
"properties" : {
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
"taglevel" : {
"type" : "long"
}
}
},
we have a social application where user can make group under some specific categories.User will have user Education , Certification ,location etc.
i want to search user on the basis of location,education etc. similarly search group based on categories . i want to use Elasticsearch
this is user mapping
"userData" : {
"dynamic" : "true",
"properties" : {
"allSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"email" : {
"type" : "string",
"index" : "not_analyzed"
},
"firstName" : {
"type" : "string"
},
"gender" : {
"type" : "object",
"enabled" : false
},
"id" : {
"type" : "string"
},
"isActive" : {
"type" : "string",
"index" : "not_analyzed"
},
"lastName" : {
"type" : "string"
},
"location" : {
"properties" : {
"country" : {
"type" : "string",
"index" : "not_analyzed"
},
"region" : {
"type" : "string"
}
}
},
"mId" : {
"type" : "object",
"enabled" : false
},
"profilePic" : {
"type" : "object",
"enabled" : false
},
"profileStatus" : {
"type" : "object",
"enabled" : false
},
"status" : {
"type" : "object",
"enabled" : false
},
"userId" : {
"type" : "object",
"enabled" : false
},
"userSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
}
}
}
group model
"groupData" : {
"dynamic" : "true",
"properties" : {
"allSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"cDate" : {
"type" : "object",
"enabled" : false
},
"categoryId" : {
"type" : "integer"
},
"groupId" : {
"type" : "object",
"enabled" : false
},
"groupName" : {
"type" : "string"
},
"groupPic" : {
"type" : "object",
"enabled" : false
},
"groupStatus" : {
"type" : "object",
"enabled" : false
},
"groupSuggest" : {
"type" : "completion",
"analyzer" : "simple",
"payloads" : true,
"preserve_separators" : true,
"preserve_position_increments" : true,
"max_input_length" : 50
},
"isActive" : {
"type" : "string",
"index" : "not_analyzed"
},
"mId" : {
"type" : "object",
"enabled" : false
}
}
}
the problem is that how can i say user is a member of the group. search members in a particular group
should i add the education details with the userData itself as nested or parent child.
Or it is difficult to handle social relations in Elasticsearch?
the problem is that how can i say user is a member of the group
For this you can have an array of group ids in user mapping itself and then use term filter to filter the group id.
search members in a particular group
This can be done using the above method itself.
should i add the education details with the userData itself as nested or parent child.
Education details should be a part of the user mapping itself. This can be made using nested array. Using parent child relationships for this will be a overkill.
I am using Elasticsearch 2.3.2, and Logstash 2.3.3. I have found from https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-movavg-aggregation.html which states that moving average can do predictions. I know it is possible to only make query in ES, but I am not sure how should I do that with logstash.
I have a logstash file which reads a csv log file storing CPU usage for every 15 seconds. Should I just include the following into the logstash output json file for the related index as an output mapping?
{
"the_movavg":{
"moving_avg":{
"buckets_path": "the_sum",
"window" : 30,
"model" : "holt_winters",
"settings" : {
"type" : "mult",
"alpha" : 0.5,
"beta" : 0.5,
"gamma" : 0.5,
"period" : 7,
"pad" : true
}
}
}
This is my json file for logstash
{
"template" : "linux_cpu-*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" }
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"#timestamp": { "type": "date" },
"#version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
}
}
}
}
}
And is it possible to have it as a graph as to be shown in Kibana?
I'm new to ElasticSearch, started working with ElasticSearch 1.7.3 as part of a Logstash-ElasticSearch-Kibana deployment.
I've defined a mapping template for my log messages, this is the interesting part:
{
"template" : "logstash-*",
"settings" : { "index.refresh_interval" : "5s" },
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}],
"properties" : {
"#version" : { "type" : "string", "index" : "not_analyzed" },
"#timestamp" : { "type" : "date", "format" : "dateOptionalTime" },
"message" : { "type" : "string" }
}
} ,
"my_log" : {
"_all" : { "enabled" : true, "omit_norms" : true },
"dynamic_templates" : [ {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}],
"properties" : {
"#timestamp" : { "type" : "date", "format" : "dateOptionalTime" },
"file" : { "type" : "string" },
"message" : { "type" : "string" }
"geolocation" : { "type" : "string" },
}
}
}
}
Although the #timestamp field is defined as doc_value:true I have an error of MemoryException because it is a fielddata:
[FIELDDATA] Data too large, data for [#timestamp] would be larger than
limit of [633785548/604.4 mb]
NOTE:
I know I can change the memory or add more nodes to the cluster, but in my point of view this is a design problem where this field should not be indexed in memory.
I have nginx logs and i have this date format [02/Mar/2015:13:02:51 +0000]
What should i use in elasticsearch and what i should put in the dateformat field of Kibana4?
curl -XGET 'http://localhost:9200/_mapping?pretty'
{
"nginx" : {
"mappings" : {
"t07_nginx" : {
"properties" : {
"#timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"body_bytes_sent" : {
"type" : "string"
},
"geoip_country_code" : {
"type" : "string"
},
"host" : {
"type" : "string"
},
"http_host" : {
"type" : "string"
},
"http_referer" : {
"type" : "string"
},
"http_user_agent" : {
"type" : "string",
"index" : "not_analyzed"
},
"http_x_forwarded_for" : {
"type" : "string"
},
"message" : {
"type" : "string"
},
"msec request_time" : {
"type" : "string"
},
"remote_addr" : {
"type" : "string"
},
"request_http_protocol" : {
"type" : "string"
},
"request_time" : {
"type" : "string"
},
"request_type" : {
"type" : "string"
},
"request_url" : {
"type" : "string"
},
"status" : {
"type" : "string"
},
"upstream_addr" : {
"type" : "string"
},
"upstream_response_time" : {
"type" : "string"
}
}
}
}
}
with the above i can't see any data(events) in Kibana
Thanks
What does the input plugin for nginx/output plugin for elasticsearch in your fluentd config file look like?
Also, make sure you have your time range setup correctly in kibana. I believe it defaults to 15 minutes.