I am new to elasticsearch and I thought I will go through the 10 minutes walk through to get started.
But I stumbled upon with some very basic doubts here. I am not able to figure out the data representation here. For eg. the tutorial mentions about creating an index
curl -XPUT http://localhost:9200/shakespeare -d '
{
"mappings" : {
"_default_" : {
"properties" : {
"speaker" : {"type": "string", "index" : "not_analyzed" },
"play_name" : {"type": "string", "index" : "not_analyzed" },
"line_id" : { "type" : "integer" },
"speech_number" : { "type" : "integer" }
}
}
}
}
';
I understand that this is a JSON string, but beyond that I am not able to understand this representation? I am not getting what is default, what is meant by not_analyzed and so on.
Is there any standard that needs to be understood on how the data is represented before proceeding with elasticsearch?
I am totally new to elasticsearch and would really appreciate if I am guided with some information/tutorial which would help me understand how to start learning this technology.
Thanks & Regards
Sunil
I think that the main aim of the 10 minutes walk through is to give a quick demo about Kibana and not a full understanding of elasticsearch (mapping,indexing,etc.)
But if you wish to understand what's happening in that example, you might want to know how to go through the documentation.
Example :
default mapping :
Often, all types in an index share similar fields and settings.
It can be more convenient to specify these common settings in
the _default_ mapping, instead of having to repeat yourself every
time you create a new type. The _default_ mapping acts as a
template for new types. All types created after the _default_
mapping will include all of these default settings, unless
explicitly overridden in the type mapping itself.
And for more details about default mapping, please refer here.
The 10 minute walk-thru is for Kibana, running on top of Elasticsearch, and IMHO is not a great place to start when getting to know ES.
Personally over the last few years I've these introductions to be helpful:
http://joelabrahamsson.com/elasticsearch-101/
http://exploringelasticsearch.com/overview.html
Overall the ES documentation is reasonably complete, looks great but can be hard to navigate thru for a novice to find exactly what you need.
Related
How to implement ElasticSearch new index creation after every fix number of days and if its possible then how to search over all the previous indexes? Currently we have only one index which has all the data. I looked at the RollOver API of ES, is this the correct way? But the problem seems when we want to search for some data in previous indexes, how this can be done? Any answers are appreciated, Thanks.
Yes, you are on the correct path, for searching into your old indices, you can link multiple indices to one alias, using alias API, and instead of searching for a single index, you need to search again the unified alias.
Refer to this official example on how to link multiple indices to the same alias(alias1 in the below example)
POST /_aliases
{
"actions" : [
{ "add" : { "index" : "test1", "alias" : "alias1" } },
{ "add" : { "index" : "test2", "alias" : "alias1" } }
]
}
I'm having trouble with what seems like a fairly basic use case, but I'm hitting certain limitations in Kibana and problems with certain geo data types. It's starting to feel like I'm just approaching it wrong.
I have a relatively large point data set (locations) of type geo_point, with a map and dashboard built. I now want to add a complex AOI. I took the shapefile, dissolved it so it became one feature instead of many, converted it to geojson and uploaded it (to create an index) via the Kibana Maps functionality. I then made it available as layer, and wanted to just allow it to be selected, show tooltip, and then Filter by Feature. Unfortunately I then received an error saying along the lines that this would be too large an operation to be posted to the URL - which I understand as there are over 2 million characters in the geojson.
Instead I thought I could write the query somehow according to the guidance on: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-geo-shape-query.html
with the pre-indexed shape.
However, it doesn't seem to work to allow geo_point to be queried against geo_shape.
e.g.
GET /locations_index/_search
{
"query": {
"geo_point": {
"geolocation": {
"relation": "within",
"indexed_shape": {
"index": "aoi_index",
"id": "GYruUnMBfgunZ6kjA8qn",
"path": "coordinates"
}
}
}
}
}
Gives an error of:
{
"error" : {
"root_cause" : [
{
"type" : "parsing_exception",
"reason" : "no [query] registered for [geo_point]",
"line" : 3,
"col" : 18
}
],
"type" : "parsing_exception",
"reason" : "no [query] registered for [geo_point]",
"line" : 3,
"col" : 18
},
"status" : 400
}
Do I need to convert my points index to be geoshape instead of geopoints? Or is there a simpler way?
I note the documentation at: https://www.elastic.co/guide/en/elasticsearch/guide/current/filter-by-geopoint.html suggests that I can query by geo_polygon, but I can't see any way of referencing my pre-indexed shape, instead of having the huge chunk of JSON in the query (as the example suggests).
Can anyone point me (even roughly) in the right direction?
Thanks in advance.
Here's how you can utilize indexed_shape. Let me know if this answer is sufficient to get you started.
I am using grafana 3.1 on maxOs Sierra and I use elasticsearch as datasource.
I have set my mapping on ES:
"properties" : {
"gasLeak" : {
"type" : "boolean"
},
"gasConsumption" : {
"type" : "boolean"
},
"electricityConsumption" : {
"type" : "boolean"
}
}
I have also a timestamp filed name alert.timestamp which is typed as a date on ES.
Everytime an alert is triggered, I add an entry into the ES index. There are 3 alerts type as shown in the mapping.
Now, on grafana, I have my dashboard with a graph panel. On this graph I want to display the annotation which represents alerts.
I have configured my alert as follow:
Grafana annotation config
But then on my graph, the annotation is displayed aheah of my measures: it is always on the right at the current datetime:
Issue on graph
Am I configuring the annoation wrongly? Do I need other configuration on ES side?
Thank you.
I found why it was not working: in the "field mapping" part, the filed "Time" has to be "timestamp"...
I used to put alert.timestamp and when I updated my ES document structure using timestamp instead of alert.timestamp as field name, after updating the field in grafana, it worked.
I also added the field "desc" and "tags" in my ES document so maybe it is also related, I did not dig that.
Hope this could help someone someday.
For the app i created, the indices are generated once in a week. And the type and nature of the data is not varying and that implies, I need the same mapping type for these indices. Is it possible in elasticsearch to apply the same mapping to all the indices as they are created?. This could avoid me the overhead of defining mapping each time the index is created.
Definitely, you can use what is called an index template. Since your mapping type is stable, that's the perfect condition for using index templates.
It's as easy as creating an index. See below, whenever you want to index a document in an index whose name matches my_*, ES will select that template and create the index for you using the given mappings, settings and aliases:
curl -XPUT localhost:9200/_template/template_1 -d '{
"template" : "my_*",
"settings" : {
"number_of_shards" : 1
},
"aliases" : {
"my_alias" : {}
},
"mappings" : {
"my_type" : {
"properties" : {
"my_field": { "type": "string" }
}
}
}
}'
It's basically the technique used by Logstash when it needs to index new logs for each new day in a new daily index.
You can employ index template to address your problem. The official documentation can be found here.
A use case of how to apply the same with examples can be found in this blog
Consider the below document mapping:
{
"admin/user" : {
"properties" : {
"message" : {"type" : "string", "store" : "yes"},
"user_id" : {"type" : "long", "included_in_all" : "false"}
}
}
}
If we have a forward slash in the document type, how do I access the user_id type in the search query? I have another document with similar user_id field so can't just directly use the user_id in the field value.
Need to use this in term filter.
I have already tried "admin/user.user_id" and "admin%2Fuser.user_id". But none seem to work in my case.
Elasticsearch version: 0.90.1
Using Tire Ruby client (version 0.4.3). I know this is pretty old version, but can't think of moving to new tire version as of now.
Using "admin/user.user_id" in term filter actually worked. I forgot to reindex the data after the change in the mapping...Thats really dumb, I know.