How to define specific field tokenization on Logstash - elasticsearch

I am using logstash to index some mysql data on elasticsearch:
input {
jdbc {
// JDBC configurations
}
}
output {
elasticsearch {
index => ""
document_type => ""
document_id => ""
hosts => [ "" ]
}
}
When checking results I found that elasticsearch automatically tokenizes the text like this:
"Foo/Bar" -> "Foo", "Bar"
"The thing" -> "The", "thing"
"Fork, Knife" -> "Fork", "Knife"
Well, that is ok for most of my fields. But there is one specific field that I'd like to have a custom tokenizer. It is a comma separated field (or semi-colon separated). So it should be:
"Foo/Bar" -> "Foo/Bar"
"The thing" -> "The thing"
"Fork, Knife" -> "Fork", "Knife"
I wander if there is a way to configure this on my logstash configuration.
UPDATE:
This is one example of the index that I have. The specific field is kind:
{
"index-name": {
"aliases": {},
"mappings": {
"My-type": {
"properties": {
"#timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"#version": {
"type": "string"
},
"kind": {
"type": "string"
},
"id": {
"type": "long"
},
"text": {
"type": "string"
},
"version": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "",
"number_of_shards": "",
"number_of_replicas": "",
"uuid": "",
"version": {
"created": ""
}
}
},
"warmers": {}
}
}

It's possible to do so by using an index template.
First delete your current index:
DELETE index_name
Then create the template for your index with the appropriate mapping for the kind field, like this:
PUT _template/index_name
{
"template": "index-name",
"mappings": {
"My-type": {
"properties": {
"#timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"#version": {
"type": "string"
},
"kind": {
"type": "string",
"index": "not_analyzed"
},
"id": {
"type": "long"
},
"text": {
"type": "string"
},
"version": {
"type": "string"
}
}
}
}
}
Then you can run Logstash again and the index will be re-created with the proper mapping.

Well, the right answer for this question is: you cannot do it by logstash. So I had to add an additional step as follow.
I finally got this done following the path showed by #Val. Thanks, pal. So, what I had to do was to create the index before the logstash ETL with a specific tokenizer:
{
"settings": {
"analysis": {
"analyzer": {
"simple_analyzer": {
"tokenizer": "simple_tokenizer"
}
},
"tokenizer": {
"simple_tokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
},
"template": "my-index",
"mappings": {
"my-type": {
"properties": {
"kind": {
"type": "string",
"analyzer": "simple_analyzer"
}
}
}
}
}
This will create a tokenizer by comma to the kind field. After that, I can perform the logstash etl and it won't overwrite the kind properties.

Related

adding a script-based field to an elasticSearch index mapping

I am following the following docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-indexed.html
I have a field which I would like to not be scripted on runtime but rather on index-time, and according to above I can do that simply by putting the field and its script inside the mapping object as normal.
Here is a simplified version of the index I'm trying to create
{
"settings": {
"analysis": {
"analyzer": {
"case_insensitive_analyzer": {
"type": "custom",
"filter": ["lowercase"],
"tokenizer": "keyword"
}
}
}
},
"mappings": {
"properties": {
"id": {
"type": "text"
},
"events": {
"properties": {
"fields": {
"type": "text"
},
"id": {
"type": "text"
},
"event": {
"type": "text"
},
"time": {
"type": "date"
},
"user": {
"type": "text"
},
"state": {
"type": "integer"
}
}
},
"eventLast": {
"type": "date",
"on_script_error": "fail",
"script": {
"source": "def events = doc['events']; emit(events[events.length-1].time.value"
}
}
}
}
}
I'm getting this 400 error back:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "unknown parameter [script] on mapper [eventLast] of type [date]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [_doc]: unknown parameter [script] on mapper [eventLast] of type [date]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "unknown parameter [script] on mapper [eventLast] of type [date]"
}
},
"status": 400
}
Essentially I'm trying to create a scripted indexed field that is calculated off the last event time in the events array of the document.
Thanks
Tldr;
As the error states, you can not define your script in here.
There is a specific way to create runtime fields in elasticsearch.
You need to put the definition at the root of the json in the runtime object.
Solution
{
"settings": {
"analysis": {
"analyzer": {
"case_insensitive_analyzer": {
"type": "custom",
"filter": ["lowercase"],
"tokenizer": "keyword"
}
}
}
},
"runtime": {
"eventLast": {
"type": "date",
"on_script_error": "fail",
"script": {
"source": "def events = doc['events']; emit(events[events.length-1].time.value"
}
}
},
"mappings": {
"properties": {
"id": {
"type": "text"
},
"events": {
"properties": {
"fields": {
"type": "text"
},
"id": {
"type": "text"
},
"event": {
"type": "text"
},
"time": {
"type": "date"
},
"user": {
"type": "text"
},
"state": {
"type": "integer"
}
}
}
}
}
}

How do I create a template that allows dynamic index alias in elasticsearch

I'm trying to follow this doc using elasticsearch 2.4 where multiple tenant data can be put into one index but by using alias and routes I can search an aliased index and only retrieve information from one particular tenant.
Briefly, I could have an index filled with widget doc types under the index search_widgets. but the template would create an alias as I enter in widget docs based on the value of the user-id. If the widget type doc has user-id = 1 it would create an index alias search_widget-1. If searched with a get-all on that index it would return widgets docs that has user-id = 1 as a field in it.
Here's a incomplete example of how it think it should work
PUT /search_widgets
{
"search_widgets": {
"settings": {
"number_of_shards": 2
},
"mappings": {
"widget": {
"properties": {
"#timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"#version": {
"type": "string"
},
"active": {
"type": "boolean"
},
"user-id": {
"type": "number",
"index": "not_analyzed"
},
"created_date": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"name": {
"type": "string"
},
"updated_date": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"id": {
"type": "string",
"index": "not_analyzed"
}
}
}
},
"aliases": {
"widget": {
"routing": "search_widget_*",
"filter": {
"term": {
"user-id": "*"
}
}
}
}
}
}

Elasticsearch.js analyzer error using custom analyzer

Using the latest version of the elasticsearch.js and trying to create a custom path analyzer when indexing and creating the mapping for some posts.
The goal is creating keywords out of each segment of the path. However as a start simply trying to get the analyzer working.
Here is the elasticsearch.js create_mapped_index.js, you can see the custom analyzer near the top of the file:
var client = require('./connection.js');
client.indices.create({
index: "wcm-posts",
body: {
"settings": {
"analysis": {
"analyzer": {
"wcm_path_analyzer": {
"tokenizer": "wcm_path_tokenizer",
"type": "custom"
}
},
"tokenizer": {
"wcm_path_tokenizer": {
"type": "pattern",
"pattern": "/"
}
}
}
},
"mappings": {
"post": {
"properties": {
"id": { "type": "string", "index": "not_analyzed" },
"titles": {
"type": "object",
"properties": {
"main": { "type": "string" },
"subtitle": { "type": "string" },
"alternate": { "type": "string" },
"concise": { "type": "string" },
"seo": { "type": "string" }
}
},
"tags": {
"properties": {
"id": { "type": "string", "index": "not_analyzed" },
"name": { "type": "string", "index": "not_analyzed" },
"slug": { "type": "string" }
},
},
"main_taxonomies": {
"properties": {
"id": { "type": "string", "index": "not_analyzed" },
"name": { "type": "string", "index": "not_analyzed" },
"slug": { "type": "string", "index": "not_analyzed" },
"path": { "type": "string", "index": "wcm_path_analyzer" }
},
},
"categories": {
"properties": {
"id": { "type": "string", "index": "not_analyzed" },
"name": { "type": "string", "index": "not_analyzed" },
"slug": { "type": "string", "index": "not_analyzed" },
"path": { "type": "string", "index": "wcm_path_analyzer" }
},
},
"content_elements": {
"dynamic": "true",
"type": "nested",
"properties": {
"content": { "type": "string" }
}
}
}
}
}
}
}, function (err, resp, respcode) {
console.log(err, resp, respcode);
});
If the call to wcm_path_analyzer is set to "non_analyzed" or index is omitted the index, mapping and insertion of posts work.
As soon as I try to use the custom analyzer on the main_taxonomy and categories path fields, like shown in the json above, I get this error:
response: '{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"wrong value for index [wcm_path_analyzer] for field [path]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [post]: wrong value for index [wcm_path_analyzer] for field [path]","caused_by":{"type":"mapper_parsing_exception","reason":"wrong value for index [wcm_path_analyzer] for field [path]"}},"status":400}',
toString: [Function],
toJSON: [Function] } { error:
{ root_cause: [ [Object] ],
type: 'mapper_parsing_exception',
reason: 'Failed to parse mapping [post]: wrong value for index [wcm_path_analyzer] for field [path]',
caused_by:
{ type: 'mapper_parsing_exception',
reason: 'wrong value for index [wcm_path_analyzer] for field [path]' } },
status: 400 } 400
Here is an example of the two objects that need the custom analyzer on the path field. I pulled this example, after inserting 15 posts into the elasticsearch index when not using the custom analyzer:
"main_taxonomies": [
{
"id": "123",
"type": "category",
"name": "News",
"slug": "news",
"path": "/News/"
}
],
"categories": [
{
"id": "157",
"name": "Local News",
"slug": "local-news",
"path": "/News/Local News/",
"main": true
},
To this point, I had googled similar questions and most said that people were missing putting the analyzers in settings and not adding the parameters to the body. I believe this is correct.
I have also reviewed the elasticsearch.js documentation and tried to create a:
client.indices.putSettings({})
But for this to be used the index needs to exist with the mappings or it throws an error 'no indices found'
Not sure where to go from here? Your suggestions are appreciated.
So the final analyzer is:
var client = require('./connection.js');
client.indices.create({
index: "wcm-posts",
body: {
"settings": {
"analysis": {
"analyzer": {
"wcm_path_analyzer": {
"type" : "pattern",
"lowercase": true,
"pattern": "/"
}
}
}
},
"mappings": {
"post": {
"properties": {
"id": { "type": "string", "index": "not_analyzed" },
"client_id": { "type": "string", "index": "not_analyzed" },
"license_id": { "type": "string", "index": "not_analyzed" },
"origin_id": { "type": "string" },
...
...
"origin_slug": { "type": "string" },
"main_taxonomies_path": { "type": "string", "analyzer": "wcm_path_analyzer", "search_analyzer": "standard" },
"categories_paths": { "type": "string", "analyzer": "wcm_path_analyzer", "search_analyzer": "standard" },
"search_tags": { "type": "string" },
// See the custom analyzer set here --------------------------^
I did determine that at least for the path or pattern analyzers that complex nested or objects cannot be used. The flattened fields set to "type": "string" was the only way to get this to work.
I ended up not needing a custom tokenizer as the pattern analyzer is full featured and already includes a tokenizer.
I chose to use the pattern analyzer as it breaks on the pattern leaving individual terms whereas the path segments the path in different ways but does not create individual terms ( I hope I'm correct in saying this. I base it on the documentation ).
Hope this helps someone else!
Steve
So I got it working ... I think that the json objects were too complex or it was the change of adding the analyzer to the field mappings that did the trick.
first I flattened out:
To:
"main_taxonomies_path": "/News/",
"categories_paths": [ "/News/Local/", "/Business/Local/" ],
"search_tags": [ "montreal-3","laval-4" ],
Then I updated the analyzer to:
"settings": {
"analysis": {
"analyzer": {
"wcm_path_analyzer": {
"tokenizer": "wcm_path_tokenizer",
"type": "custom"
}
},
"tokenizer": {
"wcm_path_tokenizer": {
"type": "pattern",
"pattern": "/",
"replacement": ","
}
}
}
},
Notice that the analyzer 'type' is set to custom.
Then when mapping theses flattened fields:
"main_taxonomies_path": { "type": "string", "analyzer": "wcm_path_analyzer" },
"categories_paths": { "type": "string", "analyzer": "wcm_path_analyzer" },
"search_tags": { "type": "string" },
which when searching yields for these fields:
"main_taxonomies_path": "/News/",
"categories_paths": [ "/News/Local News/", "/Business/Local Business/" ],
"search_tags": [ "montreal-2", "laval-3" ],
So the custom analyzer does what it was set to do in this situation.
I'm not sure if I could apply type object to the main_taxonomies_path and categories_paths, so I will play around with this and see.
I will be refining the pattern searches to format the results differently but happy to have this working.
For completeness I will put my final custom pattern analyzer, mapping and results, once I've completed this.
Regards,
Steve

Elasticsearch + Kibana, sorting on uri yields no results. (uri isn't analyzed)

I have a log of HTTP requests, one of the fields is a URI field. I want to get the average duration in ms for each URI. I set the y-axis in Kibana to
"Aggregation: Average , Field: durationInMs".
For the x-axis I have
"Aggregation: terms, Field uri, Order by: metric average durationInMs, Order: descending: 5"
Image to clarify:
This gives me a result but it doesn't use the entire URI. It instead splits up the URI and matches parts of it. After a quick google I found "Multi-fields" and I have added a URI.raw field on my index. The analyzed field warning disappeared but I get no result at all.
Any hints or tips?
lsc-logs2 mapping:
{
"lsc-logs2": {
"mappings": {
"httplogentry": {
"properties": {
"context": {
"type": "string"
},
"durationInMs": {
"type": "double"
},
"id": {
"type": "long"
},
"method": {
"type": "string"
},
"source": {
"type": "string"
},
"startTime": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"status": {
"type": "long"
},
"uri": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"username": {
"type": "string"
},
"version": {
"type": "long"
}
}
}
}
}
}
An example document:
{
"_index": "lsc-logs2",
"_type": "httplogentry",
"_id": "1148440",
"_score": 1,
"_source": {
"startTime": "2016-08-22T10:30:57.2298086+02:00",
"context": "contexturi",
"method": "GET",
"uri": "http://uri/plannings/unassigned?date=2016-08-22T03:58:57.168Z&page=1&pageSize=9999",
"username": "user",
"source": "192.168.1.82",
"durationInMs": 171.83710000000002,
"status": 200,
"id": 1148440,
"version": 1
}
}
When reindexing data, the httplogentry mapping doesn't get ported from lsc-logs to lsc-logs2, you need to create the destination index+mapping first and only then reindex.
First delete the current destination index
curl -XDELETE localhost:9200/lsc-logs2
Then create it anew by specifying the proper mapping
curl -XPUT localhost:9200/lsc-logs2 -d '{
"mappings": {
"httplogentry": {
"properties": {
"context": {
"type": "string"
},
"durationInMs": {
"type": "double"
},
"id": {
"type": "long"
},
"method": {
"type": "string"
},
"source": {
"type": "string"
},
"startTime": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
"status": {
"type": "long"
},
"uri": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"username": {
"type": "string"
},
"version": {
"type": "long"
}
}
}
}
}'
Then you can reindex your data:
curl -XPOST localhost:9200/_reindex -d '{
"source": {
"index": "lsc-logs"
},
"dest": {
"index": "lsc-logs2"
}
}'
Then refresh your the fields in your index pattern in Kibana and it should work.

ElasticSearch - how to not store fields that are not defined in the static index?

I have set a static index with user entity in ES using
{
"mappings": {
"_default_": {
"dynamic": "false"
},
"user": {
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
}
}
}
}
When I post a document with more fields than in the index it saves them to the ES.
It doesn't update the mapping but it saves the new fields.
Is there a way to remove the fields that are not in the index?
I dont want to store un-indexed fields.
In your mapping you need to use _source filtering:
{
"mappings": {
"_default_": {
"dynamic": "false"
},
"user": {
"_source": {
"includes": [
"id","name","age"
]
},
"properties": {
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
}
}
}
}

Resources