elasticsearch routing on specific field - elasticsearch

hi I want to set custom routing on specific field "userId" on my Es v2.0.
But it giving me error.I don't know how to set custom routing on ES v2.0
Please guys help me out.Thanks in advance.Below is error message, while creating custom routing with existing index.
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
}
],
"type": "mapper_parsing_exception",
"reason": "Mapping definition for [_routing] has unsupported parameters: [path : userId]"
},
"status": 400
}

In ES 2.0, the _routing.path meta-field has been removed. So now you need to do it like this instead:
In your mapping, you can only specify that routing is required (but you cannot specify path anymore):
PUT my_index
{
"mappings": {
"my_type": {
"_routing": {
"required": true
},
"properties": {
"name": {
"type": "string"
}
}
}
}
}
And then when you index a document, you can specify the routing value in the query string like this:
PUT my_index/my_type/1?routing=bar
{
"name": "foo"
}

You can still use custom routing based on a field from the data being indexed. You can setup a simple pipeline and then use the pipeline every time you index a document, or you can also change the index settings to use the pipeline whenever the index receives a document indexing request.
Read about the pipeline here
Do read above and below the docs for more clarity. It's not meant for setting custom routing, but can be used for the purpose. Custom routing was disabled for a good reason that the field to be used can turn out to have null values leading to unexpected behavior. Hence take care of that issue by yourself.
For routing, here is a sample pipeline PUT :
PUT localhost:9200/_ingest/pipeline/nameRouting
Content-Type: application/json
{
"description" : "Set's the routing value from name field in document",
"processors" : [
{
"set" : {
"field": "_routing",
"value": "{{_source.name}}"
}
}
]
}
The index settings will be :
{
"settings" : {
"index" : {
"default_pipeline" : "nameRouting"
}
}
}

Related

Disable mapping for a specific field using an Index Template Elasticsearch 6.8

I have an EFK pipeline set up. Everyday a new index is created using the logstash-* prefix. Every time a new field is sent by Fluentd, the field is added to the index pattern logstash-*. I'm trying to create an index template that will disable indexing on a specific field when an index is created. I got this to work in ES 7.1 using the PUT below:
PUT _template/logstash-test
{
"index_patterns": ["logstash-*"],
"mappings": {
"dynamic_templates" : [
{
"params" : {
"path_match" : "params",
"mapping" : {
"enabled": false
}
}
}
]
}
}
However when I try this on Elasticsearch 6.8 I get the following error:
"type": "illegal_argument_exception",
"reason": "Malformed [mappings] section for type [dynamic_templates], should include an inner object describing the mapping"
It is a little different in Elasticsearch 6.X as it had mapping types, which is not used anymore.
Try something like this:
PUT _template/logstash-test
{
"index_patterns": ["logstash-*"],
"mappings": {
"_doc": {
"dynamic_templates" : [
{
"params" : {
"path_match" : "params",
"mapping" : {
"enabled": false
}
}
}
]
}
}
}
If your index has a different custom type and is not using the _doc type, you should use that in the mapping.

How to update index mapping with dynamic as strict in future?

I am new to elastic search. I have an index named users, which has a lot of fields I know. But a few more fields can be added in the future.
So when defining my mapping, I want to include the fields that I know currently with dynamic "strict", but in the future, if I want to add the new field, how will update the new mapping and if I do it, will I have to reindex everything.
I found in the ES documents that mappings are applied only during index creation time. So I am a little confused here, what's the right way to approach this.
You can always update the mapping in future, even after keeping it strict using the put mapping api. You'll not require existing data to be re-indexed unless you want the newly added field have some value for the older documents which were added before updating the mapping with the new field.
Lets assume you already have an index test with one field say field1 of type keyword. Now in future you have a requirement to add new field say field2 of integer type. You can do so by the put mapping api as below,
PUT test/_mapping
{
"properties": {
"field2": {
"type": "integer"
}
}
}
After executing the above if you check the mapping using
GET test/_mapping
You can see the new field as well in the response,
{
"test" : {
"mappings" : {
"dynamic" : "strict",
"properties" : {
"field1" : {
"type" : "keyword"
},
"field2" : {
"type" : "integer"
}
}
}
}
}
Inner objects inherit the dynamic setting from their parent object or from the mapping type. In the following example, dynamic mapping is disabled at the type level, so no new top-level fields will be added dynamically.
However, the user.social_networks object enables dynamic mapping, so you can add fields to this inner object.
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html
PUT my-index-000001
{
"mappings": {
"dynamic": false,
"properties": {
"user": {
"properties": {
"name": {
"type": "text"
},
"social_networks": {
"dynamic": true,
"properties": {}
}
}
}
}
}
}

Elastic Search: Alternative of flattened datatype in Elastic Search 7.1

I have two Elastic Search version one is 7.3 and the second is 7.1. I am using flattened data type for Elastic Search 7.3 and I also want to use this data type in Elastic Search 7.1. So that I can store my data as I stored in Elastic Search 7.3.
I researched about flattened data type and get to know that it's supported to 7.x but when I tried in 7.1 it gives me the mapper_parsing_exception error.
What I tried is as shown below.
In Elastic Search 7.3
Index Creation
PUT demo-flattened
Response:
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "demo-flattened"
}
Insert Mapping
PUT demo-flattened/_mapping
{
"properties": {
"host": {
"type": "flattened"
}
}
}
Response:
{
"acknowledged": true
}
In Elastic Search 7.1
PUT demo-flattened
Response:
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "demo-flattened"
}
Insert Mapping
PUT demo-flattened/_mapping
{
"properties": {
"host": {
"type": "flattened"
}
}
}
Response:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "No handler for type [flattened] declared on field [host]"
}
],
"type": "mapper_parsing_exception",
"reason": "No handler for type [flattened] declared on field [host]"
},
"status": 400
}
I want to use the flattened data type in Elastic Search 7.1. Is there any alternative to use flattened data type in the 7.1 version because flattened data type is supported from Elastic Search 7.3.
Any help or suggestions will be appreciated.
First the flattened is available in 7.1 with X-pack (X-pack is paid feature),
so what I think you can use object type with enabled flag as false
This will help you store that field as it is without any indexing.
{
"properties": {
"host": {
"type": "object",
"enabled": false
}
}
}
Check the version of your ElasticSearch. If its the OSS version, then it won't work for you.
You can check it by running GET \ in the Kibana. You would get something like:
{
"version" : {
"number" : "7.10.2",
"build_flavor" : "oss",
}
}
But for ElasticSearch that does support flattened type, you would get something like:
"version" : {
"number" : "7.10.2",
"build_flavor" : "default",
}
}
You can find more details on the official Kibana Github page No handler for type [flattened] declared on field [state] #52324.
Interally, it works like this
Similarities in the way values are indexed, flattened fields share much of the same mapping and search functionality as keyword fields
Here, You have only one field called host. You can replace this with keyword.
What similarities:
Mapping:
"labels": {
"type": "flattened"
}
Data:
"labels": {
"priority": "urgent",
"release": ["v1.2.5", "v1.3.0"],
"timestamp": {
"created": 1541458026,
"closed": 1541457010
}
}
During indexing, tokens are created for each leaf value in the JSON object. The values are indexed as string keywords, without analysis or special handling for numbers or dates
To query them, you can use "term": {"labels": "urgent"} or "term": {"labels.release": "v1.3.0"}.
When it is keyword, you can have them as separate fields.
{
"host":{
"type":"keyword"
}
}
Reference

Cannot create elasticsearch mapping or get date field to work

{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
}
"status" : 400
I've read that the above is because type are deprecated in elasticsearch 7.7, is that valid for data type? I mean how am I suppose to say I want the data to be considered as a date?
My current mapping has this element:
"Time": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
I just wanted to create it again with type:"date", but I noticed even copy pasting the current mapping (which works) yields an error... The index and mappins I have are generate automatically by https://github.com/jayzeng/scrapy-elasticsearch
My goal is simply to have a date field, I have all my date in my index, but when I want to filter in kibana I can see it is not considered as a date field. And modyfing mapping doesn't seem like an option.
Obvious ELK noob here, please bare with me (:
The error is quite ironic because I pasted the mapping from an existing index/mapping...
You're experiencing this because of the breaking change in 7.0 regarding named doc types.
Long story short, instead of
PUT example_index
{
"mappings": {
"_doc_type": {
"properties": {
"Time": {
"type": "date",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
}
you should do
PUT example_index
{
"mappings": { <-- Note no top-level doc type definition
"properties": {
"Time": {
"type": "date",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
I'm not familiar with your scrapy package but at a glance it looks like it's just a wrapper around the native py ES client which should forward your mapping definitions to ES.
Caveat: if you have your fields defined as text and you intend to change it to date (or add a field of type date), you'll get the following error:
mapper [Time] of different type, current_type [text], merged_type [date]
You basically have two options:
drop the index, set the new mapping & reindex everything (easiest but introduces downtime)
extend the mapping with a new field, let's call it Time_as_datetime and update your existing docs using a script (introduces a brand new field/property -- that's somewhat verbose)

how to index a file with elasticsearch 5.5.1

I'm new to Elasticsearch. I have successfully installed Elasticsearch with Kibana, X-pack and ingest-attachment. I have both Elasticsearch and Kibana running. I have kept it simple at the moment with the install using default options on a windows 2012 server. I have a directory on another drive w\mydocs and at the moment it just has 3 plain text files in it, but I will want to add others like pdf and doc file types. So now I want to get these files into Elasticsearches index. I have tried using the following link as a guide Attaching pdf docs in Elasticsearch, however I cannot get it to work.
Here's how I have set up the index and pipeline:
PUT _ingest/pipeline/docs
{
"description": "documents",
"processors" : [
{
"attachment" : {
"field": "data",
"indexed_chars" : -1
}
}]
}
PUT myindex
{
"mappings" : {
"documents" : {
"properties" : {
"attachment.data" : {
"type": "text",
"analyzer": "standard"
}
}
}
}
}
Then to get the first document in I use the following:
PUT localhost:9200/documents/1?pipeline=docs -d #/w/mydocs/README.TXT
and the error that I receive is:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "request body is required"
}
],
"type": "parse_exception",
"reason": "request body is required"
},
"status": 400
}
you still have to send valid JSON to Elasticsearch, even when indexing binary data. This means, that you have to encode your document as base64 and then put it into a JSON document like this
{
"data" : "base64encodedcontentofyourfile"
}
I was advised not to use the ingest-attachment, but instead to use FsCrawler. I managed to get Fscrawler working without having to convert anything to base64.

Resources