After upgrading from ES 5 to ES 6 I've got error message each time i want to store something in new index. However all old indexes are working fine.
The error message is:
Rejecting mapping update to [test] as the final mapping would have more than 1 type: [user, group]
Im using elasticsearch 6.3. It works properly on production server on previously created indexes. I've tried dropping index to no avail.
My test documents are:
PUT test/group/1
{
"id": "5b29fb9aa3d24b5a2b6b8fcb",
"_mongo_id_": "5b29fb9aa3d24b5a2b6b8fcb"
}
and
PUT test/user/1
{
"id": "5ad4800ca3d24be81d7a6806",
"_mongo_id_": "5ad4800ca3d24be81d7a6806"
}
Index mapping seems ok:
{
"mapping": {
"group": {
"properties": {
"_mongo_id_": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
You're trying to add more than one type per index
PUT test/group/1
PUT test/user/1
This behaviour is not allowed from ES 6.
From breaking changes
The ability to have multiple mapping types per index has been removed
in 6.0. New indices will be restricted to a single type. This is the
first step in the plan to remove mapping types altogether. Indices
created in 5.x will continue to support multiple mapping types.
Related
I have created an index template, where I am defining an "ignore_above" mapping parameter for a "keyword" field type.
Now I want to update the mappings and want to add a new parameter "eager_global_ordinals" for the same "keyword" field using the PUT mappings API,
I can see the API returning 200OK.
So to verify the mapping when I call GET index-name, I can see the new mappings "eager_global_ordinals", but not able to see "ignore_above"?
I checked, and found out that "ignore_above" property is getting overrided with default value?
Meaning PUT mappings API overriding the mappings parameter to default value, if i dont pass any.
is there any way to avoid this behavior?
Commands I have tried in sequence:
PUT /_index_template/example_template
{
"index_patterns": [
"example*"
],
"priority": 1,
"template": {
"aliases": {
"example":{}
},
"mappings": {
"_source":
{"enabled": false},
"properties": {
"SomeID":
{ "type": "keyword", "index" : true,"store":true,"ignore_above":5}
}
}
}
}
PUT example10022021
GET example10022021
In this GET API response you can see "ignore_above" parameter in mappings
PUT /example/_mappings
{
"properties": {
"SomeID": {
"type": "keyword",
"store":true,
"eager_global_ordinals":true
}
}
}
Now again
GET example10022021
Now you can see in the response that "ignore_above" parameter is
NOT present
That's correct, for the same reason you need to re-specify store: true otherwise the update doesn't work (because it would take the default value which is false), you also need to re-specify ignore_above: 5:
So you can do it like this:
PUT /example/_mappings
{
"properties": {
"SomeID": {
"type": "keyword",
"store":true,
"ignore_above": 5,
"eager_global_ordinals":true
}
}
}
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
}
"status" : 400
I've read that the above is because type are deprecated in elasticsearch 7.7, is that valid for data type? I mean how am I suppose to say I want the data to be considered as a date?
My current mapping has this element:
"Time": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
I just wanted to create it again with type:"date", but I noticed even copy pasting the current mapping (which works) yields an error... The index and mappins I have are generate automatically by https://github.com/jayzeng/scrapy-elasticsearch
My goal is simply to have a date field, I have all my date in my index, but when I want to filter in kibana I can see it is not considered as a date field. And modyfing mapping doesn't seem like an option.
Obvious ELK noob here, please bare with me (:
The error is quite ironic because I pasted the mapping from an existing index/mapping...
You're experiencing this because of the breaking change in 7.0 regarding named doc types.
Long story short, instead of
PUT example_index
{
"mappings": {
"_doc_type": {
"properties": {
"Time": {
"type": "date",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
}
you should do
PUT example_index
{
"mappings": { <-- Note no top-level doc type definition
"properties": {
"Time": {
"type": "date",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
I'm not familiar with your scrapy package but at a glance it looks like it's just a wrapper around the native py ES client which should forward your mapping definitions to ES.
Caveat: if you have your fields defined as text and you intend to change it to date (or add a field of type date), you'll get the following error:
mapper [Time] of different type, current_type [text], merged_type [date]
You basically have two options:
drop the index, set the new mapping & reindex everything (easiest but introduces downtime)
extend the mapping with a new field, let's call it Time_as_datetime and update your existing docs using a script (introduces a brand new field/property -- that's somewhat verbose)
I typically insert data (logs) into elasticsearch via logstash plugin. Then, I
can search them from kibana.
However, if I try to intert data in elasticsearch programatically (in order to
skip filebeat and logstash), I cannot find the data in kibana.
This is what I tested:
from elasticsearch import Elasticsearch
es = Elasticsearch(["XXX"], ...)
doc = {
"#version": 1,
"#timestamp": datetime.now(),
"timestamp": datetime.now(), # Just in case this is needed too
"message": "test message"
}
res = es.index(
index="foobar-2019.05.13", doc_type='whatever', id=3, body=doc,
refresh=True
)
# Doc is indexed by above code, as proved by
# es.search(
# index="foobar-*", body={"query": {"match_all": {}}}
#)
I added the index pattern `foobar-*`` to kibana in "Index Pattern -> Create
index pattern". Then, I can use "discover" page to search for documents in that
index. But no documents are found by kibana, even if those exist in
elasticsearch.
What I am missing? Are there any mappings that should be configured for index?
(note: using 6.x versions)
UPDATE: example of doc indexed, and mapping of index
# Example of doc indexed
{'_index': 'foobar-2019.05.13', '_type': 'doc', '_id': '41', '_score': 1.0,
'_source': {'author': 'foobar', 'message': 'karsa big and crazy. icarium crazy. mappo big.',
'timestamp': '2019-05-13T15:52:19.857898',
'#version': 1, '#timestamp': '2019-05-13T15:52:19.857900'}}
# mapping of foobar-2019.05.13'
{
"mapping": {
"doc": {
"properties": {
"#timestamp": {
"type": "date"
},
"#version": {
"type": "long"
},
"author": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "date"
}
}
}
}
}
I found the issue... there was a 2 hour timezone difference between host were
python code is running and elasticsearch/kibana hosts.
So, since I was using datetime.now(), I was inserting documents with a timestamp "hours in the future", and I was searching for them "anywhere in the past".
If I look for them in the future (or, if I wait for 2 hours without updating
them), they are found.
Embarrassing mistake on my side.
Fix for me was to use datetime.now(timezone.utc)
Currently, I'm trying to implement a variation of the car example here:
https://www.elastic.co/blog/managing-relations-inside-elasticsearch
If I run:
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":"Saturn"
},
{
"key":"year",
"value":"2015"
}
]
}
the code works correctly.
But if I delete the index and change 2015 from a string to a number:
DELETE /vehicle
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":"Saturn"
},
{
"key":"year",
"value":2015
}
]
}
I get the following error message:
{ "error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "mapper [metadata.value] of different type, current_type [long], merged_type [text]"
}
],
"type": "illegal_argument_exception",
"reason": "mapper [metadata.value] of different type, current_type [long], merged_type [text]" }, "status": 400 }
How do I fix this error?
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":"Saturn"
},
{
"key":"year",
"value":2015
}
]
}
After deleting the index and then trying to index a new document as above, the following steps takes place:
When elastic could not find an index by the name of vehicle and auto index creation is enabled (which is by default enabled) it will create a new index named as vehicle.
Based on the input document elastic now tries to make best guess of the datatype for the fields of the document. This is know as dynamic field mapping.
For the above document since metadata is an array of objects the field metadata is assumed to be of object data type.
Now comes the step to decide the data type of fields of individual object. When it encounters the first object it finds two fields key and value. Both these fields have string value (make and Saturn respectively) and hence elastic identifies the datatype for both the fields as text.
Elastic then try to define the mapping as below:
{
"properties": {
"metadata": {
"properties": {
"key": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"value": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
When it encounter the second object where the value of value field is numeric (2015) which it guesses the datatype as long. This results in conflict with the previously identified datatype which was text. A field can not be of mixed data type. Data types are strict, and hence the error.
To resolve the error you need to make sure the input values for the fields are of same type for each of the object as below:
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":2016
},
{
"key":"year",
"value":2015
}
]
}
For above it could be better used as:
PUT /vehicle/_doc/1
{
"metadata":[
{
"make":"Saturn",
"year": 2015
}
]
}
I am using query_string for search. Searching is working fine but its getting all records with small letters and capital letters match.But i want to exact match with case sensitive?
For example :
Search field : "title"
Current output :
title
Title
TITLE,
I want to only first(title). How to resolved this issue.
My code in java :
QueryBuilder qbString=null;
qbString=QueryBuilders.queryString("title").field("field_name");
You need to configure your mappings / text processing so tokens are indexed without being lowercased.
The "standard"-analyzer lowercases (and removes stopwords).
Here's an example that shows how to configure an analyzer and a mapping to achieve this: https://www.found.no/play/gist/7464654
With Version 5 + on ElasticSearch there is no concept of analyzed and not analyzed for index, its driven by type !
String data type is deprecated and is replaced with text and keyword, so if your data type is text it will behave like string and can be analyzed and tokenized.
But if the data type is defined as keyword then automatically its NOT analyzed, and return full exact match.
SO you should remember to mark the type as keyword when you want to do exact match with case sensitive.
code example below for creating index with this definition:
PUT testindex
{
"mappings": {
"original": {
"properties": {
"#timestamp": {
"type": "date"
},
"#version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"APPLICATION": {
"type": "text",
"fields": {
"exact": {"type": "keyword"}
}
},
"type": {
"type": "text",
"fields": {
"exact": {"type": "keyword"}
}
}
}
}
}
}