{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
}
"status" : 400
I've read that the above is because type are deprecated in elasticsearch 7.7, is that valid for data type? I mean how am I suppose to say I want the data to be considered as a date?
My current mapping has this element:
"Time": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
I just wanted to create it again with type:"date", but I noticed even copy pasting the current mapping (which works) yields an error... The index and mappins I have are generate automatically by https://github.com/jayzeng/scrapy-elasticsearch
My goal is simply to have a date field, I have all my date in my index, but when I want to filter in kibana I can see it is not considered as a date field. And modyfing mapping doesn't seem like an option.
Obvious ELK noob here, please bare with me (:
The error is quite ironic because I pasted the mapping from an existing index/mapping...
You're experiencing this because of the breaking change in 7.0 regarding named doc types.
Long story short, instead of
PUT example_index
{
"mappings": {
"_doc_type": {
"properties": {
"Time": {
"type": "date",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
}
you should do
PUT example_index
{
"mappings": { <-- Note no top-level doc type definition
"properties": {
"Time": {
"type": "date",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
I'm not familiar with your scrapy package but at a glance it looks like it's just a wrapper around the native py ES client which should forward your mapping definitions to ES.
Caveat: if you have your fields defined as text and you intend to change it to date (or add a field of type date), you'll get the following error:
mapper [Time] of different type, current_type [text], merged_type [date]
You basically have two options:
drop the index, set the new mapping & reindex everything (easiest but introduces downtime)
extend the mapping with a new field, let's call it Time_as_datetime and update your existing docs using a script (introduces a brand new field/property -- that's somewhat verbose)
Related
When putting an anaylzer into mapping using PUT /job/_mapping/doc/ but get conflicts.
But there isn't a anaylzer in mappings.
PUT /job/_mapping/doc/
{
"properties":{
"title": {
"type": "text",
"analyzer":"ik_smart",
"search_analyzer":"ik_smart"
}
}
}
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Mapper for [title] conflicts with existing mapping in other types:\n[mapper [title] has different [analyzer]]"
}
],
"type": "illegal_argument_exception",
"reason": "Mapper for [title] conflicts with existing mapping in other types:\n[mapper [title] has different [analyzer]]"
},
"status": 400
}
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"fielddata": true
},
The output config is like this.
output {
elasticsearch {
hosts => ["<Elasticsearch Hosts>"]
user => "<user>"
password => "<password>"
index => "<table>"
document_id => "%{<MySQL_PRIMARY_KEY>}"
}
}
You cant update mapping in elasticsearch, you can add mapping but not update mapping. Elasticsearch use mapping at the indexation time, that s why you cant update mapping of an existing field. Analyzer is part of the mapping, in fact if you don't specify one es a default one, analyzer tell elastic how to index the documents.
create a new index with your new mappings (include analyzer)
reindex your documents from your existing index to the new one (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)
Updating Mapping:
Once a document is indexed to an index i.e. the mapping is generated under a given type as like in our case Mapping of EmployeeCode, EmployeeName & isDevelopers' is generated under type "customtype", we cannot modify it afterwards. In case if we want to modify it, we need to delete the index first and then apply the modified mapping manually and then re-index the data. But If you want to add an a new property under a given type, then it is feasible. For example, our document attached our index "inkashyap-1002" under type "customtype" is as follows:
{
"inkashyap-1002": {
"mappings": {
"customtype": {
"properties": {
"EmployeeCode": {
"type": "long"
},
"isDeveloper": {
"type": "boolean"
},
"EmployeeName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
now let's add another property "Grade" :
curl -XPUT localhost:9200/inkashyap-1002(IndexName)/customtype(TypeName)/2 — d '{
"EmployeeName": "Vaibhav Kashyap",
"EmployeeCode": 13629,
"isDeveloper": true,
"Grade": 5
}'
Now hit the GET mapping API. In the results, you can see there is another field added called "Grade".
Common Error:
In the index "inkashyap-1002", so far we have indexed 2 documents. Both the documents had the same type for the field "EmployeeCode" and the type was "Long". Now let us try to index a document like below:
curl -XPUT localhost:9200/inkashyap-1002/customtype/3 -d '{
"EmployeeName": "Vaibhav Kashyap",
"EmployeeCode": "onethreesixtwonine",
"isDeveloper": true,
"Grade": 5
}'
Note that here the "EmployeeCode" is given in string type, which indicates that it is a string field. The response to the above request will be like below:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "failedtoparse[
EmployeeCode
]"
}
],
"type": "mapper_parsing_exception",
"reason": "failedtoparse[
EmployeeCode
]",
"caused_by": {
"type": "number_format_exception",
"reason": "Forinputstring: \"onethreesixtwonine\""
}
},
"status": 400
}
In the above response, we can see the error "mapper_parsing_exception" on the field "EmployeeCode". This indicates that the expected field here was of another type and not string. In such cases re-index the document with the appropriate type
Currently, I'm trying to implement a variation of the car example here:
https://www.elastic.co/blog/managing-relations-inside-elasticsearch
If I run:
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":"Saturn"
},
{
"key":"year",
"value":"2015"
}
]
}
the code works correctly.
But if I delete the index and change 2015 from a string to a number:
DELETE /vehicle
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":"Saturn"
},
{
"key":"year",
"value":2015
}
]
}
I get the following error message:
{ "error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "mapper [metadata.value] of different type, current_type [long], merged_type [text]"
}
],
"type": "illegal_argument_exception",
"reason": "mapper [metadata.value] of different type, current_type [long], merged_type [text]" }, "status": 400 }
How do I fix this error?
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":"Saturn"
},
{
"key":"year",
"value":2015
}
]
}
After deleting the index and then trying to index a new document as above, the following steps takes place:
When elastic could not find an index by the name of vehicle and auto index creation is enabled (which is by default enabled) it will create a new index named as vehicle.
Based on the input document elastic now tries to make best guess of the datatype for the fields of the document. This is know as dynamic field mapping.
For the above document since metadata is an array of objects the field metadata is assumed to be of object data type.
Now comes the step to decide the data type of fields of individual object. When it encounters the first object it finds two fields key and value. Both these fields have string value (make and Saturn respectively) and hence elastic identifies the datatype for both the fields as text.
Elastic then try to define the mapping as below:
{
"properties": {
"metadata": {
"properties": {
"key": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"value": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
When it encounter the second object where the value of value field is numeric (2015) which it guesses the datatype as long. This results in conflict with the previously identified datatype which was text. A field can not be of mixed data type. Data types are strict, and hence the error.
To resolve the error you need to make sure the input values for the fields are of same type for each of the object as below:
PUT /vehicle/_doc/1
{
"metadata":[
{
"key":"make",
"value":2016
},
{
"key":"year",
"value":2015
}
]
}
For above it could be better used as:
PUT /vehicle/_doc/1
{
"metadata":[
{
"make":"Saturn",
"year": 2015
}
]
}
In my search engine, users can select to search case-sensitively or not. If they choose to, the query will search on fields which use a custom case-sensitive analyser. This is my setup:
GET /candidates/_settings
{
"candidates": {
"settings": {
"index": {
"number_of_shards": "5",
"provided_name": "candidates",
"creation_date": "1528210812046",
"analysis": {
"analyzer": {
"case_sensitive": {
"filter": [
"stop",
"porter_stem"
],
"type": "custom",
"tokenizer": "standard"
}
}
},
...
}
}
}
}
So I have created a custom analyser called case_sensitive taken from this answer. I am trying to define my mapping as follows:
PUT /candidates/_mapping/candidate
{
"properties": {
"first_name": {
"type": "text",
"fields": {
"case": {
"type": "text",
"analyzer": "case_sensitive"
}
}
}
}
}
So, when querying, for a case-sensitive match, I can do:
simple_query_string: {
query: **text to search**,
fields: [
"first_name.case"
]
}
I am not even getting to the last step as I am getting the error described in the title when I am trying to define the mapping. The full stack trace is in the image below:
I initially thought that my error was similar to this one but I think that issue is only related to using the keyword tokenizer and not the standard one
In this mapping definition, I was actually trying to adjust the mapping for several different fields and not just first_name. One of these fields has the type long and that is the mapping definition that was throwing the error. When I remove that from the mapping definition, it works as expected. However, I am unsure as to why this fails for this data type?
I am using Elasticsearch 5.4.1. Here is mapping:
{
"testi": {
"mappings": {
"testt": {
"properties": {
"last": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
When I use URI search I receive results. On the other hand during using Request Body search there is empty result in any case.
GET testi/testt/_search
{
"query" : {
"term" : { "name" : "John" }
}
}
Couple things going on here:
For both last and name, you are indexing the field itself as text and then a subfield as a keyword. Is that your intention? You want to be able to do analyzed/tokenized search on the raw field and then keyword search on the subfield?
If that is your intention, you now have two ways to query each of these fields. For example, name gives you the analyzed version of the field (you designed type text meaning Elasticsearch applied a standard analyzer on it and applied lowercase filter, some basic tokenizing and stemming, etc.) and name.keyword gives you the unaltered keyword version of this field
Therefore, your terms query expects your input string John to match on the field you're querying against. Since you used capitalization in your query input, you likely want to use the keyword subfield of name so try "term" : { "name.keyword" : "John" } instead.
As a light demonstration of what is happening to the original field, "term" : { "name.keyword" : "john" } should work as well
You are seeing results in _search because it is just executing a match_all. If you did pass a basic text parameter, it is executing against _all which is a concatenation of all the fields in each document, so both the keyword and text versions are available
I am using query_string for search. Searching is working fine but its getting all records with small letters and capital letters match.But i want to exact match with case sensitive?
For example :
Search field : "title"
Current output :
title
Title
TITLE,
I want to only first(title). How to resolved this issue.
My code in java :
QueryBuilder qbString=null;
qbString=QueryBuilders.queryString("title").field("field_name");
You need to configure your mappings / text processing so tokens are indexed without being lowercased.
The "standard"-analyzer lowercases (and removes stopwords).
Here's an example that shows how to configure an analyzer and a mapping to achieve this: https://www.found.no/play/gist/7464654
With Version 5 + on ElasticSearch there is no concept of analyzed and not analyzed for index, its driven by type !
String data type is deprecated and is replaced with text and keyword, so if your data type is text it will behave like string and can be analyzed and tokenized.
But if the data type is defined as keyword then automatically its NOT analyzed, and return full exact match.
SO you should remember to mark the type as keyword when you want to do exact match with case sensitive.
code example below for creating index with this definition:
PUT testindex
{
"mappings": {
"original": {
"properties": {
"#timestamp": {
"type": "date"
},
"#version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"APPLICATION": {
"type": "text",
"fields": {
"exact": {"type": "keyword"}
}
},
"type": {
"type": "text",
"fields": {
"exact": {"type": "keyword"}
}
}
}
}
}
}