adding a script-based field to an elasticSearch index mapping - elasticsearch

I am following the following docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime-indexed.html
I have a field which I would like to not be scripted on runtime but rather on index-time, and according to above I can do that simply by putting the field and its script inside the mapping object as normal.
Here is a simplified version of the index I'm trying to create
{
"settings": {
"analysis": {
"analyzer": {
"case_insensitive_analyzer": {
"type": "custom",
"filter": ["lowercase"],
"tokenizer": "keyword"
}
}
}
},
"mappings": {
"properties": {
"id": {
"type": "text"
},
"events": {
"properties": {
"fields": {
"type": "text"
},
"id": {
"type": "text"
},
"event": {
"type": "text"
},
"time": {
"type": "date"
},
"user": {
"type": "text"
},
"state": {
"type": "integer"
}
}
},
"eventLast": {
"type": "date",
"on_script_error": "fail",
"script": {
"source": "def events = doc['events']; emit(events[events.length-1].time.value"
}
}
}
}
}
I'm getting this 400 error back:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "unknown parameter [script] on mapper [eventLast] of type [date]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [_doc]: unknown parameter [script] on mapper [eventLast] of type [date]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "unknown parameter [script] on mapper [eventLast] of type [date]"
}
},
"status": 400
}
Essentially I'm trying to create a scripted indexed field that is calculated off the last event time in the events array of the document.
Thanks

Tldr;
As the error states, you can not define your script in here.
There is a specific way to create runtime fields in elasticsearch.
You need to put the definition at the root of the json in the runtime object.
Solution
{
"settings": {
"analysis": {
"analyzer": {
"case_insensitive_analyzer": {
"type": "custom",
"filter": ["lowercase"],
"tokenizer": "keyword"
}
}
}
},
"runtime": {
"eventLast": {
"type": "date",
"on_script_error": "fail",
"script": {
"source": "def events = doc['events']; emit(events[events.length-1].time.value"
}
}
},
"mappings": {
"properties": {
"id": {
"type": "text"
},
"events": {
"properties": {
"fields": {
"type": "text"
},
"id": {
"type": "text"
},
"event": {
"type": "text"
},
"time": {
"type": "date"
},
"user": {
"type": "text"
},
"state": {
"type": "integer"
}
}
}
}
}
}

Related

Root mapping definition has unsupported parameters issue

I am posting to my nearly created Elasticsearch service, but unable to post the mapping. I can not identify which part is unsupported. Any suggestions?
{
"mappings": {
"employees": {
"properties": {
"FirstName": {
"type": "text"
},
"LastName": {
"type": "text"
},
"Designation": {
"type": "text"
},
"Salary": {
"type": "integer"
},
"DateOfJoining": {
"type": "date",
"format": "yyyy-MM-dd"
},
"Address": {
"type": "text"
},
"Gender": {
"type": "text"
},
"Age": {
"type": "integer"
},
"MaritalStatus": {
"type": "text"
},
"Interests": {
"type": "text"
}
}
}
}
}
I consistently get this error returned.
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [employees : {properties={Designation={type=text}, Salary={type=integer}, MaritalStatus={type=text}, DateOfJoining={format=yyyy-MM-dd, type=date}, Address={type=text}, FirstName={type=text}, LastName={type=text}, Gender={type=text}, Age={type=integer}, Interests={type=text}}}]"
}
Using Version 7.6.2
As you didn't specify your Elasticsearch version, which I also asked in my comment,
I have multiple versions of Elasticsearch and as you are using text, I started with Elasticsearch 7 as, you also specified employees in your mapping, which created suspect as this is normally defined in older versions, where types are defined.
Please see removal of types official doc for more info and not it would be completely removed in coming major version of 8.X
I was able to reproduce the issue with Elasticsearch 7.6 and got the same error:
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [employees : {properties={Designation={type=text}, Salary={type=integer}, MaritalStatus={type=text}, DateOfJoining={format=yyyy-MM-dd, type=date}, Address={type=text}, FirstName={type=text}, LastName={type=text}, Gender={type=text}, Age={type=integer}, Interests={type=text}}}]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [employees : {properties={Designation={type=text}, Salary={type=integer}, MaritalStatus={type=text}, DateOfJoining={format=yyyy-MM-dd, type=date}, Address={type=text}, FirstName={type=text}, LastName={type=text}, Gender={type=text}, Age={type=integer}, Interests={type=text}}}]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [employees : {properties={Designation={type=text}, Salary={type=integer}, MaritalStatus={type=text}, DateOfJoining={format=yyyy-MM-dd, type=date}, Address={type=text}, FirstName={type=text}, LastName={type=text}, Gender={type=text}, Age={type=integer}, Interests={type=text}}}]"
}
},
"status": 400
}
Solution
Simply remove the employee from your mapping and use below mapping.
{
"mappings": {
"properties": { --> note `employees` removed.
"FirstName": {
"type": "text"
},
"LastName": {
"type": "text"
},
"Designation": {
"type": "text"
},
"Salary": {
"type": "integer"
},
"DateOfJoining": {
"type": "date",
"format": "yyyy-MM-dd"
},
"Address": {
"type": "text"
},
"Gender": {
"type": "text"
},
"Age": {
"type": "integer"
},
"MaritalStatus": {
"type": "text"
},
"Interests": {
"type": "text"
}
}
}
}
Above create index successfully
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "mustnot"
}
You seem to be using version 7.x of Elasticsearch, which doesn't support mapping types anymore. You simply need to remove employees, like this:
{
"mappings": {
"properties": {
"FirstName": {
"type": "text"
},
"LastName": {
"type": "text"
},
"Designation": {
"type": "text"
},
"Salary": {
"type": "integer"
},
"DateOfJoining": {
"type": "date",
"format": "yyyy-MM-dd"
},
"Address": {
"type": "text"
},
"Gender": {
"type": "text"
},
"Age": {
"type": "integer"
},
"MaritalStatus": {
"type": "text"
},
"Interests": {
"type": "text"
}
}
}
}

How to debug elastic search error number_format_exception with reason "empty String" but no field name

I just deployed a small application that loads a few thousand docs into an index and when working with production data i get an error in my search request.
http code is 400 and the error is
{
"error": {
"root_cause": [
{
"type": "number_format_exception",
"reason": "empty String"
}
],
"type": "number_format_exception",
"reason": "empty String"
},
"status": 400
}
Okay i kind of get it that my mapping defines some numeric field which i oviously dont store correctly, but how am i supposed to find that field?
each doc contains hundereds of fields.... i mean, really?
I tried looking in /var/log/elasticsearch but nothing useful there...
Please help me i need to get the thing going
I defined my fields as integer which should hold arrays and might be empty. Could that be a problem?
My ES Version is 6.6.0
Update:
The error does occur while searching, during index all is fine
My mapping for that index:
{
"development-object-1551202425": {
"mappings": {
"_doc": {
"dynamic": "false",
"properties": {
"accommodation": {
"properties": {
"badges": {
"properties": {
"maskedProp1": {
"type": "boolean"
},
"maskedProp2": {
"type": "boolean"
},
"maskedProp3": {
"type": "boolean"
},
"maskedProp4": {
"type": "boolean"
},
"maskedProp5": {
"type": "boolean"
},
"maskedProp6": {
"type": "boolean"
}
}
},
"businessTypes": {
"type": "integer"
},
"classification": {
"properties": {
"classification": {
"type": "keyword"
},
"classificationValue": {
"type": "short"
}
}
},
"endowments": {
"type": "integer"
},
"hasPrice": {
"type": "boolean"
},
"lowestPrice": {
"type": "float"
},
"metascore": {
"type": "short"
},
"rating": {
"type": "short"
},
"regionscore": {
"type": "short"
}
}
},
"certificates": {
"type": "integer"
},
"geoLocation": {
"type": "geo_point"
},
"id": {
"type": "text"
},
"isAccommodation": {
"type": "boolean"
},
"location": {
"properties": {
"maskedProp1": {
"type": "integer"
},
"maskedProp2": {
"type": "integer"
},
"id": {
"type": "integer"
},
"name": {
"type": "text",
"fielddata": true
},
"zipcodes": {
"type": "integer"
}
}
},
"maskedProp1": {
"type": "integer"
},
"maskedProp2": {
"type": "integer"
},
"description": {
"type": "text"
},
"sortTitle": {
"type": "keyword"
},
"title": {
"type": "text"
}
}
}
}
}
}
The name consists of an environment string (development) and a timestamp appended (i work with automatic index switching and query for the alias, that does is called {env}-{name}-current.
In my case the error was an empty "size" parameter in the query, i tried to find the error in my filters and did not see that...
A more verbose error message (at least at what property or setting the error occured) could save thousands of hours of debugging all around the world i guess xD.
For now you would have to take apart your dsl section by section to find the issue.

Add mapping on elastic search index field

I would like to add anaylyser uax_url_email for search email field in elasticsearch document. However, I get the error.
Here is the code I am using to create this index
{
"user": {
"aliases": {},
"mappings": {
"user": {
"properties": {
"created": {
"type": "date"
},
"email": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
},
"settings": {
"index": {
"number_of_shards": "5",
"provided_name": "user",
"creation_date": "1521016015646",
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "uax_url_email",
"max_token_length": "255"
}
}
},
"number_of_replicas": "1",
"uuid": "RS96V9gFQbG5UmoQ2R_gLA",
"version": {
"created": "6010099"
}
}
}
}
}
PUT /user/_mapping/email
{
"mappings": {
"_doc": {
"properties": {
"email": {
"type": "text",
"fields": {
"my_analyzer": {
"email": "text",
"analyzer": "my_analyzer"
}
}
}
}
}
}
}
I got an error stating "root_cause":
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [mappings : {_doc={properties={email={type=text, fields={my_analyzer={email=text, analyzer=my_analyzer}}}}}}]"
}
],
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [mappings : {_doc={properties={email={type=text, fields={my_analyzer={email=text, analyzer=my_analyzer}}}}}}]"
},
"status": 400
}
Nothing will be found. I want my analyzer and tokenizer work on email field any help will be highly appreciated
This should work:
PUT /user/_mapping/user
{
"properties": {
"email": {
"type": "text",
"fields": {
"my_analyzer": {
"type": "text",
"analyzer": "my_analyzer"
}
}
}
}
}
Your mistake was that you thought that the index type was _doc, but looking at your mapping, the index type is user. Basically, you have a user index with a user type.
The format of the command is PUT /[INDEX_NAME]/_mapping/[TYPE_NAME] {"properties:" { "[FIELD_TO_BE_UPDATED]": {....} } }.

Elasticsearch for multiple language support

I am using elasticsearch 5.1.1.
I have a requirement where in I want to index data in multiple languages.
I used following mapping:
PUT http://localhost:9200/movies
{
"mappings": {
"title": {
"properties": {
"title": {
"type": "string",
"fields": {
"de": {
"type": "string",
"analyzer": "german"
},
"en": {
"type": "string",
"analyzer": "english"
},
"fr": {
"type": "string",
"analyzer": "french"
},
"es": {
"type": "string",
"analyzer": "spanish"
}
}
}
}
}
}
}
when I try to insert some data as :
POST http://localhost:9200/movies/movie/1
{
"title.en" :"abc123"
}
I am getting following error:
{
"error": {
"root_cause": [
{
"type": "remote_transport_exception",
"reason": "[IQ7CUTp][127.0.0.1:9300][indices:data/write/index[p]]"
}
],
"type": "illegal_argument_exception",
"reason": "[title] is defined as an object in mapping [movie] but this name is already used for a field in other types"
},
"status": 400
}
Can someone point me what is wrong here?
The problem is that the title field is declared as a string and you're trying to access the title.en sub-field like you would do if title was and object field. You need to change your mapping like this instead and then it will work:
{
"mappings": {
"title": {
"properties": {
"title": {
"type": "object", <--- change this
"properties": { <--- and this
"de": {
"type": "string",
"analyzer": "german"
},
"en": {
"type": "string",
"analyzer": "english"
},
"fr": {
"type": "string",
"analyzer": "french"
},
"es": {
"type": "string",
"analyzer": "spanish"
}
}
}
}
}
}
}
As I can see, you have defined the title both as a type and as a property.
The error seems to state this issue.
From your post call, I see that the type is the movie.
Do you really want the title as a type?
You should define the mapping for the title inside the movie type.

Elastic bulk error: failed to parse

I tried posting a _bulk post into elastic search, but it throws:
{
"took": 1,
"errors": true,
"items": [
{
"index": {
"_index": "quick",
"_type": "parts",
"_id": "ACI250-2016",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse [part]",
"caused_by": {
"type": "number_format_exception",
"reason": "For input string: \"250-2016\""
}
}
}
}
]
}
And here's what I'm trying to post:
POST _bulk
{"index":{"_index":"quick","_type":"parts","_id":"ACI250-2016"}}
{"eMfg":"ACI","part":"250-2016"}
And the map is:
{
"quick": {
"mappings": {
"parts": {
"properties": {
"app": {
"type": "string"
},
"eMfg": {
"type": "string"
},
"fPart": {
"type": "long"
},
"oPart": {
"type": "string"
},
"ofPart": {
"type": "string"
},
"part": {
"type": "long"
},
"price": {
"type": "double"
},
"title": {
"type": "string"
}
}
}
}
}
}
According to your mapping, part has type long and you're trying to send "250-2016". The reason might be that you've sent a document at some point with a part that was coercible to a number, e.g. "250" and now you're trying to send a string and it fails.
The best way is to use the mapping above to define a new index with the correct mapping type (see below) and then you can try your bulk import again.
DELETE /quick
PUT /quick
{
"mappings": {
"parts": {
"properties": {
"app": {
"type": "string"
},
"eMfg": {
"type": "string"
},
"fPart": {
"type": "long"
},
"oPart": {
"type": "string"
},
"ofPart": {
"type": "string"
},
"part": {
"type": "string" <-- change this
},
"price": {
"type": "double"
},
"title": {
"type": "string"
}
}
}
}
}

Resources