I want to make a query in elasticsearch that returns the duplicates, but the query returns error 400 and set fieldata=True.
I need to make a query in elasticsearch,
I currently have a query:
{
"query": {
"match_all": {}
},
"aggs": {
"duplicateCount": {
"terms": {
"field": "hash_code_document",
"min_doc_count": 2
}
}
}
}
but when doing the query I get this 400 error:
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [hash_code_document] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [
{
"shard" : 0,
"index" : "curriculo-19",
"node" : "QOzYVehEQhezjq1TWxYvAA",
"reason" : {
"type" : "illegal_argument_exception",
"reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [hash_code_document] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
],
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [hash_code_document] in order to load field data by uninverting the inverted index. Note that this can use significant memory.",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [hash_code_document] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
},
"status" : 400
}
Do I need to change the mapping to make the query?
Look like your field defined as "text" in the mapping. Elasticsearch doesn't allow to do the aggregations in this type of fields.
You need to change the field type in the mapping to the "keyword" like:
"mappings": {
"properties": {
"hash_code_document":{
"type": "keyword"
}
}
}
Or if you already have field like: hash_code_document.keyword, you need to use it for aggregation
Related
I have the below index configuration for Products in Elastic Search.
{
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 0
}
},
"mappings": {
"properties": {
"name": {
"fields": {
"original": {
"type": "keyword"
}
},
"type": "text",
"fielddata": true,
"analyzer": "portuguese"
},
"product_data": {
"type": "object"
}
}
}
}
HERE I UPSERT THE DATA
http://127.0.0.1:9200/product2/_update/IMOB01
BODY
{
"doc": {
"name":"Test",
"product_data": {
"symbol": "IMOB01",
"release_date": "2013-01-01T00:00:00"
}
},
"doc_as_upsert": true
}
RESPONSE
201 created
The problem is, if I try _search and sort by name everything is OK.
BUT if for instance, I try use _Search with sort by **product_data.symbol ** I receive the error below...
localhost:9200/product/_search
BODY
{
"from": 0,
"size": 50,
"query": {
"bool": {
"must": {
"match": {
"product_data.symbol": "imob01"
}
}
}
},
"sort": [
{ "product_data.symbol": {"order": "asc", "unmapped_type" : "text"}}
]
}
REPONSE
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [product_data.symbol] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "product",
"node": "PfwDOAwzTZm63IHq_rt1TA",
"reason": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [product_data.symbol] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
],
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [product_data.symbol] in order to load field data by uninverting the inverted index. Note that this can use significant memory.",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [product_data.symbol] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
}
},
"status": 400
}
What I`m doing wrong ?
OBS: The reason that I created the product_data as object is because I have some products with different types of return, they can have symbol or other fields, symbol is just a example.
I tried change the type of product_data for fielddata:true with no success.
Text fields are not optimised for operations that require per-document
field data like aggregations and sorting, so these operations are
disabled by default. Please use a keyword field instead.
Alternatively, set fielddata=true on [product_data.symbol] in order to
load field data by uninverting the inverted index. Note that this can
use significant memory.
Try user keyword type:
"sort": [
{ "product_data.symbol.keyword": {"order": "asc", "unmapped_type" : "text"}}
]
I dont know how my field mapping changed from keyword to text but now its an issue and i need to change from text to keyword .
I have huge data so re-indexing will take 2 to 3 days time .Now we are looking way to update the index mapping so that the issue will be resolved .
In our lower environment the field data is still keyword and in prod it is changed .
We are using AWS Elastic search 7.1
Please help .
This is what we want
{
"mappings":{
"properties":{
"objectID":{
"type":"keyword"
}
}
}
}
But this gives us error
"type": "resource_already_exists_exception",
This is our search query
Finally we have upgraded our ES cluster so can that be the root cause of the issue ?
Its dynamic mapping for this filed .
As mentioned in the question, it states that now the field has the dynamic mapping. This means that your current index mapping for objectId field is of text type and multifield of keyword type
{
"<index-name>" : {
"mappings" : {
"properties" : {
"objectId" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
You cannot change the mapping of objectId field from text to keyword type using Update mapping API. If you try to do so, you will get the below error
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "mapper [name] cannot be changed from type [text] to [keyword]"
}
],
"type" : "illegal_argument_exception",
"reason" : "mapper [objectId] cannot be changed from type [text] to [keyword]"
},
"status" : 400
}
So instead you can use objectId.keyword field (that is already created using dynamic mapping as stated in the question above), or you need to use the reindex API.
With the reindex API, you have to create a new index with the required index mapping, and then reindex the old data into the new index (based on the new index mapping)
I'm trying to write a query to return unique cities. My code is:
GET /files/_doc/_search
{
"size":"0",
"aggs" : {
"uniq_cities" : {
"terms" : { "field" : "cities" }
}
}
}
I have an error message as follows:
Fielddata is disabled on text fields by default. Set fielddata=true on
[cities] in order to load fielddata in memory by
uninverting the inverted index. Note that this can however use
significant memory. Alternatively use a keyword field instead.
When I run
GET /files/_doc/_mapping
I get:
"cities" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
Based on the above, it seems that I already have a keyword field. How do I fix the error message?
The error says "Fielddata is disabled on text fields by default" because you are trying to apply aggregation on a text field cities. It also has a sub field keyword whose data type is keyword. Therefore, apply the aggregation on cities.keyword field as below:
GET /files/_doc/_search
{
"size":"0",
"aggs" : {
"uniq_cities" : {
"terms" : { "field" : "cities.keyword" }
}
}
}
I have problem with getting distinct titles from ES 5.4. It was returning only single distinct words instead of the whole titles. I have created raw multifield like suggested in this question. I have adjusted solution from there to current mapping changes but it doesn't work well as I get error:
"Fielddata is disabled on text fields by default. Set fielddata=true on [title.raw] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
Field mapping:
"title":{
"type":"text",
"fields":{
"keyword":{"type":"keyword","ignore_above":256},
"raw":{"type":"text","index":false},
"sort":{"type":"text","fielddata":true}}}
Terms aggregation with query:
{
"aggs": {
"title.raw" : {
"terms" : {
"field" : "title.raw",
"size" : 30,
"min_doc_count" : 1,
"shard_min_doc_count" : 0,
"show_term_doc_count_error" : false,
"order" : [
{
"_count" : "desc"
},
{
"_term" : "asc"
}
]
}
}
As I understand I can't use fielddata true together with index false. How should look my mapping to be able get full distinct titles sorted ascending?
I have a sample data stored similar to below in Elasticsearch 5.5. I can create index, search based on match_all, gte etc. using postman.
{
"name":"Apple",
"address": {
"city":"Cupertino",
"state":"CA",
"country":"USA"
},
"rating":"4.9"
}
I need to sort all the entities based on rating, so I am using below
{
"query":{
"match_all":{}
},
"sort" : [
{
"rating" : {
"order" : "desc"
}
}
]
}
But I see below error in postman
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [rating] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
Any suggestion?
It seems you need to change the rating field to numeric in order to perform the sort on this field.
{
"name":"Apple",
"address": {
"city":"Cupertino",
"state":"CA",
"country":"USA"
},
"rating":4.9
}
otherwise, you can enable fielddata on an existing text field using the PUT mapping API as follows:
PUT my_index/_mapping/my_type
{
"properties": {
"rating": {
"type": "text",
"fielddata": true
}
}
}