I am running elastic stack v 7.2.0 on kubernetes and I am getting this error in the elasticsearh while accessing the metricbeat dashboard
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [host.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
My questions is If I have manually added the template and the dashboards, then why am I getting this error? With the template added should not the required mapping go inside the index? I saw some answers which suggested to apply the required mapping explicitly. But then In this case how should i give the index name on which this mapping is to be applied as my metricbeat index would be created daily with each new date. How this explicit mapping will persist across all the metricbeat indices created with each date?
PUT /what-should-be-the-index-name/_mapping
Related
I am upgrading an app from Elasticsearch 2.3 to 7.9. I'm using the NEST client version 7.11.1 which shows to be compatible with ES 7.9. We are using 7.9 because that is the latest version version available on AWS server we are working with.
The old application has the following field mapping:
.String(s => s
.Name(f => f.PartDescription)
.Analyzer(Analyzers.DescriptionAnalyzer)
.Fielddata(descriptor => descriptor.Loading(FielddataLoading.Eager)));
I am using the following mapping to replace this in the new version:
.Text(t => t
.Name(ep => ep.PartDescription)
.Analyzer(Analyzer.DescriptionAnalyzer)
.Fielddata(true))
I see that in the new version the only option for Fielddata is a boolean. The Eager and other options are missing.
Is Fielddata(true) a suitable equivalent for the upgrade?
The boolean on fielddata determines whether fielddata is enabled for the field. fielddata is used when performing aggregations, sorting and for scripting, and is loaded into the heap, into the fielddata cache, on demand (not eagerly loaded).
Typically for text datatype fields, you don't want fielddata; text data types undergo analysis and the resulting tokens are stored in the inverted index. When fielddata is set to true, the inverted index is uninverted on demand to produce a columnar structure that is loaded into the heap to serve aggregations, sorting and scripting on text fields. Text analysis often produces many tokens that serve the purpose of full-text search well but don't serve the purpose of aggregation, sorting and scripting well. With many tokens and many concurrent aggregations, heap memory can grow quickly, exerting GC pressure. So, the default for text datatype fields is to have fielddata be false, and to set it to true if you know what you're doing.
Instead of setting fielddata to true on a text datatype field, a good approach is to use multi-fields and also map the field as a keyword datatype if the field is one that you want to use for aggregations, sorting and scripting, and target the keyword multi field for this purpose.
I'm using Nlog to write logs to Elasticsearch, which works just fine. The problem is that aggregations don't work because the fielddata on the property I try to aggregate is set to false by default. The error I get reads as follows:
illegal_argument_exception Reason: "Fielddata is disabled on text
fields by default. Set fielddata=true on [level] in order to load
fielddata in memory by uninverting the inverted index
Since an index is created by Nlog, I would like it to map certain properties in a way that they can be later aggregated. Is it possible to configurte Nlog so that the error is gone and aggregations start working?
After reading some Elasticsearch index tuning guides like How to Maximize Elasticsearch Index Performance and elastic's Tune for indexing speed I wanted to take a look at updating the refresh_interval.
We are using AWS Elasticsearch domains (elasticsearch version 6.2). There's no mention of refresh_interval on Cloudformation's doc site AWS::Elasticsearch::Domain
So I wanted to see what the default setting was for AWS Elasticsearch.
Using the _settings API doesn't show the refresh_interval.
GET /my_index/_settings
And specifying the refresh_interval doesn't show anything either.
GET /my_index/_settings/index.refresh_interval
Only returns an empty object.
{}
How can I find the current refresh_interval for Elasticsearch?
You need to add a parameter called include_defaults in order to also retrieve the default values:
GET /my_index/_settings?include_defaults=true
In the response, you'll get a defaults section which includes the default value of the refresh_interval setting, most probably 1s.
NOTE: The reason the refresh_interval is empty is because your index has not set the value explicitly and so your index uses the default value.
When I use elasticsearch java api to sort my document, the es has error:
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [namespaceName] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.
And my code for sorting like this:
SortBuilder sortBuilder = SortBuilders.fieldSort(sortField)
.order(SortOrder.valueOf(order.toUpperCase()));
SearchRequestBuilder srb1 = client.prepareSearch()
.setQuery(qb).setIndices(indexName)
.setTypes(type).addSort(sortBuilder);
I want to know how to set fielddata=true by java api.
For production I would like to restrict ElasticSearch automatic index creation. As per the documentation I have restricted the server elasticsearch.yml
action.auto_create_index
index.mapper.dynamic: false
However I'm stable to insert new documents with fields that do not match the custom mapping?
According to documentation, I think dynamic mapping should be set to strict.
The dynamic creation of fields within a type can be completely disabled by setting the dynamic property of the type to strict.