I'm using Nlog to write logs to Elasticsearch, which works just fine. The problem is that aggregations don't work because the fielddata on the property I try to aggregate is set to false by default. The error I get reads as follows:
illegal_argument_exception Reason: "Fielddata is disabled on text
fields by default. Set fielddata=true on [level] in order to load
fielddata in memory by uninverting the inverted index
Since an index is created by Nlog, I would like it to map certain properties in a way that they can be later aggregated. Is it possible to configurte Nlog so that the error is gone and aggregations start working?
Related
I am upgrading an app from Elasticsearch 2.3 to 7.9. I'm using the NEST client version 7.11.1 which shows to be compatible with ES 7.9. We are using 7.9 because that is the latest version version available on AWS server we are working with.
The old application has the following field mapping:
.String(s => s
.Name(f => f.PartDescription)
.Analyzer(Analyzers.DescriptionAnalyzer)
.Fielddata(descriptor => descriptor.Loading(FielddataLoading.Eager)));
I am using the following mapping to replace this in the new version:
.Text(t => t
.Name(ep => ep.PartDescription)
.Analyzer(Analyzer.DescriptionAnalyzer)
.Fielddata(true))
I see that in the new version the only option for Fielddata is a boolean. The Eager and other options are missing.
Is Fielddata(true) a suitable equivalent for the upgrade?
The boolean on fielddata determines whether fielddata is enabled for the field. fielddata is used when performing aggregations, sorting and for scripting, and is loaded into the heap, into the fielddata cache, on demand (not eagerly loaded).
Typically for text datatype fields, you don't want fielddata; text data types undergo analysis and the resulting tokens are stored in the inverted index. When fielddata is set to true, the inverted index is uninverted on demand to produce a columnar structure that is loaded into the heap to serve aggregations, sorting and scripting on text fields. Text analysis often produces many tokens that serve the purpose of full-text search well but don't serve the purpose of aggregation, sorting and scripting well. With many tokens and many concurrent aggregations, heap memory can grow quickly, exerting GC pressure. So, the default for text datatype fields is to have fielddata be false, and to set it to true if you know what you're doing.
Instead of setting fielddata to true on a text datatype field, a good approach is to use multi-fields and also map the field as a keyword datatype if the field is one that you want to use for aggregations, sorting and scripting, and target the keyword multi field for this purpose.
Elasticsearch which is running in docker is getting stopped when i run my service in docker.
if i run my service(spring boot app) first then if i run elasticsearch in docker , elasticsearch will never start in docker container? any idea about this?
is below is the problem?
{
"type":"server",
"timestamp":"2019-12-11T10:25:39,589Z",
"level":"DEBUG",
"component":"o.e.a.s.TransportSearchAction",
"cluster.name":"docker-cluster",
"node.name":"179c5890f49c",
"message":"All shards failed for phase: [query]",
"cluster.uuid":"myyvQb8oS9qFjcCL4B41Sg",
"node.id":"RVuYJJ_NQ3uygvgq3cM-dg",
"stacktrace":[
"org.elasticsearch.ElasticsearchException$1: Fielddata is disabled on text fields by default. Set fielddata=true on [type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
]
}
I am running elastic stack v 7.2.0 on kubernetes and I am getting this error in the elasticsearh while accessing the metricbeat dashboard
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [host.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
My questions is If I have manually added the template and the dashboards, then why am I getting this error? With the template added should not the required mapping go inside the index? I saw some answers which suggested to apply the required mapping explicitly. But then In this case how should i give the index name on which this mapping is to be applied as my metricbeat index would be created daily with each new date. How this explicit mapping will persist across all the metricbeat indices created with each date?
PUT /what-should-be-the-index-name/_mapping
When I use elasticsearch java api to sort my document, the es has error:
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [namespaceName] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.
And my code for sorting like this:
SortBuilder sortBuilder = SortBuilders.fieldSort(sortField)
.order(SortOrder.valueOf(order.toUpperCase()));
SearchRequestBuilder srb1 = client.prepareSearch()
.setQuery(qb).setIndices(indexName)
.setTypes(type).addSort(sortBuilder);
I want to know how to set fielddata=true by java api.
Using metricbeat 5, ES5, on Kibana5 I am getting this error in red:
Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [beat.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.
What does it mean? Where do I config it?