As per this article in elasticsearch reference. We can update the following setting dynamically for a live cluster with the cluster-update-settings.
indices.recovery.file_chunk_size
indices.recovery.translog_ops
indices.recovery.translog_size
But when I try to update any of the above I am getting the following error:
PUT /_cluster/settings
{
"transient" : {
"indices.recovery.file_chunk_size" : "5mb"
}
}
Response:
"type": "illegal_argument_exception",
"reason": "transient setting [indices.recovery.file_chunk_size], not dynamically updateable"
Have they changed this and didn't updated there reference article or am I missing something? I am using Elasticsearch 5.0.2
They have been removed in this pull request:
indices.recovery.file_chunk_size - now fixed to 512kb
indices.recovery.translog_ops - removed without replacement
indices.recovery.translog_size - now fixed to 512kb
indices.recovery.compress - file chunks are not compressed due to lucene's compression but translog operations are.
But I'm surprised it is not reflected in the documentation.
Related
I have ElasticSearch on AWS which uses Open Distro rather than Elastics ilm.
When you apply state management for indexes it causes a crazy amount of audit indexes to be created. I would like to just disable this completely.
https://opendistro.github.io/for-elasticsearch-docs/docs/ism/settings/
Apparently it's just done setting opendistro.index_state_management.history.enabled to false but if I apply it to the _cluster/settings it doesn't appear to work.
PUT _cluster/settings
{
"opendistro.index_state_management.history.enabled": false
}
Results in:
{
"Message": "Your request: '/_cluster/settings' payload is not allowed."
}
The setting is also not valid on an index template so I cannot set it there.
How can I disable this audit history?
I asked on GitHub and got an answer:
PUT _cluster/settings
{
"persistent" : {
"opendistro.index_state_management.history.enabled": false
}
}
Need to wrap it with an action of persistent.
https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/configuration/
How to delete static settings from an index if this setting is not supported/known anymore to the running ES version.
Indices created with ES 5.2.2 or 5.3.0 have been subject to shrinking with a hot-warm strategy in order to lower the number of shards.
This shrinking created two static index settings shrink.source.name and shrink.source.uuid in the newly created index.
The new index works as expected.
In the meantime I upgraded to ES 6.8.1 and I am preparing the Elasticsearch cluster for ES 7.0 as indices created with older versions are not supported anymore with ES 7.0.
Kibana offers a nice UI for the required reindexing but this fails due to these two unsupported setting.
As I have no need for these settings anyway (they are just informational for me) I want to delete them from the indices.
Deleting a static setting from an index requires the follwing steps:
close the index
set the setting to null
reopen the index
Unfortunately this does not work with settings which are not supported anymore with the current version of ES.
curl -X PUT "elk29:9200/logstash-20160915/_settings?pretty" -H 'Content-Type: application/json' -d' { "index" : { "shrink.source.uuid" : null }}'
{
"error" : {
"root_cause" : [
{
"type" : "remote_transport_exception",
"reason" : "[elk24][10.21.15.24:9300][indices:admin/settings/update]"
}
],
"type" : "illegal_argument_exception",
"reason" : "unknown setting [index.shrink.source.uuid] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
},
"status" : 400
}
I expected that the setting is simply removed.
Oviously ES emulates the removal of a setting by setting the value of a setting to null. Unfortunately this only works with explicitly supported settings but not with outdated unsupported settings.
The question remains how to remove index settings which are not supported anymore with the current version of ES?
We're ingesting data to Elasticsearch through filebeat and hit a configuration problem.
I'm trying to specify a date format for a particular field (standard #timestamp field holds indexing time and we need an actual event time). So far, I was unable to do so - I tried fields.yml, separate json template file, specifying it inline in filebeat.yml. That last option is just a guess, I haven't found any example of this particular configuration combo.
What am I missing here? I was sure this should work:
filebeat.yml
#rest of the file
template:
# Template name. By default the template name is filebeat.
#name: "filebeat"
# Path to template file
path: "custom-template.json"
and in custom-template.json
{
"mappings": {
"doc": {
"properties": {
"eventTime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss.SSSS"
}
}
}
}
}
but it didn't.
We're using Filebeat version is 6.2.4 and Elasticsearch 6.x
I couldn't get the Filebeat configuration to work. So in the end changed the time field format in our service and it worked instantly.
I found official Filebeat documentation to be lacking complete examples. May be that's just my problem
EDIT actually, it turns out you can specify a list of allowed formats in your mapping
I currently have Elasticsearch version 6.2.2 and Apache Nifi version 1.5.0 running on the same machine. I'm trying to follow the Nifi example located: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html except instead of storing to Hive, I want to store to Elasticsearch.
Initially I tried using the PutElasticsearch5 processor but I was getting the following error on Elasticsearch:
Received message from unsupported version: [5.0.0] minimal compatible version is: [5.6.0]
When I tried Googling this error message, it seemed like the consensus was to use the PutElasticsearchHttp processor. My Nifi looks like:
And the configuration for the PutElasticsearchHttp processor:
When the flowfile gets to the PutElasticsearchHttp processor, the following error shows up:
PutElasticSearchHttp failed to insert StandardFlowFileRecord into Elasticsearch due to , transferring to failure.
It seems like the reason is blank/null. There also wasn't anything in the Elasticsearch log.
After the ConvertAvroToJson, the data is a JSON array with all of the entries on a single line. Here's a sample value:
{"City": "Athens",
"Edition": 1896,
"Sport": "Aquatics",
"sub_sport": "Swimming",
"Athlete": "HAJOS, Alfred",
"country": "HUN",
"Gender": "Men",
"Event": "100m freestyle",
"Event_gender": "M",
"Medal": "Gold"}
Any ideas on how to debug/solve this problem? Do I need to create anything in Elasticsearch first? Is my configuration correct?
I was able to figure it out. After the ConvertAvroToJSON, the flow file was a single line that contained a JSON Array of JSON indices. Since I wanted to store the individual indices I needed a SplitJSON processor. Now my Nifi looks like this:
The configuration of the SplitJson looks like this:
The index name cannot contain the / character. Try with a valid index name: e.g. sports.
I had a similar flow, wherein changing the type to _doc did the trick after including splitTojSON.
I am trying to map openNLP to enable parsing of filed in a document. Using the following code:
"article":
"properties":
"content" : { "type" : "opennlp" }
Prior to create the mapping, I downloaded the named entity extraction binary file from sourceforge.net and installed/unpacked using cURL in elasticsearch plugin folders.
I get the following error message when I tried to run the above mapping code.
"error": "MapperParsingException[No handler for type [opennlp]
declared on field [content]]" "status": 400
After quick Googling I've found this: https://github.com/spinscale/elasticsearch-opennlp-plugin
I assume that you're trying to install it. However - it's outdated and probably not even supported by recent Elasticsearch versions.
The purpose of it seems to extract data from files and index them as tags. Elasticsearch Mapper Attachments Type plugin does exactly that. I would encourage you to use it instead of OnenNLP. Quick extract from documentation:
The mapper attachments plugin adds the attachment type to
Elasticsearch using Apache Tika. The attachment type allows to index
different "attachment" type field (encoded as base64), for example,
microsoft office formats, open document formats, ePub, HTML, and so on
(full list can be found here).
An example how to use map fields using it:
PUT /test/person/_mapping
{
"person" : {
"properties" : {
"my_attachment" : {
"type" : "attachment"
}
}
}
}