We're ingesting data to Elasticsearch through filebeat and hit a configuration problem.
I'm trying to specify a date format for a particular field (standard #timestamp field holds indexing time and we need an actual event time). So far, I was unable to do so - I tried fields.yml, separate json template file, specifying it inline in filebeat.yml. That last option is just a guess, I haven't found any example of this particular configuration combo.
What am I missing here? I was sure this should work:
filebeat.yml
#rest of the file
template:
# Template name. By default the template name is filebeat.
#name: "filebeat"
# Path to template file
path: "custom-template.json"
and in custom-template.json
{
"mappings": {
"doc": {
"properties": {
"eventTime": {
"type": "date",
"format": "YYYY-MM-dd HH:mm:ss.SSSS"
}
}
}
}
}
but it didn't.
We're using Filebeat version is 6.2.4 and Elasticsearch 6.x
I couldn't get the Filebeat configuration to work. So in the end changed the time field format in our service and it worked instantly.
I found official Filebeat documentation to be lacking complete examples. May be that's just my problem
EDIT actually, it turns out you can specify a list of allowed formats in your mapping
Related
I have crated a very basic grok filter to parse Cisco Syslogs:
input {
udp {
port => 5140
type => syslog
}
}
filter {
grok {
match => { "message"=> "%{TIMESTAMP_ISO8601:Timestamp_Local} %{IPV4:IP_Address} %{GREEDYDATA:Event}" }
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
index => "ciscologs-%{+YYYY.MM.dd}"
}
}
After reloading Logstash and verifying that logs show no major issues I reloaded Kibana and refreshed indexes.
When accessing the Discovery section, I saw that the index was indeed created. But looking at the fields, they were the default ones and not the ones defined in the grok filter.
The logs received after adding the filter show the following tag in Kibana:
Before adding the filter I made sure it works using Kibana's Grok debugger.
The tag states that there was a problem with the logs parsing but at this point.
Running versions are: 7.7.1 (Kibana/Elasticsearch) and 7.13.3 (Logstash)
I'm not sure where the issue might be, so, any help would be appreciated.
I found the problem. I was trying to match the logs in the order sent by the Cisco devices and not the logs in the "message" field. Once I modified that, the filter started working as expected.
I have ElasticSearch on AWS which uses Open Distro rather than Elastics ilm.
When you apply state management for indexes it causes a crazy amount of audit indexes to be created. I would like to just disable this completely.
https://opendistro.github.io/for-elasticsearch-docs/docs/ism/settings/
Apparently it's just done setting opendistro.index_state_management.history.enabled to false but if I apply it to the _cluster/settings it doesn't appear to work.
PUT _cluster/settings
{
"opendistro.index_state_management.history.enabled": false
}
Results in:
{
"Message": "Your request: '/_cluster/settings' payload is not allowed."
}
The setting is also not valid on an index template so I cannot set it there.
How can I disable this audit history?
I asked on GitHub and got an answer:
PUT _cluster/settings
{
"persistent" : {
"opendistro.index_state_management.history.enabled": false
}
}
Need to wrap it with an action of persistent.
https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/configuration/
As per this article in elasticsearch reference. We can update the following setting dynamically for a live cluster with the cluster-update-settings.
indices.recovery.file_chunk_size
indices.recovery.translog_ops
indices.recovery.translog_size
But when I try to update any of the above I am getting the following error:
PUT /_cluster/settings
{
"transient" : {
"indices.recovery.file_chunk_size" : "5mb"
}
}
Response:
"type": "illegal_argument_exception",
"reason": "transient setting [indices.recovery.file_chunk_size], not dynamically updateable"
Have they changed this and didn't updated there reference article or am I missing something? I am using Elasticsearch 5.0.2
They have been removed in this pull request:
indices.recovery.file_chunk_size - now fixed to 512kb
indices.recovery.translog_ops - removed without replacement
indices.recovery.translog_size - now fixed to 512kb
indices.recovery.compress - file chunks are not compressed due to lucene's compression but translog operations are.
But I'm surprised it is not reflected in the documentation.
I am trying to map openNLP to enable parsing of filed in a document. Using the following code:
"article":
"properties":
"content" : { "type" : "opennlp" }
Prior to create the mapping, I downloaded the named entity extraction binary file from sourceforge.net and installed/unpacked using cURL in elasticsearch plugin folders.
I get the following error message when I tried to run the above mapping code.
"error": "MapperParsingException[No handler for type [opennlp]
declared on field [content]]" "status": 400
After quick Googling I've found this: https://github.com/spinscale/elasticsearch-opennlp-plugin
I assume that you're trying to install it. However - it's outdated and probably not even supported by recent Elasticsearch versions.
The purpose of it seems to extract data from files and index them as tags. Elasticsearch Mapper Attachments Type plugin does exactly that. I would encourage you to use it instead of OnenNLP. Quick extract from documentation:
The mapper attachments plugin adds the attachment type to
Elasticsearch using Apache Tika. The attachment type allows to index
different "attachment" type field (encoded as base64), for example,
microsoft office formats, open document formats, ePub, HTML, and so on
(full list can be found here).
An example how to use map fields using it:
PUT /test/person/_mapping
{
"person" : {
"properties" : {
"my_attachment" : {
"type" : "attachment"
}
}
}
}
i have a problem when using the Stemmer Override Token Filter with a file with all the rules. It doesn´t work! When using the rules inline it works properly. Has anyone an idea why is that so?
not working example:
"protwords": { "type": "stemmer_override", "rules_path" : "analysis/protwords.txt" }
working example:
"protwords": { "type": "stemmer_override", "rules" : [ "jacke=>jacke", "jacken=>jacke", ] }
ES Version 1.7.1
thank you
Please make sure analysis/protwords.txt is inside your elasticsearch config folder and restart elasticsearch.
This is the only thing I can think of.
I hope this helps!