creating dynamic index from kafka-filebeat - elasticsearch

Question

software version: ES-OSS-7.4.2， filebeat-OSS-7.4.2
following is my filebeat.yml and grok pipeline
filebeat.inputs:
- type: kafka
hosts:
- test-bigdata-kafka0003:9092
- test-bigdata-kafka0002:9092
- test-bigdata-kafka0001:9092
topics: ["bigdata-k8s-test-serverlog"]
group_id: "filebeat-kafka-test"
setup.template.settings:
index.number_of_shards: 1
_source.enabled: true
setup.template.name: "test"
setup.template.pattern: "test-*"
setup.template.overwrite: true
setup.template.enabled: true
setup.ilm.enable: true
setup.ilm.rollover_alias: "test"
setup.kibana:
host: "https://xxx:8080"
username: "superuser"
password: "123456"
ssl.verification_mode: none
output.elasticsearch:
index: "test-%{[jiserver]}-%{+yyyy.MM.dd}"
pipeline: "test-pipeline"
hosts: ["xxx:8200"]
username: "superuser"
password: "123456"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
pipeline.json
{
"description": "Test pipeline",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{CUSTOMTIME:timestamp} (?:%{NOTSPACE:jiserver}|-) (?:%{NOTSPACE:hostname}|-) (?:%{LOGLEVEL:level}|-) (?:%{NOTSPACE:thread}|-) (?:%{NOTSPACE:class}|-) (?:%{NOTSPACE:method}|-) (?:%{NOTSPACE:line}|-) (?:%{CUSTOMDATA:message}|-)"],
"pattern_definitions": {
"CUSTOMTIME": "%{YEAR}[- ]%{MONTHNUM}[- ]%{MONTHDAY}[- ]%{TIME}",
"CUSTOMDATA": "((%{GREEDYDATA})[[:space:]]?)+"
}
}
}
],
"on_failure": [
{
"set": {
"field": "error_information",
"value": "Processor {{ _ingest.on_failure_processor_type }} with tag {{ _ingest.on_failure_processor_tag }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}"
}
}
]
}
I use grok split the message to different field, one of them is jiserver . And I want my index dynamicly name with jiserver, how to do . above setting is not work, and recevie error
[elasticsearch] elasticsearch/client.go:541 Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}

Answer 1

I found a solution 。 filebeat.yml add a script processor
processors:
- script:
lang: javascript
id: my_filter
source: >
function process(event) {
var message = event.Get("message");
var name = message.split(" ")
event.Put("jiserver", name[2])
}