How to Use a Custom Ingest Pipeline with a Filebeat Module - elasticsearch

Question

How do I use a custom ingest pipeline with a Filebeat module? In my case, I'm using the apache module.
According to multiple sources, this is supposedly configurable via output.elasticsearch.pipeline / output.elasticsearch.pipelines[pipeline]. Sources follow:
https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#pipelines-option-es
https://stackoverflow.com/a/58726519/1026263
https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#pipelines-option-es
However, after many attempts at different permutations, I have never been able to influence which ingest pipeline is used by the Filebeat; it always uses the module's stock ingest pipeline.
This is just one of the many attempts:
filebeat.config:
filebeat.modules:
- module: apache
access:
enabled: true
var.paths: ["/var/log/apache2/custom_access*"]
error:
enabled: true
var.paths: ["/var/log/apache2/custom_error*"]
filebeat.config.modules:
reload.enabled: true
reload.period: 5s
output.elasticsearch:
hosts: ["${ELASTICSEARCH_URL}"]
pipeline: "apache_with_optional_x_forwarded_for"
Running filebeat with debug (-d "*") shows the following, which, I assume, demonstrates that my specification has been ignored. (I can also tell by the resulting docs in Elasticsearch that my custom pipeline was sidestepped.)
2021-12-16T23:23:47.464Z DEBUG [processors] processing/processors.go:203 Publish event: {
"#timestamp": "2021-12-16T23:23:47.464Z",
"#metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.10.2",
"pipeline": "filebeat-7.10.2-apache-access-pipeline"
},
I have tried this in both Filebeat v6.8 and v7.10 (in the docker.elastic.co/beats/filebeat docker images).
This is similar to these threads, which never had a satisfactory conclusion:
How to use custom ingest pipelines with docker autodiscover
How to specify pipeline for Filebeat Nginx module?

Answer 1

Well, according to this PR to on the beats repository, to override the module pipeline you need to specify the custom pipeline in the input configuration, not on the output.
Try this:
filebeat.modules:
- module: apache
access:
enabled: true
input.pipeline: your-custom-pipeline
var.paths: ["/var/log/apache2/custom_access*"]
error:
enabled: true
input.pipeline: your-custom-pipeline
var.paths: ["/var/log/apache2/custom_error*"]