Installing Logstash on Kubernetes and Sending logs to AWS ElasticSearch - elasticsearch

Question

I am trying to set up a monitoring solution for apps running on Kubernetes via AWS elasticsearch.
To ship logs I am using filebeat --> Logstash --> AWS ElasticSearch and it's proving to be a nightmare so far :(
To ship logs from logstash, I need to use amazon_es output plugin but I am getting different errors
Below is the manifest file for logstash that I am using
kind: ConfigMap
metadata:
name: logstash-configmap
namespace: kube-system
data:
logstash.yml: |
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
logstash.conf: |
# all input will come from filebeat, no local logs
input {
beats {
port => 5044
}
}
filter {
if [message] =~ /^\{.*\}$/ {
json {
source => "message"
}
}
if [ClientHost] {
geoip {
source => "ClientHost"
}
}
}
output {
amazon_es {
hosts => [ "https://vpc-eks***********.es.amazonaws.com:443" ]
region => "eu-west-1"
index => "devtest-logs-%{+YYYY.MM.dd}"
}
}
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: logstash-deployment
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:7.1.0
ports:
- containerPort: 5044
volumeMounts:
- name: config-volume
mountPath: /usr/share/logstash/config
- name: logstash-pipeline-volume
mountPath: /usr/share/logstash/pipeline
volumes:
- name: config-volume
configMap:
name: logstash-configmap
items:
- key: logstash.yml
path: logstash.yml
- name: logstash-pipeline-volume
configMap:
name: logstash-configmap
items:
- key: logstash.conf
path: logstash.conf
---
kind: Service
apiVersion: v1
metadata:
name: logstash-service
namespace: kube-system
spec:
selector:
app: logstash
ports:
- protocol: TCP
port: 5044
targetPort: 5044
type: ClusterIP
│ [INFO ] 2019-08-27 13:20:50.114 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"fb22e6cb-d7bb-4735-a05a-da4a2e4dabde", :path=>"/usr/share/logstash/data/uuid"} │
│ [ERROR] 2019-08-27 13:20:55.290 [Converge PipelineAction::Create<main>] registry - Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/outputs/amazon_es>, :path=>"logstash/outp │
│ uts/amazon_es", :type=>"output", :name=>"amazon_es"} │
│ [ERROR] 2019-08-27 13:20:55.295 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Could │
│ n't find any output plugin named 'amazon_es'. Are you sure this is correct? Trying to load the amazon_es output plugin resulted in this error: no such file to load -- logstash/outputs/amazon_es", :backtrace=>["/usr/share/logst │
│ ash/logstash-core/lib/logstash/plugins/registry.rb:211:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:137:in `lookup'", "org/logstash/plugins/PluginFactoryExt.java:200:in `plugin'", "or │
│ g/logstash/plugins/PluginFactoryExt.java:137:in `buildOutput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in `initialize'", "/usr/ │
│ share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}
so far i haven't been able to find anything as the container goes into crash loop. Any help would be great

Answer 1

you need to install amazon_es plugin to fix this issue.
Include this in your dockerfile.
RUN logstash-plugin install logstash-output-amazon_es