I am working on Filebeat, where I am pushing the data from our application and system logs to ES domain on AWS. It's working fine, just that the host field as a type is shown as JSON instead of plain-text. I checked the fields.yml file, but no reference to host with JSON as the output.
filebeat.yml :
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/*.log
fields:
type: develop.gateway.nginx.log
environment: develop.gateway
service: nginx
document_type: filebeat.develop.gateway
registry: /var/lib/filebeat/registry
- type: log
paths:
- /var/www/html/api-gateway/deploy/var/log/*.log
fields:
type: develop.gateway.application.log
environment: develop.gateway.application
service: gateway
document_type: filebeat.gateway.develop
registry: /var/lib/filebeat/registry
- type : log
paths:
- /var/log/php*.log
fields:
type: develop.1c.php-fpm.log
environment: develop.1c
service: php-fpm
document_type: filebeat.php-fpm.1c
registry: /var/lib/filebeat/registry
output.elasticsearch:
hosts: ["OUR_DOMAN"]
protocol: "https"
You can pick out the hostname from the payload as i do here:
output.kafka:
...
codec.format:
string: '{"version":"1.1","timestamp":"%{[#timestamp]}","short_message":"%{[message]}","file":"%{[log.file.path]}","host":"%{[host.hostname]}"}'
...
Related
There was a problem with the Swagger document path.
build.gradle
implementation("org.springdoc:springdoc-openapi-ui:1.6.8")
application.properties
server.servlet.context-path: /poo
REQ : http://localhost:port/poo/swagger-ui/index.html
-> Failed to load remote configuration.
REQ : http://localhost:port/poo//swagger-ui/index.html
-> OK
Can be Ingress, try this
Change this
spec:
rules:
- http:
paths:
- path: /my-context/my-path/swagger
pathType: ImplementationSpecific
backend:
service:
name: my-service #.ns-config.svc.cluster.local
port:
number: 8080
For this
spec:
rules:
- http:
paths:
- backend:
service:
name: my-service #.ns-config.svc.cluster.local
port:
number: 8080
path: /my-context/my-path/swagger
pathType: ImplementationSpecific
Might check this comment describing the best approach to using spring-doc behind custom context-path or proxy.
Applies to configuration: Logging processors
This setup works:
/etc/google-cloud-ops-agent/config.yaml
logging:
receivers:
app:
type: files
include_paths: [/www/logs/app-*.log]
processors:
monolog:
type: parse_regex
field: message
regex: "^\[(?<time>[^\]]+)\]\s+(?<environment>\w+)\.(?<severity>\w+):\s+(?<msg>.*?)(?<context>{.*})?\s*$"
service:
pipelines:
default_pipeline:
receivers: [app]
processors: [monolog]
I am trying to configure time_key, but the logs do not show up in the log viewer. I call the API whether the logs are being processed, whether they are read and sent. They come out but are not in the log viewer.
logging:
receivers:
app:
type: files
include_paths: [/www/logs/app-*.log]
processors:
monolog:
type: parse_regex
field: message
regex: "^\[(?<time>[^\]]+)\]\s+(?<environment>\w+)\.(?<severity>\w+):\s+(?<msg>.*?)(?<context>{.*})?\s*$"
time_key: time
time_format: "%Y-%m-%d %H:%M:%S"
service:
pipelines:
default_pipeline:
receivers: [app]
processors: [monolog]
Log structure:
[2021-10-06 12:12:08] production.EMERGENCY: Testing {"abc":"xyz"}
Parsed (first code example):
{
jsonPayload: {
context: "{"abc":"xyz"}"
environment: "production"
msg: "Testing "
severity: "EMERGENCY"
time: "2021-10-06 12:12:08"
}
}
API call to check logs processed:
curl -s localhost:2020/api/v1/metrics | jq
From strptime(3) also tried to use "%F %H:%M:%S"
What am I doing wrong?
I wish to forward logs from remote EKS clusters to a centralised EKS cluster hosting ECK.
Versions in use:
EKS v1.20.7
Elasticsearch v7.7.0
Kibana v7.7.0
Filebeat v7.10.0
The setup is using a AWS NLB to forward requests to Nginx ingress, using host based routing.
When the DNS lookup (filebeat test output) for the Elasticsearch is tested on Filebeat, it validates the request.
But the logs for Filebeat are telling a different story.
2021-10-05T10:39:00.202Z ERROR [publisher_pipeline_output]
pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)):
Get "https://elasticsearch.dev.example.com:9200": Bad Request
The Filebeat agents can connect to the remote Elasticsearch via the NLB, when using a curl request.
The config is below. NB: dev.example.com is the remote cluster hosing ECK.
app:
name: "filebeat"
configmap:
enabled: true
filebeatConfig:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- config:
- type: container
paths:
- /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- drop_event.when.not.or:
- contains.kubernetes.namespace: "apps-"
- equals.kubernetes.namespace: "cicd"
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
overwrite_keys: true
- add_fields:
fields:
kubernetes.cluster.name: dev-eks-cluster
target: ""
processors:
- add_cloud_metadata: ~
- add_host_metadata: ~
cloud:
id: '${ELASTIC_CLOUD_ID}'
cloud:
auth: '${ELASTIC_CLOUD_AUTH}'
output:
elasticsearch:
enabled: true
hosts: "elasticsearch.dev.example.com"
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl:
verification_mode: "none"
headers:
Host: "elasticsearch.dev.example.com"
proxy_url: "https://example.elb.eu-west-2.amazonaws.com"
proxy_disable: false
daemonset:
enabled: true
version: 7.10.0
image:
repository: "docker.elastic.co/beats/filebeat"
tag: "7.10.0"
pullPolicy: Always
extraenvs:
- name: ELASTICSEARCH_HOST
value: "https://elasticsearch.dev.example.com"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
value: "remote-cluster-elasticsearch-es-elastic-user-password"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
clusterrolebinding:
enabled: true
namespace: monitoring
clusterrole:
enabled: true
serviceaccount:
enabled: true
namespace: monitoring
deployment:
enabled: false
configmap:
enabled: false
Any tips or suggestions on how to enable Filebeat forwarding, would be much appreciated :-)
#1 Missing ports:
Even with the ports added in as suggested. Filebeat is erroring with:
2021-10-06T08:34:41.355Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)): Get "https://elasticsearch.dev.example.com:9200": Bad Request
...using a AWS NLB to forward requests to Nginx ingress, using host based routing
How about unset proxy_url and proxy_disable, then set hosts: ["<nlb url>:<nlb listener port>"]
The final working config:
app:
name: "filebeat"
configmap:
enabled: true
filebeatConfig:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- config:
- type: container
paths:
- /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- drop_event.when.not.or:
- contains.kubernetes.namespace: "apps-"
- equals.kubernetes.namespace: "cicd"
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
overwrite_keys: true
- add_fields:
fields:
kubernetes.cluster.name: qa-eks-cluster
target: ""
processors:
- add_cloud_metadata: ~
- add_host_metadata: ~
cloud:
id: '${ELASTIC_CLOUD_ID}'
cloud:
auth: '${ELASTIC_CLOUD_AUTH}'
output:
elasticsearch:
enabled: true
hosts: ["elasticsearch.dev.example.com:9200"]
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl:
verification_mode: "none"
daemonset:
enabled: true
version: 7.10.0
image:
repository: "docker.elastic.co/beats/filebeat"
tag: "7.10.0"
pullPolicy: Always
extraenvs:
- name: ELASTICSEARCH_HOST
value: "https://elasticsearch.dev.example.com"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
value: "remote-cluster-elasticsearch-es-elastic-user-password"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
clusterrolebinding:
enabled: true
namespace: monitoring
clusterrole:
enabled: true
serviceaccount:
enabled: true
namespace: monitoring
deployment:
enabled: false
configmap:
enabled: false
In addition the following changes were needed:
NBL:
Add listener for 9200 forwarding to the Ingress Controller for HTTPS
SG:
Opened up port 9200 on the EKS worker nodes
I have an elastic stack on kubernetes (k8s) using ECK.
Kibana version:
7.13.2
Elasticsearch version:
7.13.2
APM Server version:
7.13.2
APM Agent language and version:
https://www.npmjs.com/package/#elastic/apm-rum - 5.9.1
Browser version:
Chrome latest
Description of the problem
Frontend apm-run agent fails to send messages to apm server. if i disable cors on the browser it works - google-chrome --disable-web-security --user-data-dir=temp then navigate to my frontend http://localhost:4201/
[Elastic APM] Failed sending events! Error: https://apm.<redacted>.com/intake/v2/rum/events HTTP status: 0
at ApmServer._constructError (apm-server.js:120)
at eval (apm-server.js:48)
POST https://apm.<acme>.com/intake/v2/rum/events net::ERR_BLOCKED_BY_CLIENT
Code:
apm.yml
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-server-prod
namespace: elastic-system
spec:
version: 7.13.2
count: 1
elasticsearchRef:
name: "elasticsearch-prod"
kibanaRef:
name: "kibana-prod"
http:
service:
spec:
type: NodePort
config:
apm-server:
rum.enabled: true
ilm.enabled: true
elastic.ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: elastic-ingress
namespace: elastic-system
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/group.name: "<redacted>"
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
alb.ingress.kubernetes.io/backend-protocol: 'HTTPS'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-2:<rd>:certificate/0250a551-8971-468d-a483-cad28f890463
alb.ingress.kubernetes.io/tags: Environment=prod,Team=dev
alb.ingress.kubernetes.io/healthcheck-path: /health
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '300'
alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=<redacted>-aws-ingress-logs,access_logs.s3.prefix=dev-ingress
spec:
rules:
- host: elasticsearch.<redacted>.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: elasticsearch-prod-es-http
port:
number: 9200
- host: kibana.<redacted>.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kibana-prod-kb-http
port:
number: 5601
- host: apm.<redacted>.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: apm-server-prod-apm-http
port:
number: 8200
frontend.js
import { init as initApm } from "#elastic/apm-rum";
import config from "<redacted>-web/config/environment";
export const apm = initApm({
serviceName: "frontend",
serverUrl: "https://apm.<redacted>.com",
environment: config.environment,
logLevel: "debug",
});
Errors in browser console:
apm server:
apm-server pod does not apear to display any errors in this case, i assume the client never reaches the server .
I was running into the same problem. Check your ad blocker. I found that UBlock was blocking requests to */rum/events.
I'm guessing that they consider this as a type of user "tracker" and that's why they're blocked, really no way around it though unless you change the endpoint path I guess.
This is my filebeat config map.
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: $${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
setup.ilm.enabled: false
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
This sends logs from every pod to AWS ElasticSearch. How I can restrict it to send logs from specific pods by name and/or by the label?