I'm using filebeat>kafka>logstash>elastiserach stack of version 7.15.0.
My logs are getting broken and not writing properly into elastic which is resulting in json.erros
error.message:Key 'log' not found
error.type: json
and
error.message:Error decoding JSON: invalid character 's' after array element error.type:json
my log contains both json and nonjson contents
my docker stdout logs>>
{"log":"{\"instant\":{\"epochSecond\":1643023707,\"nanoOfSecond\":538281000},\"thread\":\"grpc-default-executor-11\",\"level\":\"INFO\",\"loggerName\":\"com.abc.ab.ab.core.service.integration.controller.NotifyCoreGrpcController\",\"message\":\"RMQ_CORE_GRPC_NOTIFY RESP : {\\\"baseBizResponse\\\":{\\\"success\\\":true,\\\"resultCode\\\":\\\"SUCCESS\\\"}} \",\"endOfBatch\":false,\"loggerFqcn\":\"org.apache.logging.slf4j.Log4jLogger\",\"contextMap\":{\"RMQ_ID\":\"2022012401445669241212121212\",\"FLOW_TYPE\":\"RMQ_CORE_GRPC_NOTIFY\",\"MERCHANT_TRANS_ID\":\"bcd4ab1e54abaha122\",\"spanId\":\"4fa1474c078afceb\",\"traceId\":\"bcd4ab1e54abaha122\"},\"threadId\":100,\"threadPriority\":5,\"dateTime\":\"2022-01-24T16:58:27.538+0530\"}\r\n","stream":"stdout","time":"2022-01-24T11:28:27.538354156Z"}
and
[244540.169s][debug][gc,age] GC(51) Desired survivor size 80740352 bytes, new threshold 15 (max threshold 15)
Filebeat conf>>
filebeat.yml: |
filebeat.inputs:
- type: container
multiline.pattern: ^[[:space:]]'
multiline.negate: false
multiline.match: after
json.keys_under_root: true
json.message_key: log
json.add_error_key: true
enabled: true
paths:
- /var/log/containers/*.log
exclude_files: ['fluentd-*', 'istio-*', 'cluster-logging-*', 'infra-*']
processors:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
default_matchers.enabled: false
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- drop_fields:
fields:
- 'kubernetes.node.name'
output.kafka:
enabled: true
hosts: ["kafka1:9092","kafka2:9092","kafka3:9092"]
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 10000000
topics:
- topic: '%{[kubernetes.labels.app]}'
default: 'app-perf-k8s-logs'
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
filebeat.modules:
- module: nginx
- module: kafka
logging.level: debug
logstash config >>
input {
kafka {
topics_pattern => ".*."
bootstrap_servers => "kafka1:9092","kafka2:9092","kafka3:9092"
client_id => "logstash"
codec => json
decorate_events => true
consumer_threads => 10
heartbeat_interval_ms => "100000"
session_timeout_ms => "300000"
poll_timeout_ms => 300000
partition_assignment_strategy => "org.apache.kafka.clients.consumer.RoundRobinAssignor"
request_timeout_ms => "400000"
group_id => "logConsumer"
auto_offset_reset => "latest"
}
}
output {
elasticsearch {
hosts => "es-logging-perf-lb.abc.com:80"
index => "filebeat-%{[#metadata][kafka][topic]}-%{+YYYY.MM.dd}"
}
Please help and suggest.
Related
I have a nginx-ingress-controller logs in the Elasticsearch that is not able to catch/parse HTTP error 503 (Service Unavailable) and come up with it in the "Status" field, but HTTP 200 and others work.
Elasticsearch discover - currently
I have these the annotations already configured in the nginx-ingress-controller deployment:
annotations:
co.elastic.logs/processors.0.dissect.ignore_failure: "true"
co.elastic.logs/processors.0.dissect.target_prefix: dissect
co.elastic.logs/processors.0.dissect.tokenizer: '%{levelandtimestamp} %{pid} %{class} %{message}'
co.elastic.logs/processors.0.dissect.when.regexp.message: ^[IWEF][0-9]{4}.*
co.elastic.logs/processors.1.copy_fields.fields.0.from: dissect.levelandtimestamp
co.elastic.logs/processors.1.copy_fields.fields.0.to: dissect.timestamp
co.elastic.logs/processors.1.copy_fields.ignore_missing: "true"
co.elastic.logs/processors.2.dissect.field: dissect.timestamp
co.elastic.logs/processors.2.dissect.ignore_failure: "true"
co.elastic.logs/processors.2.dissect.target_prefix: dissect
co.elastic.logs/processors.2.dissect.tokenizer: '%{time_normilized}'
co.elastic.logs/processors.2.dissect.trim_chars: IWEF
co.elastic.logs/processors.2.dissect.trim_values: left
co.elastic.logs/processors.3.timestamp.field: dissect.time_normilized
co.elastic.logs/processors.3.timestamp.ignore_missing: "true"
co.elastic.logs/processors.3.timestamp.layouts: 0102 15:04:05.999
co.elastic.logs/processors.3.timestamp.when.has_fields: dissect.time_normilized
co.elastic.logs/processors.4.drop_fields.fields: message
co.elastic.logs/processors.4.drop_fields.ignore_missing: "true"
co.elastic.logs/processors.4.drop_fields.when.regexp.message: ^[IWEF][0-9]{4}.*
co.elastic.logs/processors.5.rename.fields.0.from: dissect.message
co.elastic.logs/processors.5.rename.fields.0.to: message
co.elastic.logs/processors.5.rename.fields.1.from: dissect.class
co.elastic.logs/processors.5.rename.fields.1.to: class
co.elastic.logs/processors.5.rename.ignore_missing: "true"
co.elastic.logs/processors.5.rename.when.has_fields: dissect.message
co.elastic.logs/processors.6.add_fields.fields.level: INFO
co.elastic.logs/processors.6.add_fields.target: ""
co.elastic.logs/processors.7.add_fields.fields.level: ERROR
co.elastic.logs/processors.7.add_fields.target: ""
co.elastic.logs/processors.7.add_fields.when.contains.dissect.levelandtimestamp: E
co.elastic.logs/processors.8.add_fields.fields.level: WARN
co.elastic.logs/processors.8.add_fields.target: ""
co.elastic.logs/processors.8.add_fields.when.contains.dissect.levelandtimestamp: W
co.elastic.logs/processors.9.add_fields.fields.level: FATAL
co.elastic.logs/processors.9.add_fields.target: ""
co.elastic.logs/processors.9.add_fields.when.contains.dissect.levelandtimestamp: F
co.elastic.logs/processors.10.decode_json_fields.fields: message
co.elastic.logs/processors.10.decode_json_fields.max_depth: "1"
co.elastic.logs/processors.10.decode_json_fields.overwrite_keys: "true"
co.elastic.logs/processors.10.decode_json_fields.target: ""
co.elastic.logs/processors.11.timestamp.field: time
co.elastic.logs/processors.11.timestamp.layouts: "2006-01-02T15:04:05+00:00"
co.elastic.logs/processors.11.timestamp.when.has_fields: time
co.elastic.logs/processors.12.dissect.field: request_query
co.elastic.logs/processors.12.dissect.ignore_failure: "true"
co.elastic.logs/processors.12.dissect.target_prefix: ""
co.elastic.logs/processors.12.dissect.tokenizer: '%{request_method} %{request_uri} %{request_protocol}'
co.elastic.logs/processors.12.dissect.when.regexp.message: request_query
co.elastic.logs/processors.13.drop_fields.fields: message
co.elastic.logs/processors.13.drop_fields.ignore_missing: "true"
co.elastic.logs/processors.13.drop_fields.when.has_fields: request_query
Filebeat.yaml - ConfigMap:
Data
====
filebeat.yml:
----
http:
enabled: true
host: localhost
port: 5066
filebeat.inputs:
- type: udp
max_message_size: 10MiB
host: "0.0.0.0:9999"
fields:
event_type: "vault-audit"
fields_under_root: true
processors:
- decode_json_fields:
fields: ["message"]
target: "vault"
process_array: true
overwrite_keys: false
add_error_key: true
- copy_fields:
fields:
- from: vault.response.data.username
to: vault.response.datainfo.username
fail_on_error: false
ignore_missing: true
- drop_fields:
fields: ["vault.response.data"]
ignore_missing: true
- timestamp:
field: vault.time
layouts:
- 'Y'
- type: tcp
max_message_size: 10MiB
host: "0.0.0.0:9000"
fields:
event_type: "vault-audit"
fields_under_root: true
processors:
- decode_json_fields:
fields: ["message"]
target: "vault"
process_array: true
overwrite_keys: false
add_error_key: true
- copy_fields:
fields:
- from: vault.response.data.username
to: vault.response.datainfo.username
fail_on_error: false
ignore_missing: true
- drop_fields:
fields: ["vault.response.data"]
ignore_missing: true
- timestamp:
field: vault.time
layouts:
- 'Y'
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
hints.default_config:
type: container
paths:
- /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
processors:
- add_fields:
target: kubernetes
fields:
cluster.name: "k8s-dev"
- drop_event:
when:
and:
- contains:
message: "DEBUG"
- contains:
message: "changes.SessionEntityWrapper"
- drop_event:
when:
equals:
class: "c.i.p.metrics.TelegrafMetricObserver"
- drop_event:
when:
contains:
message: "com.xxx"
- drop_event:
when:
contains:
message: "metrics.xxx.com"
- drop_event:
when:
and:
- equals:
service: "CMC"
- contains:
exception.stacktrace: "org.hibernate.HibernateException: createQuery is not valid without active transaction"
- drop_event:
when:
and:
- equals:
kubernetes.container.name: "selenoid"
- contains:
kubernetes.pod.name: "availability-tests"
- drop_event:
when:
equals:
kubernetes.labels.app: "time-nginx"
- add_cloud_metadata: ~
- rename:
ignore_missing: true
fail_on_error: false
fields:
- from: "kubernetes.labels.k8s-app"
to: "service"
- from: "kubernetes.labels.service"
to: "service"
- rename:
fields:
- from: "kubernetes.labels.tenant-alias"
to: "tenant_alias"
ignore_missing: true
fail_on_error: false
when:
not:
has_fields: ['tenant_alias']
- rename:
fields:
- from: "kubernetes.labels.tenant-id"
to: "tenant_id"
ignore_missing: true
fail_on_error: false
when:
not:
has_fields: ['tenant_id']
- script:
lang: javascript
id: lowercase
source: >
function process(event) {
var level = event.Get("level");
if(level != null) {
event.Put("level", level.toString().toLowerCase());
}
}
- drop_fields:
fields:
- dissect
- ecs
- input
- ts
- tsNs
- stream
- kubernetes.namespace_uid
- kubernetes.namespace_labels
- kubernetes.node.uid
- kubernetes.node.hostname
- kubernetes.node.labels
- kubernetes.pod.uid
- kubernetes.pod.ip
- kubernetes.statefulset
- kubernetes.replicaset
- kubernetes.container.image
- kubernetes.labels
- container.id
ignore_missing: true
logging.metrics.enabled: false
logging.json: true
logging.level: warning
output.kafka:
version: 2.0.0
codec.json:
pretty: false
# escape_html: false
client_id: "logshipper"
hosts: ["kafka-cp.xxx.com:9094"]
topic: "logging-kubernetes"
topics:
- topic: "kubernetes-audit"
when.equals:
event_type: "audit"
- topic: "vault-audit"
when.equals:
event_type: "vault-audit"
partition.round_robin:
group_events: 10
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1e+06
ssl.certificate_authorities: /tmp/ca.crt
How can I parse status error 503 into Elasticsearch Discover the same way as others?
The error was in the log format, more specifically in log-format-upstream in the nginx-ingress-controller configmap, some fields did not have "quote".
old:
log-format-upstream: '{ "time": "$time_iso8601", "remote_addr": "$remote_addr",
"x-forward-for": "$proxy_add_x_forwarded_for", "request_time": $request_time,
"status": $status, "bytes_sent": $bytes_sent, "body_bytes_sent": $body_bytes_sent,
"request_length": $request_length, "request_host": "$http_host", "request_query":
"$request", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent",
"upstream_name": "$proxy_upstream_name", "upstream_addr": "$upstream_addr", "upstream_response_length":
$upstream_response_length, "upstream_response_time": $upstream_response_time,
"upstream_status": $upstream_status, "request_id": "$req_id" }'
new:
log-format-upstream: '{ "time": "$time_iso8601", "remote_addr": "$remote_addr",
"x-forward-for": "$proxy_add_x_forwarded_for", "request_time": "$request_time",
"status": "$status", "bytes_sent": "$bytes_sent", "body_bytes_sent": "$body_bytes_sent",
"request_length": "$request_length", "request_host": "$http_host", "request_query":
"$request", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent",
"upstream_name": "$proxy_upstream_name", "upstream_addr": "$upstream_addr", "upstream_response_length":
"$upstream_response_length", "upstream_response_time": "$upstream_response_time",
"upstream_status": "$upstream_status", "request_id": "$req_id" }'
I installed Grafana and Loki on my local kubernetes cluster (minikube). I was able to add Loki as a datasource and see the logs being generated on my cluster. For context, I helm installed the loki-stack helm chart. I didn't do it by writing a yaml. I have a few questions about log storage and deletion.
• What is the default storage space for logs and how would one go about deleting them? I came across this but I'm having a hard time understanding where I would configure the retention/deletion properties mentioned in that page. Any help with this will be appreciated.
• By default, after how long do the logs get deleted from the default storage space?
Also, this is what I see when I run (indentation might be off).
helm show values loki/loki-stack
loki:
enabled: true
promtail:
enabled: true
fluent-bit:
enabled: false
grafana:
enabled: false
sidecar:
datasources:
enabled: true
image:
tag: 6.7.0
prometheus:
enabled: false
filebeat:
enabled: false
filebeatConfig:
filebeat.yml: |
# logging.level: debug
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.logstash:
hosts: ["logstash-loki:5044"]
logstash:
enabled: false
image: grafana/logstash-output-loki
imageTag: 1.0.1
filters:
main: |-
filter {
if [kubernetes] {
mutate {
add_field => {
"container_name" => "%{[kubernetes][container][name]}"
"namespace" => "%{[kubernetes][namespace]}"
"pod" => "%{[kubernetes][pod][name]}"
}
replace => { "host" => "%{[kubernetes][node][name]}"}
}
}
mutate {
remove_field => ["tags"]
}
}
outputs:
main: |-
output {
loki {
url => "http://loki:3100/loki/api/v1/push"
#username => "test"
#password => "test"
}
# stdout { codec => rubydebug }
}
I was able to send logs to Elasticsearch using Filebeat using the below configuration successfully.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
# Authentication credentials - either API key or username/password.
username: "elastic"
password: "XXXXXXXXXXXXX"
#Index name customization as we do not want 'Filebeat-" prefix for the indices that filbeat creates by default
index: "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
#Below configuration setting are mandatory when customizing index name
setup.ilm.enabled: false
setup.template:
name: 'network'
pattern: 'network-*'
enabled: false
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
# ============================= X-Pack Monitoring ==============================
#monitoring.elasticsearch:
monitoring:
enabled: true
cluster_uuid: 9ZIXSpCDBASwK5K7K1hqQA
elasticsearch:
hosts: ["http:/esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
username: beats_system
password: XXXXXXXXXXXXXX
I enabled all Cisco modules and they are able to create indices as below:
network-cisco.ios-YYYY.MM.DD
network-cisco.nexus-YYYY.MM.DD
network-cisco.asa-YYYY.MM.DD
network-cisco.ftd-YYYY.MM.DD
Until here there was no issue but it all came to a halt when I tried to introduce Logstash in between Filebeat & Elasticsearch.
Below is the network.conf file details for your analysis.
input {
beats {
port => "5046"
}
}
output {
if [event.dataset] == "cisco.ios" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cisco.nexus" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cisco.asa" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cisco.ftd" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cef.log" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "panw.panos" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
stdout {codec => rubydebug}
}
With the above configuration I am unable to connect Filbeat --> Logstash --> Elasticsearch pipeline that I am looking to achieve.
There is no data that is getting added and stdout is able to produce output when I run logstash as below:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/network.conf
Using --config_test_and_exit the config file is tested successfully, also the above line is producing stdout json lines, but in spite of that there is no document that is getting added to the existing indices (network-cisco.ios-YYYY.MM.DD, network-cisco.nexus-YYYY.MM.DD etc.).
When I tried to change the index name to 'test-%{+yyyy.MM.dd}' by testing with one elasticsearch output, I found that it creates an index with the same execution above.
Also when I take Logstash out of the equation, Filebeat is able to continue writing to the existing indices but it is not happening with the above Logstash configuration.
Any help would be greatly appreciated!
Thanks,
Arun
I have setup the version(7.3.1) of ELK. filebeat(7.3.1) on a different VM.
I have multiple logs on the VM with Filebeat installed on it.
I want to have different index name for different logs.
I have tried a way which is not working,the config files are as below
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /data01/-----/LOG1/forecaster.log
fields:
log_type: type1
- type: log
enabled: true
paths:
- /data01/-----/LOG2/forecaster.log
fields:
log_type: type2
- type: log
enabled: true
paths:
- /data01/-----/LOG3/forecaster.log
fields:
log_type: type3
logstash.conf
input {
beats {
type => "filebeat"
port => "5044"
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
}
output {
stdout {
codec => rubydebug
}
if ([fields][log_type] == "type1") {
elasticsearch {
hosts => ["IP:9200"]
index => "log1"
}
}
if ([fields][log_type] == "type2") {
elasticsearch {
hosts => ["IP:9200"]
index => "log2"
}
}
if ([fields][log_type] == "type3") {
elasticsearch {
hosts => ["IP:9200"]
index => "log3"
}
}
}
using the above configurations and after analyzing the logs of EL and filebeat, log files are fetched from filebeat and send to logstash where it is being processed but they are not sent to elastic search.
I need help figuring out what's wrong/missing in order to make this work
Thanks
It seems that the indenting in the filebeat configuration is not correct in the fields section, you're missing two space characters
filebeat.inputs:
- type: log
enabled: true
paths:
- /data01/-----/LOG1/forecaster.log
fields:
log_type: type1 <-- fis this line
- type: log
enabled: true
paths:
- /data01/-----/LOG2/forecaster.log
fields:
log_type: type2 <-- fis this line
- type: log
enabled: true
paths:
- /data01/-----/LOG3/forecaster.log
fields:
log_type: type3 <-- fis this line
I am new to ELK stack. I am trying to setup FileBeat --> Logstash --> ElasticSearch --> Kibana. Here while trying to send FileBeat output to Logstash input I am getting below error on Logstash side:
CircuitBreaker::rescuing exceptions {:name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover. {:exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
I am using Logstash 2.3.2 version with FileBeat: 1.2.2, elasticsearch: 2.2.1
my logstash config:
input {
beats {
port => 5044
# codec => multiline {
# pattern => "^%{TIME}"
# negate => true
# what => previous
# }
}
}
filter {
grok {
match => { "message" => "^%{TIME:time}\s+%{LOGLEVEL:level}" }
}
}
output {
elasticsearch {
hosts => ["host:9200"]
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
my filebeat config:
filebeat:
prospectors:
- paths: - *.log
input_type: log
tail_files: false
output:
logstash:
hosts: ["host:5044"]
compression_level: 3
shipper:
logging:
to_files: true
files:
path: /tmp
name: mybeat.log
level: error