Grafana Loki Logs Storage and Deletion - grafana-loki

I installed Grafana and Loki on my local kubernetes cluster (minikube). I was able to add Loki as a datasource and see the logs being generated on my cluster. For context, I helm installed the loki-stack helm chart. I didn't do it by writing a yaml. I have a few questions about log storage and deletion.
• What is the default storage space for logs and how would one go about deleting them? I came across this but I'm having a hard time understanding where I would configure the retention/deletion properties mentioned in that page. Any help with this will be appreciated.
• By default, after how long do the logs get deleted from the default storage space?
Also, this is what I see when I run (indentation might be off).
helm show values loki/loki-stack
loki:
enabled: true
promtail:
enabled: true
fluent-bit:
enabled: false
grafana:
enabled: false
sidecar:
datasources:
enabled: true
image:
tag: 6.7.0
prometheus:
enabled: false
filebeat:
enabled: false
filebeatConfig:
filebeat.yml: |
# logging.level: debug
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.logstash:
hosts: ["logstash-loki:5044"]
logstash:
enabled: false
image: grafana/logstash-output-loki
imageTag: 1.0.1
filters:
main: |-
filter {
if [kubernetes] {
mutate {
add_field => {
"container_name" => "%{[kubernetes][container][name]}"
"namespace" => "%{[kubernetes][namespace]}"
"pod" => "%{[kubernetes][pod][name]}"
}
replace => { "host" => "%{[kubernetes][node][name]}"}
}
}
mutate {
remove_field => ["tags"]
}
}
outputs:
main: |-
output {
loki {
url => "http://loki:3100/loki/api/v1/push"
#username => "test"
#password => "test"
}
# stdout { codec => rubydebug }
}

Related

Cisco-module (Filebeat) to Logstash - Configuration issue - Unable to write to existing indices

I was able to send logs to Elasticsearch using Filebeat using the below configuration successfully.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
# Authentication credentials - either API key or username/password.
username: "elastic"
password: "XXXXXXXXXXXXX"
#Index name customization as we do not want 'Filebeat-" prefix for the indices that filbeat creates by default
index: "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
#Below configuration setting are mandatory when customizing index name
setup.ilm.enabled: false
setup.template:
name: 'network'
pattern: 'network-*'
enabled: false
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
# ============================= X-Pack Monitoring ==============================
#monitoring.elasticsearch:
monitoring:
enabled: true
cluster_uuid: 9ZIXSpCDBASwK5K7K1hqQA
elasticsearch:
hosts: ["http:/esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
username: beats_system
password: XXXXXXXXXXXXXX
I enabled all Cisco modules and they are able to create indices as below:
network-cisco.ios-YYYY.MM.DD
network-cisco.nexus-YYYY.MM.DD
network-cisco.asa-YYYY.MM.DD
network-cisco.ftd-YYYY.MM.DD
Until here there was no issue but it all came to a halt when I tried to introduce Logstash in between Filebeat & Elasticsearch.
Below is the network.conf file details for your analysis.
input {
beats {
port => "5046"
}
}
output {
if [event.dataset] == "cisco.ios" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cisco.nexus" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cisco.asa" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cisco.ftd" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "cef.log" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
else if [event.dataset] == "panw.panos" {
elasticsearch {
hosts => ["http://esnode1.cluster.com:9200","http://esnode2.cluster.com:9200"]
index => "network-%{[event.dataset]}-%{+yyyy.MM.dd}"
user => "elastic"
password => "XXXXXXXXXXXX"
pipeline => "%{[#metadata][pipeline]}"
manage_template => "false"
ilm_enabled => "false"
}
}
stdout {codec => rubydebug}
}
With the above configuration I am unable to connect Filbeat --> Logstash --> Elasticsearch pipeline that I am looking to achieve.
There is no data that is getting added and stdout is able to produce output when I run logstash as below:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/network.conf
Using --config_test_and_exit the config file is tested successfully, also the above line is producing stdout json lines, but in spite of that there is no document that is getting added to the existing indices (network-cisco.ios-YYYY.MM.DD, network-cisco.nexus-YYYY.MM.DD etc.).
When I tried to change the index name to 'test-%{+yyyy.MM.dd}' by testing with one elasticsearch output, I found that it creates an index with the same execution above.
Also when I take Logstash out of the equation, Filebeat is able to continue writing to the existing indices but it is not happening with the above Logstash configuration.
Any help would be greatly appreciated!
Thanks,
Arun

filebeat JSON logging broken logs in Elastic search

I'm using filebeat>kafka>logstash>elastiserach stack of version 7.15.0.
My logs are getting broken and not writing properly into elastic which is resulting in json.erros
error.message:Key 'log' not found
error.type: json
and
error.message:Error decoding JSON: invalid character 's' after array element error.type:json
my log contains both json and nonjson contents
my docker stdout logs>>
{"log":"{\"instant\":{\"epochSecond\":1643023707,\"nanoOfSecond\":538281000},\"thread\":\"grpc-default-executor-11\",\"level\":\"INFO\",\"loggerName\":\"com.abc.ab.ab.core.service.integration.controller.NotifyCoreGrpcController\",\"message\":\"RMQ_CORE_GRPC_NOTIFY RESP : {\\\"baseBizResponse\\\":{\\\"success\\\":true,\\\"resultCode\\\":\\\"SUCCESS\\\"}} \",\"endOfBatch\":false,\"loggerFqcn\":\"org.apache.logging.slf4j.Log4jLogger\",\"contextMap\":{\"RMQ_ID\":\"2022012401445669241212121212\",\"FLOW_TYPE\":\"RMQ_CORE_GRPC_NOTIFY\",\"MERCHANT_TRANS_ID\":\"bcd4ab1e54abaha122\",\"spanId\":\"4fa1474c078afceb\",\"traceId\":\"bcd4ab1e54abaha122\"},\"threadId\":100,\"threadPriority\":5,\"dateTime\":\"2022-01-24T16:58:27.538+0530\"}\r\n","stream":"stdout","time":"2022-01-24T11:28:27.538354156Z"}
and
[244540.169s][debug][gc,age] GC(51) Desired survivor size 80740352 bytes, new threshold 15 (max threshold 15)
Filebeat conf>>
filebeat.yml: |
filebeat.inputs:
- type: container
multiline.pattern: ^[[:space:]]'
multiline.negate: false
multiline.match: after
json.keys_under_root: true
json.message_key: log
json.add_error_key: true
enabled: true
paths:
- /var/log/containers/*.log
exclude_files: ['fluentd-*', 'istio-*', 'cluster-logging-*', 'infra-*']
processors:
- add_kubernetes_metadata:
in_cluster: true
host: ${NODE_NAME}
default_matchers.enabled: false
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- drop_fields:
fields:
- 'kubernetes.node.name'
output.kafka:
enabled: true
hosts: ["kafka1:9092","kafka2:9092","kafka3:9092"]
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 10000000
topics:
- topic: '%{[kubernetes.labels.app]}'
default: 'app-perf-k8s-logs'
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
filebeat.modules:
- module: nginx
- module: kafka
logging.level: debug
logstash config >>
input {
kafka {
topics_pattern => ".*."
bootstrap_servers => "kafka1:9092","kafka2:9092","kafka3:9092"
client_id => "logstash"
codec => json
decorate_events => true
consumer_threads => 10
heartbeat_interval_ms => "100000"
session_timeout_ms => "300000"
poll_timeout_ms => 300000
partition_assignment_strategy => "org.apache.kafka.clients.consumer.RoundRobinAssignor"
request_timeout_ms => "400000"
group_id => "logConsumer"
auto_offset_reset => "latest"
}
}
output {
elasticsearch {
hosts => "es-logging-perf-lb.abc.com:80"
index => "filebeat-%{[#metadata][kafka][topic]}-%{+YYYY.MM.dd}"
}
Please help and suggest.

ECK Filebeat Daemonset Forwarding To Remote Cluster

I wish to forward logs from remote EKS clusters to a centralised EKS cluster hosting ECK.
Versions in use:
EKS v1.20.7
Elasticsearch v7.7.0
Kibana v7.7.0
Filebeat v7.10.0
The setup is using a AWS NLB to forward requests to Nginx ingress, using host based routing.
When the DNS lookup (filebeat test output) for the Elasticsearch is tested on Filebeat, it validates the request.
But the logs for Filebeat are telling a different story.
2021-10-05T10:39:00.202Z ERROR [publisher_pipeline_output]
pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)):
Get "https://elasticsearch.dev.example.com:9200": Bad Request
The Filebeat agents can connect to the remote Elasticsearch via the NLB, when using a curl request.
The config is below. NB: dev.example.com is the remote cluster hosing ECK.
app:
name: "filebeat"
configmap:
enabled: true
filebeatConfig:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- config:
- type: container
paths:
- /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- drop_event.when.not.or:
- contains.kubernetes.namespace: "apps-"
- equals.kubernetes.namespace: "cicd"
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
overwrite_keys: true
- add_fields:
fields:
kubernetes.cluster.name: dev-eks-cluster
target: ""
processors:
- add_cloud_metadata: ~
- add_host_metadata: ~
cloud:
id: '${ELASTIC_CLOUD_ID}'
cloud:
auth: '${ELASTIC_CLOUD_AUTH}'
output:
elasticsearch:
enabled: true
hosts: "elasticsearch.dev.example.com"
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl:
verification_mode: "none"
headers:
Host: "elasticsearch.dev.example.com"
proxy_url: "https://example.elb.eu-west-2.amazonaws.com"
proxy_disable: false
daemonset:
enabled: true
version: 7.10.0
image:
repository: "docker.elastic.co/beats/filebeat"
tag: "7.10.0"
pullPolicy: Always
extraenvs:
- name: ELASTICSEARCH_HOST
value: "https://elasticsearch.dev.example.com"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
value: "remote-cluster-elasticsearch-es-elastic-user-password"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
clusterrolebinding:
enabled: true
namespace: monitoring
clusterrole:
enabled: true
serviceaccount:
enabled: true
namespace: monitoring
deployment:
enabled: false
configmap:
enabled: false
Any tips or suggestions on how to enable Filebeat forwarding, would be much appreciated :-)
#1 Missing ports:
Even with the ports added in as suggested. Filebeat is erroring with:
2021-10-06T08:34:41.355Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)): Get "https://elasticsearch.dev.example.com:9200": Bad Request
...using a AWS NLB to forward requests to Nginx ingress, using host based routing
How about unset proxy_url and proxy_disable, then set hosts: ["<nlb url>:<nlb listener port>"]
The final working config:
app:
name: "filebeat"
configmap:
enabled: true
filebeatConfig:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- config:
- type: container
paths:
- /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- drop_event.when.not.or:
- contains.kubernetes.namespace: "apps-"
- equals.kubernetes.namespace: "cicd"
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
overwrite_keys: true
- add_fields:
fields:
kubernetes.cluster.name: qa-eks-cluster
target: ""
processors:
- add_cloud_metadata: ~
- add_host_metadata: ~
cloud:
id: '${ELASTIC_CLOUD_ID}'
cloud:
auth: '${ELASTIC_CLOUD_AUTH}'
output:
elasticsearch:
enabled: true
hosts: ["elasticsearch.dev.example.com:9200"]
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl:
verification_mode: "none"
daemonset:
enabled: true
version: 7.10.0
image:
repository: "docker.elastic.co/beats/filebeat"
tag: "7.10.0"
pullPolicy: Always
extraenvs:
- name: ELASTICSEARCH_HOST
value: "https://elasticsearch.dev.example.com"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
value: "remote-cluster-elasticsearch-es-elastic-user-password"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
clusterrolebinding:
enabled: true
namespace: monitoring
clusterrole:
enabled: true
serviceaccount:
enabled: true
namespace: monitoring
deployment:
enabled: false
configmap:
enabled: false
In addition the following changes were needed:
NBL:
Add listener for 9200 forwarding to the Ingress Controller for HTTPS
SG:
Opened up port 9200 on the EKS worker nodes

How to have different index name for different log from same filebeat to logstash

I have setup the version(7.3.1) of ELK. filebeat(7.3.1) on a different VM.
I have multiple logs on the VM with Filebeat installed on it.
I want to have different index name for different logs.
I have tried a way which is not working,the config files are as below
filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /data01/-----/LOG1/forecaster.log
fields:
log_type: type1
- type: log
enabled: true
paths:
- /data01/-----/LOG2/forecaster.log
fields:
log_type: type2
- type: log
enabled: true
paths:
- /data01/-----/LOG3/forecaster.log
fields:
log_type: type3
logstash.conf
input {
beats {
type => "filebeat"
port => "5044"
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
}
output {
stdout {
codec => rubydebug
}
if ([fields][log_type] == "type1") {
elasticsearch {
hosts => ["IP:9200"]
index => "log1"
}
}
if ([fields][log_type] == "type2") {
elasticsearch {
hosts => ["IP:9200"]
index => "log2"
}
}
if ([fields][log_type] == "type3") {
elasticsearch {
hosts => ["IP:9200"]
index => "log3"
}
}
}
using the above configurations and after analyzing the logs of EL and filebeat, log files are fetched from filebeat and send to logstash where it is being processed but they are not sent to elastic search.
I need help figuring out what's wrong/missing in order to make this work
Thanks
It seems that the indenting in the filebeat configuration is not correct in the fields section, you're missing two space characters
filebeat.inputs:
- type: log
enabled: true
paths:
- /data01/-----/LOG1/forecaster.log
fields:
log_type: type1 <-- fis this line
- type: log
enabled: true
paths:
- /data01/-----/LOG2/forecaster.log
fields:
log_type: type2 <-- fis this line
- type: log
enabled: true
paths:
- /data01/-----/LOG3/forecaster.log
fields:
log_type: type3 <-- fis this line

CircuitBreaker::rescuing exceptions {:name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}

I am new to ELK stack. I am trying to setup FileBeat --> Logstash --> ElasticSearch --> Kibana. Here while trying to send FileBeat output to Logstash input I am getting below error on Logstash side:
CircuitBreaker::rescuing exceptions {:name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}
Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover. {:exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}
I am using Logstash 2.3.2 version with FileBeat: 1.2.2, elasticsearch: 2.2.1
my logstash config:
input {
beats {
port => 5044
# codec => multiline {
# pattern => "^%{TIME}"
# negate => true
# what => previous
# }
}
}
filter {
grok {
match => { "message" => "^%{TIME:time}\s+%{LOGLEVEL:level}" }
}
}
output {
elasticsearch {
hosts => ["host:9200"]
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
my filebeat config:
filebeat:
prospectors:
- paths: - *.log
input_type: log
tail_files: false
output:
logstash:
hosts: ["host:5044"]
compression_level: 3
shipper:
logging:
to_files: true
files:
path: /tmp
name: mybeat.log
level: error

Resources