The Log entry looks like this
2022-05-12 10:32:19,659 - root:loadrequestheaders:33 - INFO: otds=XYZ, domain=XYZ, username=user, origin=10.98.76.67
I tried to add the promtail configuration to scrap log entry to get new labels :
Updated ConfigMap - loki-promtail with below content
- job_name: kubernetes-pods-static
pipeline_stages:
- match:
selector: '{container="model"}'
stages:
- regex:
expression: '\.*username=(?P<username>\S+).*origin=(?P<origin>\S+)'
- labels:
username:
origin:
I could see loki pods are running successfully with above config map update but I don't see the entries "username and origin" - in the labels list
Any suggestion on how to add new labels in loki datasource.
enter image description here
Related
I'm using Amazon OpenSearch with the engine OpenSearch 1.2.
I was working on setting up APM with the following details
Service 1 - Tomcat Application running on an EC2 server that accesses an RDS database. The server is behind a load balancer with a sub-domain mapped to it.
I added a file - setenv.sh file in the tomcat/bin folder with the following content
#!/bin/sh
export CATALINA_OPTS="$CATALINA_OPTS -javaagent:<PATH_TO_JAVA_AGENT>"
export OTEL_METRICS_EXPORTER=none
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<OTEL_COLLECTOR_SERVER_IP>:4317
export OTEL_RESOURCE_ATTRIBUTES=service.name=<SERVICE_NAME>
export OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING=<RDS_HOST_ENDPOINT>=Database-Service
OTEL Java Agent for collecting traces from the application
OTEL Collector and Data Prepper running on another server with the following configuration
docker-compose.yml
version: "3.7"
services:
data-prepper:
restart: unless-stopped
image: opensearchproject/data-prepper:1
volumes:
- ./pipelines.yaml:/usr/share/data-prepper/pipelines.yaml
- ./data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml
networks:
- apm_net
otel-collector:
restart: unless-stopped
image: otel/opentelemetry-collector:0.55.0
command: [ "--config=/etc/otel-collector-config.yml" ]
volumes:
- ./otel-collector-config.yml:/etc/otel-collector-config.yml
ports:
- "4317:4317"
depends_on:
- data-prepper
networks:
- apm_net
data-prepper-config.yaml
ssl: false
otel-collector-config.yml
receivers:
otlp:
protocols:
grpc:
exporters:
otlp/data-prepper:
endpoint: http://data-prepper:21890
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp/data-prepper]
pipelines.yaml
entry-pipeline:
delay: "100"
source:
otel_trace_source:
ssl: false
sink:
- pipeline:
name: "raw-pipeline"
- pipeline:
name: "service-map-pipeline"
raw-pipeline:
source:
pipeline:
name: "entry-pipeline"
prepper:
- otel_trace_raw_prepper:
sink:
- opensearch:
hosts:
[
<AWS OPENSEARCH HOST>,
]
# IAM signing
aws_sigv4: true
aws_region: <AWS_REGION>
index_type: "trace-analytics-raw"
service-map-pipeline:
delay: "100"
source:
pipeline:
name: "entry-pipeline"
prepper:
- service_map_stateful:
sink:
- opensearch:
hosts:
[
<AWS OPENSEARCH HOST>,
]
# IAM signing
aws_sigv4: true
aws_region: <AWS_REGION>
index_type: "trace-analytics-service-map"
The data-prepper is getting authenticated via Fine access based control with all_access role and I'm able to see the otel resources like indexes, index templates generated when running it.
On running the above setup, I'm able to see traces from the application in the Trace Analytics Dashboard of OpenSearch, and upon clicking on the individual traces, I'm able to see a pie chart with one service. I also don't see any errors in the otel-collector as well as in data-prepper. Also, in the logs of data prepper, I see records being sent to otel service map.
However, the services tab of Trace Analytics remains empty and the otel service map index also remains empty.
I have been unable to figure out the reason behind this even after going through the documentation and any help is appreciated!
I installed Loki, grafana and promtail and all three runing. on http://localhost:9080/targets Ready is True, but the logs are not displayed in Grafana and show in the explore section "No logs found"
promtail-local-config-yaml:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
host: ward_workstation
agent: promtail
__path__: D:/LOGs/*log
loki-local-config.yaml:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
How can i solve this problem?
Perhaps you are using Loki in Windows ?
In your promtail varlogs job ,the Path "D:/LOGs/*log" is obviously wrong, you cannot access the windows file from your docker directly.
You shoud mount your windows file to your docker like this:
promtail:
image: grafana/promtail:2.5.0
volumes:
- D:/LOGs:/var/log
command: -config.file=/etc/promtail/config.yml
networks:
- loki
Then everything will be ok.
Note that, in your promtail docker the config is like this:
you can adjust both to make a match...
Here's a general advice how to debug Loki according to the question's title:
(1) Check promtail logs
If you discover such as error sending batch you need to fix your Promtail configuration.
level=warn ts=2022-10-12T16:26:20.667560426Z caller=client.go:369 component=client host=monitor:3100 msg="error sending batch, will retry" status=-1 error="Post \"http://loki:3100/loki/api/v1/push\": dial tcp: lookup *Loki* on 10.96.0.10:53: no such host"
(2) Open the Promtail config page and check, if Promtail has read your given configuration: http://localhost:3101/config
(3) Open the Promtail targets page http://localhost:3101/targets and check
if your service is listed as Ready
if the log file contains the wanted contents and is readable by Promtail. If you're using docker or kubernetes I would log into the Promtail Container and would try to read the logfile manually.
To the specific problem of the questioner:
The questioner said, that the services are shown as READY in the targets page. So I recommend to check (1) Promtail configuration and (3b) access to log files (as Frank).
We deployed the metricbeat on our Kubernetes cluster. All the metrics are been pushed to the elasticsearch index without any issues. But observing below error in the metribeat logs and not sure what exactly is causing this.
2020-07-01T14:05:25.421Z INFO module/wrapper.go:259 Error fetching data for metricset kubernetes.apiserver: error getting metrics: unexpected status code 403 from server
We have explored and tried the suggested cluster-role and didn't see any success. Below is the cluster-role we are using.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:
- apiGroups:
- "extensions"
- "apps"
- "v1"
- ""
resources:
- namespaces
- pods
- events
- deployments
- nodes
- replicasets
- statefulsets
- nodes/metrics
- nodes/stats
verbs:
- get
- list
- watch
Let me know if anyone has came across the same and have fixed it. Thanks in advance.
It looks like Metricbeat cannot connect to the K8S API server, HTTP 403 is a security error.
According to the official documentation, can you make sure that you add these in the ClusterRole rules:
rules:
- nonResourceURLs:
- /metrics
verbs:
- get
I am using elasticserach 6.8 and filebeat 6.8.0 in a Kubernetes cluster. I want filebeat to ignore certain container logs but it seems almost impossible :).
This is my autodiscover config
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
templates:
- condition:
contains:
kubernetes.namespace: bagmessage
config:
- type: docker
containers.ids:
- "${data.kubernetes.container.id}"
processors:
- drop_event:
when:
or:
- contains:
kubernetes.container.name: "filebeat"
- contains:
kubernetes.container.name: "weave-npc"
- contains:
kubernetes.container.name: "bag-fluentd-es"
- contains:
kubernetes.container.name: "logstash"
- contains:
kubernetes.container.name: "billing"
I've tried many variations of this configuration but still filebeats is processing container logs that I want it to ignore.
I'd like to know if what I want to do is possible and if so, what am I doing wrong?
Thanks
The first error I see in your config is incorrect indentation of the condition section in the template. Should be:
- type: kubernetes
hints.enabled: true
templates:
- condition:
contains:
kubernetes.namespace: bagmessage
Secondly, I'm not sure the kubernetes.* is visible to the processors inside the config with type: docker. You may try to reference docker.container.name instead. Or alternatively you can move all your k8s-specific conditions to the condition section under the templates:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
templates:
- condition:
and:
- contains.kubernetes.namespace: bagmessage
- contains.container.name: billing
config:
...
Also, make sure that "container.name" (but not "pod.name") is indeed what you want.
Seems to work now. I'm not sure what the issue was.
I am using the ELK stack (elasticsearch, logsash, kibana) for log processing and analysis in a Kubernetes (minikube) environment. To capture logs I am using filebeat. Logs are propagated successfully from filebeat through to elasticsearch and are viewable in Kibana.
My problem is that I am unable to get the pod name of the actual pod issuing log records. Rather I only get the filebeat podname which is gathering log files and not name of the pod that is originating log records.
The information I can get from filebeat are (as viewed in Kibana)
beat.hostname: the value of this field is the filebeat pod name
beat.name: value is the filebeat pod name
host: value is the filebeat pod name
I can also see/discern container information in Kibana which flow through from filebeat / logstash / elasticsearch:
app: value is {log-container-id}-json.log
source: value is /hostfs/var/lib/docker/containers/{log-container-id}-json.log
As shown above, I seem to be able to get the container Id but not the pod name.
To mitigate the situation, I could probably embed the pod-name in the actual log message and parse it from there, but I am hoping there is a solution in which I can configure filebeat to emit actual pod names.
Does anyone now how to configure filebeat (or other components) to capture kubernetes (minikube) pod names in their logs?
My current filebeat configuration is listed below:
ConfigMap is shown below:
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat
namespace: logging
labels:
component: filebeat
data:
filebeat.yml: |
filebeat.prospectors:
- input_type: log
tags:
- host
paths:
- "/hostfs/var/log"
- "/hostfs/var/log/*"
- "/hostfs/var/log/*/*"
exclude_files:
- '\.[0-9]$'
- '\.[0-9]\.gz$'
- input_type: log
tags:
- docker
paths:
- /hostfs/var/lib/docker/containers/*/*-json.log
json:
keys_under_root: false
message_key: log
add_error_key: true
multiline:
pattern: '^[[:space:]]+|^Caused by:'
negate: false
match: after
output.logstash:
hosts: ["logstash:5044"]
logging.level: info
DamemonSet is shown below:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: logging
spec:
template:
metadata:
labels:
component: filebeat
spec:
containers:
- name: filebeat
image: giantswarm/filebeat:5.2.2
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 100m
requests:
cpu: 100m
volumeMounts:
- name: config
mountPath: /etc/filebeat
readOnly: true
- name: hostfs-var-lib-docker-containers
mountPath: /hostfs/var/lib/docker/containers
readOnly: true
- name: hostfs-var-log
mountPath: /hostfs/var/log
readOnly: true
volumes:
- name: config
configMap:
name: filebeat
- name: hostfs-var-log
hostPath:
path: /var/log
- name: hostfs-var-lib-docker-containers
hostPath:
path: /var/lib/docker/containers
disclaimer: I'm a beats developer
What you want to do is not yet supported by filebeat, but definitely, it's something we want to put some effort on, so you can expect future releases supporting this kind of mapping.
In the meantime, I think your approach is correct. You can append the info you need to your logs so you have it in elasticsearch
I have achieved what you looking for, by assigning a group of specific pods to a namespace, and now can query the log I look for using a combination of namespace, pod name and container name which is also included in generated log which is piped by file beat without any extra effort as you can see here
For future people coming here, it is now already in place in a filebeat processor :
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/*.log
- /var/log/messages
- /var/log/syslog
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
- drop_event:
when:
equals:
kubernetes.container.name: "filebeat"
helm chart default values : https://github.com/helm/charts/blob/master/stable/filebeat/values.yaml
doc : https://www.elastic.co/guide/en/beats/filebeat/current/add-kubernetes-metadata.html