I am having an issue with getting promtail to read and log file and extract the infomation i need to send to loki
The log line in the file looks like this
2022-11-16T16:55:35.738757+00:00 hostname-13 tracker[15857] {'arg': 'test.py', 'show': None, 'envname': None, 'ostype': 'Linux', 'hostname': 'hostname-13', 'username': 'foo', 'site': 'UK', 'app': 'test', 'ver': '2.9'}
And my promtail config scrape config looks like
scrape_configs:
- job_name: tracker
static_configs:
- targets:
- localhost
labels:
job: tracker
__path__: /var/log/apptrack.log
pipeline_stages:
- match:
selector: '{job="tracker"}'
stages:
- regex:
expression: "(?P<msg>{(.*?)})"
- json:
expressions:
args: argv
show: show
envname: envname
ostype: ostype
hostname: hostname
username: username
site: site
app: app
ver: ver
but in Grafana/loki im still getting all the log line
[enter image description here](https://i.stack.imgur.com/BTFwF.png)
Any suggestions
Thanks
Related
I'm having some challenges with coercing my log lines in a certain format.
I'm running one promtail instance on several log files, of which some are logfmt and others are free-form.
My objective is to transform the free-form ones to the same logfmt as the others, independent of any other labeling. That means the actual payload (log line) pushed to my qryn instance is then supposed to have the same format, and I woudn't even be able to "see" the original, free-form log line downstream. This should enable me to use a simple | logfmt in grafana, regardless of the log source.
I tried in several ways, but I can't get the log line replaced, i.e. while I can extract to labels in all ways conceivable, I can't replace the actual log line.
A (slightly redacted) promtail-config.yml:
server:
disable: true
positions:
filename: ${RUNDIR}/.logs/positions.yaml
clients:
- url: http://mylocalqryn:33100/loki/api/v1/push
batchwait: 5s
timeout: 30s
scrape_configs:
- job_name: consolidated-logs
# https://grafana.com/docs/loki/latest/clients/promtail/pipelines/
# https://grafana.com/docs/loki/latest/clients/promtail/stages/template/
pipeline_stages:
- match:
selector: '{ Program="freeformlog" }'
stages:
- regex:
expression: '^(?P<time>^[0-9-:TZ.+]*)\s+(?P<level>[A-z]*)\s+(?P<Function>[0-9A-z:.]*)\s+(?P<msg>.*$)'
- timestamp:
format: RFC3339
source: time
- template:
source: level
template: '{{ ToLower .Value }}'
- labels:
level:
msg:
Function:
- replace:
expression: '.*'
replace: 'time="{{ .timestamp }}" level="{{ .level }}" msg="{{ .msg }}" Host="{{ .Host }}" Program="{{ .Program }}" Function="{{ .Function }}"'
static_configs:
- targets:
- localhost
labels:
Host: ${HOST:-"_host-unknown_"}
Program: logfmtcompat
__path__: ${RUNDIR}/.logs/logfmtcompat.log
- targets:
- localhost
labels:
Host: ${HOST:-"_host-unknown_"}
Program: freeformlog
__path__: ${RUNDIR}/.logs/freeformlog.log
in our cluster some apps are sending logs as multiline, and the problem is that the log structure is different from app to app.
How can we set up an 'if' condition that will include the
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
In it?
Our code:
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_event:
when:
contains:
container.image.name: "kibana"
output.logstash:
hosts: ["logstash-listener:5044"]
You need to use auto-discovery (either Docker or Kubernetes) with template conditions.
You will probably have at least two templates, one for capturing your containers that emit multiline messages and another for other containers.
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition: <--- your multiline condition goes here
contains:
kubernetes.namespace: xyz-namespace
config:
- type: container
paths:
- /var/lib/docker/containers/${data.docker.container.id}/*.log
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_event:
when:
contains:
container.image.name: "kibana"
I'm a bit new to Grafana so this might be an easy one! I have a simple config-promtail.yaml file loading logs into Loki and everything is working, but I'd like to restrict the log rows passed to Loki to only those lines that include the word "error". Here is what I have:
server:
http_listen_port: <port #>
grpc_listen_port: <port #>
positions:
filename: /tmp/positions.yaml
clients:
- url: 'http://10.128.15.231:3100/loki/api/v1/push'
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: log_export
__path__: /path/to/log/file.log
host: host-name
pipeline_stages:
- match:
selector: '{host="host-name"} |= "error"'
action: keep
it works fine until I add the |= "error
I've also tried something like this:
pipeline_stages:
- match:
selector: '{host="host-name"}'
stages:
- regex:
expression: '.*error.*'
which also throws config errors. it seems like this should be relatively simple, but the documentation is really not clear...thanks in advance for any assistance!
I'm trying to pull source code from Github then build and push a docker image to docker hub using Tekton pipeline and Knative on Kubernetes cluster.
I'm following this link for the installation and setup of Tekton:
https://www.ibm.com/cloud/blog/build-a-knative-service-with-tekton-and-apache-openwhisk-nodejs-runtime
task-build.yaml
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
name: task-build
spec:
inputs:
resources:
- name: docker-source
type: git
params:
- name: TARGET_IMAGE_NAME
description: name of the image to be tagged and pushed
- name: TARGET_IMAGE_TAG
description: tag the image before pushing
default: "latest"
- name: DOCKERFILE
description: name of the dockerfile
- name: OW_RUNTIME_DEBUG
description: flag to indicate debug mode should be on/off
default: "false"
- name: OW_RUNTIME_PLATFORM
description: flag to indicate the platform, one of ["openwhisk", "knative", ... ]
default: "knative"
- name: OW_ACTION_NAME
description: name of the action
default: "foo"
- name: OW_ACTION_CODE
description: JavaScript source code to be evaluated
default: ""
- name: OW_ACTION_MAIN
description: name of the function in the "__OW_ACTION_CODE" to call as the action handler
default: "main"
- name: OW_ACTION_BINARY
description: flag to indicate zip function, for zip actions, "__OW_ACTION_CODE" must be base64 encoded string
default: "false"
- name: OW_HTTP_METHODS
description: list of HTTP methods, any combination of [GET, POST, PUT, and DELETE], default is [POST]
default: "[POST]"
- name: OW_ACTION_RAW
description: flag to indicate raw HTTP handling, interpret and process an incoming HTTP body directly
default: "false"
outputs:
resources:
- name: builtImage
type: image
steps:
- name: add-ow-env-to-dockerfile
image: "gcr.io/kaniko-project/executor:debug"
command:
- /busybox/sh
args:
- -c
- |
cat <<EOF >> ${inputs.params.DOCKERFILE}
ENV __OW_RUNTIME_DEBUG "${inputs.params.OW_RUNTIME_DEBUG}"
ENV __OW_RUNTIME_PLATFORM "${inputs.params.OW_RUNTIME_PLATFORM}"
ENV __OW_ACTION_NAME "${inputs.params.OW_ACTION_NAME}"
ENV __OW_ACTION_CODE "${inputs.params.OW_ACTION_CODE}"
ENV __OW_ACTION_MAIN "${inputs.params.OW_ACTION_MAIN}"
ENV __OW_ACTION_BINARY "${inputs.params.OW_ACTION_BINARY}"
ENV __OW_HTTP_METHODS "${inputs.params.OW_HTTP_METHODS}"
ENV __OW_ACTION_RAW "${inputs.params.OW_ACTION_RAW}"
EOF
- name: adapt-dockerfile-to-tekton
image: "gcr.io/kaniko-project/executor:debug"
command:
- sed
args:
- -i
- -e
- 's/COPY ./COPY .\/docker-source/g'
- ${inputs.params.DOCKERFILE}
- name: build-openwhisk-nodejs-runtime
image: "gcr.io/kaniko-project/executor:latest"
args: ["--destination=${inputs.params.TARGET_IMAGE_NAME}:${inputs.params.TARGET_IMAGE_TAG}", "--dockerfile=${inputs.params.DOCKERFILE}"]
When trying to build and push the image, am getting error:
conditions:
- lastTransitionTime: "2020-09-24T07:33:11Z"
"step-add-ow-env-to-dockerfile" exited with code 2 (image: "docker-pullable://gcr.io/kaniko-project/executor#sha256:0f27b0674797b56db08010dff799c8926c4e9816454ca56cc7844df228c53485"); for logs run: kubectl -n default logs task-run-helloworld-pod-5bbkx -c step-add-ow-env-to-dockerfile
reason: Failed
status: "False"
type: Succeeded
When checked the logs for error msg, I'm getting:
Error : /busybox/sh: syntax error: bad substitution
I have a prometheus.yml config file which have multiple k8s clusters configured for monitoring. Since the servers come and go we need to delete the servers which are deleted from our peometheus.yml config file using ansible.
I tried with the following and did not work.
https://docs.ansible.com/ansible/2.5/modules/blockinfile_module.html
- hosts: blocks
tasks:
- name: Removing a line using blockinfile
blockinfile:
dest: /home/mdtutorials2/block_output.txt
marker: <!-- {mark} Adding IP address -->
state: absent
prometheus.yml
# *************************************************************START-RAVVE*****************************************************************************************
# metrics for kubernetes scheduler and controller
- job_name: 'ravve.ntnxsherlock.com-scheduler-and-controller'
scrape_interval: 5s
static_configs:
- targets: ['ip-172-31-12-14.us-east-2.compute.internal:10251']
labels:
customer: 'RAVVE'
# metrics foom node exporter
- job_name: 'ravve.ntnxsherlock.com-nodes-exporter'
scrape_interval: 5s
static_configs:
- targets: ['ip-172-31-12-14.us-east-2.compute.internal:9100']
labels:
customer: 'RAVVE'
- targets: ['ip-172-31-13-200.us-east-2.compute.internal:9100']
labels:
customer: 'RAVVE'
# metrics from cadvisory
- job_name: 'ravve.ntnxsherlock.com-cadvisor'
scrape_interval: 10s
metrics_path: "/metrics/cadvisor"
static_configs:
- targets: ['ip-172-31-12-14.us-east-2.compute.internal:10255']
labels:
customer: 'RAVVE'
# metrics for default/kubernetes api's from the kubernetes master
- job_name: 'ravve.ntnxsherlock.com-apiservers'
kubernetes_sd_configs:
- role: endpoints
api_server: https://ip-172-31-12-14.us-east-2.compute.internal
tls_config:
insecure_skip_verify: true
basic_auth:
username: admin
password: XXXXXXXXXXXXXXXX
scheme: https
tls_config:
insecure_skip_verify: true
basic_auth:
username: admin
password: XXXXXXXXXXXXX
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
# **************************************************************END-RAVVE*****************************************************************************************
Now i need to delete the line between start and end cluster
Start *************************************************************START-RAVVE*****************************************************************************************
End
# **************************************************************END-RAVVE*****************************************************************************************