How to truncate the logs in Promtail - grafana-loki

I am using Promtail to ship some Jenkins logs to Loki and I want to truncate the logs which are longer than e.g. 12k characters.
This is the current configuration that I'm using:
clients:
- url: my-loki-instance
external_labels:
cluster: "clustername"
app: "jenkins-test-promtail"
namespace: ${JENKINS_NAMESPACE}
pod: ${JENKINS_POD_NAME}
container: ${JENKINS_CONTAINER}
node: ${JENKINS_NODE_NAME}
positions:
filename: "/var/log/jenkins/promtail/positions.yaml"
limits_config:
readline_rate_enabled: true
readline_rate_drop: false
options:
stream_lag_labels: namespace,pod,container
scrape_configs:
- job_name: master_logs
static_configs:
- labels:
component: master
__path__: /var/log/jenkins/master.0.log
pipeline_stages:
- regex:
expression: "(?P<truncated_log>.{0,12000}).*"
- output:
source: truncated_log
As expected, most of the logs are shorter than 12k characters and I get this error:
level=error ts=2022-10-06T13:48:37.260110981Z caller=main.go:117 msg="error creating promtail" error="failed to make file target manager: invalid regex stage config: could not compile regular expression: error parsing regexp: invalid repeat count: `{0,12000}`"
I have tried truncating to smaller values, like 20 characters and it works as expected because the lines are longer than 20 characters.
Any idea how to overcome this or if there's another solution for it?
Promtail version: 2.6.1
UPDATE:
I ended up using go templates to do the truncating, like this:
...
pipeline_stages
- match:
selector: '{component="master"}'
stages:
- template:
source: log_truncated
template: |
{{- if le (len .log) 12000 -}}
{{ .log }}
{{- end -}}
{{- if gt (len .log) 12000 -}}
{{ trunc 12000 .log }} TRUNCATED
{{- end -}}
- output:
source: log_truncated
...

Related

promtail: transform the whole log line based on regex

I'm having some challenges with coercing my log lines in a certain format.
I'm running one promtail instance on several log files, of which some are logfmt and others are free-form.
My objective is to transform the free-form ones to the same logfmt as the others, independent of any other labeling. That means the actual payload (log line) pushed to my qryn instance is then supposed to have the same format, and I woudn't even be able to "see" the original, free-form log line downstream. This should enable me to use a simple | logfmt in grafana, regardless of the log source.
I tried in several ways, but I can't get the log line replaced, i.e. while I can extract to labels in all ways conceivable, I can't replace the actual log line.
A (slightly redacted) promtail-config.yml:
server:
disable: true
positions:
filename: ${RUNDIR}/.logs/positions.yaml
clients:
- url: http://mylocalqryn:33100/loki/api/v1/push
batchwait: 5s
timeout: 30s
scrape_configs:
- job_name: consolidated-logs
# https://grafana.com/docs/loki/latest/clients/promtail/pipelines/
# https://grafana.com/docs/loki/latest/clients/promtail/stages/template/
pipeline_stages:
- match:
selector: '{ Program="freeformlog" }'
stages:
- regex:
expression: '^(?P<time>^[0-9-:TZ.+]*)\s+(?P<level>[A-z]*)\s+(?P<Function>[0-9A-z:.]*)\s+(?P<msg>.*$)'
- timestamp:
format: RFC3339
source: time
- template:
source: level
template: '{{ ToLower .Value }}'
- labels:
level:
msg:
Function:
- replace:
expression: '.*'
replace: 'time="{{ .timestamp }}" level="{{ .level }}" msg="{{ .msg }}" Host="{{ .Host }}" Program="{{ .Program }}" Function="{{ .Function }}"'
static_configs:
- targets:
- localhost
labels:
Host: ${HOST:-"_host-unknown_"}
Program: logfmtcompat
__path__: ${RUNDIR}/.logs/logfmtcompat.log
- targets:
- localhost
labels:
Host: ${HOST:-"_host-unknown_"}
Program: freeformlog
__path__: ${RUNDIR}/.logs/freeformlog.log

how to filter rows in promtail yaml config

I'm a bit new to Grafana so this might be an easy one! I have a simple config-promtail.yaml file loading logs into Loki and everything is working, but I'd like to restrict the log rows passed to Loki to only those lines that include the word "error". Here is what I have:
server:
http_listen_port: <port #>
grpc_listen_port: <port #>
positions:
filename: /tmp/positions.yaml
clients:
- url: 'http://10.128.15.231:3100/loki/api/v1/push'
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: log_export
__path__: /path/to/log/file.log
host: host-name
pipeline_stages:
- match:
selector: '{host="host-name"} |= "error"'
action: keep
it works fine until I add the |= "error
I've also tried something like this:
pipeline_stages:
- match:
selector: '{host="host-name"}'
stages:
- regex:
expression: '.*error.*'
which also throws config errors. it seems like this should be relatively simple, but the documentation is really not clear...thanks in advance for any assistance!

ingress variables syntax from values.yaml

I have a simple api that I've been deploying to K8s as a NodePort service via Helm. I'm working to add an ingress to the Helm chart but I'm having some difficulty getting the variables correct
values.yaml
ingress:
metadata:
name: {}
labels: {}
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: "testapi.local.dev"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: {}
port:
number: 80
templates/ingress.yaml, showing only the spec section where I'm having issues.
spec:
rules:
{{- range .Values.ingress.spec.rules }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path | quote }}
pathType: {{ .pathType | quote }}
backend:
service:
name: {{ include "testapi.service.name" . }}
port:
{{- range $key, $value := (include "testapi.deployment.ports" . | fromYaml) }}
number: {{ .port }}
{{- end}}
{{- end}}
{{- end}}
When running helm template it just leaves these values blank and I'm not sure where the syntax is wrong. Removing the {{- range .paths }} and the following .path and .pathType and replacing them with the value corrects the issue
spec:
rules:
- host: "testapi.local.dev"
http:
paths:
Comments revealed I should be using {{- range .http.paths }}.

How to avoid translate some `{{` of the helm chart?

I want to put the following CRD into helm chart, but it contains go raw template. How to make helm not translate {{ and }} inside rawTemplate. Thanks for your response.
https://github.com/kubeflow/katib/blob/master/examples/random-example.yaml
apiVersion: "kubeflow.org/v1alpha1"
kind: StudyJob
metadata:
namespace: katib
labels:
controller-tools.k8s.io: "1.0"
name: random-example
spec:
studyName: random-example
owner: crd
optimizationtype: maximize
objectivevaluename: Validation-accuracy
optimizationgoal: 0.99
requestcount: 4
metricsnames:
- accuracy
workerSpec:
goTemplate:
rawTemplate: |-
apiVersion: batch/v1
kind: Job
metadata:
name: {{.WorkerId}}
namespace: katib
spec:
template:
spec:
containers:
- name: {{.WorkerId}}
image: katib/mxnet-mnist-example
command:
- "python"
- "/mxnet/example/image-classification/train_mnist.py"
- "--batch-size=64"
{{- with .HyperParameters}}
{{- range .}}
- "{{.Name}}={{.Value}}"
{{- end}}
{{- end}}
restartPolicy: Never
In the Go template language, the expression
{{ "{{" }}
will expand to two open curly braces, for cases when you need to use Go template syntax to generate documents in Go template syntax; for example
{{ "{{" }}- if .Values.foo }}
- name: FOO
value: {{ "{{" }} .Values.foo }}
{{ "{{" }}- end }}
(In a Kubernetes Helm context where you're using this syntax to generate YAML, be extra careful with how whitespace is handled; consider using helm template to dump out what gets generated.)

Helm yaml keys from values.yaml

I want to make a yaml KEY (not the value) dynamically.
In my values.yaml
failoverip1: 0.0.0.0` (<- this is only a demo IP)
In my templates/configmap.yaml I have this:
apiVersion: v1
kind: ConfigMap
metadata:
name: vip-configmap
data:
{{- .Values.failoverip1 -}}: {{ .Release.Namespace -}}/{{- .Values.target -}}
^^^^^^^^^^^^^^^^^^^^^----> here should be an IP address from values.yaml
{{ .Release.Namespace -}}/{{- .Values.target -}} renders successfully.
But if I add {{- .Values.failoverip1 -}} to the key part, it renders nothing.
(Nothing means, the whole data: block, does not get rendered.
This is the error message when I run helm install --name hetzner-failover .
Error: YAML parse error on hetzner-failover/templates/configmap-ip.yaml: error converting YAML to JSON: yaml: line 4: mapping values are not allowed in this context
Is it not allowed to make a
key dynamic?
If not, how to drive around that?
Here is the repo I am talking of:
https://github.com/exocode/helm-charts/blob/master/hetzner-failover/templates/configmap-ip.yaml
The error seems to be, that the leading - got cut.
So the correct way is to remove that minus:
Before:
{{- .Values.failoverip1 | indent 2 -}}
After:
{{ .Values.failoverip1 | indent 2 -}}
The yaml is now:
apiVersion: v1
kind: ConfigMap
metadata:
name: vip-configmap
data:
{{ .Values.failoverip1 | indent 2 -}}: {{ .Release.Namespace -}}/{{- .Values.target -}} # add your config map here. must map the base64 encoded IP in secrets.yaml
And the rendered result is:
kubectl get configmap -o yaml
apiVersion: v1
items:
- apiVersion: v1
data:
0.0.0.0: default/nginx# add your config map here. must map the base64 encoded
IP in secrets.yaml
kind: ConfigMap

Resources