Editing inserting values in YAML based on criteria using a bash script - bash

A YAML file as follows:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'node-exporter'
relabel_configs:
- source_labels: [__address__]
target_label: instance
regex: '([^:]+)(:[0-9]+)?'
replacement: '${1}'
static_configs:
- labels:
type: 'YYY'
group: 'Bar'
targets:
- '192.168.1.134:80'
- '192.168.1.146:80'
- labels:
type: 'YYY'
group: 'Foo'
targets:
- '192.168.2.136:80'
- labels:
type: 'ZZZ'
group: 'Foo'
targets:
- '192.168.2.100:80'
I'd like to use a bash script to add the "targets" to the YAML based on "type" and "group". For example:
./add_host.sh 192.168.1.17:80 YYY Bar
Added bonus for deleting entries :)
Things I've tried (and failed):
Using jq;
Tried using sed to insert values, but I have trouble in finding the right section to insert it to (Block parsing).

You don't mean that you want to parse the YAML file with bash, do you? That would be tedious and error-prone, not to mention slow to execute. You must mean you want a command-line utility to manipulate the file.
What you want is something that parses the YAML into a data structure. If I were doing it, I'd try the Python PyYAML module. It represents the whole file as a nested data structure. Read it in, change the parts you want to change, and write it out.

Related

Promtail: How to remove timestamps from filenames?

I have a simple problem:
My logfiles have timestamps in their name, i.e.:
/var/log/html/access-2021-11-27.log
/var/log/html/access-2021-11-28.log
/var/log/html/access-2021-11-29.log
Promtail is scraping this but does not "see" that access-2021-11-28.log is a continuation of access-2021-11-27.log. So it will "detect" a log file access-2021-11-28.log on the 28th and not show the access-2021-11-27.log anymore. I would want to see just "access.log" with data for several days.
I would assume this should be a well-known scenario, but I cannot find anything on this on the Internet.
The only way is to change log configuration of the application which is generating the logs, to use a unique access.log instead of the schema of the access-xxxx-xx-xx.log files. Unfortunately, this is not always possible.
But...
The old files can still be shown, it only depends on the time range used. Here is an example:
You can use regular expressions to perform the query, like in this example:
{filename=~".*JIRA_INSTALL/logs/access_log\\..*"}
If you want to statically override the filename field you can so something as simple as this:
scrape_configs:
- job_name: system
static_configs:
- labels:
job: remotevarlogs
__path__: /var/log/html/access-*.log
pipeline_stages:
- match:
selector: '{job="remotevarlogs"}'
stages:
- static_labels:
filename: '/var/log/html/access.log'
For those of you searching how to dynamically change the filepath prefix. For example, I'm using FreeBSD jails to nullfs mount my logs from other jails into a promtail jail. I don't want the local mount location (/mnt/logs/<hostname>) to show up as part of the path. Mounting shared folder could similarly be done with NFS or Docker.
scrape_configs:
- job_name: system
static_configs:
- labels:
job: remotevarlogs
__path__: /mnt/logs/*/**/*.log
pipeline_stages:
- match:
selector: '{job="remotevarlogs"}'
stages:
- regex:
source: filename
expression: "/mnt/logs/(?P<host>\\S+?)/(?P<relativepath>\\S+)"
- template:
source: host
template: '{{ .Value }}.mylocaldomain.com'
- template:
source: relativepath
template: '/var/log/{{ .Value }}'
- labels:
host:
filename: relativepath
- labeldrop:
- job
- relativepath
/etc/fstab for loki jail to pass-in /var/log/ directory from the grafana jail:
# Device Mountpoint FStype Options Dump Pass#
...
/jails/grafana/root/var/log/ /jails/loki/root/mnt/logs/grafana nullfs ro,nosuid,noexec 0 0
...
Now when I browse the logs, instead of seeing /mnt/logs/grafana/nginx/access.log, I see /var/log/nginx/access.log from grafana.mylocaldomain.com.

Insert variables/constants in YAML file

Let's say I want to avoid inserting the same value multiple times in a YAML file:
- name: region_1
inputs:
tag_name: constant_tag_name
aws_region: us-east-1
- name: region_2
inputs:
tag_name: constant_tag_name
aws_region: us-west-2
...
So in the above example, I would like to define a variable somewhere above all sections with the value constant_tag_name and mention the variable everywhere the value is the same. If the name is to change, I would want to change it at the top and have it be reflected everywhere. I looked at YAML aliases but they appear to be for a code section, whereas here I only have one variable. Can you guide me?
You should use an alias, like so:
- const_tag: &CT "constant_tag_name"
- name: region_1
inputs:
tag_name: *CT
aws_region: us-east-1
- name: region_2
inputs:
tag_name: *CT
aws_region: us-west-2
Try it out here: yaml-online-parser.appspot.com

Ansible replace text in file

So ive been tasked with replaceing zabbix server. To do so i have to modify zabbix_agent file in all server and there are many. Tho in this job is the first time i see ansible so i need some help. And i am using ansible-playbook.
In zabbix_agentd.conf file there is the old zabbix conf:
HostMetadata=Linux
PidFile=/var/run/zabbix/zabbx_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
Server=zabbix.company.com
ServerActive=zabbix.company.com
HostnameItem=system.hostname
Include=/etc/zabbix_agentd.d/
Now i need to replace "Server" and "ServerActive" to "zabbix2.company.com"
I have tried various codes from this page to work for my needs but so far it has failed. No clue what im doing wrong
Try this one
- lineinfile:
path: /etc/zabbix_agentd.conf
regexp: '^\s*{{ key }}\s*=(.*)$'
line: '{{ key }}={{ value }}'
notify: reload zabix
loop:
- {key: 'Server', value: 'zabbix2.company.com'}
- {key: 'ServerActive', value: 'zabbix2.company.com'}
Notes
Path is required; probably /etc/zabbix_agentd.conf ?
It is not necessary to search the white-space \s* in regexp. However, it would match and fix potential spaces in the configuration.
Create and notify a handler reload zabix when anything changed. See Handlers: Running Operations On Change.
Take a look at Zabix modules.
I have manged to solve this issue using this code.
---
tasks:
- name: 'replace line'
lineinfile:
dest: /etc/zabbix/zabbix_agentd.conf
regexp: '^(.*)Server=zabbix.company.com(.*)$'
line: 'Server=zabbix2.company.com'
backrefs: yes

How create correct template file for ansible role

I need create some configuration file in yml wtith ansible role. This example what i want:
filebeat.inputs:
- type: log
paths:
- /var/log/system.log
- /var/log/wifi.log
In ansible dirctory structue in Templates dir i have file with such text:
filebeat.inputs:
- type: {{filebeat_input.type}}
{{ filebeat_input.paths | to_yaml}}
In Directory defaults i have such main.yml:
filebeat_create_config: true
filebeat_input:
type: log
paths:
- "/var/log/*.log"
- "/var/somelog/*.log"
When im run ansible-playbook with this role im got this:
filebeat.inputs:
- type: log
[/var/log/*.log, /var/sdfsadf]
Where i`m wrong ? what and where i need change to get exactly what i whant (see example above).
Thanks for any help !
filebeat_input.paths is a list, so when you pass it to to_yaml you get a list. You just need to structure your YAML template so that you're putting the list in the right place, e.g:
filebeat.inputs:
- type: {{filebeat_input.type}}
paths: {{ filebeat_input.paths | to_yaml}}
Note that:
paths: [/var/log/*.log, /var/sdfsadf]
Is exactly equivalent to:
paths:
- /var/log/*.log
- /var/sdfsadf

Retrieving contents of j2 template file on stdout

I'm attempting to use Ansible to better manage my Kubernetes configmaps in a multienvironment project (dev, stage, and prod). I've generalized each of the config maps as j2 templates, and I'll override the variables depending on how they might change in different environments (so that they aren't duplicated three times for basically the same file).
My playbook currently looks something like this:
---
- hosts: localhost
vars_files:
- "vars/{{ env }}.yml"
tasks:
- name: Generate YAML from j2 template
template:
src: templates/foo.j2
dest: output/foo.yml
And this has been working great for testing so far. However, I'm at the point where I want to incorporate this into my already existing Jenkins CI/CD, but I'm having trouble understanding how it might work with what I am doing currently.
After generating what is basically a Kuberenets ConfigMap from the j2, I'll somehow do this within Jenkins:
kubectl apply -f <yaml>
However, the playbook is creating a YAML file every time I run it, and I am wondering if there is an alternative that would allow me to pipe the contents of the YAML file or somehow retrieve it from stdout.
Basically, I want to evaluate the template and retrieve it without necessarily creating a file.
If I do this, I could do something like the following:
echo result | kubectl apply -f -
where result of course is the contents of the YAML file that results after the templating, and the short dash after the f flag specifies Kubernetes to use the process' stdout.
Sorry for so much explaining, I can clarify anything if needed.
I would like to retrieve the result of the template, and pipe it into that command, such as "echo result | kubectl apply -f -"
In which case, you'd use the stdin: parameter of the command: module:
- name: generate kubernetes yaml
command: echo "run your command that generates yaml here"
register: k8s_yaml
- name: feed the yaml to kubectl apply
command: kubectl apply -f -
stdin: '{{ k8s_yaml.stdout }}'
It isn't super clear what the relationship is in your question between the top part, dealing with the template:, and the bottom part about apply -f -, but if you mean "how can I render a template to a variable, instead of a file?" the the answer is the template lookup plugin:
- name: render the yaml
set_fact:
k8s_yaml: '{{ lookup("template", "templates/foo.j2") }}'
- name: now feed it to apply
command: kubectl apply -f -
stdin: '{{ k8s_yaml }}'
You've got a couple options here. I usually try to stay away from shelling out to command wherever possible. Check out the k8s module in ansible. Note that as long as state is present ansible will patch your object.
- name: Apply your previously generated configmap if you so choose.
k8s:
state: present
definition: "{{ lookup('file', '/output/foo.yml') }}"
Or even better, you could just directly create the configmap
- name: Create the configmap for {{ env }}
k8s:
state: present
definition:
apiVersion: v1
kind: ConfigMap
metadata:
name: ConfigMap
namespace: "{{ foo_namespace }}"
labels:
app: bar
environment: "{{ bizzbang }}"

Resources