Using shell script to parse lines from a consul template file - bash

As you can see the in the template file below , I have few variables declared (like MYSQL_ACCOUNT_PASSWORD , MYSQL_ACCOUNT_USERNAME etc) between "if" and "end" and I am looking forward to write a shell script to get all the variables in such file.
With the format of the consul template using "Golang's text/template package", I am afraid that basic tools like cut,sed or awk wont do my job. How can I do this.
{{- template "findService" "mysql-master-acc" }}
{{- if scratch.Key "mysql-master-acc-service" }}
{{- $mysql_account_db_name := printf "service/%s/mysql-master-acc/%s/config/acc_mysql_db_name" $dc ( scratch.Get "mysql-master-acc-tag" ) | key }}
{{ range scratch.Get "mysql-master-acc-service"}}
MYSQL_ACCOUNT_URL=jdbc:mysql://{{- .Address }}:{{- .Port }}/{{ $mysql_account_db_name }}?useSSL=false&autoReconnect=true&serverTimezone=America/Chicago&rewriteBatchedStatements=true
{{- end }}
{{- if eq (env "SECRETS_BACKEND") "vault" }}
{{ $vault_secret_path := printf "secret/service/%s/kos-mysql-acc/%s" $dc ( scratch.Get "mysql-master-acc-tag" ) -}}
{{ with printf "%s/%s" $vault_secret_path $mysql_account_db_name | secret -}}
MYSQL_ACCOUNT_PASSWORD={{ .Data.password }}
MYSQL_ACCOUNT_USERNAME={{ .Data.username }}
{{- end }}
{{- else if eq ( env "SECRETS_BACKEND" ) "consul" }}
{{- $secretPath := printf "secret/%s/mysql-master-acc/%s/acc_mysql_db_user_password" $dc ( scratch.Get "mysql-master-acc-tag" ) }}
{{- $keyPath := printf "service/%s/mysql-master-acc/%s/config/acc_mysql_db_user" $dc ( scratch.Get "mysql-master-acc-tag" ) }}
MYSQL_ACCOUNT_PASSWORD={{ key $secretPath }}
MYSQL_ACCOUNT_USERNAME={{ key $keyPath }}
{{- end }}
{{- end }}

Could you just use grep?
To get all lines that don't start with {{ you could use
grep -v "^{{" inputfile
If you want all lines starting with MYSQL..., use
grep "^MYSQL" inputfile
In this case, these both return:
MYSQL_ACCOUNT_URL=jdbc:mysql://{{- .Address }}:{{- .Port }}/{{ $mysql_account_db_name }}?useSSL=false&autoReconnect=true&serverTimezone=America/Chicago&rewriteBatchedStatements=true
MYSQL_ACCOUNT_PASSWORD={{ .Data.password }}
MYSQL_ACCOUNT_USERNAME={{ .Data.username }}
MYSQL_ACCOUNT_PASSWORD={{ key $secretPath }}
MYSQL_ACCOUNT_USERNAME={{ key $keyPath }}

Related

Helm template for Vault annotation

I'm trying to nest one template into another, namely I have a k8s deployment.yaml which should be templated by helm. However, it is already using Vault template in annotation field responsible for injecting secrets into the pod.
Without Helm template it looks like this:
vault.hashicorp.com/agent-inject-template-.env: |
{{- with secret (print "envs/data/test") -}}{{- range $k, $v := .Data.data -}}
{{ $k }}={{ $v }}
{{ end }}{{- end -}}
Now, I would like to make a template for Helm and replace path envs/data/test with value coming from Helm values. So I've tried to use it like that:
vault.hashicorp.com/agent-inject-template-.env: |
{{- with secret (print {{ .Values.path }}) -}}{{- range $k, $v := .Data.data -}}
{{ $k }}={{ $v }}
{{ end }}{{- end -}}
This however does not work meaning {{ .Values.path }} is outputed as is, as a string.
How can I template over another template and resolve {{ .Values.path }} as a variable?

Check if a key/value exists in helm chart yaml and add a value to it if exists, otherwise create a new key-pair value

Let's say that I have the following structure in one helm template file
{{- define "deployAnnotations" -}}
"key1" : "value1",
"key2" : "value2"
"ip_range": "20.20.20.20/32"
{{- end -}}
and the above are included in the deployment.yaml file of my helm chart
template:
metadata:
annotations:
{{ include "podAnnotations" . | nindent 8 }}
Now I want to add one more annotation that also define in a template file the following one:
{{- define "ipdeployAnnotations" -}}
ip_range: 10.10.10.10/32
{{- end -}}
My question is how can I search if ip_range exists in the podAnnotations structure. If exists I just want to add the 10.10.10.10/32 to the existing line , otherwise I would like to create a new one.
I imagine something like below but in operator is not working for me , and also I do not know how to add the additional value to ip_range
template:
metadata:
annotations:
{{ include "podAnnotations" . | nindent 8 }}
{{ - if "ip_range" in "podAnnotations" }}
// do something in order to have the following result
"ip_range": "20.20.20.20/32", "10.10.10.10/32"
{{- else }}
{{ include "ipdeployAnnotations" . | nindent 8 }}
{{- end }}
You may convert it to a struct first, and then use it in the way of manipulating dictionary.
templates/_helper.tpl
{{- define "deployAnnotations" -}}
"key1" : "value1"
"key2" : "value2"
"ip_range": "20.20.20.20/32"
{{- end -}}
{{- define "ipdeployAnnotations" -}}
ip_range: "10.10.10.10/32"
{{- end -}}
templates/xxx.yaml
...
{{- $da := (include "deployAnnotations" . | fromYaml) }}
{{- $ida := (include "ipdeployAnnotations" . | fromYaml) }}
{{- if (hasKey $da "ip_range") }}
{{- $_ := set $da "ip_range" (printf "%s, %s" $da.ip_range $ida.ip_range) }}
{{- else }}
{{- $_ := set $da "ip_range" $ida.ip_range }}
{{- end }}
{{- range $k, $v := $da }}
{{ $k }}: {{ $v }}
{{- end }}
...
output
ip_range: 20.20.20.20/32, 10.10.10.10/32
key1: value1
key2: value2

Remove new line at the beginning of the file - Go template

I'm using fallowing loop to generate credentials file in Hashicorp Vault. All works well but I'm getting one new line at the beginning of the file. How can I remove it?
vault.hashicorp.com/agent-inject-template-credentials.txt: |
{{- with secret (print "secret/data/test/config") }}{{- range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{- end }}{{- end }}
Input: map[test1:test1 test2:test2 test3:test3]
Current output:
// one empty line at the beginning
test1: test1
test2: test2
test3: test3
Your template contains a newline before rendering the elements, use the - sign to get rid of that:
{{- with secret (print "secret/data/test/config") }}{{- range $k, $v := .Data.data -}}
{{ $k }}: {{ $v }}
{{- end }}{{- end }}
Note the added - sign at the end of the first line.
This of course will render each pair on the same line. Leave the newline at the end of rendering the elements by removing the - sign from the beginning of the last line:
{{- with secret (print "secret/data/test/config") }}{{- range $k, $v := .Data.data -}}
{{ $k }}: {{ $v }}
{{ end }}{{- end }}
Alternatively you can move the first added - sign to the beginning of the second line:
{{- with secret (print "secret/data/test/config") }}{{- range $k, $v := .Data.data }}
{{- $k }}: {{ $v }}
{{ end }}{{- end }}
These templates will output (no first empty line):
test1: test1
test2: test2
test3: test3
Try it on the Go Playground.

Helm Template predifined Array Iterate for configmap

Hi we have listenerProcessor array in values.yaml like below
listenerProcessor:
- name: "nbi-sbi"
port: 99
allowedPath:
- "/sbi/test"
- "/sbi/test123"
for this we need to merge our predefined set of listenerProcessor array which is not exposed through values.yaml , which something like this below
listenerProcessor:
- name: nbinorc
port: 9910
allowedPath:
- /nbi/norc/tmf641/v4/orders/
- /nbi/norc/tmf645/v4/serviceQualification
- name: nbiuiv
port: 9920
allowedPath:
- /nbi/uiv/tmf640/v4/services/
so to merge this two as single property and iterate it over , defined a variable in _helper.tpl
{{- define "nifi.listener-processor.spec" -}}
listenerProcessor:
- name: nbinorc
port: 9910
allowedPath:
- /nbi/norc/tmf641/v4/orders/
- /nbi/norc/tmf645/v4/serviceQualification
- name: nbiuiv
port: 9920
allowedPath:
- /nbi/uiv/tmf640/v4/services/
{{ toYaml .Values.listenerProcessor.ports }}
{{- end -}}
Now i need to iterate it over .name & .port in NOTES.txt but its giving error " <.name>: can't evaluate field name in type interface {}".
In NOTES.txt i have this
{{- $listenerProcessor := (include "nifi.listener-processor.spec" . ) }}
{{- range $index, $element := fromYaml $listenerProcessor }}
{{ .name }}
{{ .port }}
{{- end }}
when i just print $element & $index those values its showing below output
NOTES:
$NIFI_CONF_DIRECTORY_PATH/extensions directory"
[map[allowedPath:[/nbi/norc/tmf641/v4/orders/ /nbi/norc/tmf645/v4/serviceQualification] name:nbinorc port:9910] map[allowedPath:[/nbi/uiv/tmf640/v4/services/] name:nbiuiv port:9920] map[allowedPath:[/sbi/test /sbi/test123] name:nbi-sbi port:99]]
listenerProcessor
Got that Working with below code, we need to have another variable and than do pass it for range over
{{- $listenerProcessor := (include "nifi.listener-processor.spec" . ) }}
{{- $test := fromYaml $listenerProcessor }}
{{- range $index, $element := $test.listenerProcessor }}
{{ .name }} {{ .port }} {{ .allowedPath }}
{{- end }}

Convert YAML string to dict in Helm template

I am creating a chart for a project that has a binary that when executed generates a configuration file in YAML format that looks like this:
---
PARAM_1: value1
PARAM_2: value2
My chart needs to read this file and and load all of its values into environment variables in a container, so I created a variable config in my values.yaml file and when the chart is installed I am passing the file content using --set-file:
helm install <CHART> --set-file config=/path/to/yaml/config/file
Next I create a ConfigMap withe the value of .Values.config:
apiVersion: v1
kind: ConfigMap
metadata:
...
data:
{{ .Values.config }}
The problem I am having is that I need to do two things with values of config:
prefix all keys with a predefined value (so in the example above I would MY_APP_PARAM_1 as key)
make sure the values are all string, otherwise the ConfigMap will fail
How can I parse the value of .Values.config in my template as a dict so that I can use a range loop do these changes?
In the end I was able to do something like this:
{{ $lines := splitList "\n" .Values.config -}}
{{- range $lines }}
{{- if not (. | trim | empty) -}}
{{- $kv := . | splitn ":" 2 -}}
{{ printf "MY_APP_%s: %s" $kv._0 ($kv._1 | trim | quote) | indent 2 }}
{{ end -}}
{{- end -}}
I had a hard time getting the {{- vs {{ right, and helm install --debug --dry-run . help a lot in this part.
It's kind of messy, so I would be very interested in seeing if anyone has a better solution.
If You need to work with more tricky YAML (map in a map) You can use this:
List Handler (list to comma separated string)
{{- define "helm-toolkit.utils.joinListWithComma" -}}
{{- $local := dict "first" true -}}
{{- range $k, $v := . -}}{{- if not $local.first -}}, {{ end -}}{{ $v -}}{{- $_ := set $local "first" false -}}{{- end -}}
{{- end -}}
Convert Yaml dict to the properties-like look
{{- define "utils.yaml2properties" }}
{{- $yaml := . -}}
{{- range $key, $value := $yaml }}
{{- if kindIs "map" $value -}}
{{ $top:=$key }}
{{- range $key, $value := $value }}
{{- if kindIs "map" $value }}
{{- $newTop := printf "%s.%s" $top $key }}
{{- include "utils.yaml2properties" (dict $newTop $value) }}
{{- else if kindIs "slice" $value }}
{{ $top }}.{{ $key }}={{ include "helm-toolkit.utils.joinListWithComma" $value }}
{{- else }}
{{ $top }}.{{ $key }}={{ $value }}
{{- end }}
{{- end }}
{{- else if kindIs "slice" $value }}
{{ $key }}={{ include "helm-toolkit.utils.joinListWithComma" $value }}
{{- else }}
{{ $key }}={{ $value }}
{{- end }}
{{- end }}
{{- end }}
Use (example from K8s configMap)
{{- define "config.yaml" }}
vault.pki:
enabled: {{.Values.vault.pki.enabled}}
role: idverify
common-name: idverify
role:
map:
list:
- one
- two
{{- end }}
data:
bootstrap.properties: |
{{ (include "utils.yaml2properties" (include "config.yaml" . | fromYaml )) | indent 4 }}

Resources