Passing YAML content to a command in a bash function - bash

I'm currently writing a bash script and struggling with something that looked fairly simple at first.
I'm trying to create a function that calls a kubectl (Kubernetes) command. The command is expecting the path to a file as an argument although I'd like to pass the content itself (multiline YAML text). It works in the shell but can't make it work in my function. I've tried many things and the latest looks like that (it's just a subset of the the YAML content):
#!/bin/bash
AGENT_NAME="default"
deploy_agent_statefulset() {
kubectl apply -n default -f - $(cat <<- END
kind: ConfigMap
metadata:
name: $AGENT_NAME
apiVersion: v1
data:
agent.yaml: |
metrics:
wal_directory: /var/lib/agent/wal
END
)
}
deploy_agent_statefulset
The initial command that works in the shell is the following.
cat <<'EOF' | NAMESPACE=default /bin/sh -c 'kubectl apply -n $NAMESPACE -f -'
kind: ConfigMap
...
I'm sure I m doing a lot of things wrong - keen to get some help
Thank you.
name: grafana-agent

In your function, you didn't contruct stdin properly :
#!/bin/bash
AGENT_NAME="default"
deploy_agent_statefulset() {
kubectl apply -n default -f - <<END
kind: ConfigMap
metadata:
name: $AGENT_NAME
apiVersion: v1
data:
agent.yaml: |
metrics:
wal_directory: /var/lib/agent/wal
END
}
deploy_agent_statefulset

this one should work:
#!/bin/bash
AGENT_NAME="default"
deploy_agent_statefulset() {
cat << EOF | kubectl apply -n default -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: $AGENT_NAME
data:
agent.yaml: |
metrics:
wal_directory: /var/lib/agent/wal
EOF
}
deploy_agent_statefulset
To point out what is wrong in your yaml which are all indentations,
you don't need to add the indentations in the beginning
name goes under metadata, so it needs to be intended.
agent.yaml is the key, for the data in the ConfigMap, so it needs to be intended as well

Related

Proper way to directly pass yaml content to kubectl patch?

Example of functional kubectl patch command:
# kubectl patch storageclass local-path \
-p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'
In certain cases the patched key/values are too numerous, so is recommended to use a file instead:
# kubectl patch storageclass local-path --patch-file=file.yaml
I would like to use an alternative of this format, which returns an error:
cat << 'EOF' | kubectl patch storageclass local-path --patch-file -
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: false
EOF
error: unable to read patch file: open -: no such file or directory
My goal is to use a dynamic way of pushing the patch data, without creating a file. What would be the correct format? Thank you.
Update: Based on provided documentation, I tried this format:
cat << 'EOF' | kubectl patch storageclass local-path --type=merge -p -
{
"metadata": {
"annotations": {
"storageclass.kubernetes.io/is-default-class": "false"
}
}
}
EOF
Error from server (BadRequest): json: cannot unmarshal array into Go value of type map[string]interface {}
Or:
kubectl patch storageclass local-path --type=merge -p << 'EOF'
{
"metadata": {
"annotations": {
"storageclass.kubernetes.io/is-default-class": "false"
}
}
}
EOF
error: flag needs an argument: 'p' in -p
What would be the correct format? I'm trying to avoid a very long line and keep a nice readable format.
If you look at the documentation of kubectl patch help that it is not a supported feature to pass the patch as you are trying to do because you either need to pass the patch as a json or from the file contains that contians the data.
You can pass something like this, but still you need to clean up the file you created here (auto.yaml).
$ cat <<EOF | echo "metadata:
> labels:
> app: testapp "> auto.yaml | kubectl patch pod pod-name --patch-file=auto.yaml
> EOF
For more information about EOF refer to the Here Document section in this document
For Updated question:
You are actually missing the ' quotation before starting the json and don't give - after -p. Give a try like this, this is working in our environment
$ cat <<EOF | kubectl patch deployments nginx --type=merge --patch '{
> "metadata":
> {
> "labels":
> {
> "check": "good"
> }
> }
> }'
> EOF

Shell variables in yq 4 using env()

I want to build a pipeline function that replaces a value in a yaml file. For that I want to make both the
pattern and the replacement value variable. I have seen the env-variables-operators article in the yq docs, however I cannot find the relevant section.
I have a yaml file with the following content:
---
spec:
source:
helm:
parameters:
- name: "image.tag"
value: "1.0.0"
I now want to build a pipeline function that will replace the value of the value key in the yaml.
I can do so with:
$ yq '.spec.source.helm.parameters[0].value = "2.0.0"' myyaml.yml
---
spec:
source:
helm:
parameters:
- name: "image.tag"
value: "2.0.0"
Now I want to make this command customizable.
What works:
$ VALUE=3.0.0
$ replacement=$VALUE yq '.spec.source.helm.parameters[0].value = env(replacement)' myyaml.yml
---
spec:
source:
helm:
parameters:
- name: "image.tag"
value: "3.0.0"
What doesn't work
$ VALUE=3.0.0
$ PATTERN=.spec.source.helm.parameters[0].value
$ replacement=$VALUE pattern=$PATTERN yq 'env(pattern) = env(replacement)'
spec:
source:
helm:
parameters:
- name: "image.tag"
value: "1.0.0"
I have also tried to use strenv and wrapping the replacement pattern in quotes, but it is not working.
Can anyone help me with the correct syntax?
You can import data with env but not code. You could inject it (note the changes in the quoting), but this is bad practice as it makes your script very vulnerable:
VALUE='3.0.0'
PATTERN='.spec.source.helm.parameters[0].value'
replacement="$VALUE" yq "${PATTERN} = env(replacement)" myyaml.yml
---
spec:
source:
helm:
parameters:
- name: "image.tag"
value: "3.0.0"
Better practice would be to import the path in a form that is interpretable by yq, e.g. as an array and using setpath:
VALUE='3.0.0'
PATTERN='["spec","source","helm","parameters",0,"value"]'
replacement="$VALUE" pattern="$PATTERN" yq 'setpath(env(pattern); env(replacement))' myyaml.yml

YAML: from string to multi-line?

I have a yaml which looks like this.
Is there a way to get the "Corefile" value to multi-line?
apiVersion: v1
data:
Corefile: ".:53 {\n rewrite name regex (.*)\\.test\\.io {1}.default.svc.cluster.local \n errors\n health {\n lameduck 5s\n }\n ready\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n pods insecure\n fallthrough in-addr.arpa ip6.arpa\n ttl 30\n }\n prometheus :9153\n forward . /etc/resolv.conf {\n max_concurrent 1000\n }\n cache 30\n loop\n reload\n loadbalance\n}\n"
kind: ConfigMap
metadata:
creationTimestamp: "2022-02-25T12:36:15Z"
name: coredns
namespace: kube-system
resourceVersion: "14874"
uid: dc352ab8-1e43-4663-8c6a-0d404f4bb4f3
I tried yq -P, but this did not help
The basic command is this (e can be omitted in newer versions):
yq e '.data.Corefile style="literal"' test.yaml
However this will not work in your case, since YAML says that trailing whitespace is ignored, and thus you cannot have data with trailing whitespace formatted as literal block scalar. Relevant part of your data is:
default.svc.cluster.local \n
^
This space does not seem to be relevant, so you can write additional code to remove it:
yq e '.data.Corefile |= sub("\s*(\n)", "${1}") | .data.Corefile style="literal"' test.yaml
(There is a curious bug where I cannot substitute with "\n" directly as that will create "\\n" in the data for some reason, so I use the captured newline instead.)
Result:
apiVersion: v1
data:
Corefile: |
.:53 {
rewrite name regex (.*)\.test\.io {1}.default.svc.cluster.local
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2022-02-25T12:36:15Z"
name: coredns
namespace: kube-system
resourceVersion: "14874"
uid: dc352ab8-1e43-4663-8c6a-0d404f4bb4f3

Split string and extract variables with shell script

Question
Given this single-line string:
PG_USER=postgres PG_PORT=1234 PG_PASS=icontain=and*symbols
What would be the right way to assign each value to its designated variable so that I can use it afterward?
Context
I'm parsing the context of a k8s secret within a CronJob so that I can periodically call a Stored Procedure in our Postgres database.
To do so, I plan on using:
PG_OUTPUT_VALUE=$(PGPASSWORD=$PG_PASSWD psql -qtAX -h $PG_HOST -p $PG_PORT -U $PG_USER -d $PG_DATABASE -c $PG_TR_CLEANUP_QUERY)
echo $PG_OUTPUT_VALUE
The actual entire helm chart I'm currently trying to fix looks like this:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: {{ template "fullname" $ }}-tr-cleanup-cronjob
spec:
concurrencyPolicy: Forbid
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
volumes:
- name: postgres
secret:
secretName: {{ template "fullname" $ }}-postgres
containers:
- name: {{ template "fullname" $ }}-tr-cleanup-pod
image: postgres:12-alpine
imagePullPolicy: Always
env:
- name: PG_PROPS
valueFrom:
secretKeyRef:
name: {{ template "fullname" $ }}-postgres
key: postgres.properties
command:
- /bin/sh
- -c
- echo "props:" && echo $PG_PROPS && PG_USER=$(grep "^PG_USER=" | cut -d"=" -f2-) && echo $PG_USER && PG_TR_CLEANUP_QUERY="SELECT something FROM public.somewhere;" && echo $PG_TR_CLEANUP_QUERY && PG_OUTPUT_VALUE=$(PGPASSWORD=$PG_PASSWD psql -qtAX -h $PG_HOST -p $PG_PORT -U $PG_USER -d $PG_DATABASE -c $PG_TR_CLEANUP_QUERY) && echo PG_OUTPUT_VALUE
volumeMounts:
- name: postgres
mountPath: /etc/secrets/postgres
Current approach
As you can see, I'm currently using:
PG_USER=$(grep "^PG_USER=" | cut -d"=" -f2-)
That is because I initially thought the secret would be output on multiple lines, but it turns out that I was wrong. The echo $PG_USER displays an empty string.
The bash declare command is appropriate here, and is safer than eval.
Suppose the input contains something potentially malicious
line='PG_USER=postgres PG_PORT=1234 PG_PASS=icontain=and*symbols`ls`'
I'm assuming none of the values contain whitespace. Let's split that string
read -ra assignments <<< "$line"
Now, declare each one
for assignment in "${assignments[#]}"; do declare "$assignment"; done
Everywhere we examine the input, we maintain double quotes.
Let's see what we ended up with:
$ declare -p PG_USER PG_PORT PG_PASS
declare -- PG_USER="postgres"
declare -- PG_PORT="1234"
declare -- PG_PASS="icontain=and*symbols\`ls\`"
Option 1
This function can be reused to assign each variable individually:
extract() {
echo "$INPUT" | grep -o "$1=.*" | cut -d" " -f1 | cut -d"=" -f2- ;
}
And to use it:
PG_USER=$(extract PG_USER)
PG_PORT=$(extract PG_PORT)
PG_PASS=$(extract PG_PASS)
Option 2
Another potential solution, with a security concern, is to simply use:
eval "$INPUT"
It should only be used if you have validated the input.
Contextual complete answer
And because I've presented the k8s context in the question, here is the answer as plugged into that solution.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: {{ template "fullname" $ }}-cronjob
spec:
concurrencyPolicy: Forbid
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
volumes:
- name: postgres
secret:
secretName: {{ template "fullname" $ }}-postgres
containers:
- name: {{ template "fullname" $ }}-cronjob-pod
image: postgres:12-alpine
imagePullPolicy: Always
env:
- name: PG_PROPS
valueFrom:
secretKeyRef:
name: {{ template "fullname" $ }}-postgres
key: postgres.properties
command:
- /bin/sh
- -c
- >-
extract() { echo "$PG_PROPS" | grep -o "$1=.*" | cut -d" " -f1 | cut -d"=" -f2- ; } &&
export PGHOST=$(extract PG_HOST) &&
export PGPORT=$(extract PG_PORT) &&
export PGDATABASE=$(extract PG_DATABASE) &&
export PGUSER=$(extract PG_USER) &&
PG_SCHEMA=$(extract PG_SCHEMA) &&
PG_QUERY="SELECT tenant_schema FROM $PG_SCHEMA.tenant_schema_mappings;" &&
PGPASSWORD=$(extract PG_PASSWD) psql --echo-all -c "$PG_QUERY"
volumeMounts:
- name: postgres
mountPath: /etc/secrets/postgres

Extract common commands into functions

I've got a file that consists of a number of configmaps.
Something like
{{- define "config1" -}}
kind: ConfigMap
metadata:
name: config-{{.Chart.Nn}}
apiVersion: v1
data:
script.sh: |-
#!/bin/bash
echo "Hello World"
echo "Hello Planet"
{{- end -}}
How do I extract echo "Hello World" and echo "Hello Planet" into a function so I can simply refer to the function within script.sh for the configmaps that need to run these particular commands?
I'm trying to avoid having to write the same code over and over.
Thanks.

Resources