Using YQ, I can merge 2 Kubernetes Helm chart yaml value files together using:
yq '. *= load("values-prod.yaml")' ../web/values.yaml
However, with values-prod.yaml, I need the contents to be dynamically generated at runtime.
I'm trying:
yq '. *= load("<(helm template -f values-prod.yaml -s templates/web.yaml . | yq .spec.source.helm.values)")' ../web/values.yaml
Error: Failed to load <(helm template -f values-prod.yaml -s templates/web.yaml . | yq .spec.source.helm.values): open <(helm template -f values-prod.yaml -s templates/web.yaml . | yq .spec.source.helm.values): no such file or directory
I've tried multiple variations of quotations with no luck.
Thanks.
load is an operator of yq. It accepts a file name as a parameter.
You cannot use process substitution here as this is shell syntax
(the error is generated by yq, not by your shell).
You need to create the dynamic content outside yq and pass it to yq.
Maybe you can just swap the two inputs:
yq '. *= load("../web/values.yaml")' <(helm template -f values-prod.yaml -s templates/web.yaml . | yq .spec.source.helm.values)
I can't test this code, but I hope the idea leads you to the solution.
Related
I try to get an information out of a helm file in Jenkins. For this I do:
helm history adt-development -n adt-development --max 1 --output yaml
to get the value ".chart" inside the helm yaml output I use yq
yq ".chart"
Because there is a bad yaml file reported, a "-" is too much at the beginning I try to cut it.
However, how to write this command in jenkins shell tofill a variable?
I tried:
helmvalue=$(helm history pro-development -n pro-development --max 1 --output yaml) | cut -c2- | yq '-chart'
But yq will read the output with the "-" so it ignore the cut output.
Any idea how to
read helm information -> cut first 2 char -> read .chart from output before.
On request additional information:
reported by helm:
- app_version: 0.6.0
chart: prof-1.0.1-2022-03-02-1616-46eb101842db6e367f3cd9ab42636ee9bf7d4912
description: Upgrade complete
revision: 355
status: deployed
updated: '"2022-03-02T16:42:52.04760689+01:00"'
Output from yq:
Error: bad file '-': yaml: mapping values are not allowed in this context
In YAML syntax, the dash sign - introduces a list of key-value objects, which by yq is converted into an array. Access it using .[].
With mikefarah/yq, use
yq '.[].chart'
With kislyuk/yq, add the -r option to generate raw text
yq -r '.[].chart'
If I'm reading the intent and syntax correctly,
helmvalue=$(helm history pro-development -n pro-development --max 1 --output yaml)
is stored in a variable; there is no output to stdout.
Which means <some cmd output> | cut -c2- has no input from stdin to the cut command and then no input to yq.
helmvalue=$(helm history pro-development -n pro-development --max 1 --output yaml| cut -c2-| yq '-chart')
Move closing bracket ")" to end and:
helm history ... -output yaml will send to stdout
cut -c2- will process stdin, cut 2 chars and send to stdout
yq '-chart should process from stdin a series of colon separated values and output a helmchart (?) to stdout
helmvalue=$(...) will contain the output from above
helmval
I am looking to add an environment variable to a yaml file from the command line. I used yq. This is the command:
cat mytemplate.yml | yq -Y '.services.samson.environment += {"newKey":"newVal"}'
I keep getting this error:
Error: unknown shorthand flag: 'Y' in -Y
Usage:
yq [flags]
yq [command]
Any work-arounds for this
You are probably confusing yq, the jq wrapper for YAML with yq, the jq reimplementation for YAML which are two different projects.
You have the second one installed but your usage of -Y indicates you need the first one, since only that one has such an option. So you need to uninstall the currently installed yq and then install the other one.
currently I call
cat my_file.json | jq
to pretty print json data. I am a bit surprised that I can't do
I would like to avoid the extra cat; i.e.,
jq my_file.json
Can I specify a file name?
You need to specify the jq program to run:
jq . my_file.json
jq -h
The usage line produced by jq -h:
Usage: jq [options] <jq filter> [file...]
Note that the summary produced by invoking jq with the -h option does not (currently) provide a complete listing of the options. For the supported options, see the jq manual: https://stedolan.github.io/jq/manual/
Two undocumented options of note are:
--debug-dump-disasm
--debug-trace
jq .
Under certain circumstances, jq . can be abbreviated to jq but it's always safe to use the full form; a good rule of thumb is: if in doubt, do so.
A complex .yaml file from this link needs to be fed into a bash script that runs as part of an automation program running on an EC2 instance of Amazon Linux 2. Note that the .yaml file in the link above contains many objects, and that I need to extract one of the environment variables defined inside one of the many objects that are defined in the file.
Specifically, how can I extract the 192.168.0.0/16 value of the CALICO_IPV4POOL_CIDR variable into a bash variable?
- name: CALICO_IPV4POOL_CIDR
value: "192.168.0.0/16"
I have read a lot of other postings and blog entries about parsing flatter, simpler .yaml files, but none of those other examples show how to extract a nested value like the value of CALICO_IPV4POOL_CIDR in this question.
As others are commenting, it is recommended to make use of yq (along with jq) if available.
Then please try the following:
value=$(yq -r 'recurse | select(.name? == "CALICO_IPV4POOL_CIDR") | .value' "calico.yaml")
echo "$value"
Output:
192.168.0.0/16
If you're able to install new dependencies, and are planning on dealing with lots of yaml files, yq is a wrapper around jq that can handle yaml. It'd allow a safe (non-grep) way of accessing nested yaml values.
Usage would look something like MY_VALUE=$(yq '.myValue.nested.value' < config-file.yaml)
Alternatively, How can I parse a YAML file from a Linux shell script? has a bash-only parser that you could use to get your value.
The right way to do this is to use a scripting language and a YAML parsing library to extract the field you're interested in.
Here's an example of how to do it in Python. If you were doing this for real you'd probably split it out into multiple functions and have better error reporting. This is literally just to illustrate some of the difficulties caused by the format of calico.yaml, which is several YAML documents concatenated together, not just one. You also have to loop over some of the lists internal to the document in order to extract the field you're interested in.
#!/usr/bin/env python3
import yaml
def foo():
with open('/tmp/calico.yaml', 'r') as fil:
docs = yaml.safe_load_all(fil)
doc = None
for candidate in docs:
if candidate["kind"] == "DaemonSet":
doc = candidate
break
else:
raise ValueError("no YAML document of kind DaemonSet")
l1 = doc["spec"]
l2 = l1["template"]
l3 = l2["spec"]
l4 = l3["containers"]
for containers_item in l4:
l5 = containers_item["env"]
env = l5
for entry in env:
if entry["name"] == "CALICO_IPV4POOL_CIDR":
return entry["value"]
raise ValueError("no CALICO_IPV4POOL_CIDR entry")
print(foo())
However, sometimes you need a solution right now and shell scripts are very good at that.
If you're hitting an API endpoint, then the YAML will usually be pretty-printed so you can get away with extracting text in ways that won't work on arbitrary YAML.
Something like the following should be fairly robust:
cat </tmp/calico.yaml | grep -A1 CALICO_IPV4POOL_CIDR | grep value: | cut -d: -f2 | tr -d ' "'
Although it's worth checking at the end with a regex that the extracted value really is valid IPv4 CIDR notation.
The key thing here is grep -A1 CALICO_IPV4POOL_CIDR .
The two-element dictionary you mentioned (shown below) will always appear as one chunk since it's a subtree of the YAML document.
- name: CALICO_IPV4POOL_CIDR
value: "192.168.0.0/16"
The keys in calico.yaml are not sorted alphabetically in general, but in {"name": <something>, "value": <something else>} constructions, name does consistently appear before value.
MYVAR=$(\
curl https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml | \
grep -A 1 CALICO_IPV4POOL_CIDR | \
grep value | \
cut -d ':' -f2 | \
tr -d ' "')
Replace curl https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml with however you're sourcing the file. That gets piped to grep -A 1 CALICO_IPV4POOL_CIDR. This gives you 2 lines of text: the name line, and the value line. That gets piped to grep value, which now gives us the line we want with just the value. That gets piped to cut -d ':' -f2 which uses the colon as a delimiter and gives us the second field. $(...) executes the enclosed script, and it is assigned to MYVAR. After this script, echo $MYVAR should produce 192.168.0.0/16.
You have two problems there:
How to read a YAML document from a file with multiple documents
How to select the key you want from that YAML document
I have guessed that you need the YAML document of kind 'DaemonSet' from reading Gregory Nisbett's answer.
I will try to only use tools that are likely to be already installed on your system because you mentioned you want to do this in a Bash script. I assume you have JQ because it is hard to do much in Bash without it!
For the YAML library I tend to use Ruby for this because:
Most systems have a Ruby
Ruby's Psych library has been bundled since Ruby 1.9
The PyYAML library in Python is a bit inflexible and sometimes broken compared to Ruby's in my experience
The YAML library in Perl is often not installed by default
It was suggested to use yq, but that won't help so much in this case because you still need a tool that can extract the YAML document.
Having extracted the document I am going to again use Ruby to save the file as JSON. Then we can use jq.
Extracting the YAML document
To get the YAML document using Ruby and save it as JSON:
url=...
curl -s $url | \
ruby -ryaml -rjson -e \
"puts YAML.load_stream(ARGF.read)
.select{|doc| doc['kind']=='DaemonSet'}[0].to_json" \
| jq . > calico.json
Further explanation:
The YAML.load_stream reads the YAML documents and returns them all as an Array
ARGF.read reads from a file passed via STDIN
Ruby's select allows easy selection of the YAML document according to its kind key
Then we take the element 4 and convert to JSON.
I pass that response through jq . so that it's formatted for human readability but that step isn't really necessary. I could do the same in Ruby but I'm guessing you want Ruby code kept to a minimum.
Selecting the key you want
To select the key you want the following JQ query can be used:
jq -r \
'.spec.template.spec.containers[].env[] | select(.name=="CALICO_IPV4POOL_CIDR") | .value' \
calico.json
Further explanation:
The first part spec.template.spec.containers[].env[] iterates for all containers and for all envs inside them
Then we select the Hash where the name key equals CALICO_IPV4POOL_CIDR and return the value
The -r removes the quotes around the string
Putting it all together:
#!/usr/bin/env bash
url='https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml'
curl -s $url | \
ruby -ryaml -rjson -e \
"puts YAML.load_stream(ARGF.read)
.select{|doc| doc['kind']=='DaemonSet'}[0].to_json" \
| jq . > calico.json
jq -r \
'.spec.template.spec.containers[].env[] | select(.name=="CALICO_IPV4POOL_CIDR") | .value' \
calico.json
Testing:
▶ bash test.sh
192.168.0.0/16
I am trying to understand a bash expression
oc process -f build/my-build-template.yaml GITSERVER=$GITSERVER | oc create -f -
found here: https://github.com/openshift/origin/issues/15474
This is an example from the OpenSift API. The first part oc process -f build/my-build-template.yaml GITSERVER=$GITSERVER returns some JSON:
I understand, that this JSON will be the input for the oc create -f <EXPECT FILE AS INPUT HERE> which expects a file.
What is the purpose of the last - in the ... | oc create -f -
Many tools allow specification of standard input as a filename of "-". This allows the pipe to work as expected without requiring a shell- or OS-based workaround.