I have a YAML file like this
apiVersion: "v1alpha1"
kind: "Druid"
metadata:
name: druid-dev-cluster
spec:
common.runtime.properties: |
# Zookeeper
druid.zk.service.host=cluster-zk-0.cluster-zk
druid.zk.paths.base=/druid
druid.zk.service.compress=false
I want to replace one of the properties in the common.runtime.properties. Is this supported using yq?
When I try normally it fails
property=.spec.common.runtime.properties.druid.zk.service.host=cluster-zk-0.cluster-zk
OVERLAY= deploy/overlays/aws/common-runtime-properties.yaml
yq e "${property}" "$OVERLAY"/"$propertyType"
Error: Parsing expression: Lexer error: could not match text starting at 1:55 failing at 1:57.
unmatched text: "ti"
This works for other properties like
apiVersion: "v1alpha1"
kind: "Druid"
metadata:
name: druid-dev-cluster
spec:
nodes:
brokers:
nodeType: "broker"
druid.port: 8088
ingressAnnotations:
kubernetes.io/ingress.class: "plb.v1"
The problems are:
| is a YAML literal block scalar. Its content is a single scalar as far as YAML is concerned and hence you cannot path-select into it from yq.
Also, the content of | is not YAML, but probably parsed as Java properties file. Therefore, you cannot parse it even with recursively calling yq on the scalar.
Also, the path to the scalar is not .spec.common.runtime.properties but .spec."common.runtime.properties" – note how the second part is written as a single scalar in the YAML content, which is not equivalent to having multiple child mappings (even though some Java folks seem to believe that).
That being said, you can of course do something like
export NEW_VALUE=droggeljug
UPDATED_CONTENT=$(\
yq e '.spec."common.runtime.properties"' test.yaml | \
sed -re 's/(druid.zk.service.host=)[^\n]*/\1'"$NEW_VALUE"'/g' \
) yq e '.spec."common.runtime.properties" = strenv(UPDATED_CONTENT)' test.yaml
This uses yq to select the scalar containing the properties (line 3), uses sed to update the value druid.zk.service.host to the content of NEW_VALUE (line 4), stores the result in UPDATED_CONTENT (line 2) and then calls yq again updating the value to be the content of UPDATED_CONTENT (line 5).
Related
I am attempting to merge two yaml files and would like any shared keys under a specific key to use values from one of the yaml files, and not merge both. This problem may be better described using an example. GIven file1.yaml and file2.yaml, I am trying to achieve the following:
file1.yaml
name: 'file1'
paths:
path1:
content: "t"
path2:
content: "b"
file2.yaml
name: 'file2'
paths:
path1:
value: "t"
My ideal result in merging is the following file:
file3.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"
Specifically, I would like to overwrite any key under paths such that if both yaml files have the same key under paths, then only use the value from file2. Is there some tool that enables this? I was looking into yq but I'm not sure if that tool would work
Please specify which implementation of yq you are using. They are quite similar, but sometimes differ a lot.
For instance, using kislyuk/yq, you can use input to access the second file, which you can provide alongside the first one:
yq -y 'input as $in | .name = $in.name | .paths += $in.paths' file1.yaml file2.yaml
name: file2
paths:
path1:
value: t
path2:
content: b
With mikefarah/yq, you'd use load with providing the second file in the code, while only the first one is your regular input:
yq 'load("file2.yaml") as $in | .name = $in.name | .paths += $in.paths' file1.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"
I'm trying to do a replace where I first escape all double quotes and then I want to use the result of this replace to updated a value. But in the last replace the backslashes are removed, why is this and how do I avoid that?
Example in bash:
>TEST_OBJECT='{"val1": "a", "val2": "b"}'
>ESCAPED_OBJ="$(echo $TEST_OBJECT | sed 's/"/\\"/g')"
>echo $ESCAPED_OBJ
{\"val1\": \"a\", \"val2\": \"b\"}
>echo 'value: "_REPLACE_ME"' | sed "s#_REPLACE_ME#$ESCAPED_OBJ#g"
value: "{"val1": "a", "val2": "b"}"
I'm expecting this on the last row:
value: "{\"val1\": \"a\", \"val2\": \"b\"}"
EDIT
I realize I presented the issue wrong, the reason why I do it in 2 steps is because the first replace happens in one step and then the second replace happens in a later step. This is part of a github workflow and the last replace actually replaces a string in a different yaml file.
sed "s#_REPLACE_ME#$ESCAPED_OBJ#g" > ${{ github.action_path }}/config/job.yml
So I don't think I can do the replace in one step, first I need to update the string and then replace a value in another file.
job.yml
...
env:
- name: CUSTOM_DATA_OBJECT
value: "_REPLACE_ME"
I need the value to be escaped so that it doesn't break the yaml.
Using sed
$ sed "s#_REPLACE_ME#${TEST_OBJECT//\"/\\\\\"}#" input_file
...
env:
- name: CUSTOM_DATA_OBJECT
value: "{\"val1\": \"a\", \"val2\": \"b\"}"
Maybe letting yq take care of the correct escaping for YAML?
TEST_OBJECT='{"val1": "a", "val2": "b"}' \
yq eval '.env[0].value = strenv(TEST_OBJECT)' file.yaml
env:
- name: CUSTOM_DATA_OBJECT
value: "{\"val1\": \"a\", \"val2\": \"b\"}"
Given yaml config file that looks like this:
key1:
key11:value1
key12:value2
key2:
key21:value3
How can I convert it in a bash script (preferable with yq) to env vars prefixed with a string? Desired output for env:
TF_VAR_key11=value1
TF_VAR_key12=value2
TF_VAR_key21=value3
Assuming the missing spaces in the input's subelements between key and value are intentional, so we are just dealing with an array of string values containing :, and separated by whitespace.
yq '.[] | split(" ") | .[] | split(":") | "TF_VAR_" + .[0] + "=" + .[1]' file.yaml
Which implementation of yq are you using? This works for mikefarah/yq. To be used with kislyuk/yq, add the -r option.
You'd first load the YAML configuration into memory.
from yaml import loads
with open("config.yaml") as f:
config = loads(f.read())
Then, iterate over the dictionary values, which appear to also be dictionaries. For each of these dictionaries, write the key=val pair to the new file.
env_str = ""
for inner_dict in config.values():
for key, val in inner_dict.items():
env_str = f"TF_VAR_{key}={val}\n"
Using python as suggested is easy and readable, if you need to do everything in bash, you can check this thread which has various solutions:
How can I parse a YAML file from a Linux shell script?
I'm trying to take a YAML style array I get from an AWS command, and assign it to a key while updating my own YAML.
This represents what I have right now:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= "'"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')"'"' -i route53.yml
This is how route53.yml looks like before I run the command:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
and this is how route53.yml looks like after:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
ResourceRecordSets: |-
- Name: a.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: b.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: c.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: d.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: e.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
As you can see there's a |- right after the key, and it seems like it is treated as multiline string instead of an array of maps. How can I avoid it and assign the array as a YAML array? When I tried manually assigning an array in the style of ['a', 'b', 'c'] and the update works as it should, adding a YAML array under the key, how can I achieve it with the output of the aws command?
The reason why yq is interpreting the output of the $(...) expression as a multiline strine is because you have quoted it; your yq expression, simplified, looks like:
yq 'ResourceRecordSets |= "some string here"'
The quotes mean "this is a string, not a structure", so that's what you get. You could try dropping the quotes, like this:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= '"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')" route53.yml
That might work, but it is fragile. A better solution is to have yq parse the output of the subexpression as a separate document, and then merge it as a structured document rather than a big string. Like this:
yq \
'(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= input.ResourceRecordSets' \
route53.yml \
<(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml)
This takes advantage of the fact that you can provide jq (and hence yq) multiple files on the command line, and then refer to the input variable (described in the IO section of the jq manual). The above command line is structured like this:
yq <expression> <file1> <file2>
Where <file1> is route53.yml, and <file2> is a bash process substitution.
This solution simplifies issues around quoting and formatting.
(You'll note I dropped your use of -i here; that seems to throw yq for a loop, and it's easy to output to a temporary file and then rename it.)
Eventually I solved it using a combination of larsks answer about the unnecessary quotes, and using jq instead of yq (no matter what I did, using solely yq wouldn't work) to assign the array to a separate variable before editing the YAML document:
record_sets=$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" | jq '.[]')
yq '(.HostedZones[] | select(.Id=="ABC123")).ResourceRecordSets |= '"$record_sets"'' -i route53.yml
When I retrieve all records for a hosted zone in Cloudflare, e.g. of response, I need to create from it the following yaml structure:
name_zones: # this line we create
.zone_name: # the value is taken from the response
auth_key: XXX # this line we create
records: # this line we create
# iterate over all records
- name: .name
type: .type
priority: .priority # create line if value set|exist
content: .content
ttl: .ttl # create line if value set|exist
e.g. jq code which almost done this:
jq '.result[] | {name: .name, type: .type, content: .content, ttl: .ttl} + if has("priority") then {priority} else null end' | jq -n '.name_zone.zone_name.auth_key.records |= [inputs]' | yq r -P -
How to pass or create the value of zone_name and auth_key: XXX?
First, your two invocations of jq can be replaced by just one, as shown in the answer below.
Second, there are currently (at least) two yqs in the wild:
python-based yq (https://kislyuk.github.io/yq) - hereafter python-yq
go-based yq (https://github.com/mikefarah/yq)
For better and/or worse, version 4 of the go-based yq is significantly different from earlier versions, so if you want to use the current go-based version you may have to make adjustments accordingly. To simplify things (at least from my point of view), I will replace your yq r -P - by:
python-jq -y .
The following produces the output shown below.
< cf_response.json \
jq '{name_zones:
{zone_name: .result[0].zone_name,
auth_key: "XXX",
records:
[.result[]
| {name, type, content, ttl}
+ if has("priority")
then {priority}
else null end] }} '|
python-yq -y .
Output
name_zones:
zone_name: test.com
auth_key: XXX
records:
- name: test.com
type: A
content: 111.111.111.111
ttl: 1
- name: test.com
type: TXT
content: google-site-verification=content
ttl: 1
- name: test.com
type: MX
content: smtp.test.com
ttl: 1
priority: 0
auth_key
If you want to pass in the value of auth_key as a parameter to jq, you could use the command-line sequence --arg auth_key XXX, and then use $auth_key in the jq program.