I need to remove an empty map from a YAML, using YQ
Sometimes this map may have values, and sometimes this will appears empty.
My YAML code looks like this:
apiVersion: route.openshift.io/v1
kind: Route
metadata:
annotations: {}
creationTimestamp: "2021-03-24T13:16:10Z"
I need to remove annotations: {}
My desired output:
apiVersion: route.openshift.io/v1
kind: Route
metadata:
creationTimestamp: "2021-03-24T13:16:10Z"
Anybody can helps me?
mikefarah/yq
For a generic approach you can use the command
yq e 'del(.. | select(tag == "!!map" and length == 0))'
to remove all empty objects in the input.
Change !!map with !!seq if you want to do the same for empty arrays.
kislyuk/yq
Remove empty objects: yq -y 'del(.. | select(objects and length == 0))'
Remove empty arrays: yq -y 'del(.. | select(arrays and length == 0))'
Remove empty objects, arrays and strings: yq -y 'del(.. | select(length == 0))'
You can delete the annotations map when its length is 0. Using mikefarah/yq, this can be done as below (verfied on yq version 4.9.6)
yq e 'del(.metadata.annotations | select(length==0))' yaml
Note: Since 4.18.1, yq's eval/e command is the default command and no longer needs to be specified.
From the #jpseng's anwser, I get the command only removes empty objects at the leaf nodes, since it may create more empty objects, like key a or d(and d eventually):
$ cat example.yaml
a:
b: {}
c:
d:
e: {}
f: "exists"
$ yq 'del(.. | select(tag=="!!map" and length == 0))' example.yaml
a: {}
c:
d: {}
f: "exists"
Remove these recursively by Bash script
$ cat delete_empty_map.sh
#!/usr/bin/env bash
result=$(yq 'del( .. | select(tag == "!!map" and length == 0))' "$1")
while [ "$(echo "$result" | yq 'map(.. | select(tag == "!!map" and length == 0)) | any')" = "true" ]
do
result=$(echo "$result" | yq 'del( .. | select(tag == "!!map" and length == 0))')
done
echo "$result" | yq
$ ./delete_empty_map.sh example.yaml
f: "exists"
Related
Using yq (or any other tool), how can I return the full YAML path of an arbitrary line number ?
e.g. with this file :
a:
b:
c: "foo"
d: |
abc
def
I want to get the full path of line 2; it should yield: a.b.c. Line 0 ? a, Line 4 ? a.d (multiline support), etc.
Any idea how I could achieve that?
Thanks
I have coded two solutions that differ slightly in their behaviour (see remarks below)
Use the YAML processor mikefarah/yq.
I have also tried to solve the problem using kislyuk/yq, but it is not suitable,
because the operator input_line_number only works in combination with the --raw-input option
Version 1
FILE='sample.yml'
export LINE=1
yq e '[..
| select(line == env(LINE))
| {"line": line,
"path": path | join("."),
"type": type,
"value": .}
]' $FILE
Remarks
LINE=3 returns two results, because line 3 contains two nodes
the key 'c' of map 'a.b'
the string value 'foo' of key 'c'.
LINE=5 does not return a match, because the multiline text node starts in line 4.
the results are wrapped in an array, as multiple nodes can be returned
Output for LINE=1
- line: 1
path: ""
type: '!!map'
value:
a:
b:
c: "foo"
d: |-
abc
def
Output for LINE=2
- line: 2
path: a
type: '!!map'
value:
b:
c: "foo"
Output for LINE=3
- line: 3
path: a.b
type: '!!map'
value:
c: "foo"
- line: 3
path: a.b.c
type: '!!str'
value: "foo"
Output for LINE=4
- line: 4
path: d
type: '!!str'
value: |-
abc
def
Output for LINE=5
[]
Version 2
FILE='sample.yml'
export LINE=1
if [[ $(wc -l < $FILE) -lt $LINE ]]; then
echo "$FILE has less than $LINE lines"
exit
fi
yq e '[..
| select(line <= env(LINE))
| {"line": line,
"path": path | join("."),
"type": type,
"value": .}
]
| sort_by(.line, .type)
| .[-1]' $FILE
Remarks
at most one node is returned, even if there are more nodes in the selected row. So the result does not have to be wrapped in an array.
Which node of one line is returned can be controlled by the sort_by function, which can be adapted to your own needs.
In this case, text nodes are preferred over maps because "!!map" is sorted before "!!str".
LINE=3 returns only the text node of line 3 (not node of type "!!map")
LINE=5 returns the multiline text node starting at line 4
LINE=99 does not return the last multiline text node of sample.yaml because the maximum number of lines is checked in bash beforehand
Output for LINE=1
line: 1
path: ""
type: '!!map'
value:
a:
b:
c: "foo"
d: |-
abc
def
Output for LINE=2
line: 2
path: a
type: '!!map'
value:
b:
c: "foo"
Output for LINE=3
line: 3
path: a.b.c
type: '!!str'
value: "foo"
Output for LINE=4
line: 4
path: d
type: '!!str'
value: |-
abc
def
Output for LINE=5
line: 4
path: d
type: '!!str'
value: |-
abc
def
Sharing my findings since I've spent too much time on this.
As #Inian mentioned line numbers won't necessary be accurate.
YQ does provides us with the line operator, but I was not able to find a decent way of mapping that from an input.
That said, if you're sure the input file will not contain any multi-line values, you could do something like this
Use awk to get the key of your input line, eg 3 --> C
This assumes the value will never contain :, the regex can be edited if needed to go around this
Select row in awk
Trim leading and trailing spaces from a string in awk
export searchKey=$(awk -F':' 'FNR == 3 { gsub(/ /,""); print $1 }' ii)
Use YQ to recursive (..) loop over the values, and create each path using (path | join("."))
yq e '.. | (path | join("."))' ii
Filter the values from step 2, using a regex where we only want those path's that end in the key from step 1 (strenv(searchKey))
yq e '.. | (path | join(".")) | select(match(strenv(searchKey) + "$"))' ii
Print the path if it's found
Some examples from my local machine, where your input file is named ii and both awk + yq commands are wrapped in a bash function
$ function getPathByLineNumber () {
key=$1
export searchKey="$(awk -v key=$key -F':' 'FNR == key { gsub(/ /, ""); print $1 }' ii)"
yq e '.. | (path | join(".")) | select(match(strenv(searchKey) + "$"))' ii
}
$
$
$
$
$ yq e . ii
a:
b:
c: "foo"
$
$
$ getPathByLineNumber 1
a
$ getPathByLineNumber 2
a.b
$ getPathByLineNumber 3
a.b.c
$
$
I would like to replace a single value inside a yaml file (file1.yml) with the value of another yaml file (file2.yml). The key inside both file:
app:
key: 12345
So far here is what I have done using sed with no success:
#!/bin/bash
old_value=`cat file1.yml | grep key | head -n1 | cut -f 2 -d ':'`
new_value=`cat file2.yml | grep key | head -n1 | cut -f 2 -d ':'`
sed "s/$old_value/$new_value/g;" file1.yml
I guess I should not be sending the standard output with sed and should be using awk.
To manipulate yaml files, you should employ a yaml processor, like mikefarah/yq or kislyuk/yq.
Using mikefarah/yq:
new="$(yq '.key' file2.yml)" yq -i '.key = env(new)' file1.yml
Using kislyuk/yq:
yml="$(yq -y -s '.[0].key = .[1].key | .[0]' file1.yml file2.yml)"
cat <<< "$yml" > file1.yml
Because in a yaml file the same value may exist in multiple places you want to use sed to perform a full search and then replace the value you are looking for; or use an specialized tool like yq (a jq wrapper)
For example, this yaml file is valid
app1:
key: "1234"
app2:
key: "1234"
with sed you will run the following to change key: "1234" to key: "5678" in just app2
sed '/^app2:/{n;s/key:.*/key: "5678"/;}' file.yaml
But doing the same using yq using in-place edit would look like:
yq -i -y '.app2.key = "5678"' file.yml
I have two yaml files I am using. The first yaml file looks like this:
spring.yml
spring:
cloud:
gateway:
routes:
- id: someid
uri: someUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someOtherPath
I have another file that just contains routes and looks like this:
routes.yml
routes:
- id: someid
uri: someOtherUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someNewPath
My goal is to update the route in the first file with the value of the route in the second file. Note that the first file in reality will have many routes but for demonstration purposes I am only showing the first in this example. I have the following script which loops through to update the routes as necessary when the id's match:
#!/bin/sh
OVERRIDE_ROUTE_IDS=$(yq eval '.routes.[].id' routes.yml)
GENERATED_ROUTE_IDS=$(yq eval '.spring.cloud.gateway.routes.[].id' spring.yml)
SAVEIFS=$IFS # Save current IFS (Internal Field Separator)
IFS=$'\n' # Change IFS to newline char
OVERRIDE_ROUTE_IDS=($OVERRIDE_ROUTE_IDS) # split the `OVERRIDE_ROUTE_IDS` string into an array by the same name
GENERATED_ROUTE_IDS=($GENERATED_ROUTE_IDS) # split the `GENERATED_ROUTE_IDS` string into an array by the same name
IFS=$SAVEIFS # Restore original IFS
for (( i=0; i<${#OVERRIDE_ROUTE_IDS[#]}; i++ ))
do
if [[ "${GENERATED_ROUTE_IDS[*]}" =~ "${OVERRIDE_ROUTE_IDS[$i]}" ]]
then
echo "route ID ${OVERRIDE_ROUTE_IDS[$i]} exists in generated routes"
for (( j=0; j<${#GENERATED_ROUTE_IDS[#]}; j++ ))
do
if [[ "${GENERATED_ROUTE_IDS[$j]}" == "${OVERRIDE_ROUTE_IDS[$i]}" ]]
then
echo "index of route ${GENERATED_ROUTE_IDS[$j]} is $j"
echo "$i"
ROUTE_TO_USE=$(yq eval ".routes.[$i]" routes.yml)
$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" spring.yml)
fi
done
else
echo "no match so add to top of routes"
fi
done
My assumption is this command should update spring.yml file with the new route in place of the one that was identified with the same id:
$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" application.yml)
But I am getting the following error
Error: Parsing expression: Lexer error: could not match text starting at 1:37 failing at 1:39 unmatched text: "id"
I'm stumped on this and not sure what I'm doing wrong at this point. For reference I am using yq version 4.17.2.
Be aware that yq does not emit a data structure, it emits a string. $ROUTE_TO_USE would be, for example,
id: someid
uri: someOtherUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someNewPath
This is YAML source. Pasting this into the following yq command leads to invalid syntax; yq's expression syntax is not literal YAML. This is what the error tries to tell you.
What you want to do is to process both inputs in a single yq command:
yq ea "select(fi==0).spring.cloud.gateway.routes.[$j] = "`
`"select(fi==1).routes.[$i] | select(fi==0)" spring.yml routes.yml
ea is shorthand for eval-all which you need for processing multiple input files at the same time. fi is a shorthand for fileIndex, which is used to select the appropriate file. Piping the result to | select(fi==0) ensures that only the first (modified) file is written out. I split the long string into multiple lines using backticks for readability.
I ended up getting a solution from the creator of yq. The solution below is what I used:
yq ea '
(select(fi==0) | .spring.cloud.gateway.routes.[].id) as $currentIds |
(select(fi==1) | [.routes.[] | select( [$currentIds != .id] | all )] ) as $newRoutes |
( $newRoutes + .spring.cloud.gateway.routes + .routes) as $routesToMerge |
(
(($routesToMerge | .[] | {.id: .}) as $item ireduce ({}; . * $item )) as $uniqueMap
| ( $uniqueMap | to_entries | .[]) as $item ireduce([]; . + $item.value)
) as $mergedArray
| select(fi == 0) | .spring.cloud.gateway.routes = $mergedArray
' spring.yml routes.yml
This matches on id. If there is a match it uses the value of what's in routes.yml. If there is no match it add it the top top of the routes.
I have this below YAML input and I am trying to extract shown output using yq. I want to remove pairs where key name (VAR-A) in value {{a.b.VAR-A}} (after a.b.) matches and If I have more than one {{a.b.VAR-A}} in values separated by - , I want to keep them.
VAR-A: '{{a.b.VAR-A}}'
VAR-B: '{{a.b.VAR-B}}'
VAR-C: v0.0
VAR-D: '{{a.b.VAR-D}}-{{a.b.VAR-A}}'
VAR-E: '{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}'
VAR-F: True
Expected Output:
VAR-C: v0.0
VAR-D: '{{a.b.VAR-D}}-{{a.b.VAR-A}}'
VAR-E: '{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}'
VAR-F: True
This question works if I have all strings, but it fails when I have boolean value in yaml. Extract Key Value pairs which matches the regex in YAML using yq/sed/grep
I get below error:
Error: cannot substitute with !!bool, can only substitute strings. Hint: Most often you'll want to use '|=' over '=' for this operation.
There are at least two very different extant "yq" projects: a Python-based one, which is the focus of Part 1 below, and a Go-based one, which is the focus of Part 2.
Part 1
python-yq 'del(.[] | select( ( type == "string" and test("^{{a[.]b[.][^}]*}}$" ))))' so-vars.yaml
or
python-yq 'map_values( select( ( type == "string" and test("^{{a[.]b[.][^}]*}}$" )) | not))' so-vars.yaml
Output:
{
"VAR-C": "v0.0",
"VAR-D": "{{a.b.VAR-D}}-{{a.b.VAR-A}}",
"VAR-E": "{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}",
"VAR-F": true
}
Part 2
The Go-based version of yq that I have (4.6.3) might not be able to handle your requirements directly, but here's a solution that uses this yq to translate to and from JSON, and jq to do the rest:
yq -j eval . input.yaml |
jq 'del(.[] | select(( type == "string" and test("^{{a[.]b[.][^}]*}}$" ))))' > tmp.json
yq -P eval . tmp.json
The del-free version of the jq program:
map_values( select( type == "string" and test("^{{a[.]b[.][^}]*}}$" | not)
Output:
VAR-C: v0.0
VAR-D: '{{a.b.VAR-D}}-{{a.b.VAR-A}}'
VAR-E: '{{a.b.VAR-C}}-{{a.b.VAR-B}}-{{a.b.VAR-A}}'
VAR-F: true
I have a yq read command as below,
groups=$(yq read generated/identity-mapping.yaml "iamIdentityMappings.[0].groups")
It reads iamIdentityMappings from below yaml:
iamIdentityMappings:
- groups:
- Appdeployer
- Moregroups
It stores group as below,
- Appdeployer
- Moregroups
But I want to store groups as below.(comma separated values)
groups="Appdeployer","Moregroups"
How to do this in bash?
yq is just a wrapper for jq, which supports CSV output:
$ groups="$(yq -r '.iamIdentifyMappings[0].groups | #csv' generated/identity-mapping.yaml)"
$ echo "$groups"
"Appdeployer","Moregroups"
The yq invocation in your question just causes an error. Note the fixed version.
Use mapfile and format a null delimited list with yq:
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
typeset -p groups
Output:
declare -a groups=([0]="Appdeployer" [1]="Moregroups")
And now you can fulfill this second part of your question:
Construct a command based upon a count variable in bash
# Prepare eksctl's arguments into an array
declare -a eksctl_args=(create iamidentitymapping --cluster "$name" --region "$region" --arn "$rolearn" )
# Read the groups from the yml into an array
mapfile -d '' -t groups < <(
yq -j '.iamIdentityMappings[0].groups[]+"\u0000"' \
generated/identity-mapping.yaml
)
# Add arguments per group
for group in "${groups[#]}"; do
eksctl_args+=(--group "$group")
done
# add username argument
eksctl_args+=(--username "$username")
# call eksctl with its arguments
eksctl "${eksctl_args[#]}"
yq 4.16+ now has a built in #csv operator:
yq e '.iamIdentityMappings.[0].groups | #csv' file.yaml
Note that #csv will only wrap values in quotes if needed (e.g. they have a comma).
If you want quotes, then sub then in and join with commas:
yq e '
.iamIdentityMappings.[0].groups |
(.[] |= sub("(.*)", "\"${1}\""))
| join(",")'
Disclaimer: I wrote yq.
yq version 3 is deprecated now and you can achieve the same output using version 4
#!/bin/bash
while IFS= read -r value; do
groups_array+=($value)
done < <(yq eval '.iamIdentityMappings.[0].groups.[]' generated/identity-mapping.yaml)
printf -v comma_seperated '%s,' "${groups_array[#]}"
echo "${comma_seperated%,}"
This code prints the comma seperated values as you wanted