Assume I have a file root.yml
keyA: valA
keyB: valB
myList:
Then I receive some yml file, such as
1.yml
project_id: abc
description: xyz
2.yml
project_id: cba
description: zyx
And so on (they may stored in same folder)
Now I want to append the content of 1.yml, 2.yml (and so on) to the myList of root.yml and output to console
Expected:
keyA: valA
keyB: valB
myList:
- project_id: abc
description: xyz
- project_id: cba
description: zyx
- (so on...)
I have searched some examples but they hard code the list item in the yq command, like this post: Stack Overflow
But I want it load from files, not from hard code
Please forgive for my bad english
With mikefarah/yq, you could use the load function with the filename of the YAML files to be included, i.e. with your example
yq '.myList += [ load("1.yaml"), load("2.yaml") ]' root.yml
producing a YAML result as
keyA: valA
keyB: valB
myList:
- project_id: abc
description: xyz
- project_id: cba
description: zyx
As indicated in your comment, if one of the object has a parent structure and you want to extract the element from it, you can do
yq 'load("1.yaml") as $f | .myList += [ $f.config[], load("2.yaml") ]' root.yml
Tested on yq version 4.27.2
Related
I am attempting to merge two yaml files and would like any shared keys under a specific key to use values from one of the yaml files, and not merge both. This problem may be better described using an example. GIven file1.yaml and file2.yaml, I am trying to achieve the following:
file1.yaml
name: 'file1'
paths:
path1:
content: "t"
path2:
content: "b"
file2.yaml
name: 'file2'
paths:
path1:
value: "t"
My ideal result in merging is the following file:
file3.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"
Specifically, I would like to overwrite any key under paths such that if both yaml files have the same key under paths, then only use the value from file2. Is there some tool that enables this? I was looking into yq but I'm not sure if that tool would work
Please specify which implementation of yq you are using. They are quite similar, but sometimes differ a lot.
For instance, using kislyuk/yq, you can use input to access the second file, which you can provide alongside the first one:
yq -y 'input as $in | .name = $in.name | .paths += $in.paths' file1.yaml file2.yaml
name: file2
paths:
path1:
value: t
path2:
content: b
With mikefarah/yq, you'd use load with providing the second file in the code, while only the first one is your regular input:
yq 'load("file2.yaml") as $in | .name = $in.name | .paths += $in.paths' file1.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"
Using yq (or any other tool), how can I return the full YAML path of an arbitrary line number ?
e.g. with this file :
a:
b:
c: "foo"
d: |
abc
def
I want to get the full path of line 2; it should yield: a.b.c. Line 0 ? a, Line 4 ? a.d (multiline support), etc.
Any idea how I could achieve that?
Thanks
I have coded two solutions that differ slightly in their behaviour (see remarks below)
Use the YAML processor mikefarah/yq.
I have also tried to solve the problem using kislyuk/yq, but it is not suitable,
because the operator input_line_number only works in combination with the --raw-input option
Version 1
FILE='sample.yml'
export LINE=1
yq e '[..
| select(line == env(LINE))
| {"line": line,
"path": path | join("."),
"type": type,
"value": .}
]' $FILE
Remarks
LINE=3 returns two results, because line 3 contains two nodes
the key 'c' of map 'a.b'
the string value 'foo' of key 'c'.
LINE=5 does not return a match, because the multiline text node starts in line 4.
the results are wrapped in an array, as multiple nodes can be returned
Output for LINE=1
- line: 1
path: ""
type: '!!map'
value:
a:
b:
c: "foo"
d: |-
abc
def
Output for LINE=2
- line: 2
path: a
type: '!!map'
value:
b:
c: "foo"
Output for LINE=3
- line: 3
path: a.b
type: '!!map'
value:
c: "foo"
- line: 3
path: a.b.c
type: '!!str'
value: "foo"
Output for LINE=4
- line: 4
path: d
type: '!!str'
value: |-
abc
def
Output for LINE=5
[]
Version 2
FILE='sample.yml'
export LINE=1
if [[ $(wc -l < $FILE) -lt $LINE ]]; then
echo "$FILE has less than $LINE lines"
exit
fi
yq e '[..
| select(line <= env(LINE))
| {"line": line,
"path": path | join("."),
"type": type,
"value": .}
]
| sort_by(.line, .type)
| .[-1]' $FILE
Remarks
at most one node is returned, even if there are more nodes in the selected row. So the result does not have to be wrapped in an array.
Which node of one line is returned can be controlled by the sort_by function, which can be adapted to your own needs.
In this case, text nodes are preferred over maps because "!!map" is sorted before "!!str".
LINE=3 returns only the text node of line 3 (not node of type "!!map")
LINE=5 returns the multiline text node starting at line 4
LINE=99 does not return the last multiline text node of sample.yaml because the maximum number of lines is checked in bash beforehand
Output for LINE=1
line: 1
path: ""
type: '!!map'
value:
a:
b:
c: "foo"
d: |-
abc
def
Output for LINE=2
line: 2
path: a
type: '!!map'
value:
b:
c: "foo"
Output for LINE=3
line: 3
path: a.b.c
type: '!!str'
value: "foo"
Output for LINE=4
line: 4
path: d
type: '!!str'
value: |-
abc
def
Output for LINE=5
line: 4
path: d
type: '!!str'
value: |-
abc
def
Sharing my findings since I've spent too much time on this.
As #Inian mentioned line numbers won't necessary be accurate.
YQ does provides us with the line operator, but I was not able to find a decent way of mapping that from an input.
That said, if you're sure the input file will not contain any multi-line values, you could do something like this
Use awk to get the key of your input line, eg 3 --> C
This assumes the value will never contain :, the regex can be edited if needed to go around this
Select row in awk
Trim leading and trailing spaces from a string in awk
export searchKey=$(awk -F':' 'FNR == 3 { gsub(/ /,""); print $1 }' ii)
Use YQ to recursive (..) loop over the values, and create each path using (path | join("."))
yq e '.. | (path | join("."))' ii
Filter the values from step 2, using a regex where we only want those path's that end in the key from step 1 (strenv(searchKey))
yq e '.. | (path | join(".")) | select(match(strenv(searchKey) + "$"))' ii
Print the path if it's found
Some examples from my local machine, where your input file is named ii and both awk + yq commands are wrapped in a bash function
$ function getPathByLineNumber () {
key=$1
export searchKey="$(awk -v key=$key -F':' 'FNR == key { gsub(/ /, ""); print $1 }' ii)"
yq e '.. | (path | join(".")) | select(match(strenv(searchKey) + "$"))' ii
}
$
$
$
$
$ yq e . ii
a:
b:
c: "foo"
$
$
$ getPathByLineNumber 1
a
$ getPathByLineNumber 2
a.b
$ getPathByLineNumber 3
a.b.c
$
$
I'm trying to take a YAML style array I get from an AWS command, and assign it to a key while updating my own YAML.
This represents what I have right now:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= "'"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')"'"' -i route53.yml
This is how route53.yml looks like before I run the command:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
and this is how route53.yml looks like after:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
ResourceRecordSets: |-
- Name: a.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: b.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: c.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: d.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: e.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
As you can see there's a |- right after the key, and it seems like it is treated as multiline string instead of an array of maps. How can I avoid it and assign the array as a YAML array? When I tried manually assigning an array in the style of ['a', 'b', 'c'] and the update works as it should, adding a YAML array under the key, how can I achieve it with the output of the aws command?
The reason why yq is interpreting the output of the $(...) expression as a multiline strine is because you have quoted it; your yq expression, simplified, looks like:
yq 'ResourceRecordSets |= "some string here"'
The quotes mean "this is a string, not a structure", so that's what you get. You could try dropping the quotes, like this:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= '"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')" route53.yml
That might work, but it is fragile. A better solution is to have yq parse the output of the subexpression as a separate document, and then merge it as a structured document rather than a big string. Like this:
yq \
'(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= input.ResourceRecordSets' \
route53.yml \
<(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml)
This takes advantage of the fact that you can provide jq (and hence yq) multiple files on the command line, and then refer to the input variable (described in the IO section of the jq manual). The above command line is structured like this:
yq <expression> <file1> <file2>
Where <file1> is route53.yml, and <file2> is a bash process substitution.
This solution simplifies issues around quoting and formatting.
(You'll note I dropped your use of -i here; that seems to throw yq for a loop, and it's easy to output to a temporary file and then rename it.)
Eventually I solved it using a combination of larsks answer about the unnecessary quotes, and using jq instead of yq (no matter what I did, using solely yq wouldn't work) to assign the array to a separate variable before editing the YAML document:
record_sets=$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" | jq '.[]')
yq '(.HostedZones[] | select(.Id=="ABC123")).ResourceRecordSets |= '"$record_sets"'' -i route53.yml
When I retrieve all records for a hosted zone in Cloudflare, e.g. of response, I need to create from it the following yaml structure:
name_zones: # this line we create
.zone_name: # the value is taken from the response
auth_key: XXX # this line we create
records: # this line we create
# iterate over all records
- name: .name
type: .type
priority: .priority # create line if value set|exist
content: .content
ttl: .ttl # create line if value set|exist
e.g. jq code which almost done this:
jq '.result[] | {name: .name, type: .type, content: .content, ttl: .ttl} + if has("priority") then {priority} else null end' | jq -n '.name_zone.zone_name.auth_key.records |= [inputs]' | yq r -P -
How to pass or create the value of zone_name and auth_key: XXX?
First, your two invocations of jq can be replaced by just one, as shown in the answer below.
Second, there are currently (at least) two yqs in the wild:
python-based yq (https://kislyuk.github.io/yq) - hereafter python-yq
go-based yq (https://github.com/mikefarah/yq)
For better and/or worse, version 4 of the go-based yq is significantly different from earlier versions, so if you want to use the current go-based version you may have to make adjustments accordingly. To simplify things (at least from my point of view), I will replace your yq r -P - by:
python-jq -y .
The following produces the output shown below.
< cf_response.json \
jq '{name_zones:
{zone_name: .result[0].zone_name,
auth_key: "XXX",
records:
[.result[]
| {name, type, content, ttl}
+ if has("priority")
then {priority}
else null end] }} '|
python-yq -y .
Output
name_zones:
zone_name: test.com
auth_key: XXX
records:
- name: test.com
type: A
content: 111.111.111.111
ttl: 1
- name: test.com
type: TXT
content: google-site-verification=content
ttl: 1
- name: test.com
type: MX
content: smtp.test.com
ttl: 1
priority: 0
auth_key
If you want to pass in the value of auth_key as a parameter to jq, you could use the command-line sequence --arg auth_key XXX, and then use $auth_key in the jq program.
Suppose I have two similar files:
a.yaml
data:
- name: a1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: a2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-a"]
values: ["else-in-a"]
substitutions:
key1: a-value
key2: a-value
key3: a-value
b.yaml
data:
- name: b1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: b2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-b"]
values: ["else-in-b"]
substitutions:
key1: b-value
key2: b-value
key3: b-value
My goal is to combine parts of a and b file such that I have a new file which consists of file content before substitutions: from b.yaml and content including and after substitutions: from a.yaml
So in this case, my desired output would be like this:
c.yaml
data:
- name: b1
args: ["cmd", "something"]
config:
- name: some
val: thing
- name: b2
args: ["cmd2", "else"]
[...other array values...]
tags: ["something-in-b"]
values: ["else-in-b"]
substitutions:
key1: a-value
key2: a-value
key3: a-value
The parts before and after substitutions: in both file contents might have different lengths.
Currently, my method is like this:
head -q -n `awk '/substitution/ {print FNR-1}' b.yaml` b.yaml >! c.yaml ; \
tail -q -n `awk '/substitution/ {ROWNUM=FNR} END {print NR-ROWNUM+1}' a.yaml` a.yaml >> c.yaml; \
rm a.yaml b.yaml; mv c.yaml a.yaml; # optional newfile renaming to original
But I wonder if there's an alternative or better method for combining parts of different files based on a common text string in bash?
Use awk, you just need to flag the flow based on the string:
awk '$1 == "substitutions:"{skip = FNR==NR ? 1:0}!skip' b.yaml a.yaml
Explaination:
FNR==NR: if true, process lines in the first file b.yaml, otherwise the 2nd file a.yaml
!skip: if TRUE, print the line, otherwise skip the line.
{
head -B9999 'substitutions:' a.yaml | head -n -1
head -A9999 'substitutions:' b.yaml
} > c.yaml
A oneliner:
{ head -B9999 'substitutions:' a.yaml | head -n -1; head -A9999 'substitutions:' b.yaml; } > c.yaml
The -A9999 and -B9999 are a bit dirty, here's a solution with sed's:
{
sed '/substitutions:/,$d' a.yaml
echo substitutions:
sed '1,/substitutions:/d' b.yaml
} > c.yaml