Convert Cloudflare API response to yaml - bash

When I retrieve all records for a hosted zone in Cloudflare, e.g. of response, I need to create from it the following yaml structure:
name_zones: # this line we create
.zone_name: # the value is taken from the response
auth_key: XXX # this line we create
records: # this line we create
# iterate over all records
- name: .name
type: .type
priority: .priority # create line if value set|exist
content: .content
ttl: .ttl # create line if value set|exist
e.g. jq code which almost done this:
jq '.result[] | {name: .name, type: .type, content: .content, ttl: .ttl} + if has("priority") then {priority} else null end' | jq -n '.name_zone.zone_name.auth_key.records |= [inputs]' | yq r -P -
How to pass or create the value of zone_name and auth_key: XXX?

First, your two invocations of jq can be replaced by just one, as shown in the answer below.
Second, there are currently (at least) two yqs in the wild:
python-based yq (https://kislyuk.github.io/yq) - hereafter python-yq
go-based yq (https://github.com/mikefarah/yq)
For better and/or worse, version 4 of the go-based yq is significantly different from earlier versions, so if you want to use the current go-based version you may have to make adjustments accordingly. To simplify things (at least from my point of view), I will replace your yq r -P - by:
python-jq -y .
The following produces the output shown below.
< cf_response.json \
jq '{name_zones:
{zone_name: .result[0].zone_name,
auth_key: "XXX",
records:
[.result[]
| {name, type, content, ttl}
+ if has("priority")
then {priority}
else null end] }} '|
python-yq -y .
Output
name_zones:
zone_name: test.com
auth_key: XXX
records:
- name: test.com
type: A
content: 111.111.111.111
ttl: 1
- name: test.com
type: TXT
content: google-site-verification=content
ttl: 1
- name: test.com
type: MX
content: smtp.test.com
ttl: 1
priority: 0
auth_key
If you want to pass in the value of auth_key as a parameter to jq, you could use the command-line sequence --arg auth_key XXX, and then use $auth_key in the jq program.

Related

How to merge two .yaml files such that shared keys between the files uses only one of their values?

I am attempting to merge two yaml files and would like any shared keys under a specific key to use values from one of the yaml files, and not merge both. This problem may be better described using an example. GIven file1.yaml and file2.yaml, I am trying to achieve the following:
file1.yaml
name: 'file1'
paths:
path1:
content: "t"
path2:
content: "b"
file2.yaml
name: 'file2'
paths:
path1:
value: "t"
My ideal result in merging is the following file:
file3.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"
Specifically, I would like to overwrite any key under paths such that if both yaml files have the same key under paths, then only use the value from file2. Is there some tool that enables this? I was looking into yq but I'm not sure if that tool would work
Please specify which implementation of yq you are using. They are quite similar, but sometimes differ a lot.
For instance, using kislyuk/yq, you can use input to access the second file, which you can provide alongside the first one:
yq -y 'input as $in | .name = $in.name | .paths += $in.paths' file1.yaml file2.yaml
name: file2
paths:
path1:
value: t
path2:
content: b
With mikefarah/yq, you'd use load with providing the second file in the code, while only the first one is your regular input:
yq 'load("file2.yaml") as $in | .name = $in.name | .paths += $in.paths' file1.yaml
name: 'file2'
paths:
path1:
value: "t"
path2:
content: "b"

Assign YAML array from input to key using `yq`

I'm trying to take a YAML style array I get from an AWS command, and assign it to a key while updating my own YAML.
This represents what I have right now:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= "'"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')"'"' -i route53.yml
This is how route53.yml looks like before I run the command:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
and this is how route53.yml looks like after:
HostedZones:
- CallerReference: abc-123
Id: /hostedzone/ABC123
Name: domain.name.com.
ResourceRecordSetCount: 5
ResourceRecordSets: |-
- Name: a.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: b.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: c.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: d.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
- Name: e.domain.name.com.
ResourceRecords:
- Value: some.value.com
TTL: 300
Type: CNAME
As you can see there's a |- right after the key, and it seems like it is treated as multiline string instead of an array of maps. How can I avoid it and assign the array as a YAML array? When I tried manually assigning an array in the style of ['a', 'b', 'c'] and the update works as it should, adding a YAML array under the key, how can I achieve it with the output of the aws command?
The reason why yq is interpreting the output of the $(...) expression as a multiline strine is because you have quoted it; your yq expression, simplified, looks like:
yq 'ResourceRecordSets |= "some string here"'
The quotes mean "this is a string, not a structure", so that's what you get. You could try dropping the quotes, like this:
yq '(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= '"$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml | yq '.ResourceRecordSets')" route53.yml
That might work, but it is fragile. A better solution is to have yq parse the output of the subexpression as a separate document, and then merge it as a structured document rather than a big string. Like this:
yq \
'(.HostedZones[] | select(.Id=="/hostedzone/ABC123")).ResourceRecordSets |= input.ResourceRecordSets' \
route53.yml \
<(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" --output yaml)
This takes advantage of the fact that you can provide jq (and hence yq) multiple files on the command line, and then refer to the input variable (described in the IO section of the jq manual). The above command line is structured like this:
yq <expression> <file1> <file2>
Where <file1> is route53.yml, and <file2> is a bash process substitution.
This solution simplifies issues around quoting and formatting.
(You'll note I dropped your use of -i here; that seems to throw yq for a loop, and it's easy to output to a temporary file and then rename it.)
Eventually I solved it using a combination of larsks answer about the unnecessary quotes, and using jq instead of yq (no matter what I did, using solely yq wouldn't work) to assign the array to a separate variable before editing the YAML document:
record_sets=$(aws route53 list-resource-record-sets --hosted-zone-id "ABC123" | jq '.[]')
yq '(.HostedZones[] | select(.Id=="ABC123")).ResourceRecordSets |= '"$record_sets"'' -i route53.yml

Use yq to update array value

I have two yaml files I am using. The first yaml file looks like this:
spring.yml
spring:
cloud:
gateway:
routes:
- id: someid
uri: someUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someOtherPath
I have another file that just contains routes and looks like this:
routes.yml
routes:
- id: someid
uri: someOtherUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someNewPath
My goal is to update the route in the first file with the value of the route in the second file. Note that the first file in reality will have many routes but for demonstration purposes I am only showing the first in this example. I have the following script which loops through to update the routes as necessary when the id's match:
#!/bin/sh
OVERRIDE_ROUTE_IDS=$(yq eval '.routes.[].id' routes.yml)
GENERATED_ROUTE_IDS=$(yq eval '.spring.cloud.gateway.routes.[].id' spring.yml)
SAVEIFS=$IFS # Save current IFS (Internal Field Separator)
IFS=$'\n' # Change IFS to newline char
OVERRIDE_ROUTE_IDS=($OVERRIDE_ROUTE_IDS) # split the `OVERRIDE_ROUTE_IDS` string into an array by the same name
GENERATED_ROUTE_IDS=($GENERATED_ROUTE_IDS) # split the `GENERATED_ROUTE_IDS` string into an array by the same name
IFS=$SAVEIFS # Restore original IFS
for (( i=0; i<${#OVERRIDE_ROUTE_IDS[#]}; i++ ))
do
if [[ "${GENERATED_ROUTE_IDS[*]}" =~ "${OVERRIDE_ROUTE_IDS[$i]}" ]]
then
echo "route ID ${OVERRIDE_ROUTE_IDS[$i]} exists in generated routes"
for (( j=0; j<${#GENERATED_ROUTE_IDS[#]}; j++ ))
do
if [[ "${GENERATED_ROUTE_IDS[$j]}" == "${OVERRIDE_ROUTE_IDS[$i]}" ]]
then
echo "index of route ${GENERATED_ROUTE_IDS[$j]} is $j"
echo "$i"
ROUTE_TO_USE=$(yq eval ".routes.[$i]" routes.yml)
$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" spring.yml)
fi
done
else
echo "no match so add to top of routes"
fi
done
My assumption is this command should update spring.yml file with the new route in place of the one that was identified with the same id:
$(yq ".spring.cloud.gateway.routes.[$j] = $ROUTE_TO_USE" application.yml)
But I am getting the following error
Error: Parsing expression: Lexer error: could not match text starting at 1:37 failing at 1:39 unmatched text: "id"
I'm stumped on this and not sure what I'm doing wrong at this point. For reference I am using yq version 4.17.2.
Be aware that yq does not emit a data structure, it emits a string. $ROUTE_TO_USE would be, for example,
id: someid
uri: someOtherUri
predicates:
- Path=/somePath
filters:
- RewritePath=/someNewPath
This is YAML source. Pasting this into the following yq command leads to invalid syntax; yq's expression syntax is not literal YAML. This is what the error tries to tell you.
What you want to do is to process both inputs in a single yq command:
yq ea "select(fi==0).spring.cloud.gateway.routes.[$j] = "`
`"select(fi==1).routes.[$i] | select(fi==0)" spring.yml routes.yml
ea is shorthand for eval-all which you need for processing multiple input files at the same time. fi is a shorthand for fileIndex, which is used to select the appropriate file. Piping the result to | select(fi==0) ensures that only the first (modified) file is written out. I split the long string into multiple lines using backticks for readability.
I ended up getting a solution from the creator of yq. The solution below is what I used:
yq ea '
(select(fi==0) | .spring.cloud.gateway.routes.[].id) as $currentIds |
(select(fi==1) | [.routes.[] | select( [$currentIds != .id] | all )] ) as $newRoutes |
( $newRoutes + .spring.cloud.gateway.routes + .routes) as $routesToMerge |
(
(($routesToMerge | .[] | {.id: .}) as $item ireduce ({}; . * $item )) as $uniqueMap
| ( $uniqueMap | to_entries | .[]) as $item ireduce([]; . + $item.value)
) as $mergedArray
| select(fi == 0) | .spring.cloud.gateway.routes = $mergedArray
' spring.yml routes.yml
This matches on id. If there is a match it uses the value of what's in routes.yml. If there is no match it add it the top top of the routes.

kubectl YML : what type of file is this and how to run it

I can see a way to create kubectl command like this where I can pass some parameterize values ad well.
My question is, with what type of file we will save this, Is this a bash script? and how to run and supply the parameter?
export SECRET_NAME="my-app.secret"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: $SECRET_NAME
type: Opaque
data:
password: $(echo -n "s33msi4" | base64 -w0)
username: $(echo -n "jane" | base64 -w0)
EOF
Yes, it can be treated as a bash script. As Jetchisel already mentioned in his comment, it contains a structure called Here Document used with cat command but it also contains export command which sets and exports a variable. So as a whole it can be treated as a simple bash script with 2 instructions.
In order to run it and create a new Secret object (which is your ultimate goal), follow these steps:
Fix the indentation which is crucial in yaml files:
export SECRET_NAME="my-app.secret"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: $SECRET_NAME
type: Opaque
data:
password: $(echo -n "s33msi4" | base64 -w0)
username: $(echo -n "jane" | base64 -w0)
EOF
Save the above content as a file. You can call it secret.sh.
Source it (source and . are the same command):
. secret.sh
You should see the following message:
secret/my-app.secret created
Alternatvely you can paste it directly into the console. As you can see, it also works:
### don't copy it, this is the example console output
### you will see once you paste the above script in your bash shell
$ export SECRET_NAME="my-app.secret"
$
$ cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Secret
> metadata:
> name: $SECRET_NAME
> type: Opaque
> data:
> password: $(echo -n "s33msi4" | base64 -w0)
> username: $(echo -n "jane" | base64 -w0)
> EOF
secret/my-app.secret created

Probable quoting issue using the stdin argument of ansible's shell module

I've got a playbook with the following tasks:
- set_fact:
asg_filter: >
.AutoScalingGroups[] |
select(.Tags[] | select(.Key == "Role").Value == "myrole")
- shell: aws autoscaling --region us-west-2 describe-auto-scaling-groups | jq --compact-output "{{ asg_filter }}"
register: asgs_result
- set_fact:
stale_instance_filter: >
.LaunchConfigurationName as $lc |
.Instances[] |
select(.LaunchConfigurationName != $lc) |
.InstanceId
Now I want to use stale_instance_filter on asgs_result.stdout. The following works:
- shell: echo '{{ asgs_result.stdout }}' | jq -r '{{ stale_instance_filter }}'
But this doesn't:
- shell: jq -r '{{ stale_instance_filter }}'
args:
stdin: "{{ asgs_result.stdout }}"
I get the following error message: parse error: Invalid numeric literal at line 1, column 23 (which I believe is from the account number in the ARN for the ASG.) I think it's a quoting issue (maybe something about the double quotes in the JSON), but I've also tried asgs_result.stdout | quote to no avail. I also tried the command module; it didn't help either. Of course this all works if I do it directly on the CLI.
I realize I could combine the two jq filters but I want to reuse asgs_result for other things and don't want to have to make the query multiple times. How can I fix this so I can use the stdin argument?
Edit: I was asked to provide an example of the value of asgs_result, well here you go, here's the stdout attribute in it (since I don't use anything else):
"stdout": "{\"AutoScalingGroupARN\":\"arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev\",\"TargetGroupARNs\":[],\"SuspendedProcesses\":[],\"DesiredCapacity\":4,\"Tags\":[{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"dev\",\"Key\":\"Dimension\"},{\"ResourceType\":\"auto-scaling-group\",\"ResouJceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"true\",\"Key\":\"Monitored\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"myrole\",\"Key\":\"Name\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"myrole\",\"Key\":\"Role\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"2035-09-30 18:55:31 +0000\",\"Key\":\"cleaner-destroy-after\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"vpce-2c23ca45\",\"Key\":\"force_s3_endpoint_dependency\"},{\"ResourceType\":\"auto-scaling-group\",\"ResourceId\":\"myrole-dev\",\"PropagateAtLaunch\":true,\"Value\":\"owned\",\"Key\":\"kubernetes.io/cluster/dev\"}],\"EnabledMetrics\":[],\"LoadBalancerNames\":[],\"AutoScalingGroupName\":\"myrole-dev\",\"DefaultCooldown\":300,\"MinSize\":4,\"Instances\":[{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2b\",\"InstanceId\":\"i-0141fd35e3cf3ad0a\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2c\",\"InstanceId\":\"i-01aec2b3546d75190\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2a\",\"InstanceId\":\"i-0830b227f034d2859\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"},{\"ProtectedFromScaleIn\":false,\"AvailabilityZone\":\"us-east-2b\",\"InstanceId\":\"i-0f7d847e8c168040b\",\"HealthStatus\":\"Healthy\",\"LifecycleState\":\"InService\",\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\"}],\"MaxSize\":4,\"VPCZoneIdentifier\":\"subnet-c348988e,subnet-79743210,subnet-156ee36e\",\"HealthCheckGracePeriod\":300,\"TerminationPolicies\":[\"Default\"],\"LaunchConfigurationName\":\"dev_myrole_20180511171410107500000002\",\"CreatedTime\":\"2018-02-20T22:35:32.183Z\",\"AvailabilityZones\":[\"us-east-2a\",\"us-east-2b\",\"us-east-2c\"],\"HealthCheckType\":\"EC2\",\"NewInstancesProtectedFromScaleIn\":false}"
Sorry that it is all on one line but I don't want to make anyone think there is a newline in there, because there isn't.
The JSON content seems to be interpreted before sent to the stdin, so looks like simple quotes are sent (seen in verbose mode with -vvv):
"stdin": "{'AutoScalingGroupARN': 'arn:aws:autoscaling:us-east-2:123456:autoScalin
gGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev', ...,
'AvailabilityZones': ['us-east-2a', 'us-east-2b', 'us-east-2c']}"
Which is not JSON valid:
$ echo "{'AutoScalingGroupARN': 'arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev', 'HealthCheckGracePeriod': 300}" | jq
parse error: Invalid numeric literal at line 1, column 23
$ echo '{"AutoScalingGroupARN": "arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev", "HealthCheckGracePeriod": 300}' | jq
{
"AutoScalingGroupARN": "arn:aws:autoscaling:us-east-2:123456:autoScalingGroup:e75a213b-75fe-467c-8cf5-d7c51f76c471:autoScalingGroupName/myrole-dev",
"HealthCheckGracePeriod": 300
}
So, you need to "escape" it.
Unfortunately, the to_json filter, escape to much:
"stdin": "\"{\\\"AutoScalingGroupARN\\\":\\\"arn:aws:autosca...
But the string filter fits perfectly:
"stdin": "{\"AutoScalingGroupARN\":\"arn:aws:autosca...
So, the correct way with stdin is this:
- shell: jq -r '{{ stale_instance_filter }}'
args:
stdin: "{{ asgs_result.stdout | string }}"

Resources