Hello currently I try to find a tool (I'm pretty sure yq does not the magic for me) to remove some content from a yaml file. My file looks as following:
paths:
/entity/{id}:
get:
tags: a
summary: b
...
So its a typical openapi-specification. I would like to add a magic property for example 'env: prod' so that some endpoints look as following:
paths:
/entity/{id}:
get:
env: prod
tags: a
summary: b
...
Is there a solution to remove all endpoints, which contain env: prod?
I am also free to change the concept. If there would be some transformation with a if else I would be very happy.
Using kislyuk/yq:
yq -y '.[][][] |= ({env: "prod"} + .)'
Using mikefarah/yq:
yq '.[][][] |= ({"env": "prod"} + .)'
Both produce:
paths:
/entity/{id}:
get:
env: prod
tags: a
summary: b
This adds env: prod to every object that is three levels deep. If you want the criteria be more sophisticated, you will have to adapt .[][][] accordingly.
yq 'del(.. | select(.env == "prod"))' file.yaml
Explanation:
You want to delete all the nodes that have a child 'env' property set to 'prod'.
.. recursively matches all nodes
select(.env == "prod") select the ones that have a env property equal to "prod"
del(.. | select(.env == "prod") delete those nodes :)
Disclaimer: I wrote mikefarah/yq
Related
I have a strange problem that I can't seem to get my head around. I am trying to define some variables for use as part of the job that will deploy bicep files via Azure CLI and execute PowerShell tasks.
I get this validation error when I try and execute the pipeline: While parsing a block mapping, did not find expected key
The line that it refers to is: - name: managementResourceDNSPrivateResolverName
On the research that I have done on this problem, it sounds like an indentation problem but on the face of it, it seems to look fine.
jobs:
- job: 'Deploy_Management_Resources'
pool:
vmImage: ${{ parameters.buildAgent }}
variables:
- name: managementResourceDNSPrivateResolverName
value: 'acme-$[ lower(parameters['environmentCode']) ]-$[ lower(variables['resourceLocationShort']) ]-private-dns-resolver'
- name: managementResourceGroupManagement
value: 'acme-infrastructure-rg-management'
- name: managementResourceRouteTableName
value: 'acme-$[ lower(variables['subscriptionCode']) ]-$[ lower(variables['resourceLocationShort']) ]-route-table'
- name: managementResourceVirtualNetworkName
value: 'acme-$[ lower(variables['subscriptionCode']) ]-$[ lower(variables['resourceLocationShort']) ]-vnet-internal-mng'
Thanks!
The error message ...parsing a block mapping, did not find expected key is usually a side-effect of malformed yaml. You'll see if often with variables if you have mixed formats of arrays and property elements
variables: # an array of objects
# variable group reference object
- group: myvariablegroup
# variable template reference object
- template: my-variables.yml
# variable object
- name: myVariable
value: 'value1'
# variable shorthand syntax
myVariable: 'value1' # this fails because it's a property instead of an array element
While it doesn't appear that the sample you've provided is malformed, I am curious about the use of $[ ] which is a runtime expression. The expression $[ lower(parameters['environmentcode']) ] refers to parameters which is are only available at compile time.
Change:
$[ lower(parameters['environmentCode']) ] to ${{ lower(parameters.environmentCode) }}
For readability purpose when displaying informations using yaml format, I'd like to be able to replace a yaml array with its json equivalent
The thing is that I may have several instances to replace in the yaml file/output with different paths
Example:
objects:
- object:
name: objA
inputs:
- dims:
- 1
- 3
- object:
name: objB
outputs:
- dims:
- 5
but I'd like the output to be json format for dims arrays, like
objects:
- object:
name: objA
inputs:
- dims: [1,3]
- object:
name: objB
outputs:
- dims: [5]
converting the value from yaml to json is easy, modifying the value of the yaml nodes is easy, but I don't see how I can get the value for a "dims" node, convert it to a json value string, a put it back in the node (I mean without searching explicitly all instances)
in general, I'm looking for a way to replace the value of a node, with the result of a process on the value of the node (other example, replacing an id with the name of the corresponding object retrieved through a REST api request)
objects:
- object:
name: objA
dependency: 3fc4bd5b-a6ee-4469-946d-5f780476784e
would be displayed as
objects:
- object:
name: objA
dependency: name-of-dependency
where the id is replaced by the friendly name of the dependency
thanks
With mikefarah's yq
yq e '.objects[].object["inputs","outputs"][].dims? |= "["+join(",")+"]"' data.yml
You could use tojson and the update operator |=. This will encode your arrays as JSON, which is a string and therefore itself enclosed in quotes:
yq -y '(.. | .dims? | arrays) |= tojson'
objects:
- object:
name: objA
inputs:
- dims: '[1,3]'
- object:
name: objB
outputs:
- dims: '[5]'
Run with Python yq
I need to remove the tags field from each of the methods in my OpenAPI spec.
The spec must be in YAML format, as converting to JSON causes issues later on when publishing.
I couldn't find a ready tool for that, and my programming skills are insufficient. I tried Python with ruamel.yaml, but could not achieve anything.
I'm open to any suggestions how to approach this - a repo with a ready tool somewhere, a hint on what to try in Python... I'm out of my own ideas.
Maybe a regex that catches all cases all instances of tags so I can do a search and replace with Python, replacing them with nothing? Empty lines don't seem to break the publishing engine.
Here's a sample YAML piece (I know this is not a proper spec, just want to show where in the YAML tags sits)
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint
get:
tags:
-
tag1
-
tag3
#rest of GET definition
post:
tags:
- tag2
/anotherEndpoint
post:
tags:
- tag1
I need to get rid of all tags arrays entirely (not just make them empty)
I am not sure why you couldn't achieve anything with Python + ruamel.yaml. Assuing your spec
is in a file input.yaml:
import sys
from pathlib import Path
import ruamel.yaml
in_file = Path('input.yaml')
out_file = Path('output.yaml')
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(in_file)
# if you only have the three instances of 'tags', you can hard-code them:
# del data['paths']['/endpoint']['get']['tags']
# del data['paths']['/endpoint']['post']['tags']
# del data['paths']['/anotherEndpoint']['post']['tags']
# generic recursive removal of any key names 'tags' in the datastructure:
def rm_tags(d):
if isinstance(d, dict):
if 'tags' in d:
del d['tags']
for k in d:
rm_tags(d[k])
elif isinstance(d, list):
for item in d:
rm_tags(item)
rm_tags(data)
yaml.dump(data, out_file)
which gives as output.yaml:
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint:
get: {}
post: {}
/anotherEndpoint:
post: {}
You can write back data to input.yaml if you need that.
Please note that normally the comment #rest of GET definition would be preserved, but
not now as it is associated during loading with the key before it and that key gets deleted.
I am looking for a way to dynamically set the key using the path of the file below.
For example if I have this YAML:
prospectors.config:
- fields:
queue_name: <somehow get the globbed string below in here>
paths:
- /var/log/casino/*.log
type: log
output.redis:
hosts:
- "producer:6379"
key: "%{[fields.queue_name]}"
And then I had a file called /var/log/casino/test.log, then key would become test.
Im not sure that what you want is possible.
You could use the source field and configure your Redis output using that as the key:
output.redis:
hosts:
- "producer:6379"
key: "%{source}"
This would have the disadvantage of being the absolute path of the source file, not the basename as your question asks for.
If you have a small number of possible basename patterns, and want a queue for each. For example, you have files:
/common/path/test-1.log
/common/path/foo-0.log
/common/path/01-bar.log
/common/path/test-3.log
...
and wanted to have three queues in redis test, foo and bar you could use the source field and the conditionals available in the keys configuration of redis output something like this
output.redis:
hosts:
- "producer:6379"
key: "default_key"
keys:
- key: "test_key"
when.contains:
source: "test"
- key: "foo_key"
when.contains:
source: "foo"
- key: "bar_key"
when.contains:
source: "bar"
Suppose I have:
base_array:
-1
-2
how could I do something like:
my_array: << base_array
-3
so that my_array was [1,2,3]
Update: I should specify that I want the extending to occur inside the YAML itself.
Since the already commented issue#35 exists, merge-keys << doesn't help you. It only merges/inserts referenced keys into a map (see YAML docs merge). Instead you should work with sequences and use anchor & and alias *.
So your example should look like this:
base_list: &base
- 1
- 2
extended: &ext
- 3
extended_list:
[*base, *ext]
Will give result in output like this (JSON):
{
"base_list": [
1,
2
],
"extended": [
3
],
"extended_list": [
[
1,
2
],
[
3
]
]
}
Although not exactly what you expected, but maybe your parsing/loading environment can flatten the nested array/list to a simple array/list.
You can always test YAML online, for example use:
http://ben-kiki.org/ypaste
Online YAML Parser
I needed to do the same thing but to run on Azure DevOps Pipeline. In particular, I had to update the stage dependency dynamically. How I did it:
dependents: [Stage_A, Stage_B]
otherDependents: [Stage_C] # This needed to be added by policy to the pipeline's execution
dependsOn:
- ${{ each dependent in dependents }}:
- ${{ dependent }}
- ${{ each dependent in otherDependents }}:
- ${{ dependent }}
Doing so resulted in the required setup:
dependents: [Stage_A, Stage_B]
otherDependents: [Stage_C] # This needed to be added by policy to the pipeline's execution
dependsOn:
- Stage_A
- Stage_B
- Stage_C
I say dynamically because the variable dependents came from a template to which I had to append Stage_C.