Say I have a YAML like:
Resources:
AlarmTopic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- !If
- ShouldAlarm
Protocol: email
How do I get each key and value of all the children if I'm walking over each resource and I want to know if one of the values may contain a certain string? I'm using PyYAML but I'm also open to using some other library.
You can use the low-level event API if you only want to inspect scalar values:
import yaml
import sys
input = """
Resources:
AlarmTopic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- !If
- ShouldAlarm
- Protocol: email
"""
for e in yaml.parse(input):
if isinstance(e, yaml.ScalarEvent):
print(e.value)
(I fixed your YAML because it had a syntax error.) This yields:
Resources
AlarmTopic
Type
AWS::SNS::Topic
Properties
Subscription
ShouldAlarm
Protocol
email
Related
I have created Cognito UserPool and UserpoolClient via Resources in serverless.yml file like this -
CognitoUserPool:
Type: AWS::Cognito::UserPool
Properties:
AccountRecoverySetting:
RecoveryMechanisms:
- Name: verified_email
Priority: 2
UserPoolName: ${self:provider.stage}-user-pool
UsernameAttributes:
- email
MfaConfiguration: OFF
Policies:
PasswordPolicy:
MinimumLength: 8
RequireLowercase: True
RequireNumbers: True
RequireSymbols: True
RequireUppercase: True
CognitoUserPoolClient:
Type: AWS::Cognito::UserPoolClient
Properties:
ClientName: ${self:provider.stage}-user-pool-client
UserPoolId:
Ref: CognitoUserPool
ExplicitAuthFlows:
- ALLOW_USER_PASSWORD_AUTH
- ALLOW_REFRESH_TOKEN_AUTH
GenerateSecret: true
Now I can pass the Userpool and UserpoolClient as environment variables to the lambda functions like this -
my_function:
package: {}
handler:
events:
- http:
path:<path>
method: post
cors: true
environment:
USER_POOL_ID: !Ref CognitoUserPool
USER_POOL_CLIENT_ID: !Ref CognitoUserPoolClient
I can access these IDs in my code as -
USER_POOL_ID = os.environ['USER_POOL_ID']
USER_POOL_CLIENT_ID = os.environ['USER_POOL_CLIENT_ID']
I have printed the values and they are being printed correctly. However, UserpoolClient also generates one AppClient secret which I need to use while generating secret hash. How shall I access app client secret (UserpoolClient's secret) in my lambda?
Probably now what you hoped for, but you cannot export client secret in CloudFormation explicitly. Take a look at the return values from AWS::Cognito::UserPoolClient. There you can only get the client ID.
What you could do is to create the client in another CF template and either create there a custom resource to read the secret and output it, or have an intermediate step where you get this value with CLI and then pass it into serverless.
There is currently no other option.
I tried to separate my resources field in multiple files in project. Most of then worked fine, but only this file, where I declare CognitoUserPool + CognitoUserPoolClient trowed this exception:
Unhandled rejection YAMLException: duplicated mapping key in "/home/uriel/Desktop/Foo/Backend/MultipleFile/backend-foo/resources/cognitoUserPoolFoo.yml" at line 20, column 3:
Type: AWS::Cognito::UserPoolClient
^
at generateError (/home/uriel/.nvm/versions/node/v10.16.0/lib/node_modules/serverless/node_modules/js-yaml/lib/js-yaml/loader.js:167:10)
I've already checked logical and indentation problems. This same lines worked on single file, they only throw this error when I move them to another YML file and import it.
Main YML file importing the another one
plugins:
- serverless-webpack
- serverless-python-requirements
- serverless-offline
resources:
- ${file(resources/cognitoUserPoolFoo.yml)}
File imported, the one that throw error.
Resources:
CognitoUserPoolFoo:
Type: AWS::Cognito::UserPool
Properties:
MfaConfiguration: OFF
UserPoolName: foo-${self:provider.stage}
EmailConfiguration:
ReplyToEmailAddress: foo#test.com
SourceArn: "arn:aws:ses:us-east-1:123456789012:identity/foo#test.com"
AutoVerifiedAttributes:
- email
Policies:
PasswordPolicy:
MinimumLength: 6
RequireLowercase: True
RequireNumbers: True
RequireSymbols: False
RequireUppercase: True
FooCognitoUserPoolClient:
Type: AWS::Cognito::UserPoolClient
Properties:
ClientName: FooWebApp-${self:provider.stage}
GenerateSecret: false
UserPoolId:
Ref: "CognitoUserPoolFoo"
I need to remove the tags field from each of the methods in my OpenAPI spec.
The spec must be in YAML format, as converting to JSON causes issues later on when publishing.
I couldn't find a ready tool for that, and my programming skills are insufficient. I tried Python with ruamel.yaml, but could not achieve anything.
I'm open to any suggestions how to approach this - a repo with a ready tool somewhere, a hint on what to try in Python... I'm out of my own ideas.
Maybe a regex that catches all cases all instances of tags so I can do a search and replace with Python, replacing them with nothing? Empty lines don't seem to break the publishing engine.
Here's a sample YAML piece (I know this is not a proper spec, just want to show where in the YAML tags sits)
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint
get:
tags:
-
tag1
-
tag3
#rest of GET definition
post:
tags:
- tag2
/anotherEndpoint
post:
tags:
- tag1
I need to get rid of all tags arrays entirely (not just make them empty)
I am not sure why you couldn't achieve anything with Python + ruamel.yaml. Assuing your spec
is in a file input.yaml:
import sys
from pathlib import Path
import ruamel.yaml
in_file = Path('input.yaml')
out_file = Path('output.yaml')
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=4, sequence=4, offset=2)
yaml.preserve_quotes = True
data = yaml.load(in_file)
# if you only have the three instances of 'tags', you can hard-code them:
# del data['paths']['/endpoint']['get']['tags']
# del data['paths']['/endpoint']['post']['tags']
# del data['paths']['/anotherEndpoint']['post']['tags']
# generic recursive removal of any key names 'tags' in the datastructure:
def rm_tags(d):
if isinstance(d, dict):
if 'tags' in d:
del d['tags']
for k in d:
rm_tags(d[k])
elif isinstance(d, list):
for item in d:
rm_tags(item)
rm_tags(data)
yaml.dump(data, out_file)
which gives as output.yaml:
openapi: 3.0.0
info:
title: ""
description: ""
paths:
/endpoint:
get: {}
post: {}
/anotherEndpoint:
post: {}
You can write back data to input.yaml if you need that.
Please note that normally the comment #rest of GET definition would be preserved, but
not now as it is associated during loading with the key before it and that key gets deleted.
I am trying to update the yaml file using ruamel python.
proc=subprocess.Popen(['kubectl','get','pod','web3','-o','yaml','--export'], stdout=subprocess.PIPE)
rein=proc.stdout.read()
result, indent, block_seq_indent = ruamel.yaml.util.load_yaml_guess_indent(rein, preserve_quotes=True)
So far I have tried :
result['spec'].append('nodeSelector')
which gives ERROR :
result['spec'].append('nodeSelector')
AttributeError: 'CommentedMap' object has no attribute 'append'
Also tried like this :
result['spec']['nodeSelector']['kubernetes.io/hostname']='kubew1'
gives :
result['spec']['nodeSelector']['kubernetes.io/hostname']='kubew1'
File "/usr/local/lib/python3.6/dist-packages/ruamel/yaml/comments.py", line 752, in __getitem__
return ordereddict.__getitem__(self, key)
KeyError: 'nodeSelector'
My Intial Yaml File is :
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
app: demo
name: web
name: web3
selfLink: /api/v1/namespaces/default/pods/web3
spec:
containers:
- image: aexlab/flask-sample-one
imagePullPolicy: Always
name: web
ports:
- containerPort: 5000
name: http
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-7bcc9
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
And Expected fields I want to add inside 'spec' is :
nodeSelector:
kubernetes.io/hostname: kubew1
Any Ideas how to achieve this with ruamel library.
In your YAML file your root level collection is a mapping and the value for the key spec in that mapping is itself a mapping. Both of those mappings get loaded as dict-like objects using ruamel.yaml named CommentedMap.
As with normal dicts you can add key-value pairs, deleted keys (and their values), and update values for a key, but there is no .append() method, as there is with a list (i.e. appending an extra item to a list).
Your output is a bit terse, but of course you cannot just add nodeSelector to anything (list/sequence nor dict/mapping) and expect that to add kubernetes.io/hostname: kubew1 (a mapping in its own right) automatically.
Your try of:
result['spec']['nodeSelector']['kubernetes.io/hostname'] = 'kubew1'
cannot work because there is no dict result['spec']['nodeSelector'] where you can add the key kubernetes.io/hostname.
You would either first have to create a key with an emtpy dict as value:
result['spec']['nodeSelector'] = {}
result['spec']['nodeSelector']['kubernetes.io/hostname'] = 'kubew1'
or do
result['spec']['nodeSelector'] = {'kubernetes.io/hostname': 'kubew1'}
Please note that the above has nothing much to do with ruamel.yaml, that is just basic Python data structure manipulation. Also note that there are over 100 libraries in the ruamel namespace, out of which ruamel.yaml is just one of several published as open source, so using ruamel is not very clear statement, although of course the context often provides enough information on which library you actually use.
I am looking for a way to dynamically set the key using the path of the file below.
For example if I have this YAML:
prospectors.config:
- fields:
queue_name: <somehow get the globbed string below in here>
paths:
- /var/log/casino/*.log
type: log
output.redis:
hosts:
- "producer:6379"
key: "%{[fields.queue_name]}"
And then I had a file called /var/log/casino/test.log, then key would become test.
Im not sure that what you want is possible.
You could use the source field and configure your Redis output using that as the key:
output.redis:
hosts:
- "producer:6379"
key: "%{source}"
This would have the disadvantage of being the absolute path of the source file, not the basename as your question asks for.
If you have a small number of possible basename patterns, and want a queue for each. For example, you have files:
/common/path/test-1.log
/common/path/foo-0.log
/common/path/01-bar.log
/common/path/test-3.log
...
and wanted to have three queues in redis test, foo and bar you could use the source field and the conditionals available in the keys configuration of redis output something like this
output.redis:
hosts:
- "producer:6379"
key: "default_key"
keys:
- key: "test_key"
when.contains:
source: "test"
- key: "foo_key"
when.contains:
source: "foo"
- key: "bar_key"
when.contains:
source: "bar"