Updating the Yaml file, with new fields using ruamel - yaml

I am trying to update the yaml file using ruamel python.
proc=subprocess.Popen(['kubectl','get','pod','web3','-o','yaml','--export'], stdout=subprocess.PIPE)
rein=proc.stdout.read()
result, indent, block_seq_indent = ruamel.yaml.util.load_yaml_guess_indent(rein, preserve_quotes=True)
So far I have tried :
result['spec'].append('nodeSelector')
which gives ERROR :
result['spec'].append('nodeSelector')
AttributeError: 'CommentedMap' object has no attribute 'append'
Also tried like this :
result['spec']['nodeSelector']['kubernetes.io/hostname']='kubew1'
gives :
result['spec']['nodeSelector']['kubernetes.io/hostname']='kubew1'
File "/usr/local/lib/python3.6/dist-packages/ruamel/yaml/comments.py", line 752, in __getitem__
return ordereddict.__getitem__(self, key)
KeyError: 'nodeSelector'
My Intial Yaml File is :
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
app: demo
name: web
name: web3
selfLink: /api/v1/namespaces/default/pods/web3
spec:
containers:
- image: aexlab/flask-sample-one
imagePullPolicy: Always
name: web
ports:
- containerPort: 5000
name: http
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-7bcc9
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
And Expected fields I want to add inside 'spec' is :
nodeSelector:
kubernetes.io/hostname: kubew1
Any Ideas how to achieve this with ruamel library.

In your YAML file your root level collection is a mapping and the value for the key spec in that mapping is itself a mapping. Both of those mappings get loaded as dict-like objects using ruamel.yaml named CommentedMap.
As with normal dicts you can add key-value pairs, deleted keys (and their values), and update values for a key, but there is no .append() method, as there is with a list (i.e. appending an extra item to a list).
Your output is a bit terse, but of course you cannot just add nodeSelector to anything (list/sequence nor dict/mapping) and expect that to add kubernetes.io/hostname: kubew1 (a mapping in its own right) automatically.
Your try of:
result['spec']['nodeSelector']['kubernetes.io/hostname'] = 'kubew1'
cannot work because there is no dict result['spec']['nodeSelector'] where you can add the key kubernetes.io/hostname.
You would either first have to create a key with an emtpy dict as value:
result['spec']['nodeSelector'] = {}
result['spec']['nodeSelector']['kubernetes.io/hostname'] = 'kubew1'
or do
result['spec']['nodeSelector'] = {'kubernetes.io/hostname': 'kubew1'}
Please note that the above has nothing much to do with ruamel.yaml, that is just basic Python data structure manipulation. Also note that there are over 100 libraries in the ruamel namespace, out of which ruamel.yaml is just one of several published as open source, so using ruamel is not very clear statement, although of course the context often provides enough information on which library you actually use.

Related

Access a property in the body of a kubernetes resource using a field path

I want to get the values of the fields declared in the downwardAPI section of a Pod.
apiVersion: v1
kind: Pod
metadata:
name: sample
namespace: default
spec:
containers:
- image: rpa
imagePullPolicy: Always
name: testbot
volumeMounts:
- mountPath: /etc/pod-info
name: pod-info
volumes:
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.labels
path: labels
- fieldRef:
apiVersion: v1
fieldPath: metadata.name
path: pod-name
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: pod-namespace
name: pod-info
Using client-go, I can use pod.Spec.Volumes[0].DownwardAPI.Items to get the item slice including the fieldPath. But I would now need to dynamically able to fetch whatever values has been declared in the fieldPath. So, from the first item, I would like to access the value of metadata.labels. I could do pod.ObjectMeta.Labels but I would like to access the field dynamically. In terms of Javascript it would have been something like
var foo="metadata.labels"
var fooarr = foo.split(".")
var bar={
metadata:{
labels: "foobar"
}
}
var temp = oof
for(lm of lmaoarr){
temp = temp[lm]
}
console.log(temp)
How do I do something similar using client-go?
The standard kubelet code has logic to translate the downward API fields into environment variables. It is neither simple nor generic, though: at the bottom of the stack, only the specific fields listed in the Kubernetes documentation are supported. It would be incomplete, but not wrong or inconsistent with standard Kubernetes, to just match on these specific fields:
for _, item := range downwardApiObject.Items {
switch item.FieldPath.FieldRef {
case "metadata.name":
return pod.ObjectMeta.Name
}
}
The actual code:
Calls pods.ConvertDownwardAPIFieldLabel which does some very lightweight normalization and validation: subscripts are only allowed on metadata.annotations and metadata.labels, only the dozen or so specific field names are allowed, and spec.host is rewritten to spec.nodeName.
Handles the spec.* and status.* variables that depend on non-metadata fields in the pod spec or runtime data.
Delegates to fieldpath.ExtractFieldPathAsString which knows how to handle the metadata.* variables.
The k8s.io/kubernetes/pkg/fieldpath package contains a couple of helpers that are used in processing the downward API, and a really short answer to your specific question could be just to call fieldpath.ExtractFieldPathAsString passing it the Pod object, which will handle the metadata fields but nothing else.

How to parse PodSpec.spec.imagePullSecrets from a yaml file?

I want to parse the following structure using go:
---
prjA:
user1:
metadata:
namespace: prj-ns
spec:
containers:
- image: some-contaner:latest
name: containerssh-client-image
resources:
limits:
ephemeral-storage: 4Gi
requests:
ephemeral-storage: 2Gi
securityContext:
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
imagePullSecrets:
- docker-registry-secret
I'm using sigs.k8s.io/yaml to unmarshal YAML:
var userConfig map[string]map[string]kubernetes.PodConfig
err = yaml.UnmarshalStrict(yamlFile, &userConfig)
where kubernetes is imported from github.com/containerssh/kubernetes. Everything works fine - except the immagePullSecrets which gives the following error:
ERROR unmarshal user config file; error [error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field PodSpec.spec.imagePullSecrets of type v1.LocalObjectReference]
What is the correct way to specify / parse an imagePullSecrets in go?
This is a problem with the input - and maybe not a very clear error message.
The imagePullSecrets must be specified using the key name like:
imagePullSecrets:
- name: docker-registry-secret
I leave the question as it might help other people who run in the same problem.

Get all children key values in a YAML with PyYAML

Say I have a YAML like:
Resources:
AlarmTopic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- !If
- ShouldAlarm
Protocol: email
How do I get each key and value of all the children if I'm walking over each resource and I want to know if one of the values may contain a certain string? I'm using PyYAML but I'm also open to using some other library.
You can use the low-level event API if you only want to inspect scalar values:
import yaml
import sys
input = """
Resources:
AlarmTopic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- !If
- ShouldAlarm
- Protocol: email
"""
for e in yaml.parse(input):
if isinstance(e, yaml.ScalarEvent):
print(e.value)
(I fixed your YAML because it had a syntax error.) This yields:
Resources
AlarmTopic
Type
AWS::SNS::Topic
Properties
Subscription
ShouldAlarm
Protocol
email

Unable to dump as "pure" YAML

ruamel.yaml==0.15.37
Python 3.6.2 :: Continuum Analytics, Inc.
Current code:
from ruamel.yaml import YAML
import sys
yaml = YAML()
kube_context = yaml.load('''
apiVersion: v1
clusters: []
contexts: []
current-context: ''
kind: Config
preferences: {}
users: []
''')
kube_context['users'].append({'name': '{username}/{cluster}'.format(username='test', cluster='test'), 'user': {'token': 'test'}})
kube_context['clusters'].append({'name': 'test', 'cluster': {'server': 'URL:443'}})
kube_context['contexts'].append({'name': 'test', 'context': {'user': 'test', 'cluster': 'test'}})
yaml.dump(kube_context, sys.stdout)
My yaml.dump() is producing output that contains the list and dict objects, instead of being fully expanded.
Current output:
apiVersion: v1
clusters: [{name: test, cluster: {server: URL:443}}]
contexts: [{name: test, context: {user: test, cluster: test}}]
current-context: ''
kind: Config
preferences: {}
users: [{name: test/test, user: {token: test}}]
What do I need to do in order to have yaml.dump() output fully expanded?
Expected output:
apiVersion: v1
clusters:
- name: test
cluster:
server: URL:443
contexts:
- name: test
context:
user: test
cluster: test
current-context: ''
kind: Config
preferences: {}
users:
- name: test/test
user:
token: test
ruamel.yaml, when using the default YAML() or YAML(typ='rt') will preserve the flow- or block style of sequences and mappings. There is no way to make a block style empty sequence or empty mapping and your [] and {} are therefore tagged as flow style when loaded.
Flow style can only contain flow style (whereas block style can contain block style or flow style) (YAML 1.2 spec 8.2.3):
YAML allows flow nodes to be embedded inside block collections (but not vice-versa).
Because of that, the dict/mapping data that you insert in the (flow-style) list/sequence will also be represented as flow-style.
If you want everything to be block style (what you call "expanded" mode), you can explicitly set that by calling the .set_block_style() method on the .fa attribute (which is only available on the collections, hence the try/except):
from ruamel.yaml import YAML
import sys
yaml = YAML()
kube_context = yaml.load('''
apiVersion: v1
clusters: []
contexts: []
current-context: ''
kind: Config
preferences: {}
users: []
''')
kube_context['users'].append({'name': '{username}/{cluster}'.format(username='test', cluster='test'), 'user': {'token': 'test'}})
kube_context['clusters'].append({'name': 'test', 'cluster': {'server': 'URL:443'}})
kube_context['contexts'].append({'name': 'test', 'context': {'user': 'test', 'cluster': 'test'}})
for k in kube_context:
try:
kube_context[k].fa.set_block_style()
except AttributeError:
pass
yaml.dump(kube_context, sys.stdout)
this gives:
apiVersion: v1
clusters:
- name: test
cluster:
server: URL:443
contexts:
- name: test
context:
user: test
cluster: test
current-context: ''
kind: Config
preferences: {}
users:
- name: test/test
user:
token: test
Please note that it is not necessary to set yaml.default_flow_style = False in the default round-trip-mode; and that although block-style has been set for the value of key preferences, it is represented flow style as there is no other way to represent an empty mapping.
The output is „pure“ YAML. You want the nodes to be presented in block style (indentation-based) as opposed to the current flow style ([]{}-based). Here's how to do that:
yaml = YAML(typ="safe")
yaml.default_flow_style = False
(Note Athon's comment on the typ below; you need to set it to safe or unsafe so that the RoundTripLoader does not set the style of the empty sequences)

Use globbed string in YAML

I am looking for a way to dynamically set the key using the path of the file below.
For example if I have this YAML:
prospectors.config:
- fields:
queue_name: <somehow get the globbed string below in here>
paths:
- /var/log/casino/*.log
type: log
output.redis:
hosts:
- "producer:6379"
key: "%{[fields.queue_name]}"
And then I had a file called /var/log/casino/test.log, then key would become test.
Im not sure that what you want is possible.
You could use the source field and configure your Redis output using that as the key:
output.redis:
hosts:
- "producer:6379"
key: "%{source}"
This would have the disadvantage of being the absolute path of the source file, not the basename as your question asks for.
If you have a small number of possible basename patterns, and want a queue for each. For example, you have files:
/common/path/test-1.log
/common/path/foo-0.log
/common/path/01-bar.log
/common/path/test-3.log
...
and wanted to have three queues in redis test, foo and bar you could use the source field and the conditionals available in the keys configuration of redis output something like this
output.redis:
hosts:
- "producer:6379"
key: "default_key"
keys:
- key: "test_key"
when.contains:
source: "test"
- key: "foo_key"
when.contains:
source: "foo"
- key: "bar_key"
when.contains:
source: "bar"

Resources