I need some help understanding why, when I create a YAML file for an Ansible playbook that exactly mirrors what is specified in the module documentation, lists and list_items aren't being parsed correctly when the YAML is read by Python.
My understanding is that when reading through documentation about Ansible modules on Github, items are displayed within columns to denote their relationship relative to items above-and-below them. Further, if an item is defined as "list / elements=dictionary" then that means the following:
Typing of the item must be finished by adding a colon-and-space (this
defines "dictionary" items)
The next/subsequent line must begin at
the same indentation level (or greater) and start with a
hyphen-and-space (this denotes a "list" item)
The order of items in a list must be followed exactly.
Subsequent list items must be at the same level of indentation as the first item in the list
If any given sub-item in the list is ITSELF another "list / elements=dictionary" item repeat
step-2 again.
(EXAMPLE: Taken from the cisco.ios.ios_ospfv2 module)
So if the main item of "processes" is a "list / elements=dictionary" composed of the following:
areas (also a "list / elements=dictionary")
network (also a "list /
elements=dictionary")
process_id (integer)
Then the correct YAML syntax for the above would be:
processes:
- areas:
- area_id: 0 ##dictionary_item
- authentication: ##dictionary_item
message_digest: true ##boolean
- network:
- address: '10.1.1.0' ##string
- area: '0' ##string
- wildcard_bits: 0.0.0.255. ##integer
- process_id: 1
However, even though I don't get any YAML errors when I start my Playbook, I do get a bunch of obscure Python tracebacks. But if I change my code to THIS (see below) it works!
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
network:
address: 0.0.0.0
area: 0
wildcard_bits: 255.255.255.255
My questions:
If the order of list_items is important, why did my file NOT work when I followed the order specified in the module documentation...but it DOES work when that order is ignored ("process_id" should NOT be first according to the docs for this module)?
I understand that not EVERY list_item needs to be preceded with a hyphen-space. But I DID think that the FIRST list_item required this. So why (in my first example) does my playbook fail when I denote the sub-items under "network" with hyphens...but when I remove ALL of the hyphens from this list, it DOES work? However it DOES require a hyphen for the first list_item under "areas"??
Python Errors
(sorry for the long post...I didn't know to condense it any further than this)
Here is for a general explanation of this YAML:
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
network:
address: 0.0.0.0
area: 0
wildcard_bits: 255.255.255.255
processes: you may have 1 to N process, so it is a list, and in this case, a list of dictionary.
You should expect something like:
processes:
- foo: bar
baz: qux
- foo: some
baz: thing
process_id: is an int, you should expect:
processes:
- foo: bar
baz: qux
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
process_id: 1
- foo: some
baz: thing
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
process_id: 2
So this is the exact same YAML
processes:
- process_id: 1
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
foo: bar
baz: qux
- process_id: 2
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
foo: some
baz: thing
areas: is a list of 1 to N area, under the key areas of a dictionary, nested in a list, of key processes
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
- area_id: '1'
authentication:
message_digest: true
- process_id: 2
areas:
- area_id: '0'
authentication:
message_digest: true
network: is a dictionary in a dictionary
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
- area_id: '1'
authentication:
message_digest: true
- process_id: 2
areas:
- area_id: '0'
authentication:
message_digest: true
network:
address: 0.0.0.0
area: 0
wildcard_bits: 255.255.255.255
So in general, pay attention to how Ansible is also giving you hints here: areas is pluralised for a good reason: because it is a list of area. Same goes for processes. While network is a singular, so, as expected it will be a dictionary and not a list.
list / elements=dictionary
Means that processes must be a list where each element is a dictionary.
In YAML this is a dictionary:
foo:
some: bar
property: bar
of: bar
the: bar
dictionary: bar
of: bar
key: bar
foo: bar
And this is a list:
fruits:
- banana
- apple
- pear
- peach
Then a list of dictionary would be:
- the:
red: fox
jump: over
- the:
lazy: dog
Mind that, because this is a list, then the same key can be reproduced in the different elements of the list but cannot be seen twice in the same dictionary.
So to answer your second question, you are correct that in a dictionary the order of keys do not matter:
the:
red: fox
jump: over
Is indeed the exact same dictionary as
the:
jump: over
red: fox
But since here you are making a list of what should be a dictionary, that's where your issue reside.
A little note on the hyphen (-), too, because this seems to bug you:
If you prefer it, write your lists like this:
-
foo: bar
bar: foo
-
baz: qux
qux: baz
This is stricly equivalent to:
- foo: bar
bar: foo
- baz: qux
qux: baz
This would help you see that the hyphen just does delimit the fact that a new element of the list begins there, now you can "condense" it and have a key of your dictionary on the same line as the "new item of list" delimiter (-), but you are not forced to.
Maybe for an object oriented programming language parallel, if that could be of any help for you, you should think
a dictionary is like an object, it has multiple properties, but all the properties are actually defining the same object
a list is a collection of things, it can be a list of object, of integer, or even of object.
So when the documentation says list / elements=dictionary they actually want you do construct a list (or collection) where all the elements would be dictionaries (or objects).
Maybe the layout on the documentation page itself is easier to read than the table on GitHub.
But there, you can clearly see that the config dictionary holds a processes list that, itself contains a lot of different dictionaries, among which address_family, adjacency, ..., areas, network, ...
Thank you everyone for your insights. I've conferred with others and I think this is an issue of a documentation error for this module. The item of "network" is documented as being "list / elements=dictionary".
In reality the underlying Python code expects to see it as a dictionary (not a list). When added as a dictionary it works fine. Please see my link in which I provide a screenshot of the (incorrect) documentation as well as the Python error that results when "network" is configured as a "list / elements=dictionary".
enter image description here
Update: I was previously using the Ansible cisco.ios collection version 1.1.0 (which requires that the "network" parameter be entered as a dictionary in the python code for the module, "ios_ospfv2").
Upon upgrading to version 1.2.0 of the cisco.ios collection the "network" parameter no longer worked as a dictionary and needed to be changed back to a list (which matches the documentation for this module).
So apparently the version of the Ansible Collection one uses may change how parameters are defined within modules. Moral of the story? Ensure you're using the latest version of whatever Collection you need.
Determining Most Recent Collection Version
Related
How can I configure Ansible to assert duplicate keys?
I don't want to rely on ANSIBLE_DUPLICATE_YAML_DICT_KEY , I'd like to assert this with a task.
Imagine this dict, which contains duplicate keys:
my_dict:
one:
one:
I've tried:
- assert:
that:
- my_dict | unique == my_dict
But this fails even when there are no duplicate keys.
This is not possible. Duplicate keys are discarded during parsing, so there is no way to detect that they were present once parsing is finished. Setting DUPLICATE_YAML_DICT_KEY to error is the only way to turn this into a failure.
I'm trying to define an OpenAPI interface in a .yaml file.
The interface specification says that one of the parameters will be a data structure foo containing, among other things, one or more members of a second data structure bar. In C++ I would define a structure containing a vector of bar but I cannot figure out how to encode this in YAML. I tried making a list with just one item, and Swagger Editor complained:
# ... details omitted
- in: header
name: foo
required: true
schema:
type: object
properties:
identity:
type: integer
version:
type: integer
- bar: # <<<<< bad indentation of a mapping entry
type: object
properties:
key:
type: string
value:
type: string
This doesn't look like it's really an indentation problem. I say this because the only level of indentation that removes the error is back out level with the in. Note that, if I remove the dash and following space, the editor shows no errors, so I'm pretty sure there are no other issues here.
main question How can I define this parameter as something containing one or more structures of type bar?
bonus question Is there a way to name the structures I'm defining, e.g. KeyValuePair for bar?
This doesn't look like it's really an indentation problem.
No, but it is a syntactic error that has nothing to do with OpenAPI (or JSON Schema, which is what is most relevant for your problem). YAML defines the structure of your document, and you have made a structural error.
This snippet is enough to see the error:
version:
type: integer
- bar:
The key version: shows that we are inside a mapping here, i.e. a list of key-value pairs. What follows is a nested mapping with the single entry type: integer. Then however, a - follows on the indentation level of version:. In YAML, this is a sequence indicator. But there is no sequence at the indentation level of version:; instead, this indentation level holds a mapping. You cannot have a sequence item as part of a mapping, since a mapping contains key-value pairs. The error message could of course be a better one.
Now what you actually want to do is to define an array of bar (JSON term for a vector of bar):
version:
type: integer
bar:
type: array
items:
type: object
properties:
key:
type: string
value:
type: string
Be aware of the important difference between the syntactic YAML structure of your input, and the desired semantic of your input. You want to describe an array in your JSON Schema, but that does not mean that you have to use a YAML sequence for that. You also don't write 42 if you want to define an integer-type field.
Also be aware that bar is the name of the field, not the name of the type. You can reference previously defined types with $ref.
I have a YAML file which has several different keys that I want to provide the same value for. Additionally, I want this value to be easily configurable.
See YAML for my specific use-case below:
---
# Options for attribute_value
att_value_1: &att_value_1
att_value_2: &att_value_2
# There could be more options...
# This is where the constant is being set.
attribute_value: &attribute_value *att_value1
items:
- name: item1
attributes:
attribute_for_item1: *attribute_value
- name: item2
attributes:
attribute_for_item2: *attribute_value
Here is a simplified YAML which demonstrates the problem:
---
foo: &foo "Hello World!"
bar: &bar *foo
Error (it's complaining about the first line that has "Hello World!" on it):
(<unknown>): did not find expected key while parsing a block mapping at line 2 column 1
I expect the value to propagate.
Error (it's complaining about the first line that has "Hello World!" on it):
You have to tell us which YAML implementation you're using. PyYAML and NimYAML both correctly report that the error is in the third line; the second line is okay.
I expect the value to propagate.
There is nothing in the specification that backs this expectation. The spec says:
Note that an alias node must not specify any properties or content, as these were already specified at the first occurrence of the node.
Properties are anchors and tags. You cannot put an anchor on an alias node since it already has an anchor.
In your example,
---
foo: &foo "Hello World!"
bar: &bar *foo
&bar is completely superfluous since you already can refer to the "Hello World" node via *foo and therefore, there is no point in introducing &bar.
To answer your question: You cannot reference a YAML anchor from another YAML anchor, because a YAML anchor is not a node. YAML represents a graph, i.e. nodes and directed edges. Anchors and aliases are used to reference the same node multiple times (not to copy values as you might think). This also means that everything within a YAML file is content. There is no such thing as variable declarations.
It seems you are using the wrong tool for your use-case. To configure YAML files with external values, you would typically use a templating engine (SaltStack and Ansible use Jinja for example) and generate your YAML file from a template. You could supply your options in another YAML file. This would clearly separate the configuration options from the content.
Say I have a YML file original.yml with an array of objects
array_in_yml:
- start: 1
- middle: 2
- end: 3
I included it in modified.yml
!include "original.yml"
array_in_yml: []
I am expecting this array to be empty when I load the modified.yml, but it seems to have 3 values as original.yml. How do I force/override the array to be empty?
The discussion about !include seems to lead a bit away from the actual question. Let's assume that in some unknown way, the !include line gets replaced with the content in original.yml. We would have:
array_in_yml:
- start: 1
- middle: 2
- end: 3
array_in_yml: []
This is not valid YAML, since every key in a dictionary must be unique, but you use the key array_in_yml twice. Your YAML processor might ignore this and simply assign the first value (which is a sequence of three items) to the key array_in_yml.
Now the important part: There is no way in YAML to modify previously given values. You cannot override a value given previously with a different one. What you want to do is outside the YAML spec and you would need some merging tool that does such replacements for you.
Is there a way to make the following YAML shorter so that the same resources aren't repeated?
---
classes:
- roles::storage::nfs
samba::config::includeconf:
- alpha
- beta
- charlie
- delta
- echo
- foxtrot
smb_shares:
alpha:
name: alpha
beta:
name: beta
charlie:
name: charlie
delta:
name: delta
echo:
name: echo
path: /path/to/file
foxtrot:
name: foxtrot
If there's a way to reduce any of the repetition, that would be great. Ideally, each resource name would only appear once.
Yes you can vastly reduce this with two optimisations, one of which nullifies most of the effect of the other. You will however have to change your program from reading in simple sequences and mappings to create smarter objects ( which I called ShareInclude, Shares and Share):
---
classes:
- roles::storage::nfs
samba::config::includeconf: !ShareInclude
- alpha
- beta
- charlie
- delta
- echo
- foxtrot
smb_shares: !Shares
echo: !Share
path: /path/to/file
When creating ShareInclude, you should create a Share for each sequence element with an initial name being the same as the scalar value and insert this in some global list.
The above takes care of most of the Share object, execept info. When
echo: !Share
path: /path/to/file
is processed a temporary anonymous Share should be created with path set as an attribute or other retrievable value (if the name would be different that could be stored as well). Then once Shares is created it will know the name of the share to look up (echo from the key of the mapping) and take one of two actions:
If the name can be looked up, update the Share object with the information from the anonymous Share
If the name cannot be found, promote the anonymous share by providing the key value as its name, and store it.
This way you have to specify echo twice, otherwise there is no way to associate the explicit path with the specific Share object created when processing ShareInclude. If that is still too much you can approach it from the other way and leave ShareInclude empty and implicitly make those entries when dealing with Shares:
---
classes:
- roles::storage::nfs
samba::config::includeconf: !ShareInclude
smb_shares: !Shares
alpha:
beta:
charlie:
delta:
echo:
path: /path/to/file
foxtrot:
Although this is somewhat shorter, depending on your YAML parser, you might no longer have a guaranteed ordering in the creation of the Share object. And if you have to make Shares into a sequence of key-value pairs the shortening advantage is gone.