Is there a way to make the following YAML shorter so that the same resources aren't repeated?
---
classes:
- roles::storage::nfs
samba::config::includeconf:
- alpha
- beta
- charlie
- delta
- echo
- foxtrot
smb_shares:
alpha:
name: alpha
beta:
name: beta
charlie:
name: charlie
delta:
name: delta
echo:
name: echo
path: /path/to/file
foxtrot:
name: foxtrot
If there's a way to reduce any of the repetition, that would be great. Ideally, each resource name would only appear once.
Yes you can vastly reduce this with two optimisations, one of which nullifies most of the effect of the other. You will however have to change your program from reading in simple sequences and mappings to create smarter objects ( which I called ShareInclude, Shares and Share):
---
classes:
- roles::storage::nfs
samba::config::includeconf: !ShareInclude
- alpha
- beta
- charlie
- delta
- echo
- foxtrot
smb_shares: !Shares
echo: !Share
path: /path/to/file
When creating ShareInclude, you should create a Share for each sequence element with an initial name being the same as the scalar value and insert this in some global list.
The above takes care of most of the Share object, execept info. When
echo: !Share
path: /path/to/file
is processed a temporary anonymous Share should be created with path set as an attribute or other retrievable value (if the name would be different that could be stored as well). Then once Shares is created it will know the name of the share to look up (echo from the key of the mapping) and take one of two actions:
If the name can be looked up, update the Share object with the information from the anonymous Share
If the name cannot be found, promote the anonymous share by providing the key value as its name, and store it.
This way you have to specify echo twice, otherwise there is no way to associate the explicit path with the specific Share object created when processing ShareInclude. If that is still too much you can approach it from the other way and leave ShareInclude empty and implicitly make those entries when dealing with Shares:
---
classes:
- roles::storage::nfs
samba::config::includeconf: !ShareInclude
smb_shares: !Shares
alpha:
beta:
charlie:
delta:
echo:
path: /path/to/file
foxtrot:
Although this is somewhat shorter, depending on your YAML parser, you might no longer have a guaranteed ordering in the creation of the Share object. And if you have to make Shares into a sequence of key-value pairs the shortening advantage is gone.
Related
I recently started working with Packetbeat.
For my use-case, I only need some specific fields (to the point where if I could I would completely rewrite the mapping, but am leaving that as a last resort).
I tried removing some of the fields from the "dns.answers" array of objects, but what I did doesn't seem to have any effect:
- include_fields:
fields:
- dns.question.name
- dns.question.type
- dns.answers
- dns.answers_count
- dns.resolved_ip
- drop_fields:
fields:
- dns.answers.name
In addition, I also tried including only the fields I want but that didn't seem to work either, e.g:
- include_fields:
fields:
- dns.question.name
- dns.question.type
- dns.answers.data
- dns.answers_count
- dns.resolved_ip
Any ideas? If rewriting the template/mapping of the index is the best choice, or perhaps using the Ingest Node Pipelines is a better approach, I'd love to hear it.
Thanks
I need some help understanding why, when I create a YAML file for an Ansible playbook that exactly mirrors what is specified in the module documentation, lists and list_items aren't being parsed correctly when the YAML is read by Python.
My understanding is that when reading through documentation about Ansible modules on Github, items are displayed within columns to denote their relationship relative to items above-and-below them. Further, if an item is defined as "list / elements=dictionary" then that means the following:
Typing of the item must be finished by adding a colon-and-space (this
defines "dictionary" items)
The next/subsequent line must begin at
the same indentation level (or greater) and start with a
hyphen-and-space (this denotes a "list" item)
The order of items in a list must be followed exactly.
Subsequent list items must be at the same level of indentation as the first item in the list
If any given sub-item in the list is ITSELF another "list / elements=dictionary" item repeat
step-2 again.
(EXAMPLE: Taken from the cisco.ios.ios_ospfv2 module)
So if the main item of "processes" is a "list / elements=dictionary" composed of the following:
areas (also a "list / elements=dictionary")
network (also a "list /
elements=dictionary")
process_id (integer)
Then the correct YAML syntax for the above would be:
processes:
- areas:
- area_id: 0 ##dictionary_item
- authentication: ##dictionary_item
message_digest: true ##boolean
- network:
- address: '10.1.1.0' ##string
- area: '0' ##string
- wildcard_bits: 0.0.0.255. ##integer
- process_id: 1
However, even though I don't get any YAML errors when I start my Playbook, I do get a bunch of obscure Python tracebacks. But if I change my code to THIS (see below) it works!
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
network:
address: 0.0.0.0
area: 0
wildcard_bits: 255.255.255.255
My questions:
If the order of list_items is important, why did my file NOT work when I followed the order specified in the module documentation...but it DOES work when that order is ignored ("process_id" should NOT be first according to the docs for this module)?
I understand that not EVERY list_item needs to be preceded with a hyphen-space. But I DID think that the FIRST list_item required this. So why (in my first example) does my playbook fail when I denote the sub-items under "network" with hyphens...but when I remove ALL of the hyphens from this list, it DOES work? However it DOES require a hyphen for the first list_item under "areas"??
Python Errors
(sorry for the long post...I didn't know to condense it any further than this)
Here is for a general explanation of this YAML:
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
network:
address: 0.0.0.0
area: 0
wildcard_bits: 255.255.255.255
processes: you may have 1 to N process, so it is a list, and in this case, a list of dictionary.
You should expect something like:
processes:
- foo: bar
baz: qux
- foo: some
baz: thing
process_id: is an int, you should expect:
processes:
- foo: bar
baz: qux
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
process_id: 1
- foo: some
baz: thing
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
process_id: 2
So this is the exact same YAML
processes:
- process_id: 1
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
foo: bar
baz: qux
- process_id: 2
# since you are in a dictionary,
# process_id can be anywhere in the dictionary,
# as the first, last, or anywhere in between key
foo: some
baz: thing
areas: is a list of 1 to N area, under the key areas of a dictionary, nested in a list, of key processes
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
- area_id: '1'
authentication:
message_digest: true
- process_id: 2
areas:
- area_id: '0'
authentication:
message_digest: true
network: is a dictionary in a dictionary
processes:
- process_id: 1
areas:
- area_id: '0'
authentication:
message_digest: true
- area_id: '1'
authentication:
message_digest: true
- process_id: 2
areas:
- area_id: '0'
authentication:
message_digest: true
network:
address: 0.0.0.0
area: 0
wildcard_bits: 255.255.255.255
So in general, pay attention to how Ansible is also giving you hints here: areas is pluralised for a good reason: because it is a list of area. Same goes for processes. While network is a singular, so, as expected it will be a dictionary and not a list.
list / elements=dictionary
Means that processes must be a list where each element is a dictionary.
In YAML this is a dictionary:
foo:
some: bar
property: bar
of: bar
the: bar
dictionary: bar
of: bar
key: bar
foo: bar
And this is a list:
fruits:
- banana
- apple
- pear
- peach
Then a list of dictionary would be:
- the:
red: fox
jump: over
- the:
lazy: dog
Mind that, because this is a list, then the same key can be reproduced in the different elements of the list but cannot be seen twice in the same dictionary.
So to answer your second question, you are correct that in a dictionary the order of keys do not matter:
the:
red: fox
jump: over
Is indeed the exact same dictionary as
the:
jump: over
red: fox
But since here you are making a list of what should be a dictionary, that's where your issue reside.
A little note on the hyphen (-), too, because this seems to bug you:
If you prefer it, write your lists like this:
-
foo: bar
bar: foo
-
baz: qux
qux: baz
This is stricly equivalent to:
- foo: bar
bar: foo
- baz: qux
qux: baz
This would help you see that the hyphen just does delimit the fact that a new element of the list begins there, now you can "condense" it and have a key of your dictionary on the same line as the "new item of list" delimiter (-), but you are not forced to.
Maybe for an object oriented programming language parallel, if that could be of any help for you, you should think
a dictionary is like an object, it has multiple properties, but all the properties are actually defining the same object
a list is a collection of things, it can be a list of object, of integer, or even of object.
So when the documentation says list / elements=dictionary they actually want you do construct a list (or collection) where all the elements would be dictionaries (or objects).
Maybe the layout on the documentation page itself is easier to read than the table on GitHub.
But there, you can clearly see that the config dictionary holds a processes list that, itself contains a lot of different dictionaries, among which address_family, adjacency, ..., areas, network, ...
Thank you everyone for your insights. I've conferred with others and I think this is an issue of a documentation error for this module. The item of "network" is documented as being "list / elements=dictionary".
In reality the underlying Python code expects to see it as a dictionary (not a list). When added as a dictionary it works fine. Please see my link in which I provide a screenshot of the (incorrect) documentation as well as the Python error that results when "network" is configured as a "list / elements=dictionary".
enter image description here
Update: I was previously using the Ansible cisco.ios collection version 1.1.0 (which requires that the "network" parameter be entered as a dictionary in the python code for the module, "ios_ospfv2").
Upon upgrading to version 1.2.0 of the cisco.ios collection the "network" parameter no longer worked as a dictionary and needed to be changed back to a list (which matches the documentation for this module).
So apparently the version of the Ansible Collection one uses may change how parameters are defined within modules. Moral of the story? Ensure you're using the latest version of whatever Collection you need.
Determining Most Recent Collection Version
I have several YAML files representing the pieces of a whole. I want to merge them under a new field ("guests") that declares the whole.
file1.yml
name: johnny
age: 23
file2.yml
name: sally
age: 21
output.yml
guests:
- name: johnny
age: 23
- name: sally
age: 21
tools like yq make merging/overwriting easy, but I can't find any that helps me nest values under new fields.
The tool you are looking for comes with several different names and
are called programming languages or scripting languages. I recommend
you use Python with ruamel.yaml installed. (disclaimer: I am the author of
that package).
Once you have that you can do:
python -c "import sys, ruamel.yaml; yaml=ruamel.yaml.YAML(); yaml.indent(sequence=4, offset=2); yaml.dump(dict(guest=[yaml.load(open(f)) for f in sys.argv[1:]]), sys.stdout)" file*.yml > output.yml
To get the desired output.
A few notes:
YAML files should have the .yaml extension unless your filesystem doesn't support that.
By default sequence elements are indented two spaces and the dash has no offset within that (i.e. would align with the g of guests. Hence the yaml.indent() call.
Any comments on the key-value files of your input file would be preserved but not automatically pushed out to the right from their original starting column unless necessary because of a mapping value getting in the way. Adjusting that is possible, but I would not recommend trying that in a one-liner.
If you need to preserve quotes add yaml.preserve_quotes = True; in the one-liner
If any of your YAML files contain multiple YAML documents, the above will fail. You would need to think about how to combine the documents, and use a try except clause to fall back to yaml.load_all() for documents that do (it would be a good idea to abandon the one-liner in favour of a multiline Python program at that point).
You can also do the above using the yaml commandline utility (installable with pip install ruamel.yaml.cmd>=0.5.0):
yaml from-dirs --sequence ./*.yml | yaml map --indent 2,4,2 guest - > output.yml
but this is a two step process (first combining multiple yaml files as root level sequence, then pushing that sequence to be a value for mapping), and thus twice as slow as the one-liner.
I have inventory with a very complicated structure. For my specific installation I want to override only some values. For example, I have structure:
---
System:
atr1: 47
config:
- nodes:
- logger:
id: 'all'
svr: 'IEW'
- Database:
constr: 'login/pass#db'
atr2: 'some value'
I want to override severity of the logger, i.e. add statistic information
svr: 'IEWS'. I want to provide an override within --extra-vars parameter.
In ansible.cfg -> hash_behaviour = merge
I don't want to use something like - svr: "{{ svr_custom | default('IEW') }}", because there are too many parameters, and thus it will be difficult to write the entire inventory in such way.
I read about combine filter, but I can't use it, when I had to override only one item in hash.
How can I achieve my goal?
The way you found is the simplest one. It's verbose to write but very easy to debug and to fix.
If you REALLY want to shrink this job, you can write your own lookup plugin. (https://docs.ansible.com/ansible/2.5/plugins/lookup.html).
From my experience, I really want to say that direct and dumb approach (write verbose) is much better for overall maintainability. A next person will see a dumb dump (pun intended) which is easy to fix, not a some obscure python snippet.
To make life easier you may want to store this configuration as a separate file (with all jinja pieces) and use lookup (st right from docs):
# Since 2.4, you can pass in variables during evaluation
- debug: msg="{{ lookup('template', './some_template.j2', template_vars=dict(x=42)) }} is evaluated with x=42"
Moreover, you can use Jinja's |from_yaml (or from_json) to convert loaded and processed template into data structure.
I read about combine filter, but I can't use it, when I had to override only one item in hash.
Why is that? Wouldn't new_svr defined in --extra-vars achieve what you want?
- set_fact:
System: "{{ System | combine({'config':[{'nodes':[{'logger':{'svr':new_svr }}]}]}, recursive=True) }}"
I have a Jekyll project where two separate pages (A.html and B.html) are displaying different content based on data in YAML files A.yml and B.yml respectively. Each yml file has a bunch of variables that are defined identically. I'd prefer to keep this common list of variables in a third file C.yml and include it into both A.yml and B.yml. How can I do this?
Things I've tried:
Using * to reference global data like *site.data.C.vars - this didn't parse.
Put C.yml in _includes directory and use front matter treat the page like a liquid template and call {% include C.yml %} - this compiled, but the emitted html page had no content whatsoever.
Edit - example usage
Using a common data file in multiple views isn't quite sufficient for me because there would also need to be name-resolution logic in liquid to accompany it. Here's an example of what my data might look like:
A.yml
ingredients:
- avacado: &avacado
name: Avacado
color: Green
foods:
- *octopus_ceviche
B.yml
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
C.yml
foods:
- octopus_ceviche: &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
If there's no way to include C.yml in A and B, then all the foods need to be shared in both places. If food is used in the md/html page entries need to be accessed by direct hash access (e.g. {{ site.data.foods[octopus_ceviche] }}), which a) doesn't seem to work and b) feels like too much logic for a view.
To have a common list of key-value variables define a third data file _data/common.yml.
Then in A.html and B.html you can access all the common.yml variables with:
{{ site.data.common.myvar }}
New answer based on the edited question:
*octupus_ceviche is a YAML alias and has nothing to do with Liquid. As I said, YAML files are not processed with Liquid. YAML, however, defines that aliases must point to achors in the same document. One YAML document must reside in one stream which for most YAML processors means that it cannot be split into multiple files.
That being said, a valid option would be to place all data into a single YAML file:
C:
foods:
- octopus_ceviche: &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
A:
ingredients:
- avacado: &avacado
name: Avacado
color: Green
foods:
- *octopus_ceviche
B:
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
You may leave out A, B and C if their child keys are disjoint as they are in this example. Note however that the anchor must always be located in front of the alias (textually), even though YAML defines that mapping keys have no order. That's why I moved C in front.
Nota Bene: Anchors and aliases in YAML have been designed to serialize cyclic structures. Using them as named, reusable values is generally fine. But actually, you do not need a list with all defined “variables”, you can also just define them on first occurrence. Example:
A:
ingredients:
- avocado: &avocado
name: Avocado
color: Green
foods:
- &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
B:
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
But this can be less readable of course. ymmv.
Since Jekyll does not process data files with Liquid when it loads them, it is not possible to include one YAML file in another with {% include %}. YAML itself does not have the ability to include other files (because it is stream-based, not file-based).
However, it should not be necessary. If you move all common variables to C.yml, you can just access them via {{ site.data.C.myvar }} in both your HTML files and do not need to include anything in A.yml or B.yml.
As the question above is worded, #flyx’s is the most appropriate answer, however given external constraints (see my other question) I ended writing my own plugin to let data files textually include one another through liquid.
The goals of this plugin are to let the data be:
DRY - (don’t repeat yourself) each model should be defined only once.
Grouped - all similar data should be defined in the same format next to each other.
Separated - different data should be defined in different places.
#flyx’s solutions here fail goals #2 and #3, requiring all different types of data to be defined in the same place, and in the case of the second suggestion intermixing the definitions of foods and ingredients.
My proposed solution allows textual inclusion of one data file into another. This allows different models to be defined in different files, yet referenced from other files as if they were defined in the same place, in an arbitrary order. Applied to this problem, my solution would like this:
A.yml
{% include_relative_once C.yml %}
ingredients:
- avacado: &avacado
name: Avacado
color: Green
foods:
- *octopus_ceviche
B.yml
{% include_relative_once C.yml %}
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
C.yml
foods:
- octopus_ceviche: &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
For the plugin itself, see this gist