Jekyll Include one YAML file in another YAML file - yaml

I have a Jekyll project where two separate pages (A.html and B.html) are displaying different content based on data in YAML files A.yml and B.yml respectively. Each yml file has a bunch of variables that are defined identically. I'd prefer to keep this common list of variables in a third file C.yml and include it into both A.yml and B.yml. How can I do this?
Things I've tried:
Using * to reference global data like *site.data.C.vars - this didn't parse.
Put C.yml in _includes directory and use front matter treat the page like a liquid template and call {% include C.yml %} - this compiled, but the emitted html page had no content whatsoever.
Edit - example usage
Using a common data file in multiple views isn't quite sufficient for me because there would also need to be name-resolution logic in liquid to accompany it. Here's an example of what my data might look like:
A.yml
ingredients:
- avacado: &avacado
name: Avacado
color: Green
foods:
- *octopus_ceviche
B.yml
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
C.yml
foods:
- octopus_ceviche: &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
If there's no way to include C.yml in A and B, then all the foods need to be shared in both places. If food is used in the md/html page entries need to be accessed by direct hash access (e.g. {{ site.data.foods[octopus_ceviche] }}), which a) doesn't seem to work and b) feels like too much logic for a view.

To have a common list of key-value variables define a third data file _data/common.yml.
Then in A.html and B.html you can access all the common.yml variables with:
{{ site.data.common.myvar }}

New answer based on the edited question:
*octupus_ceviche is a YAML alias and has nothing to do with Liquid. As I said, YAML files are not processed with Liquid. YAML, however, defines that aliases must point to achors in the same document. One YAML document must reside in one stream which for most YAML processors means that it cannot be split into multiple files.
That being said, a valid option would be to place all data into a single YAML file:
C:
foods:
- octopus_ceviche: &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
A:
ingredients:
- avacado: &avacado
name: Avacado
color: Green
foods:
- *octopus_ceviche
B:
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
You may leave out A, B and C if their child keys are disjoint as they are in this example. Note however that the anchor must always be located in front of the alias (textually), even though YAML defines that mapping keys have no order. That's why I moved C in front.
Nota Bene: Anchors and aliases in YAML have been designed to serialize cyclic structures. Using them as named, reusable values is generally fine. But actually, you do not need a list with all defined “variables”, you can also just define them on first occurrence. Example:
A:
ingredients:
- avocado: &avocado
name: Avocado
color: Green
foods:
- &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
B:
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
But this can be less readable of course. ymmv.

Since Jekyll does not process data files with Liquid when it loads them, it is not possible to include one YAML file in another with {% include %}. YAML itself does not have the ability to include other files (because it is stream-based, not file-based).
However, it should not be necessary. If you move all common variables to C.yml, you can just access them via {{ site.data.C.myvar }} in both your HTML files and do not need to include anything in A.yml or B.yml.

As the question above is worded, #flyx’s is the most appropriate answer, however given external constraints (see my other question) I ended writing my own plugin to let data files textually include one another through liquid.
The goals of this plugin are to let the data be:
DRY - (don’t repeat yourself) each model should be defined only once.
Grouped - all similar data should be defined in the same format next to each other.
Separated - different data should be defined in different places.
#flyx’s solutions here fail goals #2 and #3, requiring all different types of data to be defined in the same place, and in the case of the second suggestion intermixing the definitions of foods and ingredients.
My proposed solution allows textual inclusion of one data file into another. This allows different models to be defined in different files, yet referenced from other files as if they were defined in the same place, in an arbitrary order. Applied to this problem, my solution would like this:
A.yml
{% include_relative_once C.yml %}
ingredients:
- avacado: &avacado
name: Avacado
color: Green
foods:
- *octopus_ceviche
B.yml
{% include_relative_once C.yml %}
chefs:
- anthony_bourdain: &anthony_bourdain
name: Anthony Bourdain
hobby: Brazilian Jiu-Jitsu
foods:
- *octopus_ceviche
C.yml
foods:
- octopus_ceviche: &octopus_ceviche
name: Octopus Ceviche
taste: Delicious
For the plugin itself, see this gist

Related

Ansible - recursive argument_spec

I'm trying to create a role with argument validation via the meta/arguments_spec.yml file, and I'd like a user to be able to pass in a tree of arbitrary depth as one of the arguments. In other words, a valid family_tree argument might look like:
family_tree:
name: Sarah
children:
- name: John
- name: Jimmy
- name: Billy
But it may also look like:
family_tree:
name: Bob
children:
- name: Bob, Jr.
children:
- name: Bob, III
children:
- name: Bob, IV
children:
# and so on...
This would require a recursive reference of some kind, however, I can't seem to find anything that supports this in the native argument_spec options. I know how to create recursive references via JSON Schema, and I'm aware the ansible.utils.validate supports JSON Schema as well, but as far as I can tell, this would mean manually adding a task to the beginning of my role's tasks list, which defeats the purpose of a dedicated argument_spec file and makes things harder to reason about, so not my preferred option.

The right way to extract a list of attributes from a list of maps in Ansible

I'm looking for a non-ugly way to extract a list of particular values from a list of maps in Ansible. I can find some ways to do it, for example here: here, but everything I have seen is very ugly, and it feels like there should be a way where it is clearer what is being done to someone reading it in future. I could write a filter, but it feels like this should be unnecessary since this must come up relatively regularly.
I have a data structure like so in Ansible:
interfaces:
- name: eth0
subnet: 192.168.2
netmask: 255.255.255.0
static_dhcp_hosts:
- {name: "hosta", mac: "00:01:02:03:04:05", ip: "192.168.2.20"}
- name: eth1
subnet: 192.168.5
netmask: 255.255.255.0
static_dhcp_hosts:
- {name: "hostb", mac: "00:02:03:04:05:06", ip: "192.168.5.20"}
- {name: "hostc", mac: "00:03:04:05:06:07", ip: "192.168.5.21"}
I'd like to get a space separated list of the interface names, so
eth0 eth1
Obviously this is just example data, the actual top level list has 10 elements for one host. I know that I can use the join filter to get from a list of interfaces to the string I want and how to do that.
Can anyone suggest a nice way to make the list, that is readable to future maintainers (code/configuration should be self-documenting as far as possible (and no further))?
I am looking to do something like
{% for interface in interfaces %}{{ interface.name }} {% endfor %}
or
" ".join([ interface['name'] for interface in interfaces ])
in Python.
but, AFAIK, you can't, or it is considered bad practice to, use jinja2 loops like this in a role's task/main.yml, and, as I said, it feels like it shouldn't be necessary to use a custom filter for this.
(This role isn't just configuring a DHCP server, so please don't just suggest a pre-existing role that does that, that wouldn't solve my issue).
Any non-ugly way to do this would be much appreciated, as would confirmation from people that there is no non-ugly way.
I am using Ansible 2.3, but I'm interested in answers still even if they only work in later versions.
Edit:
The following:
"{{ internal_interfaces | items2dict(key_name='name',value_name='name') | list | join(\" \") }}"
works, and is the least ugly I can think of. It makes a dictionary from the list, with both the key and values being from the name attribute of the dictionaries in the list, and then casts this dictionary to a list, which just gives a list of keys. I'd still like something less obtuse and ugly if anyione can think of anything, or for any Ansible gurus to reply if they think there is nothing nicer.
Map and join is what you require:
- set_fact:
interface_names: "{{ interfaces | map(attribute='name') | join(' ') }}"
OK. I am being stupid. There is a nice way to do this:
"{{ interfaces | map(attribute='name') | join(\" \") }}"
The output from map is a generator which generates a list of the interface names, and join turns this into the string I want. Perfect.

Changing list value in Ansible

I have inventory with a very complicated structure. For my specific installation I want to override only some values. For example, I have structure:
---
System:
atr1: 47
config:
- nodes:
- logger:
id: 'all'
svr: 'IEW'
- Database:
constr: 'login/pass#db'
atr2: 'some value'
I want to override severity of the logger, i.e. add statistic information
svr: 'IEWS'. I want to provide an override within --extra-vars parameter.
In ansible.cfg -> hash_behaviour = merge
I don't want to use something like - svr: "{{ svr_custom | default('IEW') }}", because there are too many parameters, and thus it will be difficult to write the entire inventory in such way.
I read about combine filter, but I can't use it, when I had to override only one item in hash.
How can I achieve my goal?
The way you found is the simplest one. It's verbose to write but very easy to debug and to fix.
If you REALLY want to shrink this job, you can write your own lookup plugin. (https://docs.ansible.com/ansible/2.5/plugins/lookup.html).
From my experience, I really want to say that direct and dumb approach (write verbose) is much better for overall maintainability. A next person will see a dumb dump (pun intended) which is easy to fix, not a some obscure python snippet.
To make life easier you may want to store this configuration as a separate file (with all jinja pieces) and use lookup (st right from docs):
# Since 2.4, you can pass in variables during evaluation
- debug: msg="{{ lookup('template', './some_template.j2', template_vars=dict(x=42)) }} is evaluated with x=42"
Moreover, you can use Jinja's |from_yaml (or from_json) to convert loaded and processed template into data structure.
I read about combine filter, but I can't use it, when I had to override only one item in hash.
Why is that? Wouldn't new_svr defined in --extra-vars achieve what you want?
- set_fact:
System: "{{ System | combine({'config':[{'nodes':[{'logger':{'svr':new_svr }}]}]}, recursive=True) }}"

How do I concatenate all items from listB onto all items from listA?

I'm trying to figure out how to create an Ansible list that is the result of appending every string from listB onto every string in listA, effectively multiplying two lists of strings together.
In python I'd do this:
["-".join((x, y)) for x in listA for y in listB]
In other languages I'd nest one for loop inside another.
I can't figure out an analogue to this in Ansible.
My reason for doing this is to allow a role to automatically determine the site a host is in. All of my hosts are in at least one group named for the physical location and the type of site (development, staging, production). So for example, the New York production site's group would be "nyc-prod". I need my playbook to be able to pick out the site name from the complete list of groups the host is in.
Given a list of all possible sites, I can intersect that list with the list of groups a host is in, and the resulting single-element list would contain the current hosts's site.
I could brute force this by hand-writing a list of all possible site-type combinations into group_vars/all or vars/main.yml in a role, but with a large number of sites this multiplies out to a massive list that would have to be maintained. So my approach has been to try to programmatically construct the list from the much shorter list of physical sites and the list of types.
If I could embed arbitrary python in a jinja template I could do something like this in tasks/main.yml:
# sites contains a list of all physical locations ['nyc', 'sfo', ...]
- name: get groups
debug:
var: group_names
register: groups
- name: find my site group
set_fact:
my_site: "{% site for site in {{groups}} if site in ['-'.join((x, y)) for x in {{sites}} for y in ['dev', 'stage', 'prod']] }%"
- name: print the group
debug:
msg: "My site is {{ my_site }}"
That obviously doesn't work, however.
Has anyone solved this problem before?
Note: I'm using Ansible 2.2.1.0
Could be achieved in several ways probably.
Here's a fix for the Jinja2 template (you should use Jinja2 syntax not Python inside):
vars:
list1:
- list1element1
- list1element2
list2:
- list2element1
- list2element2
tasks:
- set_fact:
list3: "{% for prefix in list1 %}{% for postfix in list2 %}{{ prefix }}-{{ postfix }} {% endfor %}{% endfor %}"
- debug:
var: list3.split(' ')

Is there a way to shorten this YAML?

Is there a way to make the following YAML shorter so that the same resources aren't repeated?
---
classes:
- roles::storage::nfs
samba::config::includeconf:
- alpha
- beta
- charlie
- delta
- echo
- foxtrot
smb_shares:
alpha:
name: alpha
beta:
name: beta
charlie:
name: charlie
delta:
name: delta
echo:
name: echo
path: /path/to/file
foxtrot:
name: foxtrot
If there's a way to reduce any of the repetition, that would be great. Ideally, each resource name would only appear once.
Yes you can vastly reduce this with two optimisations, one of which nullifies most of the effect of the other. You will however have to change your program from reading in simple sequences and mappings to create smarter objects ( which I called ShareInclude, Shares and Share):
---
classes:
- roles::storage::nfs
samba::config::includeconf: !ShareInclude
- alpha
- beta
- charlie
- delta
- echo
- foxtrot
smb_shares: !Shares
echo: !Share
path: /path/to/file
When creating ShareInclude, you should create a Share for each sequence element with an initial name being the same as the scalar value and insert this in some global list.
The above takes care of most of the Share object, execept info. When
echo: !Share
path: /path/to/file
is processed a temporary anonymous Share should be created with path set as an attribute or other retrievable value (if the name would be different that could be stored as well). Then once Shares is created it will know the name of the share to look up (echo from the key of the mapping) and take one of two actions:
If the name can be looked up, update the Share object with the information from the anonymous Share
If the name cannot be found, promote the anonymous share by providing the key value as its name, and store it.
This way you have to specify echo twice, otherwise there is no way to associate the explicit path with the specific Share object created when processing ShareInclude. If that is still too much you can approach it from the other way and leave ShareInclude empty and implicitly make those entries when dealing with Shares:
---
classes:
- roles::storage::nfs
samba::config::includeconf: !ShareInclude
smb_shares: !Shares
alpha:
beta:
charlie:
delta:
echo:
path: /path/to/file
foxtrot:
Although this is somewhat shorter, depending on your YAML parser, you might no longer have a guaranteed ordering in the creation of the Share object. And if you have to make Shares into a sequence of key-value pairs the shortening advantage is gone.

Resources