I'm having some issues with passing a hash from hiera through to a resource creation.
vhosts:
project_1:
name: project_1
project_name: project_1
project_2:
name: project_2
project_name: project_2
$vhosts = hiera('vhosts', [])
create_resources(project_vhosts::vhosts, $vhosts)
Ignore the hidden project names :) but you get the gist. My resource looks like this:
define project_vhosts::vhosts(
$vhosts = []
){
notice($vhosts)
}
I get these errors after my puppet run
Error: Invalid parameter project_name on project_vhosts::Vhosts[project_1] on node *
Wrapped exception:
Invalid parameter project_name
Error: Invalid parameter project_name on project_vhosts::Vhosts[project_1] on *
I get that it wants me to implement the parameters directly into the class. However what I really want is the hash available as a whole to me in the resource. What am I doing wrong here?
First off, please don't use [] to denote an empty hash. It's not. [] is the empty array, and {} is the empty hash.
To do what you want, your data just need one more layer of hashing.
vhost_data:
vhosts:
project_1:
name: project_1
project_name: project_1
project_2:
name: project_2
project_name: project_2
Then
$data = hiera('vhost_data', {})
create_resources(project_vhosts::vhosts, $vhosts)
Of course, there is yet a simpler way to do all of that with your data.
project_vhosts::vhosts {
'meaningless-resource-title':
vhosts => hiera('vhosts', {})
}
Related
I have a strange problem that I can't seem to get my head around. I am trying to define some variables for use as part of the job that will deploy bicep files via Azure CLI and execute PowerShell tasks.
I get this validation error when I try and execute the pipeline: While parsing a block mapping, did not find expected key
The line that it refers to is: - name: managementResourceDNSPrivateResolverName
On the research that I have done on this problem, it sounds like an indentation problem but on the face of it, it seems to look fine.
jobs:
- job: 'Deploy_Management_Resources'
pool:
vmImage: ${{ parameters.buildAgent }}
variables:
- name: managementResourceDNSPrivateResolverName
value: 'acme-$[ lower(parameters['environmentCode']) ]-$[ lower(variables['resourceLocationShort']) ]-private-dns-resolver'
- name: managementResourceGroupManagement
value: 'acme-infrastructure-rg-management'
- name: managementResourceRouteTableName
value: 'acme-$[ lower(variables['subscriptionCode']) ]-$[ lower(variables['resourceLocationShort']) ]-route-table'
- name: managementResourceVirtualNetworkName
value: 'acme-$[ lower(variables['subscriptionCode']) ]-$[ lower(variables['resourceLocationShort']) ]-vnet-internal-mng'
Thanks!
The error message ...parsing a block mapping, did not find expected key is usually a side-effect of malformed yaml. You'll see if often with variables if you have mixed formats of arrays and property elements
variables: # an array of objects
# variable group reference object
- group: myvariablegroup
# variable template reference object
- template: my-variables.yml
# variable object
- name: myVariable
value: 'value1'
# variable shorthand syntax
myVariable: 'value1' # this fails because it's a property instead of an array element
While it doesn't appear that the sample you've provided is malformed, I am curious about the use of $[ ] which is a runtime expression. The expression $[ lower(parameters['environmentcode']) ] refers to parameters which is are only available at compile time.
Change:
$[ lower(parameters['environmentCode']) ] to ${{ lower(parameters.environmentCode) }}
When I run puppet agent --test I have no errors output but the user did not create.
My puppet hira.yaml configuration is:
---
version: 5
datadir: "/etc/puppetlabs/code/environments"
data_hash: yaml_data
hierarchy:
- name: "Per-node data (yaml version)"
path: "%{::environment}/nodes/%{::trusted.certname}.yaml"
- name: "Common YAML hierarchy levels"
paths:
- "defaults/common.yaml"
- "defaults/users.yaml"
users.yaml is:
accounts::user:
joed:
locked: false
comment: System Operator
uid: '1700'
gid: '1700'
groups:
- admin
- sudonopw
sshkeys:
- ssh-rsa ...Hw== sysop+moduledevkey#puppetlabs.com
I use this module
Nothing in Hiera data itself causes anything to be applied to target nodes. Some kind of declaration is required in a manifest somewhere or in the output of an external node classifier script. Moreover, the puppetlabs/accounts module provides only defined types, not classes. You can store defined-type data in Hiera and read it back, but automated parameter binding via Hiera applies only to classes, not defined types.
In short, then, no user is created (and no error is reported) because no relevant resources are declared into the target node's catalog. You haven't given Puppet anything to do.
If you want to apply the stored user data presented to your nodes, you would want something along these lines:
$user_data = lookup('accounts::user', Hash[String,Hash], 'hash', {})
$user_data.each |$user,$props| {
accounts::user { $user: * => $props }
}
That would go into the node block matched to your target node, or, better, into a class that is declared by that node block or an equivalent. It's fairly complicated for so few lines, but in brief:
the lookup function looks up key 'accounts::user' in your Hiera data
performing a hash merge of results appearing at different levels of the hierarchy
expecting the result to be a hash with string keys and hash values
and defaulting to an empty hash if no results are found;
the mappings in the result hash are iterated, and for each one, an instance of the accounts::user defined type is declared
using the (outer) hash key as the user name,
and the value associated with that key as a mapping from parameter names to parameter values.
There are a few problems here.
You are missing a line in your hiera.yaml namely the defaults key. It should be:
---
version: 5
defaults: ## add this line
datadir: "/etc/puppetlabs/code/environments"
data_hash: yaml_data
hierarchy:
- name: "Per-node data (yaml version)"
path: "%{::environment}/nodes/%{::trusted.certname}.yaml"
- name: "Common YAML hierarchy levels"
paths:
- "defaults/common.yaml"
- "defaults/users.yaml"
I detected that using the puppet-syntax gem (included if you use PDK, which is recommended):
▶ bundle exec rake validate
Syntax OK
---> syntax:manifests
---> syntax:templates
---> syntax:hiera:yaml
ERROR: Failed to parse hiera.yaml: (hiera.yaml): mapping values are not allowed in this context at line 3 column 10
Also, in addition to what John mentioned, the simplest class to read in your data would be this:
class test (Hash[String,Hash] $users) {
create_resources(accounts::user, $users)
}
Or if you want to avoid using create_resources*:
class test (Hash[String,Hash] $users) {
$users.each |$user,$props| {
accounts::user { $user: * => $props }
}
}
Note that I have relied on the Automatic Parameter Lookup feature for that. See the link below.
Then, in your Hiera data, you would have a key named test::users to correspond (class name "test", key name "users"):
---
test::users: ## Note that this line changed.
joed:
locked: false
comment: System Operator
uid: '1700'
gid: '1700'
groups:
- admin
- sudonopw
sshkeys:
- ssh-rsa ...Hw== sysop+moduledevkey#puppetlabs.com
Use of automatic parameter lookup is generally the more idiomatic way of writing Puppet code compared to calling the lookup function explicitly.
For more info:
PDK
Automatic Parameter Lookup
create_resources
(*Note that create_resources is "controversial". Many in the Puppet community prefer not to use it.)
I have a playbook creating EC2 by using a dictionary declared in vars: then registering the IPs into a group to be used later on.
The dict looks like this:
servers:
serv1:
name: tag1
type: t2.small
region: us-west-1
image: ami-****
serv2:
name: tag2
type: t2.medium
region: us-east-1
image: ami-****
serv3:
[...]
I would like to apply tags to this playbook in the simplest way so I can create just some of them using tags. For example, running the playbook with --tags tag1,tag3 would only start EC2 matching serv1 and serv3.
Applying tags on the dictionary doesn't seem possible and I would like to avoid doing multiplying tasks like:
Creatinge EC2
Register infos
Getting private IP from previously registered infos
adding host to group
While I already have a working loop for the case I want to create all EC2 at once, is there any way to achieve that (without relying on --extra-vars, which would need key=value) ? For example, filtering out the dictionary by keeping only what is tagged before running the EC2 loop ?
I doubt you can do this out of the box. And not sure this is good idea at all.
Because tags are used to filter tasks in Ansible, so you will have to mark all tasks with tags: always.
You can accomplish this with custom filter plugin, for example (./filter_plugins/apply_tags.py):
try:
from __main__ import cli
except ImportError:
cli = False
def apply_tags(src):
if cli:
tags = cli.options.tags.split(',')
res = {}
for k,v in src.iteritems():
keep = True
if 'name' in v:
if v['name'] not in tags:
keep = False
if keep:
res[k] = v
return res
else:
return src
class FilterModule(object):
def filters(self):
return {
'apply_tags': apply_tags
}
And in your playbook:
- debug: msg="{{ servers | apply_tags }}"
tags: always
I found a way to match my needs without touching to the rest so I'm sharing it in case other might have a similar need.
I needed to combine dictionaries depending on tags, so my "main" dictionary wouldn't be static.
Variables became :
- serv1:
- name: tag1
type: t2.small
region: us-west-1
image: ami-****
- serv2:
- name: tag2
type: t2.medium
region: us-east-1
image: ami-****
- serv3:
[...]
So instead of duplicating my tasks, I used set_fact with tags like this:
- name: Combined dict
# Declaring empty dict
set_fact:
servers: []
tags: ['always']
- name: Add Server 1
set_fact:
servers: "{{ servers + serv1 }}"
tags: ['tag1']
- name: Add Server 2
set_fact:
servers: "{{ servers + serv2 }}"
tags: ['tag2']
[..]
20 lines instead of multiply tasks for each server, change vars from dictionary to lists, a few tags and all good :) Now if I add a new server it will only take a few lines.
I have defined a mapping in yaml that looks like:
default: &DEFAULT
bucket: &bucket default_path
# Make sure that the second parameter of join doesn't start with a /
# otherwise it is interpreted as an absolute path and join won't work
path1: !!python/object/apply:os.path.join [*bucket, work_area/test1]
path2: !!python/object/apply:os.path.join [*bucket, work_area/test2]
I need to define more keys where the only value to be overwritten is bucket, sth like:
production:
<<: *DEFAULT
bucket: "s3://production-bucket"
but I still get
conf['production']['path1'] => 'default_path/work_area/test1'
instead of
conf['production']['path1'] => 's3://production-bucket/work_area/test1'.
Is there any way to do this in yaml?
As obvious from the syntax, I use pyyaml to parse the file.
YAML interpreters should take the most recent definition of an anchor:
An alias node is denoted by the “*” indicator. The alias refers to the most recent preceding node having the same anchor. It is an error for an alias node to use an anchor that does not previously occur in the document. It is not an error to specify an anchor that is not used by any alias node.
So even if PyYAML (3.10/3.11) would not throw a ComposerError if you try to parse:
default: &DEFAULT
bucket: &bucket default_path
# Make sure that the second parameter of join doesn't start with a /
# otherwise it is interpreted as an absolute path and join won't work
path1: !!python/object/apply:os.path.join [*bucket, work_area/test1]
path2: !!python/object/apply:os.path.join [*bucket, work_area/test2]
production:
<<: *DEFAULT
bucket: &bucket "s3://production-bucket"
inserting the path1 and path2 keys with <<: *DEFAULT* would give you their expanded versions with default_path as that is the definition available to the parser when reading [*bucket, work_area/test1]
The "expansion" of the alias is done as soon as the alias is read in from the YAML source, not at some point at the end of the file, when all anchored data has been read in.
In you updated example, there is no other anchor bucket defined than the one for the scalar "default_path". You are confusing yourself by using the same name for the anchor and the keys (bucket), but the key names are completely irrelevant for resolving the alias *bucket.
If you can rearrange your YAML you might get something acceptable to your use case by doing ¹:
import ruamel.yaml
yaml_str = """\
default: &DEFAULT
bucket: &klm default_path
production:
&klm "s3://production-bucket"
result:
<<: *DEFAULT
# Make sure that the second parameter of join doesn't start with a /
# otherwise it is interpreted as an absolute path and join won't work
path1: !!python/object/apply:os.path.join [*klm, work_area/test1]
path2: !!python/object/apply:os.path.join [*klm, work_area/test2]
"""
conf = ruamel.yaml.load(yaml_str)
print(conf['result']['path1'])
which will give you:
s3://production-bucket/work_area/test1
¹ This was done using ruamel.yaml of which I am the author.
Suppose I have:
base_array:
-1
-2
how could I do something like:
my_array: << base_array
-3
so that my_array was [1,2,3]
Update: I should specify that I want the extending to occur inside the YAML itself.
Since the already commented issue#35 exists, merge-keys << doesn't help you. It only merges/inserts referenced keys into a map (see YAML docs merge). Instead you should work with sequences and use anchor & and alias *.
So your example should look like this:
base_list: &base
- 1
- 2
extended: &ext
- 3
extended_list:
[*base, *ext]
Will give result in output like this (JSON):
{
"base_list": [
1,
2
],
"extended": [
3
],
"extended_list": [
[
1,
2
],
[
3
]
]
}
Although not exactly what you expected, but maybe your parsing/loading environment can flatten the nested array/list to a simple array/list.
You can always test YAML online, for example use:
http://ben-kiki.org/ypaste
Online YAML Parser
I needed to do the same thing but to run on Azure DevOps Pipeline. In particular, I had to update the stage dependency dynamically. How I did it:
dependents: [Stage_A, Stage_B]
otherDependents: [Stage_C] # This needed to be added by policy to the pipeline's execution
dependsOn:
- ${{ each dependent in dependents }}:
- ${{ dependent }}
- ${{ each dependent in otherDependents }}:
- ${{ dependent }}
Doing so resulted in the required setup:
dependents: [Stage_A, Stage_B]
otherDependents: [Stage_C] # This needed to be added by policy to the pipeline's execution
dependsOn:
- Stage_A
- Stage_B
- Stage_C
I say dynamically because the variable dependents came from a template to which I had to append Stage_C.