Here's my setup
cookbooks /
supervisord /
definitions /
supervisord_group.erb
templates /
process_group.conf.erb
my_app /
recipes /
default.rb
In cookbooks/supervisord/definitions/supervisord_group.erb I have this:
define :supervisord_group, :programs => [], :enable => true do
include_recipe "supervisord::install"
if params[:enable]
template "#{node[:supervisord][:conf_dir]}/#{params[:name]}_group.conf" do
source "process_group.conf.erb"
variables({
:name => params[:name],
:programs => params[:programs].join(",")
})
owner node[:user]
group node[:group]
mode 0755
end
end
end
In cookbooks/my_app/recipes/default.rb I have this:
supervisord_group "myapps" do
programs ["test1", "test2"]
end
The problem is that when I run it I get an error that it's unable to find the template process_group.erb.conf. Here's the output:
Cookbook 'my_app' (0.0.0) does not contain a file at any of these locations:
templates/ubuntu-10.04/process_group.conf.erb
templates/ubuntu/process_group.conf.erb
templates/default/process_group.conf.erb
If my supervisord_group definition is the one referencing the template why is it expecting the my_app cookbook to have it? Any ideas?
So it looks like there is a not-so-clearly-documented property of the template provider that allows you to specify the cookbook that the template lives in. You have to add cookbook "supervisord" to the template setup for it to find it properly.
Thanks to this question
Related
I have a fairly basic puppet module for a webservice running tomcat. I want to setup logrotate on Tomcat's catalina.out file, and I want to start by writing a test that confirms logrotate is included in the module and setup with the correct settings.
Here's a stripped down version of my webservice.pp, for example:
class my_module::webservice (
...
){
include ::tomcat_server
...
logrotate::rule { 'tomcat':
path => '/var/log/tomcat/catalina.out',
rotate => 1,
rotate_every => 'day',
copytruncate => true,
missingok => true,
compress => true,
delaycompress => true,
}
}
and I have included the logrotate forge module in my .fixtures.yml like so:
fixtures:
forge_modules:
logrotate:
repo: 'puppet-logrotate'
ref: '3.2.1'
...
But I can only write a test that confirms that logrotate is included in the module like so:
require 'spec_helper'
describe 'my_module::webservice' do
on_supported_os.each do |os, os_facts|
context "on #{os}" do
let(:facts) { os_facts }
it { is_expected.to compile }
it { is_expected.to contain_class('logrotate') }
end
end
end
This doesn't work (if I remove the logrotate block from init.pp then the tests still pass):
it { is_expected.to contain_class('logrotate::conf') }
nor does asking for with:
it { is_expected.to contain_class('logrotate') \
.with('path' => '/var/log/tomcat/catalina.out',
'rotate' => 1,
'rotate_every' => 'day',
'copytruncate' => true,
'missingok' => true,
'compress' => true,
'delaycompress' => true,
)
}
and nor does a separate/nested describe block:
describe 'logrotate::rule' do
let(:title) { 'tomcat' }
let(:params) do
{
'path' => '/var/log/tomcat/catalina.out',
'rotate' => 1,
'rotate_every' => 'day',
'copytruncate' => true,
'missingok' => true,
'compress' => true,
'delaycompress' => true,
}
end
end
I can't find anything in the rspec docs that mention anything other than testing the class is defined. Is it even possible to do what I am trying to do?
Here is my directory layout:
puppet
`- modules
`- my_module
|- data
|- manifests
| |- init.pp
| `- webservice.pp
|- spec
| |- classes
| | `- webservice_spec.rb
| `- spec_helper.rb
|- .fixtures.yml
|- Gemfile
|- hiera.yaml
|- metadata.json
`- Rakefile
I have a fairly basic puppet module for a webservice running tomcat. I want to setup logrotate on Tomcat's catalina.out file, and I want to start by writing a test that confirms logrotate is included in the module and setup with the correct settings.
That sounds very reasonable. However, this ...
Here's a stripped down version of my init.pp, for example:
class my_module::webservice (
...
){
... is at best poor practice. If it exists at all then the init.pp manifest of module my_module should define only class my_module. A class named my_module::webservice should instead be defined in a manifest named webservice.pp in module my_module. The expectations for module layout are documented in the Puppet online documentation. Although you might be able to get away with certain discrepancies from those specifications, there is only downside to doing so.
At this point I observe that "inner class" is not idiomatic Puppet terminology, and it suggests a misunderstanding of what you're working with. Specifically, this ...
logrotate::rule { 'tomcat':
[...]
... does not declare a class at all, but rather declares a resource of type logrotate::rule, which is apparently a defined type provided by the puppet/logrotate module. In general, declaring a resource does not imply anything about classes from the module (if any) that provides the resource's type.
Furthermore, although it is entirely possible that declaring a logrotate::rule resource does cause class logrotate to be included in the catalog too, that would be an implementation detail of logrotate::rule, and as such, your spec tests should not be testing for it. Only if my_module::webservice is expected to itself declare class logrotate should its tests be checking for that.
You go on to say:
This doesn't work (if I remove the logrotate block from init.pp then
the tests still pass):
it { is_expected.to contain_class('logrotate::conf') }
You haven't presented enough code for us to determine why the tests pass when that is included in them, but something is very strange if ever that expectation is satisfied. logrotate::conf is also a defined (resource) type, not a class, so that expectation should never succeed. And following a theme I introduced above, if class my_module::webservice does not declare any logrotate::conf resource directly then its tests should not be checking for one.
nor does asking for with:
it { is_expected.to contain_class('logrotate') \
.with('path' => '/var/log/tomcat/catalina.out',
'rotate' => 1,
'rotate_every' => 'day',
'copytruncate' => true,
'missingok' => true,
'compress' => true,
'delaycompress' => true,
)
}
Of course that doesn't succeed. It expresses an expectation of a declaration of class logrotate, but what you've actually declared is a resource of type logrotate::rule. Even if logrotate::rule did declare logrotate, one would not expect it to pass on its own parameter list.
and nor does a separate/nested describe block:
describe 'logrotate::rule' do
[...]
Again, that's not surprising. Such a describe block tells RSpec that logrotate::rule is the class under test. Not only is it not the class under test (that is of course my_module::webservice), but, again, logrotate::rule is not a class at all. RSpec can certainly test defined types, too, but that's not what you're after here.
To test whether a resource is declared by the class under test, one uses a predicate of the form contain_type(title), where any namespace separators (::) in the type name are replaced by double underscores. For example:
it do
is_expected.to contain_logrotate__rule('tomcat')
end
It is permitted, but optional, to include one or more with clauses to specify expectations of the declared parameters of the designated resource. Following what you appear to have been trying to do, then, maybe this would more fully express what you're looking for:
require 'spec_helper'
describe 'my_module::webservice' do
on_supported_os.each do |os, os_facts|
context "on #{os}" do
let(:facts) { os_facts }
it do
is_expected.to compile
is_expected.to contain_logrotate__rule('tomcat')
.with(
path: '/var/log/tomcat/catalina.out',
rotate: 1,
rotate_every: 'day',
copytruncate: true,
missingok: true,
compress: true,
delaycompress: true
)
end
end
end
end
Do note, by the way, that when you want to test multiple predicates against the same example, it is substantially more efficient to group them together in the same it block, as demonstrated above, than to put each in its own it block. As in, you will probably notice the difference in test running time even from combining just two it blocks into one.
Additionally, my example above demonstrates coding style close to that required to avoid warnings from pdk validate, which brings us to an additional point: it is always useful to verify that pdk validate completes without errors or warnings before trying the unit tests. You will probably find that it is excessively picky about both Puppet and Ruby code style, but it will also pick up some issues that lead to mysterious test failures. Also, it runs much faster than the tests do, and it will pick up substantially all syntax errors in both Puppet and Ruby code. It's frustrating to have your tests take a long time to fail on account of a minor syntax error.
I need to make a drop down / Choice selection for one of my automated builds.
Here's the finished XML that I can observe when I configure the job on-the-fly from the front end (I need to translate this into the Chef recipe that I am creating):
<property>
<parameterDefinition>
<defaultParameterValue>
<name>DATA_BAG_NAME</name>
<value>X</value>
</defaultParameterValue>
<description>Select the data bag that contains the job above.</description>
<name>DATA_BAG_NAME</name>
<type>ChoiceParameterDefinition</type>
<choice>X</choice>
<choice>Y</choice>
<choice>Z</choice>
</parameterDefinition>
</property>
I have also attached a screenshot of what the manual configuration looks like.
Finally,
Here is the non working code I have written up for my Chef Recipe:
:build_params => [
{ 'name' => 'JENKINS_ID_TO_ADD', 'type' => 'String', 'default' => '',
'description' => ' Enter the JenkinsID you want to add, example: PRCalculator' },
{ 'name' => 'DATA_BAG_NAME', 'type' => 'Choice', 'choices' =>
'description' => ' Select the data bag that contains the job above.' }
],
Please help me determine what type of Ruby Syntax I need to make the drop down list actually contain values, instead of "'choices' => " which is what I have right now. Any permutation I try results in failure to configure, or success with 0 elements in the dropdown.
~EDIT: Adding in further code to help troubleshoot:
config_name = 'free-style'
job_name = "flag-chef-add-jenkins-id"
job_config = File.join(Chef::Config[:file_cache_path], "#{job_name}-config.xml")
template job_config do
source File.join('jenkins',"job-#{config_name}-config.xml.erb")
variables :job_name => job_name,
:max_builds => '15',
:build_params => [
{ 'name' => 'JENKINS_ID_TO_ADD', 'type' => 'String', 'default' => '',
'description' => ' Enter the JenkinsID you want to add, example: MyJobID' },
{ 'name' => 'DATA_BAG_NAME', 'type' => 'Choice', 'choice' => '1', 'choice' => '2',
'description' => ' Select the data bag that contains the job above.' }
],
:command => "
if [ \"$JENKINS_ID_TO_ADD\" != \"\" ]; then
# run gimmicky update that wont work!
echo \"Jenkins ID to add or update: \"
echo \"Running Jenkins_ID_To_Add job for the following ID: $JENKINS_ID_TO_ADD .'\n\" >> jenkinsIDTest.txt
echo recipe['jenkins::master'],recipe[\"flag_utils::data_bags::promo_lps::$JENKINS_ID_TO_ADD\"] >> jenkinsIDTest.txt
sudo -u root -i chef-client -o recipe['jenkins::master'],recipe[\"flag_utils::data_bags::promo_lps::$JENKINS_ID_TO_ADD\"] --force-formatter >> jenkinsIDTest.txt
fi
",
:email_release_subject => 'Flag Utils Chef add Jenkins ID run! $JENKINS_ID_TO_ADD',
:admin_emails => admin_emails,
:notification_emails => notification_emails
end
jenkins_job job_name do
config job_config
end
Using Chef you can generate the jobs's config.xml file using a chef template resource.
Better still I recommend using the Jenkins cookbook which has a jenkins_job resource that takes a template as a parameter.
Update - Cookbook example
I have created the following "demo" cookbook to illustrate how to install Jenkins and configure a parametrized job.
├── Berksfile
├── metadata.rb
├── recipes
│ └── default.rb
├── templates
│ └── default
│ └── choice-job.xml.erb
└── attributes
├── java.rb
└── jenkins.rb
recipes/default.rb
#
# Cookbook Name:: demo
# Recipe:: default
#
# Copyright (c) 2016 The Authors, All Rights Reserved.
#
include_recipe "apt"
include_recipe "java"
include_recipe "jenkins::master"
#
# Jenkins job
#
jobxml = File.join(Chef::Config[:file_cache_path], 'choice-job.xml')
template jobxml do
source "choice-job.xml.erb"
variables :choices => ["X", "Y", "Z"]
end
jenkins_job "Choice Demo" do
config jobxml
end
Note:
An xml file is generated by the template and passed to the Jenkins job resource.
Passing in 3 parameters that will appear as choices in the Jenkins UI
templates/default/choice-job.xml.erb
<?xml version='1.0' encoding='UTF-8'?>
<project>
<actions/>
<description>Demo job</description>
<keepDependencies>false</keepDependencies>
<properties>
<hudson.model.ParametersDefinitionProperty>
<parameterDefinitions>
<hudson.model.ChoiceParameterDefinition>
<name>DATA_BAG_NAME</name>
<description></description>
<choices class="java.util.Arrays$ArrayList">
<a class="string-array">
<% #choices.each do |choice| -%>
<string><%= choice %></string>
<% end -%>
</a>
</choices>
</hudson.model.ChoiceParameterDefinition>
</parameterDefinitions>
</hudson.model.ParametersDefinitionProperty>
</properties>
<scm class="hudson.scm.NullSCM"/>
<canRoam>true</canRoam>
<disabled>false</disabled>
<blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
<blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
<triggers/>
<concurrentBuild>false</concurrentBuild>
<builders>
<hudson.tasks.Shell>
<command>env</command>
</hudson.tasks.Shell>
</builders>
<publishers/>
<buildWrappers/>
</project>
Note the following fragment:
<% #choices.each do |choice| -%>
<string><%= choice %></string>
<% end -%>
The "choices" are passed into the template
metadata.rb
name 'demo'
maintainer 'The Authors'
maintainer_email 'you#example.com'
license 'all_rights'
description 'Installs/Configures demo'
long_description 'Installs/Configures demo'
version '0.1.0'
depends "apt"
depends "java"
depends "jenkins"
Berksfile
source 'https://supermarket.chef.io'
metadata
Note:
Dependencies pulled from supermarket by Berkshelf
attributes/java.rb
normal['java']['jdk_version'] = '7'
Note:
Jenkins needs at least Java7
attributes/jenkins.rb
normal['jenkins']['master']['install_method'] = "war"
normal['jenkins']['master']['version'] = "1.655"
normal['jenkins']['master']['checksum'] = "0cee889af697c115961ce50229cc5e39d1b798c0a0a689687b745c0a938c8547"
Note:
Specifying the version and checksum avoids the need to check for the latest version
xml = File.join(Chef::Config[:file_cache_path], 'rconfig.xml')
template xml do
source 'jconfig.xml.erb'
end
jenkins_job 'project' do
config xml
action :create
end
This is the code example. This might help. "jconfig.xml" is my config.xml file for jenkins job
How would you make an erb template that has human readable json?
The following code works, but it makes a flat json file
default.rb
default['foo']['bar'] = { :herp => 'true', :derp => 42 }
recipe.rb
template "foo.json" do
source 'foo.json.erb'
variables :settings => node['foo']['bar'].to_json
action :create
end
foo.json.erb
<%= #settings %>
Similar SO questions
Chef and ruby templates - how to loop though key value pairs?
How can I "pretty" format my JSON output in Ruby on Rails?
As pointed out by this SO Answer .erb templates are great for HTML, and XML, but is not good for json.
Luckily, CHEF uses its own json library which has support for this using .to_json_pretty
#coderanger in IRC, pointed out that you can use this library right inside the recipe. This article shows more extensively how to use chef helpers in recipes.
default.rb
# if ['foo']['bar'] is null, to_json_pretty() will error
default['foo']['bar'] = {}
recipe/foo.rb
pretty_settings = Chef::JSONCompat.to_json_pretty(node['foo']['bar'])
template "foo.json" do
source 'foo.json.erb'
variables :settings => pretty_settings
action :create
end
Or more concise as pointed out by YMMV
default.rb
# if ['foo']['bar'] is null, to_json_pretty() will error
default['foo']['bar'] = {}
recipe/foo.rb
template "foo.json" do
source 'foo.json.erb'
variables :settings => node['foo']['bar']
action :create
end
templates/foo.json.erb
<%= Chef::JSONCompat.to_json_pretty(#settings) %>
Something like this would also work:
file "/var/my-file.json" do
content Chef::JSONCompat.to_json_pretty(node['foo']['bar'].to_hash)
end
<%= Chef::JSONCompat.to_json_pretty(#settings) %> Works like Charm !!
I've got a Vagrant setup in which I'm trying to use Chef-solo to generate an conf file which loops though defined variables to pass to the application. Everything is working except the loop and I'm not familiar enough with Ruby/Chef to spot the error.
I'm going to lay out the whole chain of events in case there is something along the way that is the problem, but the first portions of this process seem to work fine.
A config file is written in yaml and includes env variable definitions to be passed:
...
variables:
- DEBUG: 2
...
The config file is read in by the Vagrantfile into a ruby hash and used to create the Chef json nodes:
...
settings = YAML::load(File.read("config.yaml"))
# Provision The Virtual Machine Using Chef
config.vm.provision "chef_solo" do |chef|
chef.json = {
"mysql" => {"server_root_password" => "secret"},
"postgresql" => {"password" => {"postgres" => "secret"}},
"nginx" => {"pid" => "/run/nginx.pid"},
"php-fpm" => {"pid" => "/run/php5-fpm.pid"},
"databases" => settings["databases"] || [],
"sites" => settings["sites"] || [],
"variables" => settings["variables"] || []
}
...
A bunch of chef cookbooks are run (apt, php, nginx, mysql etc) and finally my custom cookbook which is whats giving me grief. The portion of the cookbook responsible for creating a the conf file is shown here:
# Configure All Of The Server Environment Variables
template "#{node['php-fpm']['pool_conf_dir']}/vars.conf" do
source "vars.erb"
owner "root"
group "root"
mode 0644
variables(
:vars => node['variables']
)
notifies :restart, "service[php-fpm]"
end
And the vars.erb is just a one-liner
<%= #vars.each {|key, value| puts "env[" + key + " = " + value } %>
So, when I run all this chef spits out an error about not being able to convert a hash to a string.
can't convert Chef::Node::immutableMash into String
So for some reason this is coming across as an immutableMash and the value of key ends up being the hash [{"DEBUG"=>2}] and value ends up a nil object, but I'm not sure why or how to correct it.
The hash is ending up as the value of key in your example because the YAML file declares DEBUG: 2 as a list member of variables. This translates to variables being an array with a single hash member.
Try changing the template code to this:
<%= #vars[0].each {|key, value| puts "env[" + key + " = " + value } %>
Or try changing the YAML to this and not changing the template code:
variables:
DEBUG: 2
Either change will get your template loop iterating over the hash that you are expecting.
I have a Chef recipe for a multi-node web service, each node of which needs to get the hostname and IP of the other nodes, to put it into its own local configuration.
The code is shown below. The problem is that when the node.set[][] assignments are made in the ruby_block as shown, the values are empty when the template that relies upon them is created. If I want to create that template, I have to move all of the ruby_block code outside, and have it "loose" in the recipe. Which makes it harder to do unit-testing with Chefspec and the like.
Can any Chef guru set me straight? Is it just impossible to do node.set[] like this inside of a ruby_block? And if so, why doesn't it say so in the docs?
$cm = { :name => "web", :hostname => "" , :ip_addr => "" }
$ca = { :name => "data", :hostname => "" , :ip_addr => "" }
$cg = { :name => "gateway", :hostname => "" , :ip_addr => "" }
$component_list = [$cm, $ca, $cg]
ruby_block "get host addresses" do
block do
for cmpnt in $component_list
# do REST calls to external service to get cmpnt.hostname, ip_addr
# .......
node.set[cmpnt.name]['name'] = cmpnt.name
node.set[cmpnt.name]['host'] = cmpnt.hostname
node.set[cmpnt.name]['ip'] = cmpnt.ip_addr
end
end
end
template "/etc/app/configuration/config.xml" do
source "config.xml.erb"
variables( :dataHost => node['data']['host'],
:webHost => node['web']['host'],
:gatewayHost => node['gateway']['host'] )
action :create
end
I also added
subscribes :create, "ruby_block[get host addresses]", :immediately
to the template definition to ensure that the ruby_block ran before the template was created. This didn't make a difference.
I realize this is an old post, however for future reference, I just ran across this gist which gives a nice example of node variable assignments in the Compile vs. Converge phases. To adapt the gist to your example, you'll need to add code like the following to your ruby_block:
template_r = run_context.resource_collection.find(:template => "/etc/app/configuration/config.xml")
template_r.content node['data']['host']
template_r.content node['web']['host']
template_r.content node['gateway']['host']
For Chef 11, also see Lazy Attribute Evaluation.
The problem seems to be that attribute values inside your template resource definition get evaluated before actually invoking any resources.
I.e. the file is first executed as simple Ruby, compiling the resources, and only the the resource actions gets invoked. By that time, it is too late already.
I ran into the same problem when trying to encapsulate certain attribute manipulations into a resource. It simply does not work. Should anyone know a solution to this problem, I would appreciate it very much.
EDIT:
b = ruby_block...
...
end
b.run_action(:create)
Could possibly do the trick. It invokes the resource immediately.
The simplest answer to this is to not use chef attributes and not use ruby_block to do the work of talking to the REST API. The code can also be moved to a custom resource for better reuse:
unified_mode true
provides :my_resource
action :run do
cm = { :name => "web", :hostname => "" , :ip_addr => "" }
ca = { :name => "data", :hostname => "" , :ip_addr => "" }
cg = { :name => "gateway", :hostname => "" , :ip_addr => "" }
component_list = [cm, ca, cg]
hash = {}
for cmpnt in component_list
# do REST calls to external service to get cmpnt.hostname, ip_addr
# .......
hash[cmpnt.name] = {}
hash[cmpnt.name]['name'] = cmpnt.name
hash[cmpnt.name]['host'] = cmpnt.hostname
hash[cmpnt.name]['ip'] = cmpnt.ip_addr
end
template "/etc/app/configuration/config.xml" do
source "config.xml.erb"
variables( :dataHost => hash['data']['host'],
:webHost => hash['web']['host'],
:gatewayHost => hash['gateway']['host'] )
action :create
end
end
By using unified_mode and moving into a custom resource, it also makes it easier to use a node attribute without requiring the use of lazy {} or ruby_blocks. It also still allows chef configuration (like setting up resolv.conf or other network requirements before doing the REST calls) prior to calling this code while not having to think about compile/converge two pass issues in recipe context.
There is also no reason to use a resource like ruby_block to do pure ruby processing which does not change the system under management. In this case the ruby_block is hitting a REST service purely to collect data. That does not need to be placed into a Chef resource. It isn't clear from the question if that was being done because the questioner though it was a "best practice" (in this case it is not), or if it was being done to move execution to compile time in order to allow other chef resources that aren't part of the question to fire first (in which case using a custom resource is a much better solution than using a ruby_block).
It's been a while since this question, but in case someone is still looking for it, lazy evaluate is your friend:
template '/tmp/sql_file.sql' do
source "sql_file.sql.erb"
mode 0700
variables lazy {
# Create a new instance of MySQL library
mysql_lib = Acx::MySQL.new(
'127.0.0.1', 'root', node['mysql']['service']['pass']
)
password = node['mysql']['service']['support_admin']['ct_password']
# It returns the encrypted password after evaluate it, to
# be used in template variables
{ admin_password: mysql_lib.encrypted_password(password) }
}
end
https://docs.chef.io/resource_common.html#lazy-evaluation