I'm attempting to create a Chef cookbook that, for now, is mostly just a wrapper cookbook for another cookbook (the audit cookbook). I'm still learning Chef, but from what I can gather from the About Recipes documentation and the Resources Reference documentation, Chef recipes should execute in the order that they're defined (via Ruby code and/or Chef resources).
In the About Recipes documentation, it mentions that
When a recipe is included, the resources found in that recipe will be
inserted (in the same exact order) at the point where the
include_recipe keyword is located.
In the Resources Reference documentation, they have an apt_update resource that presumably executes before the include_recipe method due to the fact that it's defined earlier in the recipe.
My wrapper cookbook has a single recipe, default.rb, which is literally these two lines:
package 'ruby-dev'
include_recipe 'audit'
However, during a chef-client or chef-solo run I see that the audit::inspec recipe runs before the security::default recipe which causes things to break because InSpec has some other dependencies that need to be installed beforehand. Before I used the package resource I was using the execute resource to explicitly run apt-get install ruby-dev or yum install ruby-dev depending on the platform using a case statement, but same problem (all that code was skipped and the include_recipe method called first).
In case it's useful, I'm using Chef 12 which I realize is EOL but I have other dependencies that require me to stick with this version of Chef for now.
I may very well just be misunderstanding how Chef converges work and the order in which execution occurs, but it's causing me a lot of grief so I'd really appreciate some pointers! Does include_recipe always occur before other code within your recipe? Is there any way around this? Am I missing something?
-- EDIT --
I was able to get the desired functionality (install other packages and gems before an include_recipe call triggered installation of a dependency gem) using the following code in my cookbook recipe:
package 'build-essential' do
action :nothing
end.run_action(:install)
chef_gem 'train' do
version "1.4.4"
action :install
end
chef_gem 'signet' do
version "0.11.0"
action :install
end
include_recipe 'audit'
Note that I ended up installing the build-essential package rather than the ruby-dev package from my original code snippet, and also installed two gems for Chef client to use. This all gets installed in the order I expected during the compile phase of the Chef run.
Sources:
https://docs.chef.io/resource_reference.html#run-in-compile-phase
https://docs.chef.io/resource_chef_gem.html
if you would examine the audit::inspec rescipe, you will find that it uses a compile time installation of the inspec rubygem (see the last line)
inspec_gem 'inspec' do
version node['audit']['inspec_version']
source node['audit']['inspec_gem_source']
action :nothing
end.run_action(:install)
from chef documentation:
run_action
Use .run_action(:some_action) at the end of a resource block to run the specified action during the compile phase.
Related
Some Background
I am using Vagrant and chef-zero; chef server is not part of my
equation
I have two roles: base and infrastructure, and two
cookbooks: apt and php. The purpose of the base role is to do some basic provisioning and package management and to clone a repository from github, etc. The apt cookbook has a recipe, update_upgrade.rb which runs an execute resource to effectively call apt update, apt upgrade, apt-get autoremove and so forth
The infrastructure role sets up PHP via the php cookbook. It'll include more later, but it serves as a good example of the need I have.
The php cookbook has two recipes: add_repository.rb and install.rb. The add_repository recipe runs an execute resource to add the ondrej/php repository to apt. The install recipe iterates over a list of packages in order to install php from the added repository. This is where the need for the utility comes into play.
The crux is, I need to run apt update after I add the PHP repo and before I install its packages. What's the best way to achieve this?
What I've Tried
Initially, I had my add_repository and install resources combined into a single install recipe and had planned to use include_recipe apt::update_upgrade to insert my apt cookbook's execute resource into that recipe. However, after scouring the chef docs, and this handy article: Should I use include_recipe or add the recipe to run_list?, it seems that include_recipe doesn't run immediately where it's placed. The suggestion in said article was to use a run list to dictate precise execution of recipes, and the chef docs do suggest that the recipes execute in the order they are listed.
This led me to break up the php resources into add_repository and install and to use the following role run list for infrastructure:
"run_list": [
"recipe[php::add_repository]",
"recipe[apt::update_upgrade]",
"recipe[php::install]"
]
My php cookbook's metadata.rb has depends apt, and Chef doesn't error out. However, it also doesn't trigger the update_upgrade recipe after the add_repository one, either. Instead, the install recipe runs after add_repository which causes an error.
This all leads me to assume that role run lists are also executed in stages similar to include_recipe. Is this accurate?
My Conclusions
If these assumptions are accurate, then it seems the only sure way to update apt with the new repository is to duplicate my execute resource from my apt cookbook into my `php cookbook. The DRY-ist in me screams at this; hence the question of a utility recipe/resource.
EDIT
Using the apt Resource
As the original answerer indicates, for this specifically-described use case is to use a combination of apt_package, apt_update, and apt_repository. Here's how I've set it up to get what I want:
In the afore-mentioned apt cookbook, I now have two recipes, install.rb and update.rb. The key recipe is update:
apt_update 'update_packages' do
action :update
end
According to the documentation (https://docs.chef.io/resource_apt_update.html), the :update action will update the Apt repository at the start of a Chef Infra Client run. This is useful for me, as I want to update my OS packages when my VM is first spun up.
Here's my base role's run list which makes use of the apt::update recipe:
"run_list": [
"recipe[apt::install]",
"recipe[apt::update]",
...
]
Next, PHP. First, I want to add the ondrej/php repository, then loop through a list of packages and install them. Lastly, I want to run apt update after installing the new packages.
The first thing I do is to ensure, again, that my php cookbook's metadata.rb has:
depends 'apt'
which will allow me to reuse the apt_update['update_packages'] resource I defined in my apt cookbook.
My install.rb recipe in my php cookbook now contains the following:
# Add PHP repository
apt_repository 'php7' do
uri "ppa:#{node[:infrastructure][:php][:repository][:name]}"
action :add # I know this is unnecessary, but I like to be explicit
end
# Install PHP
node[:infrastructure][:php][:extensions][:list].each do |package|
apt_package package do
action :install
notifies :update, 'apt_update[update_packages]', :delayed
end
end
Since I loop through a list of packages stored in attributes, I notify the apt_update[update_packages] resource after all of the php packages are installed with the :delayed timer.
Finally, here's a look at the infrastructure role which handles the PHP cookbook execution:
"run_list": [
"recipe[php::install]"
]
This explanation handles the above use case beautifully and now hopefully makes use of the appropriate Chef resources. There are, however, some additional Apt actions which aren't covered with resources and which may still require some specific execute resources. Consider the recipes in this apt_cleanup cookbook which cover things like:
Cleaning up apt-cache
Purging leftovers from removed packages
Removing old unused kernels (for a cleaner /boot)
Removes dependencies dragged in by already deleted packages
Looking at those recipes, I see a collection of execute resources, which implies that for these actions, the execute approach may still be necessary. (Albeit within a cookbook).
the best practice is to use the appropriate chef resource in conjunction with notification.
since you are using apt, use the appropriate apt resources, such as apt_update (to make sure apt repositories are up to date), apt_package (to manipulate packages using apt, apt_repository (to add apt respository) and try to avoid using execute resource to do all of that - this will take advantage of chef idompotance (you might like reading Thinking Like a Chef)
each chef resource, can notify and\or subscribe to notifications. but you might not need to use it if you write the resources in order.
if you these advises won't lead you the right way, please post a relevant snippet where you have an issue and i will try to help you farther.
UPDATE
since apt suffers from many failures during update, i would give it few retries and also a periodic update.
apt_update 'update_packages' do
retries 3
action [:update, :periodic]
end
depending of the chef-client version that you are using, apt resouce has been bundles into chef standard resource collection, rather than using the apt cookbook.
so you might not need to specify the dependency on apt cookbook in the metadata.rb file. also, make sure you have no collision between the builtin apt and the apt cookbook functionality.
if you like all your nodes to have the apt_update step at boot time, then place the recipe which utilizes apt_update at your base chef role and make sure that each node runs this recipe (by running the base role) first in the node run-list.
when using apt_repository you did not specify any options for the resource, like arch and others. make sure that this is what you want.
when php packages are being installed using apt_package, there is no need to update apt repositories again. thus, there is no need to notify apt_update each time apt_package is invoked.
We are facing a situation, where end users Windows VM does not have internet connectivity, but only have access to file store.
We are using windows_zipfile resource in one of our recipe. So cookbook execution failed in Windows cookbook, due to the reason that, it is not able to download rubyzip from "rubygems.org" site.
We are thinking of solving the issue in either of these two ways,
Replace the windows_zipfile code with powershell_script and implement the code using Powershell commands
Load the rubyzip gem and its dependency in file store and install the gems before calling windows_zipfile resource.
Please provide suggestions to handle the scenario. Also let me know, is there any other way to solve the issue.
You should be able to install a chef_gem from a local path, after downloading it from a source inside your network (just replace the URL of https://rubygems.org):
{"httpclient" => "2.7.1", "rubyzip" => "1.1.7"}.each do |gem,version|
filename = "#{gem}-#{version}.gem"
remote_file File.join(Chef::Config[:file_cache_path], filename) do
source "https://rubygems.org/downloads/#{filename}"
end
chef_gem gem do
source File.join(Chef::Config[:file_cache_path], filename)
version version
end
end
As the Gem is used by Chef's ruby, make sure to use the chef_gem resource.
I have followed below mentioned article to configure apache, mysql, php using chef cookbook.
http://gettingstartedwithchef.com/first-steps-with-chef.html
My purpose was to write a standalone application, which will do all needed setup for mysql, apache, php etc.
By following the article, I observed they are executing the chef commands to install stuffs & getting modifying the configuration files manually.
for example for the following command I wrote respective ruby equivalent:
rvm install ruby-2.1.2
ruby equivalent : `rvm install ruby-2.1.2`
Similar way, I found alternatives for the shell commands & prepared my ruby code.
I am not sure whether that's the right approach.
2nd: how we can do custom functionality, let's say bundling a gemfile inside a project repo or running rails migrations.
How to perform those operations, please guide.
There're many community created cookbooks for many applications.
You can find many cookbooks on chef supermarket.
In general, you can install packet by use your packet manager this way:
For another supported install options you can look to cookbook.
package "#{res_name} :create nginx" do
package_name 'nginx'
action :install
end
If you want install ruby rvm package, you can use gem_package statement:
gem_package 'name' do
clear_sources TrueClass, FalseClass
gem_binary String
notifies # see description
options String
package_name String, Array # defaults to 'name' if not specified
provider Chef::Provider::Package::Rubygems
source String
subscribes # see description
timeout String, Integer
version String, Array
action Symbol # defaults to :install if not specified
end
I have a single cookbook with two recipes: recipe/install.rb and recipe/configure.rb.
In install.rb, I install all packages, gem_packages and configure, using them in configure.rb. I am installing one gem_package in install.rb, and using it as require <that-gem> in configure.rb.
Regardless of the order of runlist, chef-client execution fails mentioning <that-gem> is missing. If I run with install recipe first and then run configure recipe, things work fine. Isn't there a way where a single chef-run runs both the recipes in order? I tried mentioning include_recipe and all possible solutions.
There are two phases to a Chef client run.
First is the resource compilation stage, where all the ruby is
loaded, node attributes are resolved and a run list generated.
Next is the execution phase, where the recipes in the run list
are executed.
Gems that your recipes rely on need to be available by the execution stage which presents a bit of a chicken or the egg problem, so you need to use a few tricks to get the gems installed first.
When installing gems for Chef to use, install them with chef_gem.
chef_gem "your_gem"
require 'your_gem'
This can get a bit tricky if you start using gems in libraries, as libraries are required at compile time and you can't get actions to occur before compile time. In this case you need to fail nicely on the load so the gem can be installed.
begin
require 'your_gem'
rescue LoadError
warn "Load of [your_gem] failed. Should be installed by install.rb"
end
Then be sure not to use any of the objects the gem provides in the Chef run before the install can occur.
I'm working with someone else's chef recipe and it consists of these references to the process of installing 1.9.3p0 on my server:
package 'ruby1.9.3'
package 'ruby1.9.1-dev'
# set ruby 1.9 to be default
execute 'update-alternatives --set ruby /usr/bin/ruby1.9.1'
execute 'update-alternatives --set gem /usr/bin/gem1.9.1'
ohai "reload" do
action :reload
end
I'm new to chef so I'm not sure where these packages reside, but seeing no other reference them to them in the repo of recipes, I'm guessing it's referring to a central repo. In that case, how could I modify this recipe to get chef (solo) to prepare my servers with a different patch level?
The documentation may clear things up a little here:
package tells the chef-client to use one of sixteen different
providers during the chef-client run, where the provider that is used
by chef-client depends on the platform of the machine on which the
chef-client run is taking place
So on Debian-based systems like the one that recipe was written for, Chef will automatically resolve the package resource to an apt_package resource, which will call apt-get to install ruby1.9.3.
Now, given none of the mainstream Linux distros or FreeBSD package up multiple patchlevels of Ruby (and, in some cases, stated patchlevels are not what they seem), you probably don't want to use package to get Ruby. Most likely you'll end up wanting to build it from source using something like the bash resource.
package will still be useful for installing the Ruby prerequisites, which you can use from your vendor's package repository without issue.