What's the best way to set up "utility" resources in Chef - ruby

Some Background
I am using Vagrant and chef-zero; chef server is not part of my
equation
I have two roles: base and infrastructure, and two
cookbooks: apt and php. The purpose of the base role is to do some basic provisioning and package management and to clone a repository from github, etc. The apt cookbook has a recipe, update_upgrade.rb which runs an execute resource to effectively call apt update, apt upgrade, apt-get autoremove and so forth
The infrastructure role sets up PHP via the php cookbook. It'll include more later, but it serves as a good example of the need I have.
The php cookbook has two recipes: add_repository.rb and install.rb. The add_repository recipe runs an execute resource to add the ondrej/php repository to apt. The install recipe iterates over a list of packages in order to install php from the added repository. This is where the need for the utility comes into play.
The crux is, I need to run apt update after I add the PHP repo and before I install its packages. What's the best way to achieve this?
What I've Tried
Initially, I had my add_repository and install resources combined into a single install recipe and had planned to use include_recipe apt::update_upgrade to insert my apt cookbook's execute resource into that recipe. However, after scouring the chef docs, and this handy article: Should I use include_recipe or add the recipe to run_list?, it seems that include_recipe doesn't run immediately where it's placed. The suggestion in said article was to use a run list to dictate precise execution of recipes, and the chef docs do suggest that the recipes execute in the order they are listed.
This led me to break up the php resources into add_repository and install and to use the following role run list for infrastructure:
"run_list": [
"recipe[php::add_repository]",
"recipe[apt::update_upgrade]",
"recipe[php::install]"
]
My php cookbook's metadata.rb has depends apt, and Chef doesn't error out. However, it also doesn't trigger the update_upgrade recipe after the add_repository one, either. Instead, the install recipe runs after add_repository which causes an error.
This all leads me to assume that role run lists are also executed in stages similar to include_recipe. Is this accurate?
My Conclusions
If these assumptions are accurate, then it seems the only sure way to update apt with the new repository is to duplicate my execute resource from my apt cookbook into my `php cookbook. The DRY-ist in me screams at this; hence the question of a utility recipe/resource.
EDIT
Using the apt Resource
As the original answerer indicates, for this specifically-described use case is to use a combination of apt_package, apt_update, and apt_repository. Here's how I've set it up to get what I want:
In the afore-mentioned apt cookbook, I now have two recipes, install.rb and update.rb. The key recipe is update:
apt_update 'update_packages' do
action :update
end
According to the documentation (https://docs.chef.io/resource_apt_update.html), the :update action will update the Apt repository at the start of a Chef Infra Client run. This is useful for me, as I want to update my OS packages when my VM is first spun up.
Here's my base role's run list which makes use of the apt::update recipe:
"run_list": [
"recipe[apt::install]",
"recipe[apt::update]",
...
]
Next, PHP. First, I want to add the ondrej/php repository, then loop through a list of packages and install them. Lastly, I want to run apt update after installing the new packages.
The first thing I do is to ensure, again, that my php cookbook's metadata.rb has:
depends 'apt'
which will allow me to reuse the apt_update['update_packages'] resource I defined in my apt cookbook.
My install.rb recipe in my php cookbook now contains the following:
# Add PHP repository
apt_repository 'php7' do
uri "ppa:#{node[:infrastructure][:php][:repository][:name]}"
action :add # I know this is unnecessary, but I like to be explicit
end
# Install PHP
node[:infrastructure][:php][:extensions][:list].each do |package|
apt_package package do
action :install
notifies :update, 'apt_update[update_packages]', :delayed
end
end
Since I loop through a list of packages stored in attributes, I notify the apt_update[update_packages] resource after all of the php packages are installed with the :delayed timer.
Finally, here's a look at the infrastructure role which handles the PHP cookbook execution:
"run_list": [
"recipe[php::install]"
]
This explanation handles the above use case beautifully and now hopefully makes use of the appropriate Chef resources. There are, however, some additional Apt actions which aren't covered with resources and which may still require some specific execute resources. Consider the recipes in this apt_cleanup cookbook which cover things like:
Cleaning up apt-cache
Purging leftovers from removed packages
Removing old unused kernels (for a cleaner /boot)
Removes dependencies dragged in by already deleted packages
Looking at those recipes, I see a collection of execute resources, which implies that for these actions, the execute approach may still be necessary. (Albeit within a cookbook).

the best practice is to use the appropriate chef resource in conjunction with notification.
since you are using apt, use the appropriate apt resources, such as apt_update (to make sure apt repositories are up to date), apt_package (to manipulate packages using apt, apt_repository (to add apt respository) and try to avoid using execute resource to do all of that - this will take advantage of chef idompotance (you might like reading Thinking Like a Chef)
each chef resource, can notify and\or subscribe to notifications. but you might not need to use it if you write the resources in order.
if you these advises won't lead you the right way, please post a relevant snippet where you have an issue and i will try to help you farther.
UPDATE
since apt suffers from many failures during update, i would give it few retries and also a periodic update.
apt_update 'update_packages' do
retries 3
action [:update, :periodic]
end
depending of the chef-client version that you are using, apt resouce has been bundles into chef standard resource collection, rather than using the apt cookbook.
so you might not need to specify the dependency on apt cookbook in the metadata.rb file. also, make sure you have no collision between the builtin apt and the apt cookbook functionality.
if you like all your nodes to have the apt_update step at boot time, then place the recipe which utilizes apt_update at your base chef role and make sure that each node runs this recipe (by running the base role) first in the node run-list.
when using apt_repository you did not specify any options for the resource, like arch and others. make sure that this is what you want.
when php packages are being installed using apt_package, there is no need to update apt repositories again. thus, there is no need to notify apt_update each time apt_package is invoked.

Related

Chef recipe 'include_recipe' takes precedence over other code and resources

I'm attempting to create a Chef cookbook that, for now, is mostly just a wrapper cookbook for another cookbook (the audit cookbook). I'm still learning Chef, but from what I can gather from the About Recipes documentation and the Resources Reference documentation, Chef recipes should execute in the order that they're defined (via Ruby code and/or Chef resources).
In the About Recipes documentation, it mentions that
When a recipe is included, the resources found in that recipe will be
inserted (in the same exact order) at the point where the
include_recipe keyword is located.
In the Resources Reference documentation, they have an apt_update resource that presumably executes before the include_recipe method due to the fact that it's defined earlier in the recipe.
My wrapper cookbook has a single recipe, default.rb, which is literally these two lines:
package 'ruby-dev'
include_recipe 'audit'
However, during a chef-client or chef-solo run I see that the audit::inspec recipe runs before the security::default recipe which causes things to break because InSpec has some other dependencies that need to be installed beforehand. Before I used the package resource I was using the execute resource to explicitly run apt-get install ruby-dev or yum install ruby-dev depending on the platform using a case statement, but same problem (all that code was skipped and the include_recipe method called first).
In case it's useful, I'm using Chef 12 which I realize is EOL but I have other dependencies that require me to stick with this version of Chef for now.
I may very well just be misunderstanding how Chef converges work and the order in which execution occurs, but it's causing me a lot of grief so I'd really appreciate some pointers! Does include_recipe always occur before other code within your recipe? Is there any way around this? Am I missing something?
-- EDIT --
I was able to get the desired functionality (install other packages and gems before an include_recipe call triggered installation of a dependency gem) using the following code in my cookbook recipe:
package 'build-essential' do
action :nothing
end.run_action(:install)
chef_gem 'train' do
version "1.4.4"
action :install
end
chef_gem 'signet' do
version "0.11.0"
action :install
end
include_recipe 'audit'
Note that I ended up installing the build-essential package rather than the ruby-dev package from my original code snippet, and also installed two gems for Chef client to use. This all gets installed in the order I expected during the compile phase of the Chef run.
Sources:
https://docs.chef.io/resource_reference.html#run-in-compile-phase
https://docs.chef.io/resource_chef_gem.html
if you would examine the audit::inspec rescipe, you will find that it uses a compile time installation of the inspec rubygem (see the last line)
inspec_gem 'inspec' do
version node['audit']['inspec_version']
source node['audit']['inspec_gem_source']
action :nothing
end.run_action(:install)
from chef documentation:
run_action
Use .run_action(:some_action) at the end of a resource block to run the specified action during the compile phase.

How to use a class from a module immediately after installing it with Puppet?

I am experimenting with Puppet using Vagrant. I'm new to Puppet.
I'm installing modules in my Puppet manifest using the approach suggested at: Can I install puppet modules through puppet manifest?
My default.pp contains something like:
$dsesterojava = 'dsestero-java'
exec { 'dsestero-java':
command => "puppet module install ${dsesterojava}",
unless => "puppet module list | grep ${dsesterojava}",
path => ['/usr/bin', '/bin']
}
include java::java_7
I'm trying to import a module and then immediately use the classes defined in it.
Currently, I get:
Error: Could not find class java::java_7
If I comment out the include line and re-run it. The module installs. If I then removed the comment and run the provisioning again then it works.
There is some kind of "chicken and egg" situation here. Can I use a module in the same Puppet manifest that installs it?
How should I solve it?
No, you cannot do this. When your catalog is compiled, Puppet will search in the appropriate directories for all of the required code and data. Since the java module does not exist until catalog application, the compilation of a catalog (occurs prior to application) depending upon it will fail. You are absolutely dealing with a "chicken and egg" situation here. I highly recommend against using Puppet code to install Puppet code.
Alternatively, the recommended approach to install and manage your Puppet modules is to use one of these solutions:
librarian-puppet: http://librarian-puppet.com/
r10k: https://github.com/puppetlabs/r10k
code-manager (PE only): https://puppet.com/docs/pe/2017.3/code_management/code_mgr.html
These will also solve the problem for you within the Vagrant if you are using the agent provisioner and subscribing the Vagrant instance to a Puppet Master.
If you are using the apply provisioner inside of Vagrant, then you will need to go a different route. The simplest solution is to use the shell provisioner to install Puppet modules via module install after the Puppet installation (unless you are using a Vagrant box with Puppet baked in, in which case you are probably not installing Puppet on it). Alternatively, you could share a directory with the host where your modules are installed, or install the librarian-puppet or r10k gems onto the Vagrant box and then use them to install into the appropriate path. I can go into more detail on these upon request.

using omnibus build tool to package a full app

So I am tasked with packaging a Ruby application. The idea is to eliminate the need to apt-get any packages or gem install anything. the package is all-inclusive.
I've gotten as far as finding Omnibus (there is also fmp-cookery but it doesnt seem to be as well known)
I've figured out the general structure of omnibus projects but now getting problems with specifics:
1) I have a bunch of recipies that run with Chef AT INSTALL TIME. These setup the DB and nginx configs. Whats the best way to run through these on target machine?
2) I added a bundle install line in my build scripts, but this now dubplicates my requirements. For example nokogiri gets installed twice. Once as my actual project Gem and another time as another requirement of omnibus. So then I end up with 2 binaries one in /#{instal_dir}/embedded/bin and one in ${install_dir}/embedded/lib/ruby/.../gems any way to prevent this?
Building an installer is bit more involved than creating a chef cookbook.
User needs to write a DSL for each module (this has instructions on gathering the artifacts and installing it on specific OS).
After creating the omnibus project, refer to the README file to get started with basics.
Refer to this github project for details on how to build your own DSLs. Gitlab omnibus project has DSLs for some modules like postgresql etc.
https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master
Example DSL for embedding nginx module in your installer looks like this:
name "nginx"
default_version "1.9.10"
dependency "pcre"
dependency "openssl"
source url: "http://nginx.org/download/nginx-#{version}.tar.gz",
md5: "64cc970988356a5e0fc4fcd1ab84fe57"
relative_path "nginx-#{version}"
build do
command ["./configure",
"--prefix=#{install_dir}/embedded",
"--with-http_ssl_module",
"--with-http_stub_status_module",
"--with-http_gzip_static_module",
"--with-http_v2_module",
"--with-ipv6",
"--with-debug",
"--with-ld-opt=-L#{install_dir}/embedded/lib",
"--with-cc-opt=\"-L#{install_dir}/embedded/lib -I#{install_dir}/embedded/include\""].join(" ")
command "make -j #{workers}", :env => {"LD_RUN_PATH" => "#{install_dir}/embedded/lib"}
command "make install"
end

Order of executing Chef recipes in a single cookbook

I have a single cookbook with two recipes: recipe/install.rb and recipe/configure.rb.
In install.rb, I install all packages, gem_packages and configure, using them in configure.rb. I am installing one gem_package in install.rb, and using it as require <that-gem> in configure.rb.
Regardless of the order of runlist, chef-client execution fails mentioning <that-gem> is missing. If I run with install recipe first and then run configure recipe, things work fine. Isn't there a way where a single chef-run runs both the recipes in order? I tried mentioning include_recipe and all possible solutions.
There are two phases to a Chef client run.
First is the resource compilation stage, where all the ruby is
loaded, node attributes are resolved and a run list generated.
Next is the execution phase, where the recipes in the run list
are executed.
Gems that your recipes rely on need to be available by the execution stage which presents a bit of a chicken or the egg problem, so you need to use a few tricks to get the gems installed first.
When installing gems for Chef to use, install them with chef_gem.
chef_gem "your_gem"
require 'your_gem'
This can get a bit tricky if you start using gems in libraries, as libraries are required at compile time and you can't get actions to occur before compile time. In this case you need to fail nicely on the load so the gem can be installed.
begin
require 'your_gem'
rescue LoadError
warn "Load of [your_gem] failed. Should be installed by install.rb"
end
Then be sure not to use any of the objects the gem provides in the Chef run before the install can occur.

In a Chef recipe, how do you specify a particular patch level for a Ruby?

I'm working with someone else's chef recipe and it consists of these references to the process of installing 1.9.3p0 on my server:
package 'ruby1.9.3'
package 'ruby1.9.1-dev'
# set ruby 1.9 to be default
execute 'update-alternatives --set ruby /usr/bin/ruby1.9.1'
execute 'update-alternatives --set gem /usr/bin/gem1.9.1'
ohai "reload" do
action :reload
end
I'm new to chef so I'm not sure where these packages reside, but seeing no other reference them to them in the repo of recipes, I'm guessing it's referring to a central repo. In that case, how could I modify this recipe to get chef (solo) to prepare my servers with a different patch level?
The documentation may clear things up a little here:
package tells the chef-client to use one of sixteen different
providers during the chef-client run, where the provider that is used
by chef-client depends on the platform of the machine on which the
chef-client run is taking place
So on Debian-based systems like the one that recipe was written for, Chef will automatically resolve the package resource to an apt_package resource, which will call apt-get to install ruby1.9.3.
Now, given none of the mainstream Linux distros or FreeBSD package up multiple patchlevels of Ruby (and, in some cases, stated patchlevels are not what they seem), you probably don't want to use package to get Ruby. Most likely you'll end up wanting to build it from source using something like the bash resource.
package will still be useful for installing the Ruby prerequisites, which you can use from your vendor's package repository without issue.

Resources