My organization works entirely within the firewall (i.e, no machines have internet access). Whenever chef cookbooks are update and require gems the recipes fail due to the fact they cannot download and install gems from rubygems.org.
We self host copies of required gems - how can we add a custom internal source to chef, so that we don't have to deal with failures?
You can manage the gemrc for the embedded Ruby with Chef Client.
chef_etc_dir = Chef::Util::PathHelper.join(Chef::Config.embedded_dir, 'etc')
chef_gemrc = Chef::Util::PathHelper.join(chef_etc_dir, 'gemrc')
directory chef_etc_dir do
owner "root"
group "root"
mode "0755"
end.run_action(:create)
file chef_gemrc do
owner "root"
group "root"
mode "0644"
content <<EOF
---
:sources:
- https://path.to.internal.repo/
- https://rubygems.org
:update_sources: true
EOF
end.run_action(:create)
The PathHelper methods should give appropriate paths for either Windows or Linux.
The ".run_action(:create)" addition executes the resources at compile time, not converge time, to ensure subsequent chef_gem resources will have access to the newly-managed gemrc at converge time.
Update: Chef appears to recommend the rubygems cookbook to perform this functionality.
gemrc :global do
values(
sources: %w{ https://path.to.internal.repo https://rubygems.org }
)
end
Related
I'm trying to make a cookbook that has some dependencies, but it doesn't work
/recipe
ls
default.rb
Recipe:
include_recipe 'sudo::default'
include_recipe 'user'
def will_create_user?(username, user)
return false if node['etc']['passwd'].key?(username)
return true if user['action'] == 'create' || !user.key?('action')
false
end
node['authorization']['sudo']['groups'].each do |group|
group group
end
node['vms']['users'].each do |username, user|
send_notification = will_create_user? username, user
user_account username do
comment user['comment'] if user['comment']
ssh_keys user['ssh_keys']
groups user['groups'] if user['groups']
if send_notification
notifies :run, "execute[reset_password_#{username}]", :delayed
end
end
execute "reset_password_#{username}" do
command "passwd -d #{username} && chage -d 0 #{username}"
action :nothing
end
end
Metadata.rb
...
version '0.1.0'
chef_version '>= 14.0'
depends "sudo"
depends "user"
Berksfile.lock
DEPENDENCIES
vms-users
path: .
metadata: true
GRAPH
sudo (5.4.5)
user (0.7.0)
vms-users (0.1.0)
sudo (>= 0.0.0)
user (>= 0.0.0)
Attributes/default.rb
{
"vms": {
"users": {
'magrini' => {
'comment' => 'Bruna Magrini',
'groups' => ['sysadmin'],
'ssh_keys' => ['chave ssh'],
},
}
}
}
I'm executing using chef-client --local-mode default.rb
Error: Chef::Exceptions::CookbookNotFound: Cookbook sudo not found
Recipe `sudo::default` is not in the run_list, and cookbook 'sudo'
is not a dependency of any cookbook in the run_list. To load this recipe,
first add a dependency on cookbook 'sudo' in the cookbook you're
including it from in that cookbook's metadata.
Running handlers:
[2019-12-19T20:42:12+00:00] ERROR: Running exception handlers
Running handlers complete
[2019-12-19T20:42:12+00:00] ERROR: Exception handlers complete
Chef Infra Client failed. 0 resources updated in 01 seconds
[2019-12-19T20:42:12+00:00] FATAL: Stacktrace dumped to /home/chef-repo/.chef/local-mode-cache/cache/chef-stacktrace.out
[2019-12-19T20:42:12+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-12-19T20:42:12+00:00] FATAL: Chef::Exceptions::CookbookNotFound: Cookbook sudo not found. If you're loading sudo from another cookbook, make sure you configure the dependency in your metadata
i have a feeling that you are confusing something, since the title of your question mentions chef-solo when you are really using chef-zero (--local-mode).
you should definitely favor chef-zero over chef-solo (but i won't get into the reasons why).
in both cases (using chef-zero or chef-solo), you will have to download all the cookbooks and make verify chef-client knows where are the cookbooks located.
if you ase using chef-zero, here are some references:
Local mode does not require a configuration file, instead it will look for a directory named /cookbooks and will set chef_repo_path to be just above that. (Local mode will honor the settings in a configuration file, if desired.) If the client.rb file is not found and no configuration file is specified, local mode will search for a config.rb file.
client.rb settings:
chef_repo_path: The path to the chef-repo containing cookbooks and other files, such as environments or data bags, when running Chef Infra Client in local mode.
cookbook_path: The sub-directory for Chef Infra Client cookbooks. This value can be a string or an array of file system locations, processed in the specified order. The last cookbook is considered to override local modifications.
since i see that you are using berkshelf, you can use vendor sub-command to download all the cookbooks dependencies and place them in the same directory. then, have a custom configuration for chef-client, that sets the value of cookbook_path to the same directory which you used in conjunction with berks vendor, and finally execute chef-client.
Thanks for taking a look at this question. Any help is appreciated.
I am provisioning a virtual machine with a GUI using vagrant and chef.
Goal: to download IntelliJ IDE and then install it so that it is available to my user when I log in.
The cookbook cookbook 'idea', '~> 0.4.0'achieves the download but a user must manually complete the install on the guest.
I am having trouble with my custom recipe to complete the configuration with chef. As it is written, the recipe completes if I add it to the run list after the machine is provisioned but fails in the initial run because files are not yet installed.
I tried using the only_if method within the relevant blocks and on the entire recipe, but couldn't get it to work. I also messed with the subscribe method but couldn't get that to work either.
I'm sure this has an easy solution, but Googling and trial and error are not getting me any closer. I would appreciate any help to achieve the goal. Thanks!
Current recipe
# Configure IntelliJ Idea.
file '/opt/idea/idea.desktop' do
content '[Desktop Entry]
Name=IntelliJ IDEA
Type=Application
Exec=idea
Terminal=false
Icon=idea
Comment=Integrated Development Environment
NoDisplay=false
Categories=Development;IDE;
Name[en]=IntelliJ IDEA'
mode '644'
owner 'root'
group 'root'
end
bash 'install idea desktop' do
code <<-EOH
cd /opt/idea
sudo desktop-file-install idea.desktop
EOH
end
file '/usr/share/pixmaps/idea.png' do
owner 'root'
group 'root'
mode '0644'
content ::File.open('/opt/idea/bin/idea.png').read
action :create
end
link '/usr/local/bin/idea' do
to '/opt/idea/bin/idea.sh'
link_type :symbolic
end
Failed efforts:
Wrapping the entire script
# Configure IntelliJ Idea.
execute 'configure idea' do
only_if { ::File.exist?("/opt/idea") }
continues...
end
Using only_if in the blocks
file '/usr/share/pixmaps/idea.png' do
action :create
only_if { ::File.exist?('/opt/idea/bin/idea.png') }
owner 'root'
group 'root'
mode '0644'
content ::File.open('/opt/idea/bin/idea.png').read
end
link '/usr/local/bin/idea' do
to '/opt/idea/bin/idea.sh'
only_if { ::File.exist?('/opt/idea/bin/idea.sh') }
link_type :symbolic
end
What you probably want is a lazy evaluated property:
content lazy { ::File.open('/opt/idea/bin/idea.png').read }
That will delay the file read until converge time instead of compile time.
I have to make a REST-Client in Ruby.
The client must be runnable from the command-line like a binary and also it must be "requirable" in a ruby script and provide different functions.
My gemspec does exactly what it should.
But i have no idea how to install a configuration file (YAML) in the user-home folder?
The config file should be in the user directory to provide easy access for the user.
Is this even possible?
Should i check on the first run if there is a config file and create it?
Can i execute an own installation routine while installing a gem?
I did exactly the same thing in Python and it worked fine, so the Ruby client should behave similar.
For such decisions, I wrote gem persey. If you look at the description of the use of this gem, you can see that it provides what you expect:
# Rails.root are not initialized here
app_path = File.expand_path('../../', __FILE__)
# ...
# config with secret keys
# you don't want store this config in repository and copy to secret folder on host machine
my_secret_key_config = '/home/user/secret/keys.yml'
# ...
# Persey.init ENV["environment"] do # set current environment
Persey.init Rails.env do # set current environment
source :yaml, my_secret_key_config, :secret # no comments. It's secret!
env :production do
# ...
end
env :development, :parent => :production do
# ...
end
end
I have built a cookbook for installing Jenkins CI. It uses the key and repository resources from the yum cookbook, so I end up with the following recipe:
yum_key "RPM-GPG-KEY-jenkins" do
url "http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key"
action :add
end
yum_repository "jenkins" do
description "Jenkins-CI 3rd party repository"
url "http://pkg.jenkins-ci.org/redhat"
key "RPM-GPG-KEY-jenkins"
action :add
end
When I include this recipe in another recipe:
include_recipe 'sp_jenkins::default'
and I test this with the following ChefSpec test
it 'includes the `sp_jenkins::default` recipe' do
expect(chef_run).to include_recipe('sp_jenkins::install')
end
my ChefSpec test fails with the following output:
NameError:
Cannot find a resource for yum_key on chefspec version 0.6.1
(I'm not sure why it says version 0.6.1, gem list tells me it's using 3.0.2)
The sp_jenkins cookbook does depend on the yum cookbook (metadata.rb), and runs fine, however, the cookbook I'm currently writing does not depend on the yum cookbook and therefore doesn't have the yum_key and yum_repository methods available.
Is there a way to prevent ChefSpec from 'descending' into included recipes/cookbooks and just test the current cookbook?
Ohai! Julian is correct - ChefSpec actually does a Chef Solo run in memory on your local machine. It rewrites the provider actions to be a noop, but creates a registry of all the actions taken (including those that would be taken if notifications were executed).
So just like you need the yum cookbook to converge this recipe on a real node, you need it to converge during your unit tests with ChefSpec. The easiest way to accomplish this is by using the Berkshelf or Librarian resolvers. To use the Berkshelf resolver, simply require 'chefspec/berkshelf' after requiring chefspec:
# spec_helper.rb
require 'chefspec'
require 'chefspec/berkshelf'
If you have Berkshelf installed on your system, it will pull all the cookbooks into a temporary directory and run ChefSpec for you.
You may also want to take a look at Strainer, which aims to solve a similar problem.
On a somewhat unrelated note, I am working on a fairly large refactor to the Jenkins cookbook that may better suit your needs.
Sources:
I wrote it...
No, there's no way to prevent it from descending, because it's trying to converge an entire Chef run in memory.
However, if you use the Berkshelf functionality in ChefSpec, the Berkshelf dependency resolver will feed all dependent cookbooks to the in-memory Chef run, and you'll be golden.
It is absolutely valid to expect to test your cookbook in isolation, and not include other projects' code into the scope of your tests. Unfortunately there appears to be no supported, "clean" way to do this, that I can find. I was able to achieve this, but it comes at a price.
To use this technique, do not require 'chefspec/berkshelf' anywhere in your test code, only chefspec itself, as you are intentionally not gathering other cookbook source. Here is a template of my working test module (not my complete test code, as I have omitted RSpec config options):
describe 'mycookbook::recipe' do
let(:chef_run) do
ChefSpec::SoloRunner.new(platform: 'x', version: 'x') {
# ...
}.converge(described_recipe)
end
before :each do
allow_any_instance_of(Chef::RunContext::CookbookCompiler).to receive(:cookbook_order) do
Chef::Log.debug 'Attempt to source external cookbooks blocked'
[described_cookbook]
end
allow_any_instance_of(Chef::Recipe).to receive(:include_recipe) do |recipe|
Chef::Log.debug "Attempt to include #{recipe} blocked"
end
end
it 'works' do
# ...
end
end
You need both of these in your before. The one I had to work for is the intercept of the :cookbook_order method. I had to drill down into the Chef internals to discover this. Keep in mind, this worked for me using Chef 14, but there is no guarantee that this will be future-safe. After upgrading Chef you might have to find another solution, if the implementation of CookbookCompiler ever changes. (The intercept of Chef::Recipe.include_recipe however is a supported API and therefore should be at least somewhat future-safe.)
And, I mention that this comes at a price. (Other than using an unsupported hack!) You will not be able to do any expects for your recipe or attribute includes, except within your own cookbook. A test case like this will fail, because the recipe can't actually be included, as you are preventing that:
it 'includes othercookbook::recipe' do
expect_any_instance_of(Chef::Recipe).to receive(:include_recipe).with('othercookbook::recipe')
end
Also, you must now satisfy in your before blocks all attributes and other preconditions that might otherwise be fulfilled by other recipes in your run list. So you may be signing yourself up for considerable pain by doing this. But, once you have finished, you will have much less brittle tests. (Although to achieve 100% purity regarding external dependencies, you must also surrender fauxhai, which will be even more painful.)
So I use the following recipe:
include_recipe "build-essential"
node_packages = value_for_platform(
[ "debian", "ubuntu" ] => { "default" => [ "libssl-dev" ] },
[ "amazon", "centos", "fedora", "centos" ] => { "default" => [ "openssl-devel" ] },
"default" => [ "libssl-dev" ]
)
node_packages.each do |node_package|
package node_package do
action :install
end
end
bash "install-node" do
cwd Chef::Config[:file_cache_path]
code <<-EOH
tar -xzf node-v#{node["nodejs"]["version"]}.tar.gz
(cd node-v#{node["nodejs"]["version"]} && ./configure --prefix=#{node["nodejs"]["dir"]} && make && make install)
EOH
action :nothing
not_if "#{node["nodejs"]["dir"]}/bin/node --version 2>&1 | grep #{node["nodejs"]["version"]}"
end
remote_file "#{Chef::Config[:file_cache_path]}/node-v#{node["nodejs"]["version"]}.tar.gz" do
source node["nodejs"]["url"]
checksum node["nodejs"]["checksum"]
notifies :run, resources(:bash => "install-node"), :immediately
end
It successfully installed nodejs on my Vagrant VM but on restart it's getting executed again. How do I prevent this? I'm not that good in reading ruby code.
To make the remote_file resource idempotent (i.e. to not download a file already present again) you have to correctly specify the checksum of the file. You do this in your code using the node["nodejs"]["checksum"] attribute. However, this only works, if the checksum is correctly specified as the SHA256 hash of the downloaded file, no other algorithm (esp. not MD5) is supported.
If the checksum is not correct, your recipe will still work. However, on the next run, Chef will notice that the checksum of the existing file is different from the one you specified and will download the file again, thus notify the install node ressource and do the whole compile stuff.
With chef, it's important that recipes be idempotent. That means that they should be able to run over and over again without changing the outcome. Chef expects to be able to run all the recipes on a node periodically, and that should be ok.
Do you have a way of knowing which resource within that recipe is causing you problems? The remote_file one is the only one I'm suspicious of being non-idempotent, but I'm not sure offhand.
Looking at the Chef wiki, I find this:
Deprecated Behavior In Chef 0.8.x and earlier, Remote File is also
used to fetch files from the files/ directory in a cookbook. This
behavior is now provided by #Cookbook File, and use of Remote File for
this purpose is deprecated (though still valid) in Chef 0.9.0 and
later.
Anyway, the way chef tends to work, it will look to see if whatever "#{Chef::Config[:file_cache_path]}/node-v#{node["nodejs"]["version"]}.tar.gz" resolves to exists, and if it does, it should skip that resource. Is it possible that install-node deletes that file when it's finished installing? If so, chef will re-fetch it every time.
You can run a recipe only once overriding the run-list with -o modifier.
sudo chef-client -o "recipe[cookbook::recipe]"
-o RunlistItem,RunlistItem..., Replace current run list with specified items
--override-runlist
In my experience remote_file always runs when executing chef-client, even if the target file already exists. I'm not sure why (haven't dug into the Chef code to find the exact cause of the bug), though.
You can always write a not_if or only_if to control the execution of the remote_file resource, but usually it's harmless to just let it run every time.
The rest of your code looks like it's already idempotent, so there's no harm in running the client repeatedly.
There's an action you can specify for remote_file that will make it run conditionally:
remote_file 'target' do
source 'wherever'
action :create_if_missing
end
See the docs.
If you want to test whether your recipe is idempotent, you may be interested in ToASTER, a framework for systematic testing of Chef scripts.
http://cloud-toaster.github.io/
Chef recipes are executed with different configurations in isolated container environments (Docker VMs), and ToASTER reports various metrics such as system state changes, convergence properties, and idempotence issues.