Install gem from S3 in OpsWorks Chef recipe - ruby

I need net-ssh and net-scp as part of a custom OpsWorks chef recipe.
Getting occasional failures from rubygems.org failing to provide the gems, so I would like to host them myself on S3.
chef_gem has the 'source' argument, but it seems to require the local file exist prior to chef being started (so I can't download the file immediately before chef_gem using remote_file)
$gemSsh = "#{Chef::Config[:file_cache_path]}/net-ssh.gem"
$gemScp = "#{Chef::Config[:file_cache_path]}/net-scp.gem"
remote_file $gemSsh do
source "https://s3-us-west-2.amazonaws.com/****/net-ssh-2.9.1.gem"
action :nothing
end.run_action(:create)
remote_file $gemScp do
source "https://s3-us-west-2.amazonaws.com/****/net-scp-1.2.1.gem"
action :nothing
end.run_action(:create)
chef_gem "net-ssh" do
action :nothing
source $gemSsh
end.run_action(:install)
chef_gem "net-scp" do
action :nothing
source $gemScp
end.run_action(:install)
(Note: the run_action(:install) is based on comments here https://tickets.opscode.com/browse/CHEF-4843)
This fails with the following error:
NoMethodError
-------------
undefined method `name' for "/var/lib/aws/opsworks/cache.stage2/net-scp.gem":String
Cookbook Trace:
---------------
/var/lib/aws/opsworks/cache.stage2/cookbooks/opsworks_commons/libraries/monkey_patch_rubygems_provider.rb:55:in `install'
/var/lib/aws/opsworks/cache.stage2/cookbooks/****/recipes/default.rb:24:in `from_file'

You can use "--local" flag which is provided by gem install (You can find other options with gem install --help).
Basic command would be somethig like gem install --local path_to_gem/filename.gem. So your recipe in that case would be:
....
chef_gem "net-ssh" do
action :nothing
options("--local #{$gemSsh}")
end.run_action(:install)
chef_gem "net-scp" do
action :nothing
options("--local #{$gemScp}")
end.run_action(:install)

Related

How would I include and configure a ruby library in a Chef recipe?

I want to include the Diplomat gem in my Chef cookbook so that I can perform Consul variable lookups in .erb templates.
I need to configure the Consul URL:
irb(main):015:0> require 'diplomat'
irb(main):016:0> Diplomat.configure do |config|
irb(main):017:1* config.url = "consulurl:80"
irb(main):018:1> end
Set a variable as the URL path:
irb(main):020:0> kv_path = "path/to/variable"
=> "path/to/variable"
And finally, perform the lookup within the templates.
irb(main):022:0> foo = Diplomat::Kv.get(kv_path + '/test_foo_123')
=> "bar"
Where in the cookbook would I need to write the configuration code above such that I can perform variable lookups within .erb templates?
You want to use the chef_gem resource, but make sure to run it during the compile phase:
chef_gem 'diplomat' do
action :nothing
compile_time false
end.run_action(:install)
require 'diplomat'
Installing gems with Chef is relatively painless. Most of the time, you can use the gem_package resource, which behaves very similarly to the native package resource:
gem_package 'httparty'
You can even specify the gem version to install:
gem_package 'httparty' do
version '0.12.0'
end
You may have also seen the chef_gem resource. What's the difference?
The chef_gem and gem_package resources are both used to install Ruby
gems. For any machine on which the chef-client is installed, there are
two instances of Ruby. One is the standard, system-wide instance of
Ruby and the other is a dedicated instance that is available only to
the chef-client. Use the chef_gem resource to install gems into the
instance of Ruby that is dedicated to the chef-client. Use the
gem_package resource to install all other gems (i.e. install gems
system-wide).
source: https://sethvargo.com/using-gems-with-chef/

Local ruby gems fail to install on Windows 2008R2: Errno::EADDRNOTAVAIL

Edit: Bug in rubygems 2.4.4. (fixed in 2.4.5)
I'm having trouble installing gems with the embedded ruby that comes with Chef Client v12.2.1, using the chef_gem resource:
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
chef_gem[zabbixapi] (generic_server_win::libzabbix-deps line 6) had an error: Mixlib::ShellOut::ShellCommand Failed: Expected process to exit with [0], but received '1'
---- Begin output of C:/opscode/chef/embedded/bin/gem install c:/chef/cache/zabbixapi-2.2.2.gem -q --no-rdoc --no-ri -v "2.2.2" ----
STDOUT:
STDERR: ERROR: While executing gem ... (Errno::EADDRNOTAVAIL)
The requested address is not valid in its context. - connect(2)
---- End output of C:/opscode/chef/embedded/bin/gem install c:/chef/cache/zabbixapi-2.2.2.gem -q --no-rdoc --no-ri -v "2.2.2" ----
Ran C:/opscode/chef/embedded/bin/gem install c:/chef/cache/zabbixapi-2.2.2.gem -q --no-rdoc --no-ri -v "2.2.2" returned 1
Also:
Same result when running the command manually on the command-line as an Administrator with the --local option
Same error occurs for other gems.
The servers that I'm trying to run this on have no internet access
I'm unable to reproduce the problem on a newly installed test machine (with internet access)
The version of ruby used is 2.0.0: ruby 2.0.0p451 (2014-02-24) [i386-mingw32]
Here is my Chef recipe:
cookbook_file "#{Chef::Config[:file_cache_path]}/zabbixapi-2.2.2.gem" do
source 'zabbixapi-2.2.2.gem'
end
chef_gem "zabbixapi" do
source "#{Chef::Config[:file_cache_path]}/zabbixapi-2.2.2.gem"
end
You're problem is that the chef_gem resource is special in the way it enforce the use of the embedded ruby in chef installation and that it is run before convergence to allow the gem to be required in recipes. documentation about it here
To use a local source deployed with chef you have to ensure the file is present before, if not the gem command will try to download it (and fail with no internet access).
To ensure your local file is present before the chef_gem call you have to ensure the cookbook_file resource is called at compile time with this trick
in your specific case this should do:
cookbook_file "#{Chef::Config[:file_cache_path]}/zabbixapi-2.2.2.gem" do
action :nothing
source 'zabbixapi-2.2.2.gem'
end.run_action(:create)
chef_gem "zabbixapi" do
source "#{Chef::Config[:file_cache_path]}/zabbixapi-2.2.2.gem"
end
The action nothing in the resource is to avoid having it called twice (once in compile phase and once in converge phase, even if the later won't have any impact, it's better to save time no evaluating it twice) then calling the action :create at end of the definition will trigger the action in the compile phase and the file will be present for the chef_gem call later.

chef gem_package not using the right binary

I'm trying to install the gems "sensu_plugin" and "mixlib-shellout", on a windows server 2008r2, from a recipe. In this last one, I have the following code :
%w{ sensu-plugin mixlib-shellout }.each do |gem_plugin|
gem_package gem_plugin do
gem_binary("C:/opt/sensu/embedded/bin/gem")
options("--no-rdoc --no-ri")
action :install
end
end
Yet, when running the recipe on my machine, I receive the error "ERROR: While executing gem ... (Errno::ENOENT) No such file or directory - U:/". What I found out is that whenever I try installing a gem package using the gem binary located in the chef-client directory (C:\opscode\chef\embedded\bin\gem), it raised this last error.
The thing is that the gem I wish to use to installed the gem packages is actually located in C:/opt/sensu/embedded/bin/gem, which I have declared in the gem_binary option.
In the end it looks like chef is ignoring the gem_binary option and trying to install it from its own gem binary.
I can't see what's wrong with this config. Is the gem_binary really indicating what binary to use?

Chef does non sequential recipe execution

I followed tutorial http://gettingstartedwithchef.com/, chapter 1.
My run list is
"run_list": [ "recipe[apt]", "recipe[phpap]" ]
My default recipe of phpap cookbook
include_recipe "apache2"
include_recipe "build-essential"
include_recipe "openssl"
include_recipe "mysql::client"
include_recipe "mysql::server"
include_recipe "php"
include_recipe "php::module_mysql"
include_recipe "apache2::mod_php5"
include_recipe "mysql::ruby"
Dependencies of my cookbook
depends "apache2"
depends "mysql"
depends "php"
depends "database"
My repo has following downloaded cookbooks
apache2 aws database openssl phpap xml
apt build-essential mysql php postgresql xfs
I use chef-solo. My host has outdated apt repo info.
Old apt repo should not be a problem because the first recipe in my run list
updates it. But chef ignores apt recipe and starts from mysql one.
See log
dan#mywp3:~/chef-repo$ sudo chef-solo -c solo.rb -j web.json
Starting Chef Client, version 11.6.2
Compiling Cookbooks...
[2013-10-27T00:59:28+04:00] WARN: Cloning resource attributes for service[apache2] from prior resource (CHEF-3694)
[2013-10-27T00:59:28+04:00] WARN: Previous service[apache2]: /home/dan/chef-repo/cookbooks/apache2/recipes/default.rb:24:in `from_file'
[2013-10-27T00:59:28+04:00] WARN: Current service[apache2]: /home/dan/chef-repo/cookbooks/apache2/recipes/default.rb:210:in `from_file'
[2013-10-27T00:59:28+04:00] WARN: Cloning resource attributes for directory[/var/cache/local/preseeding] from prior resource (CHEF-3694)
[2013-10-27T00:59:28+04:00] WARN: Previous directory[/var/cache/local/preseeding]: /home/dan/chef-repo/cookbooks/apt/recipes/default.rb:76:in `block in from_file'
[2013-10-27T00:59:28+04:00] WARN: Current directory[/var/cache/local/preseeding]: /home/dan/chef-repo/cookbooks/mysql/recipes/server.rb:44:in `from_file'
[2013-10-27T00:59:28+04:00] WARN: Cloning resource attributes for directory[/var/lib/mysql] from prior resource (CHEF-3694)
[2013-10-27T00:59:28+04:00] WARN: Previous directory[/var/lib/mysql]: /home/dan/chef-repo/cookbooks/mysql/recipes/server.rb:117:in `block in from_file'
[2013-10-27T00:59:28+04:00] WARN: Current directory[/var/lib/mysql]: /home/dan/chef-repo/cookbooks/mysql/recipes/server.rb:117:in `block in from_file'
[2013-10-27T00:59:28+04:00] WARN: Cloning resource attributes for template[/etc/mysql/my.cnf] from prior resource (CHEF-3694)
[2013-10-27T00:59:28+04:00] WARN: Previous template[/etc/mysql/my.cnf]: /home/dan/chef-repo/cookbooks/mysql/recipes/server.rb:134:in `from_file'
[2013-10-27T00:59:28+04:00] WARN: Current template[/etc/mysql/my.cnf]: /home/dan/chef-repo/cookbooks/mysql/recipes/server.rb:194:in `from_file'
Recipe: mysql::client
* package[mysql-client] action install
================================================================================
Error executing action `install` on resource 'package[mysql-client]'
================================================================================
Chef::Exceptions::Exec
----------------------
apt-get -q -y install mysql-client=5.5.32-0ubuntu0.12.04.1 returned 100, expected 0
Cookbook Trace:
---------------
/home/dan/chef-repo/cookbooks/mysql/recipes/ruby.rb:44:in `block in from_file'
/home/dan/chef-repo/cookbooks/mysql/recipes/ruby.rb:43:in `each'
/home/dan/chef-repo/cookbooks/mysql/recipes/ruby.rb:43:in `from_file'
/home/dan/chef-repo/cookbooks/phpap/recipes/default.rb:20:in `from_file'
Resource Declaration:
---------------------
# In /home/dan/chef-repo/cookbooks/mysql/recipes/client.rb
46: package name
47: end
Compiled Resource:
------------------
# Declared in /home/dan/chef-repo/cookbooks/mysql/recipes/client.rb:46:in `block in from_file'
package("mysql-client") do
action :install
retries 0
retry_delay 2
package_name "mysql-client"
version "5.5.32-0ubuntu0.12.04.1"
cookbook_name :mysql
recipe_name "client"
end
================================================================================
Recipe Compile Error in /home/dan/chef-repo/cookbooks/phpap/recipes/default.rb
================================================================================
Chef::Exceptions::Exec
----------------------
package[mysql-client] (mysql::client line 46) had an error: Chef::Exceptions::Exec: apt-get -q -y install mysql-client=5.5.32-0ubuntu0.12.04.1 returned 100, expected 0
Cookbook Trace:
---------------
/home/dan/chef-repo/cookbooks/mysql/recipes/ruby.rb:44:in `block in from_file'
/home/dan/chef-repo/cookbooks/mysql/recipes/ruby.rb:43:in `each'
/home/dan/chef-repo/cookbooks/mysql/recipes/ruby.rb:43:in `from_file'
/home/dan/chef-repo/cookbooks/phpap/recipes/default.rb:20:in `from_file'
Relevant File Content:
----------------------
/home/dan/chef-repo/cookbooks/mysql/recipes/ruby.rb:
37: when 'rhel'
38: resources('yum_key[RPM-GPG-KEY-percona]').run_action(:add)
39: resources('yum_repository[percona]').run_action(:add)
40: end
41: end
42:
43: node['mysql']['client']['packages'].each do |name|
44>> resources("package[#{name}]").run_action(:install)
45: end
46:
47: chef_gem 'mysql'
48:
[2013-10-27T00:59:30+04:00] ERROR: Running exception handlers
[2013-10-27T00:59:30+04:00] ERROR: Exception handlers complete
[2013-10-27T00:59:30+04:00] FATAL: Stacktrace dumped to /home/dan/chef-solo/chef-stacktrace.out
Chef Client failed. 0 resources updated
[2013-10-27T00:59:30+04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
The host runs ubuntu 12.04.
I tried to put my own recipes before phpap in the run list but without any success.
I can workaround this with manual placing apt-get update in bash script right before chef-solo.
Is it mysql cookbook so bad? As far as I know chef ALWAYS follows run-list strait forward.
In general chef will always honor the run list. However sometimes you might want to run a resource before any others (like to configure a package manager, apt/yum/etc). What seems to be happening in the mysql ruby recipe is that the recipe is explicitly running some package installs at compile time (before chef has begun to execute resources). They fail as the apt recipe hasn't been run yet (none have).
There's a good opscode post explaining compile/execute phases here.
The offending code in mysql::ruby is
node['mysql']['client']['packages'].each do |name|
resources("package[#{name}]").run_action(:install)
end
The run_action(:install) is telling chef to run this now (at compile time).
I got phpap recipe without failure on bare ubuntu 12.04 (without any manual pre update) with following fix.
I removed recipes "build-essential" and "openssl" from phpap one.
After that change chef executes recipes as expected.
But weird behavior left unexplained.
vagrant destroy -f && vagrant up // destroy everything and start over => OK

How do I install dependencies for a chef handler?

I am trying to install a chef handler via the chef_handler lwrp. This handler (chef-handler-email) comes bundled in a gem. I am trying to install the gem then turn on the handler from within a single recipe that looks like:
chef_gem "chef-handler-mail"
chef_handler "MailHandler" do
source 'chef/handler/mail'
arguments :to_address => "root"
action :nothing
supports :exception => true, :report => false
end.run_action(:enable)
This works fine if the gem is already installed. However, if the Gem is not already installed I receive this error:
[2012-12-09T20:47:56-05:00] FATAL: LoadError: chef_handler[MailHandler] (chef_handler::email line 13) had an error: LoadError: no such file to load -- chef/handler/mail.rb
It appears as though the chef_handler resource is trying to load the handler before chef_gem has executed and installed the gem for the handler. I can obviously do this in a two step manual process where I have a separate recipe for installing the gem, then flip over to another recipe that configures the handler, but I'm hoping to avoid multi-step manual processes. Can it be done via single recipe?
I have a similar recipe for chef minitest-chef-handler:
chef_gem 'minitest'
chef_gem 'minitest-chef-handler'
require 'rubygems'
require 'minitest-chef-handler'
[... some unrelated code ...]
chef_handler "MiniTest::Chef::Handler" do
source "minitest-chef-handler"
arguments :verbose => true
action :nothing
end.run_action( :enable )
Try requiring your gem before creating chef_handler resource, or may be source should be different...
The #run_action call causes the chef_handler resource to be run immediately at "compile" phase while the chef_gem resource is run during the "execute" phase as normally.
So also the gem needs to be installed at compile phase. And it seems that a require statement is also needed (as suggested in another answer) for Chef to load the gem.
chef_gem 'chef-handler-mail' do
action :nothing
end.run_action(:install)
require 'chef/handler/mail'
chef_handler 'MailHandler' do
source 'chef/handler/mail'
# ... other attributes
action :nothing
end.run_action(:enable)

Resources