Custom package provider to work around namespace conflicts - ruby

I have a set of RHEL / CentOS systems where I am using the NetBSD pkgsrc system
to install various packages into a certain tree (e.g. /opt/local). I want to
manage this with Puppet. I started out simply using the pkgin provider, but
I quickly ran into namespace conflicts. For example, both the yum and pkgin
providers were adding Package[openssl], and they conflicted.
To fix this, I have been trying to use a static prefix on my pkgsrc packages,
then strip it off when it is actually passed to the pkgin command for handling.
First, I wrote a define to add the prefix and set the provider.
define mypkgin::package($ensure='latest') {
package { "mypkgin_${title}":
ensure => $ensure,
provider => 'mypkgin',
}
}
This will show up in the namespace as Mypkgin::Package[openssl], which does
not itself conflict with anything. And its goal is to add a resource named
Package[mypkgin_openssl], to not conflict with Package[openssl] from yum.
Then, I copied the pkgin provider code and modified it to provide a thing
named mypkgin. I have a small method in here to strip the prefix from the
name, and that is used in a few places so that the pkgin command receives
the pkgsrc package name instead of the prefixed version.
require "puppet/provider/package"
Puppet::Type.type(:package).provide :mypkgin, :parent => Puppet::Provider::Package do
desc "Package management using pkgin/pkgsrc, my local edition."
# Specify full path since it's not in Puppet's exec path
commands :pkgin => "/opt/local/bin/pkgin"
has_feature :installable, :uninstallable, :upgradeable, :versionable
# Strip prefix off of package name
def my_pkgname(package)
package.gsub(/^mypkgin_/, '')
end
def self.parse_pkgin_line(package)
# e.g.
# vim-7.2.446 = Vim editor (vi clone) without GUI
match, name, version, status = *package.match(/(\S+)-(\S+)(?: (=|>|<))?\s+.+$/)
if match
{
:name => name,
:status => status,
:ensure => version
}
end
end
def self.prefetch(packages)
super
# Without -f, no fresh pkg_summary files are downloaded
pkgin("-yf", :update)
end
def self.instances
pkgin(:list).split("\n").map do |package|
new(parse_pkgin_line(package))
end
end
def query
packages = parse_pkgsearch_line
if packages.empty?
if #resource[:ensure] == :absent
notice "declared as absent but unavailable #{#resource.file}:#{resource.line}"
return false
else
#resource.fail "No candidate to be installed"
end
end
packages.first.update( :ensure => :absent )
end
def parse_pkgsearch_line
packages = pkgin(:search, my_pkgname(resource[:name])).split("\n")
return [] if packages.length == 1
packages.slice!(-4, 4)
pkglist = packages.map{ |line| self.class.parse_pkgin_line(line) }
pkglist.select{ |package| my_pkgname(resource[:name]) == package[:name] }
end
def install
if String === #resource[:ensure]
pkgin("-y", :install, "#{my_pkgname(resource[:name])}-#{resource[:ensure]}")
else
pkgin("-y", :install, my_pkgname(resource[:name]))
end
end
def uninstall
pkgin("-y", :remove, my_pkgname(resource[:name]))
end
def latest
package = parse_pkgsearch_line.detect{ |package| package[:status] == '<' }
return properties[:ensure] if not package
return package[:ensure]
end
def update
pkgin("-y", :install, my_pkgname(resource[:name]))
end
end
Every time I run puppet agent --test, it tells me that it has added all of these
packages. So clearly I have some namespace bit wrong. It thinks on each run that
the package either needs to be installed, or upgraded, and it tries to do that.
I assume that in one or more places, I should add or remove calls to
my_pkgname(), but I can't seem to figure out where I've gone wrong.

Your provider determines which packages are already available on the system by means of prefetching. It looks like you have not replaced or overridden the bits that will result in that process ultimately relying on the instances() method.
Looking at that method, it appears that it will generate instances exactly as the pkgin provider would do, right down to the generated package names. That's a problem for you, because your scheme involves translating package names. It's not enough just to translate the names Puppet uses to the native ones -- you must also translate the native ones to the ones you want Puppet to see when you create instances on the provider side.
It looks like self.parse_pkgin_line() could be modified pretty easily to prepend the needed prefix to your package names. That seems like it would be the right thing for it to do, but I'm uncertain whether it would have any undesirable side effects. It is possible that additional changes will be required as well, such as in parse_pkgsearch_line(), but that's not immediately clear to me.

Related

Chef - run install block based on variable condition

Background: our systems are setup in a way that I will only be able to see the local chef log and will have no access to the Chef server console or any other sysadmin privileges. Hence I have a need to log locally if I want to see if or why something failed.
I can hear you asking " If you don't trust the pkg or Chef to install it correctly, then..." My answer is that while you are correct, I still want to be covered by the occasional anomaly.
My goal is to install a pkg, check to see that it installed correctly than go on to the next pkg.
On to the question:
I would like to set a variable that checks for the existence of a directory that was created by the first package using the following code:
mycond = ::File.directory?('/opt/MyPkg/conf')
Chef::Log.fatal("MyPkg package not installed ? conf dir is missing") unless mycond
the next stage in the recipee is to run the next install block checking to see if the variable has been set.
yum_package 'OtherPkg' do
action :install
only_if { mycond }
end
My question is since the only_if is failing, I was wondering if there was something wrong with the way I am setting the mycond variable ? perhapes {} braces are needed somewhere in the code ?
Total Chef newbie so please be specific with your answer.
Thanks !
Full code below:
yum_package 'MyPkg' do
flush_cache [ :before ]
action :install
end
mycond = ::File.directory?('/opt/MyPkg/conf')
Chef::Log.fatal("MyPkg package not installed ? conf dir is missing") unless mycond
yum_package 'OtherPkg' do
action :install
only_if { mycond }
end
The problem is Chef's two-pass model. See https://coderanger.net/two-pass/ for the full explanation for for this you just need to move the condition check in to the only_if block itself since that is delayed until converge time: only_if { ::File.directory?('/opt/MyPkg/conf') }.
Using the fatal log level is also probably not a good idea as this isn't actually a fatal error as written.
Chef has an order of precidance that controls the flow of execution.
Code inside resource blocks (e.g. 'yum_package') will execute AFTER any loose code in your recipe.
The following lines are being executed FIRST, before your 'yum_package' blocks:
mycond = ::File.directory?('/opt/MyPkg/conf')
Chef::Log.fatal("MyPkg package not installed ? conf dir is missing") unless mycond
I believe you can nest resource blocks. You cold be able to combind all this code in a 'ruby_block' and it should execute in order as you'd expect.

Add a custom function in puppet

I'm trying to port a Ruby script in puppet. As far as I know, the only way to achieve that is by creating a custom function in a module. Feel free to tell me if there's another way. I tried my first test as shown on https://docs.puppetlabs.com/guides/custom_functions.html , I declared a new module in:
/etc/puppet/modules/custom_module
and edited a new function file called /etc/puppet/modules/custom_module/lib/puppet/parser/functions/newfunction with this code:
module Puppet::Parser::Functions
newfunction(:write_line_to_file) do |args|
filename = args[0]
str = args[1]
File.open(filename, 'a') {|fd| fd.puts str }
end
end
Then I used in this manifest /etc/puppet/environments/desarrollo/manifests/des.pp with this content:
node "develserver" {
write_line_to_file('/tmp/some_file', "Hello world!")
}
And finally when I run a puppet agent -tod it shows me the following error:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Unknown function write_line_to_file at /etc/puppet/environments/desarrollo/manifests/des.pp:5 on node develserver
What am I doing wrong?
The immediate problem is likely the missing .rb extension on your function file name.
Keep in mind that the Ruby code is run on the master during catalog compilation. If you want Puppet to take action on the agent side, you cannot use custom functions. You will have to write a custom Resource Type (and often a Provider) to do that.

Using File::read in a provider's default.rb in Chef

I am trying to create an LWRP that will call the resource that is defined within itself. My cookbook's structure is as follows:
In the machine cookbook's provider, I have a code snippet as follows:
require 'chef/provisioning' # driver for creating machines
require '::File'
def get_environment_json
##environment_template = JSON.parse(File::read(new_resource.template_path + "environment.json"))
return ##environment_template
end
The code is only trying to read a json file and I am using File::read for it.
I keep getting an error as follows:
LoadError
cannot load such file -- ::File
Does anyone know how I can use File::read inside my LWRP's provider?
OK, so the prior two answers are both half right. You have two problems.
First, you can't require ::File as it's already part of Ruby. This is the cause of your error.
Second, if you call File.read you will grab Chef's File not ruby's. You need to do a ::File.read to use Ruby's File class.
require '::File'
Is incorrect and is causing the LoadError. Delete this line. You don't need it. File is part of the Ruby core and doesn't need to be required.
To further explain, the string argument to require represents the file name of the library you want to load. So, it should look like require "file", or require "rack/utils".
It happens becuase Chef already has a file resource. We have to use the Ruby File class in a recipe.We use ::File to use the Ruby File class to fix this issue. For example:
execute 'apt-get-update' do
command 'apt-get update'
ignore_failure true
only_if { apt_installed? }
not_if { ::File.exist?('/var/lib/apt/periodic/update-success-stamp') }
end
Source: https://docs.chef.io/ruby.html#ruby-class

Ruby require a code snippet from github

Is there anyway to get Ruby's require statement to download a file from somewhere like github rather than just the local file system?
Update: Sorry I should have made the question clearer. I want to download a file that contains Ruby module and import it into my script rather than just downloading an image or some other arbitrary file within my script.
In other words something like this
require 'http:\\github.com\myrepo\snippet.rb'
puts 'hi'
By default, this is not possible. Also, it's not a good idea for security reasons.
You have a couple of alternatives. If the file you want to include is a Gem and Git repository, then you can use Bundler to download and package the dependency in your project. Then you'll be able to require the file directly in your source code.
This is the best and safest way to include an external dependency.
If you trust the source and you really know what you are doing, you can download the file using Net::HTTP (or any other HTTP library) and eval the body directly in your Ruby code.
You can package everything in a custom require_remote function.
You could download and eval it
require "open-uri"
alias :require_old :require
def require(path)
return false if $".include?(path)
unless path=~ /\Ahttp:\/\/
return require_old(path)
end
eval(open(path).read)
$"<< path
true
end
Be aware, this code has no error checking for network outages nonexisting files, ... . I also believe it is in general not a good idea to require libraries this way, there are security and reliability problems in this approach. But maybe you have a valid usecase for this.
you can include a remote gem from within Gemfiles then it will download when you run bundle install
After reading this question and answers I wanted something a little more bullet proof and verbose that used a paradigm of creating a local file from a repo and then requiring it, only if it didn't already exist locally already. The request for the repo version is explicit via the method repo_require. Used on files you control, this approach improves security IMO.
# try local load
def local_require(filename, relative_path)
relative_flname = File.join(relative_path, filename)
require_relative(relative_flname)
end
# try loading locally first, try repo version on load error
# caution: only use with files you control access to!
def repo_require(raw_repo_prefix, filename, relative_path = '')
local_require(filename, relative_path)
rescue LoadError => e
puts e.message
require 'open-uri'
tempdir = Dir.mktmpdir("repo_require-")
temp_flname = File.join(tempdir, File.basename(filename))
return false if $LOADED_FEATURES.include?(temp_flname)
remote_flname = File.join(raw_repo_prefix, filename)
puts "file not found locally, checking repo: #{remote_flname}"
begin
File.open(temp_flname, 'w') do |f|
f.write(URI.parse(remote_flname).read)
end
rescue OpenURI::HTTPError => e
raise "Error: Can't load #{filename} from repo: #{e.message} - #{remote_flname}"
end
require(temp_flname)
FileUtils.remove_entry(tempdir)
end
Then you could call repo_require like this:
repo_require('https://raw.githubusercontent.com/username/reponame/branch',
'filename', 'relative_path')
The relative_path would the the relative path you would use for the file if the repo was locally installed. For example, you may have something like require_relative '../lib/utils.rb'. In this example filename='lib/utils.rb' and relative_path='..'. This information allows the repo url to be constructed correctly as it does not use the relative path portion.

Broken precompiled assets in Rails 3.1 when deploying to a sub-URI

I'm in the process of updating a Rails 3 app to use Rails 3.1 and as part of that, am making use of the new asset pipeline. So far, I've got everything working apart from one rather annoying problem I can't solve.
The application and all its assets works fine in development, but in production it is deployed to a sub-URI using Passenger (http://the-host/sub-uri/). The problem with this is that the assets are pre-compiled during deployment and one of my CSS (well, it's a .css.scss file) files is making use of the image-url helper from the sass-rails gem. Since during the pre-compilation process, the paths are hard-coded into the precompiled CSS file, the sub-uri is not taken account of:
In my .css.scss file:
body { background-image: image-url("bg.png"); }
The result in the compiled application-<md5-hash-here>.css file:
body { background-image: url(/assets/bg.png); }
What it should be to make it work correctly:
body { background-image: url(/sub-uri/assets/bg.png); }
Is this scenario just asking too much? If so, I'll have to switch back to the old non-asset-pipelined way and just serve my images and CSS from public. However it seems like something which should have been thought about and solved...? Am I missing the solution?
Edit 1: I should note that using the erb solution instead yields the same result, as one would expect.
Edit 2: in response to Benoit Garret's comment
No, the problem isn't related to the config.assets.prefix. I tried setting that (to /sub-uri/assets rather than the default of /assets) but it turned out that was the wrong thing to do - it seems like this setting is already in relation to the root of the Rails app, not the server. Removing that (and thus returning to the default) has fixed all the weird issues that caused (and there were many, all the assets ended up in /sub-uri/sub-uri/assets - it was all very strange). The only problem is that the image-url helper and friends do not pick up the sub-URI when they are pre-compiled. Needless to say, this is logical since when it is pre-compiled, it couldn't possibly know that when it's running under Passenger, it'll be configured in this way. My question is how to inform it of this and thus end up with the correct paths in the precompiled result. If indeed it can be done.
My current workaround is to reference the iamge in the CSS like this: url(../images/bg.png) and place it in the non-pipelined public/images location. Hardly ideal since it doesn't benefit from the fingerprinting and everything which the pipeline provides.
Finally I've worked out a couple of workarounds/solutions.
1) From https://github.com/rails/sass-rails/issues/17 it looks like this could get fixed in sass-rails. I've monkey-patched helpers.rb myself along the lines of the proposed patch in the link above. I simply set the required environment variable in the asset precompile line in deploy.rb.
I do all my monkey patching in a single file config/initializers/gem_patches.rb. In this file I patched this method as:
module Sass
module Rails
module Helpers
protected
def public_path(asset, kind)
path = options[:custom][:resolver].public_path(asset, kind.pluralize)
path = ENV['PRODUCTION_URI'] + path if ENV['PRODUCTION_URI']
path
end
end
end
end
2) Alternatively if you are ok to embed images in the CSS, changing the stylesheet to have a .erb extension, and replacing the image-url("bg.png") with url(<%= asset_data_uri "bg.png" %>) will work without any need to change sass-rails. asset-data-uri doesn't exist as a pure Sass function so you have to use the Rails helper asset_data_uri.
In the latest Rails 3.1.3 you need to monkey patch a different module now, for it to work
This is what I did
module Sprockets
module Helpers
module RailsHelper
def asset_path(source, options = {})
source = source.logical_path if source.respond_to?(:logical_path)
path = asset_paths.compute_public_path(source, asset_prefix, options.merge(:body => true))
path = options[:body] ? "#{path}?body=1" : path
if !asset_paths.send(:has_request?)
path = ENV['RAILS_RELATIVE_URL_ROOT'] + path if ENV['RAILS_RELATIVE_URL_ROOT']
end
path
end
end
end
end
And in my deploy.rb I have:
desc "precompile the assets"
namespace :assets do
task :precompile_assets do
run "cd #{release_path} && rm -rf public/assets/* && RAILS_ENV=production bundle exec rake assets:precompile RAILS_RELATIVE_URL_ROOT='/my_sub_uri'"
end
end
before "deploy:symlink", "assets:precompile_assets"
I'm using Rails 3.1.3 and deploying to a sub-URI successfully.
I have NOT monkey-patched anything.
The key problems with this setup have been better discussed here. As you can see, the solution was applied to Rails 3.2 and never backPorted to 3.1.4.
But, I have came to a solution using Rails 3.1.3 that works for my setup.
Try this: (I'm no expert, just trying to contribute to solve a problem that hassled me for hours...)
environment.rb:
#at top:
ENV['RAILS_RELATIVE_URL_ROOT'] = '/rais'
production.rb:
config.assets.prefix = ENV['RAILS_RELATIVE_URL_ROOT'] ? ENV['RAILS_RELATIVE_URL_ROOT'] + '/assets' : '/assets'
routes.rb:
Rais::Application.routes.draw do
scope ENV['RAILS_RELATIVE_URL_ROOT'] || '/' do #see config/environment.rb
<<resources here>>
end
end
As you can see, I've put assets.prefix inside production.rb, not in application.rb
After that you do:
rake assets:clear
rake assets:precompile
and than, test with the console:
RAILS_ENV=production rails console
Results:
foo = ActionView::Base.new
foo.stylesheet_link_tag 'application'
=> "<link href=\"/rais/assets/layout.css?body=1\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\" />\n<link href=\"/rais/assets/application.css?body=1\" media=\"screen\" rel=\"stylesheet\" type=\"text/css\" />"
foo.image_tag('arrow-up.png')
=> "<img alt=\"Arrow-up\" src=\"/rais/assets/arrow-up-ca314ad9b991768ad2b9dcbeeb8760de.png\" />"
After a bit of digging around, I have found the issue. The issue is in Rails, specifically Sprockets::Helpers::RailsHelper::AssetPaths#compute_public_path. Sprockets::Helpers::RailsHelper::AssetPaths inherits from ActionView::AssetPaths and overrides a number of methods. When compute_public_path is called through the Sass::Rails::Resolver#public_path method is sass-rails, the rails sprocket helper picks up the task of resolving the asset. Sprockets::Helpers::RailsHelper::AssetPaths#compute_public_path defers to super which is ActionView::AssetPaths#compute_public_path. In this method there is a condition of has_request? on rewrite_relative_url_root as seen below:
def compute_public_path(source, dir, ext = nil, include_host = true, protocol = nil)
...
source = rewrite_relative_url_root(source, relative_url_root) if has_request?
...
end
def relative_url_root
config = controller.config if controller.respond_to?(:config)
config ||= config.action_controller if config.action_controller.present?
config ||= config
config.relative_url_root
end
If you look at the internals of rewrite_relative_url_root it relies on a request to be present and the ability to derive it from the controller variable in order to resolve the relative url root. The issue is that when sprockets resolves these assets for sass it does not have a controller present and therefore no request.
The solution above didn't work in development mode for me. Here is the solution that I am using to make it work for now:
module Sass
module Rails
module Helpers
protected
def public_path(asset, kind)
resolver = options[:custom][:resolver]
asset_paths = resolver.context.asset_paths
path = resolver.public_path(asset, kind.pluralize)
if !asset_paths.send(:has_request?) && ENV['RAILS_RELATIVE_URL_ROOT']
path = ENV['RAILS_RELATIVE_URL_ROOT'] + path
end
path
end
end
end
end

Resources