I'm building a Ruby gem file that has various 3rd party executable requirements (some non-ruby tools that the gem will then use through system calls).
The requirement list is specified using the gemspec informative requirements list, and I was tasked with creating a set up where that requirement list is verified during installation - essentially converting the requirements list from being informative to being authoritative.
My initial approach is to use extensions, like in the extconf.rb pattern, to run some ruby code during installation that will verify the list of requirements and fail the installation if the requirements are not met.
So my current implementation is using a ext/Rakefile, that reads gemspec file using a hard-coded path, parses the requirements list using a custom (but straight-forward and simple) syntax and run some simple tests on the required executables.
I would really want to factor out the dependency on knowing the path to the gemspec file, so I can reuse the code in a modular way on other projects.
For completion (and in the hopes that it will be useful for someone), here's my code - which uses the Gem::Ext::Builder "Rakefile mode" to just run some arbitrary code.
require 'rubygems'
require 'mkmf'
def verify_requirement(reqspec)
exe, version_spec, version_regex = reqspec.split(/\s*,\s*/,3)
exe = $1 if exe =~ /\[([^\]]+)\]/ # support human readable alias for executable name
abort "Missing required executable #{exe}!" unless find_executable(exe)
return unless version_spec # no version is specified, so executable existing is good enough
version = %x|#{exe} --version 2>&1|.chomp
version = %x|#{exe} -version 2>&1|.chomp unless $?.success?
abort "Error checking for version of #{exe}: (#{$?.exitstatus}) #{version}" unless $?.success?
if version_regex
abort "Version test '#{version_regex}' failed to match '#{version}'!" unless version =~ /#{version_regex}/m
abort "Version test '#{version_regex}' failed to generate a version number!" unless $1
version = $1
end
abort "Failed to locate a valid version number for #{exe}, found '#{version}'" unless version =~ /^[0-9\.]+/
abort "Insufficient version '#{version}' for #{exe} - requires #{version_spec}" unless Gem::Dependency.new(exe, version_spec).match? exe, version
end
task :default do
gemspec = File.absolute_path('../my.gemspec')
Gem::Specification::load(gemspec).requirements.each do |req|
verify_requirement(req)
end
end
This code requires that the requirements uses either of these syntaxes:
exe-file[, <semantic-version-spec>[, <version-garabbing-regex>]]
alias '[' exe-file ']'[, <semantic-version-spec>[, <version-garabbing-regex>]]
For example:
spec.requirements << 'sox'
spec.requirements << 'ffmpeg, >= 2.5, ffmpeg version ([\d\.]+)'
spec.requirements << 'ImageMagick [convert], >= 6.7.5, ImageMagick ([\d\.]+)'
The Gem::Specification class can be used to load and parse the gemspec. I'm using the following code in ext/Rakefile to perform tasks based on the gemspec file:
task :default do
gemspec = File.absolute_path('../gemname.gemspec')
Gem::Specification::load(gemspec).requirements.each do |req|
verify_requirement(req)
end
end
Related
I've seen similar questions regarding this error, but all of them rails-related. I'm not using rails; I'm working on a local rake task that reads from a yaml file and then does stuff with the data. I'd rather not install bundler for this (the solutions for the similar rails issues suggest prepending with bundle exec), since this script is simple and thus shouldn't need it.
Here's the simplified code, (which gets the same error as the code I'm working on):
require 'FileUtils'
require 'yaml'
SOME_FILE = "#{Dir.pwd}/some_file.yaml"
task default: :foo
task :foo do
bar = File.open(SOME_FILE) { |yf| YAML::load( yf ) }
bar.each {|k,v| puts k}
end
And here's the list of errors:
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/FileUtils.rb:93: warning: already initialized constant FileUtils::OPT_TABLE
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/fileutils.rb:93: warning: previous definition of OPT_TABLE was here
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/FileUtils.rb:1272: warning: already initialized constant FileUtils::Entry_::S_IF_DOOR
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/fileutils.rb:1272: warning: previous definition of S_IF_DOOR was here
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/FileUtils.rb:1535: warning: already initialized constant FileUtils::Entry_::DIRECTORY_TERM
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/fileutils.rb:1535: warning: previous definition of DIRECTORY_TERM was here
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/FileUtils.rb:1537: warning: already initialized constant FileUtils::Entry_::SYSCASE
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/fileutils.rb:1537: warning: previous definition of SYSCASE was here
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/FileUtils.rb:1656: warning: already initialized constant FileUtils::LOW_METHODS
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/fileutils.rb:1656: warning: previous definition of LOW_METHODS was here
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/FileUtils.rb:1662: warning: already initialized constant FileUtils::METHODS
/Users/jpalmieri/.rbenv/versions/2.0.0-p353/lib/ruby/2.0.0/fileutils.rb:1662: warning: previous definition of METHODS was here
The script will run fine despite the warnings; the above code would puts the keys as expected, right after the warnings.
This warning shows up when I write require 'FileUtils'. If I write require 'fileutils' (all lower case) warning disappears.
This link may be helpful explaining the behavior. I think in essence ruby thinks FileUtils and fileutils are different modules, therefore imports it twice. Then the redeclaration of constants give warning messages.
Wanted to answer this clearly (two years after it was asked) in case anyone wanders in here.
First, note that require in Ruby does not load a Module, as in the object FileUtils that is in memory. It loads the file "fileutils.rb" from your hard drive. The ".rb" is omitted by convention but you could write require 'fileutils.rb'.
The purpose of require in Ruby is to load a file only once, as opposed to load which will reload the file every time it is used. The way require avoids loading a file multiple times is by recording the filename argument and skipping it if it if that filename is passed again.
When you first require a file, Ruby responds with true to indicate that it was loaded. If you require the same file again it will return false to indicate that it was already loaded:
> require 'fileutils'
=> true
> require 'fileutils'
=> false
Since the filename stored by require is case-sensitive, but the actual file lookup is not, fileutils.rb will still be found if you use caps in the name:
> require 'FileUtils'
=> true
But if something in your Ruby program already loaded that file without caps (in your case "yaml.rb" probably requires "fileutils" as well) you will reload the file and may see warnings:
> require 'fileutils'
=> true
> require 'FileUtils'
/bin/ruby/lib/ruby/2.3.0/FileUtils.rb:96: warning: already initialized constant FileUtils::OPT_TABLE
etc.
By convention Ruby files should be named in lowercase with underscores, e.g. "my_class.rb", so you would always use require 'my_class'.
Things get a little trickier if you are requiring using absolute or relative paths, e.g. require 'special_classes/my_class'. I suggest reading about require_relative and the Ruby load path ($LOAD_PATH).
I solved this similar issue when I list my gem items which named "fileutils" has two versions
fileutils (1.1.0, default: 1.0.2)
then I run
sudo gem uninstall fileutils -v 1.1.0
and solved
I found that these warnings don't appear and the script runs perfectly if I simply comment out or remove line 1 of my original code (require 'FileUtils'). Although I haven't browsed the code for Rake, it must already include FileUtils (which makes sense).
For the sake of completeness, here is my revised code (note that I removed the require 'FileUtils' line:
require 'yaml'
SOME_FILE = "#{Dir.pwd}/some_file.yaml"
task default: :foo
task :foo do
bar = File.open(SOME_FILE) { |yf| YAML::load( yf ) }
bar.each {|k,v| puts k}
end
I was having the same issue with Travis and the problem was that I forgot to use bundle exec rake db:setup instead of rake db:setup. Hope it helps someone :)
Here My Problem: Warbler doesn't support gems which are included via path in the gemfile. nevertheless i need to do that for my work. it is important that the included gem is packed and handled as a simple rubygem in the war archive. until now i have tried to manipulate the bundler so when the spec arrives in the warbler/traits/bundler.rb (that's where the specs are packed to the archive) it already has 'Bundler::Source::Rubygems' as source. the problem is that it sill needs to be build and installed from the path, but i can't handle it to pass the path anywhere in the spec or in the source. it already works that the gem is built, installed and packed into the archive as a rubygem and listed under GEM in the Lockfile but only with bad coding (it is all with reference to a specific gem and the path is typed in clearly)
Here my code:
warbler/lib/warbler/traits/bunlder.rb line: 60
case spec.source
when ::Bundler::Source::Git
config.bundler[:git_specs] ||= []
config.bundler[:git_specs] << spec
when ::Bundler::Source::Path
$stderr.puts("warning: Bundler `path' components are not currently supported.",
"The `#{spec.full_name}' component was not bundled.",
"Your application may fail to boot!")
else
##################################################################### MINE
if spec.name == "charla_common"
path = ::Bundler::GemHelper.new("../../common/trunk", spec.name).install_gem
end
##################################################################### MINE END
config.gems << spec
end
This is where the gem is installed path
Bundler/lib/bundler/dsl.rb line: 120
def source(source, options = {})
############################################################### MINE
if source.class == Bundler::Source::Path
options[:path] = source.options["path"]
source = "https://rubygems.org"
end
############################################################### MINE END
case source
when :gemcutter, :rubygems, :rubyforge then
Bundler.ui.warn "The source :#{source} is deprecated because HTTP " \
"requests are insecure.\nPlease change your source to 'https://" \
"rubygems.org' if possible, or 'http://rubygems.org' if not."
#rubygems_source.add_remote "http://rubygems.org"
return
when String
# ensures that the source in the lockfile is shown only once
unless options[:prepend]
#rubygems_source.add_remiote source
end
return
else
#source = source
if options[:prepend]
#sources = [#source] | #sources
else
#sources = #sources | [#source]
end
yield if block_given?
return #source
end
ensure
#source = nil
end
I'm not sure if this is an option for you, but in order to work around this problem, I create symbolic links in vendor/cache to the path with the included gem.
E.g. vendor/cache/gem_name -> ../../../gem_name.
I am trying to use the main gem for making command line utilities. This was presented in a recent Ruby Rogues podcast.
If I put all the code in one file and require that file, then rspec gives me an error, as the main dsl regards rpsec as a command line invocation of the main utility.
I can break out a method into a new file and have rspec require that file. Suppose you have this program, but want to put the do_something method in a separate file to test with rspec:
require 'main'
def do_something(foo)
puts "foo is #{foo}"
end
Main {
argument('foo'){
required # this is the default
cast :int # value cast to Fixnum
validate{|foo| foo == 42} # raises error in failure case
description 'the foo param' # shown in --help
}
do_something(arguments['foo'].value)
}
What is the convenient way to distribute/deploy a ruby command line program with multiple files? Maybe create a gem?
You are on the right track for testing - basically you want your "logic" in separate files so you can unit test them. You can then use something like Aruba to do an integration test.
With multiple files, your best bet is to distribute it as a RubyGem. There's lots of resources out there, but the gist of it is:
Put your executable in bin
Put your files in lib/YOUR_APP/whatever.rb where "YOUR_APP" is the name of your app. I'd also recommend namespacing your classes with modules named for your app
In your executable, require the files in lib as if lib were in the load path
In your gemspec, make sure to indicate what your bin files are and what your lib files are (if you generate it with bundle gem and are using git, you should be good to go)
This way, your app will have access to the files in lib at runtime, when installed with RubyGems. In development, you will need to either do bundle exec bin/my_app or RUBYLIB=lib bin/my_app. Point is, RubyGems takes care of the load path at runtime, but not at development time.
So here's what I'm attempting to do. I'm building an ember.js application, with a java backend running on GAE.
I'm using handlebars, but I want them divided up into separate files, not just all pasted into the index.html.
Via the ember.js irc I was turned on to rake-pipeline along with minispade
Along with the web filters and a custom handlebars filter I started building the assetfile. I don't know Ruby, or gem files, etc.
So I'm trying to figure out the best way to be able to compile my coffeescript/handlebars files on the fly, minispade them, but keep the individual files accessible while in dev mode so I can debug them. What makes that hard is that the rake pipeline is running on a different port than GAE. So I'm not sure exactly how to handle this. Do I make my index file in GAE point to individual files at the 9292 port (rakep) during development, but in production mode point to the fully concatenated version? I'm not sure.
So I was attempting to do that here: https://gist.github.com/1495740 by having only one section that was triggered by the 'build' flag. Not even sure if that works that way.
I know there's a lot of confusion here. Apologies, like I said I'm not even remotely familiar with the Ruby style of doing things.
Since you're not a Ruby person, here are the most reliable steps for getting a stock OSX environment set up with rake pipeline:
Step 1: Install bundler
# on OSX, using built-in Ruby
$ sudo gem install bundler --pre
Step 2: Create a Gemfile
# inside your app directory
$ bundle init
# will create a file named Gemfile in the root
Step 3: Add rake-pipeline to the Gemfile
# inside the Gemfile
gem "rake-pipeline-web-filters"
Step 4: Install your gems
$ bundle install --binstubs
Step 5: Set up Assetfile
However you were already doing it...
Step 6: Run Rake::Pipeline
# to run the preview server
$ bin/rakep
# to build your assets
$ bin/rakep build
Rake::Pipeline.build is the method that evaluates an Assetfile. You can imagine that your entire Assetfile is wrapped inside a Rake::Pipeline.build {} block; you shouldn't ever need to write one inside an Assetfile.
Some of the filters in the docs are hypothetical, most of those docs were written before there were any filters at all. A CoffeeScript compiler has been recently added, though.
As to your main question, I'm not sure there's a clean way to do it with the current rakep implementation. An Assetfile is just Ruby, though, so it's possible to hack something together that should work. Here's how I would write yours:
require "json"
require "rake-pipeline-web-filters"
require "rake-pipeline-web-filters/helpers"
class HandlebarsFilter < Rake::Pipeline::Filter
def initialize(&block)
block ||= proc { |input| input.sub(/\.handlebars$/, '.js') }
super(&block)
end
def generate_output(inputs, output)
inputs.each do |input|
output.write "return Ember.Handlebars.compile(#{input.read.to_json})"
end
end
end
# process all js, css and html files in app/assets
input "assets"
# processed files should be outputted to public
output "public"
# process all coffee files
match "**/*.coffee" do
# compile all CoffeeScript files. the output file
# for the compilation should be the input name
# with the .coffee extension replaced with .js
coffee_script
# The coffee_script helper is exactly equivalent to:
# filter Rake::Pipeline::Web::Filters::CoffeeScriptCompiler
end
match "**/*.js" do
minispade
if ENV['RAKEP_ENV'] == "production"
concat "application.js"
else
concat
end
end
match "**/*.handlebars" do
filter HandlebarsFilter
minispade
concat "templates.js"
end
The if ENV['RAKEP_ENV'] bit reads an environment variable to decide whether to concatenate your JS to a single file.
So now you can run RAKEP_ENV="production" rakep build for a concatenated build, or just rakep build for a development build.
Is there anyway to get Ruby's require statement to download a file from somewhere like github rather than just the local file system?
Update: Sorry I should have made the question clearer. I want to download a file that contains Ruby module and import it into my script rather than just downloading an image or some other arbitrary file within my script.
In other words something like this
require 'http:\\github.com\myrepo\snippet.rb'
puts 'hi'
By default, this is not possible. Also, it's not a good idea for security reasons.
You have a couple of alternatives. If the file you want to include is a Gem and Git repository, then you can use Bundler to download and package the dependency in your project. Then you'll be able to require the file directly in your source code.
This is the best and safest way to include an external dependency.
If you trust the source and you really know what you are doing, you can download the file using Net::HTTP (or any other HTTP library) and eval the body directly in your Ruby code.
You can package everything in a custom require_remote function.
You could download and eval it
require "open-uri"
alias :require_old :require
def require(path)
return false if $".include?(path)
unless path=~ /\Ahttp:\/\/
return require_old(path)
end
eval(open(path).read)
$"<< path
true
end
Be aware, this code has no error checking for network outages nonexisting files, ... . I also believe it is in general not a good idea to require libraries this way, there are security and reliability problems in this approach. But maybe you have a valid usecase for this.
you can include a remote gem from within Gemfiles then it will download when you run bundle install
After reading this question and answers I wanted something a little more bullet proof and verbose that used a paradigm of creating a local file from a repo and then requiring it, only if it didn't already exist locally already. The request for the repo version is explicit via the method repo_require. Used on files you control, this approach improves security IMO.
# try local load
def local_require(filename, relative_path)
relative_flname = File.join(relative_path, filename)
require_relative(relative_flname)
end
# try loading locally first, try repo version on load error
# caution: only use with files you control access to!
def repo_require(raw_repo_prefix, filename, relative_path = '')
local_require(filename, relative_path)
rescue LoadError => e
puts e.message
require 'open-uri'
tempdir = Dir.mktmpdir("repo_require-")
temp_flname = File.join(tempdir, File.basename(filename))
return false if $LOADED_FEATURES.include?(temp_flname)
remote_flname = File.join(raw_repo_prefix, filename)
puts "file not found locally, checking repo: #{remote_flname}"
begin
File.open(temp_flname, 'w') do |f|
f.write(URI.parse(remote_flname).read)
end
rescue OpenURI::HTTPError => e
raise "Error: Can't load #{filename} from repo: #{e.message} - #{remote_flname}"
end
require(temp_flname)
FileUtils.remove_entry(tempdir)
end
Then you could call repo_require like this:
repo_require('https://raw.githubusercontent.com/username/reponame/branch',
'filename', 'relative_path')
The relative_path would the the relative path you would use for the file if the repo was locally installed. For example, you may have something like require_relative '../lib/utils.rb'. In this example filename='lib/utils.rb' and relative_path='..'. This information allows the repo url to be constructed correctly as it does not use the relative path portion.