I'm developing a gem, and I'd like to simulate how it will be actually used in the wild. I would like to automate testing it with any satisfactory version of its dependencies.
For example, if my gemspec claims spec.add_dependency "nokogiri", '~> 1.2' then I'd want to automate installing anywhere from 1.2.0 to the latest 1.x. Ideally it would do the same for nokogiri's dependencies.
If you are actually looking for a way to list every possible dependency of a Gem and every possible dependency of those dependencies, and so on.
The following should technically work
def gem_list(gem)
gem.runtime_dependencies.each_with_object({}) do |dep,obj|
# considerations for platform may be required
# check the output from the gem list command
obj[dep.name] = `gem list #{dep.name} -r -e -a`.scan(/\d+\.\d+.\d+/)
.filter_map do |v|
if dep.requirement.satisfied_by?(Gem::Version.create(v))
# load spec from the yaml output of the gem specification command
spec = Gem::Specification.from_yaml(`gem specification #{dep.name} -r -v #{v}`)
# recursively load the spec dependency chains for each dependent gem
{ v=> gem_list(spec)}
end
end
end
end
spec = Gem::Specification.load(PATH_TO_YOUR_GEMSPEC)
dependency_trees = gem_list(spec)
Example using nokogiri.gemspec (which is a fairly short list)
spec.add_runtime_dependency("mini_portile2", "~> 2.8.0")
spec.add_runtime_dependency("racc", "~> 1.4")
Output
{"mini_portile2"=>[{"2.8.0"=>{}}],
"racc"=>
[{"1.6.1"=>{}},
{"1.6.0"=>{}},
{"1.5.2"=>{}},
{"1.5.1"=>{}},
{"1.5.0"=>{}},
{"1.4.16"=>{}},
{"1.4.15"=>{}},
{"1.4.14"=>{}},
{"1.4.13"=>{}},
{"1.4.12"=>{}},
{"1.4.11"=>{}},
{"1.4.10"=>{}},
{"1.4.9"=>{}},
{"1.4.8"=>{}},
{"1.4.7"=>{}},
{"1.4.6"=>{}}]}
Please note this could go on for a very long time depending on the number of gems that satisfy the requirement and the number of gems that are in the dependency chain.
This will not actually install the gems. So my "recommended" process would be:
Build this list
Then traverse the deeply nested stacks and flatten each into a dependency chain
Install a single dependency chain
Test
Clear all gems
Repeat 3-5 until no chains are left
Related
I am using rbenv and bundler on macos.
I need to maintain multiple versions of Ruby for my application. Specifically 2.2.4 and 2.5.5. Different users will have different versions.
Currently I am only working with 2.2.4 so I do the following. My gemfile looks like this
source "https://rubygems.org"
ruby '2.2.4'
gem "net-ssh", "4.1.0"
gem "net-scp", "2.0.0"
.....
To install I run bundle install and then to deploy with my application I run
bundle install --deployment --path src/mct-tools/ext/gems
This generates a folder src/mct-tools/ext/gems/ruby/2.2.0 in my application that I then distribute.
Now I would like to add ruby 2.5.5 with updated gems so I was wondering whether it would be acceptable to have a gemfile looking like this one
source "https://rubygems.org"
ruby '2.2.4'
gem "net-ssh", "4.1.0"
gem "net-scp", "2.0.0"
.....
ruby '2.5.5'
gem "net-ssh", "6.1.0"
gem "net-scp", "3.0.0"
.....
so that when I deploy with bundle install --deployment --path src/mct-tools/ext/gems the two versions are added to the gems folder.
Is this the correct approach? An alternative would be to specify two different gemfiles for each version and then switch between them.
Gemfiles Declare Dependencies
A Gemfile declares a dependency on a Ruby version, with or without semantic versioning constraints. It is not meant to control multiple build targets for your application. It simply enforces that the Ruby version available to your app/gem is whatever you've defined. For example:
# Will fail if run with a different RUBY_VERSION.
ruby '2.2.4'
# Allows RUBY_VERSION >= 2.2.4, but <= 2.3.
ruby '~> 2.2.4'
# Allows either Ruby 2.2.4+ or 2.5.5+, with
# a minimum (but no maximum) patch version.
ruby '~> 2.2.4', '~> 2.5.5'
However, it won't install a given Ruby, nor do anything other than raise an error and a non-zero exit status when running bundler install. You need to take a different approach to test multiple targets.
Changing Build Targets with Continuous Integration (CI) Tools
If you're using an external CI like TravisCI, you can create a build matrix that targets multiple Ruby versions to test against. Whether you remove the Ruby version constraint altogether, or specify a supported range, is up to you. Leveraging your CI tool to build against the versions of Ruby you plan to support is really the best approach, though, whether or not you constrain your Ruby runtime in a Gemfile.
For example, you might use a matrix in your travis.yml like so:
language: ruby
rvm:
- 2.2.4
- 2.5.5
Switching Gemfiles
If you insist on doing it the way you're doing it, with a singular Ruby version allowed in your Gemfile, then you might consider having two separate gemfiles with different names in your source tree, such as Gemfile-2.2.4 and Gemfile-2.5.5. You can then specify which Gemfile to use with Bundler's --gemfile flag , or by symlinking a custom Gemfile to the canonical Gemfile for your project.
Here are some examples to consider:
# Resolve against a specific Gemfile with
# hard-coded Ruby version.
$ ls Gemfile*
Gemfile-2.2.4 Gemfile-2.5.5
$ bundle install --gemfile="Gemfile-2.2.4"
# Resolve against whatever custom file is
# symlinked to your ./Gemfile.
$ ln -sf Gemfile{-2.5.5,}
$ ls -F Gemfile*
Gemfile# Gemfile-2.2.4 Gemfile-2.5.5
$ bundle install
Both approaches work, but the former is more flexible at the cost of needing to specify your chosen Gemfile each time, while the latter can help when you have a development/testing workflow that doesn't support Bundler's --gemfile flag.
Changing Rubies with Ruby Managers
If you have multiple Ruby versions in development, best practice is to use a version manager such as rvm, rbenv, or chruby. You can use your version manager to change rubies back and forth manually as needed.
You might also check whether your version manager supports auto-switching on .ruby-version or other configuration files. You'd still have to update that file each time you want to build or test against a different Ruby, but you wouldn't have to keep changing your Gemfile contents, re-pointing the Gemfile symlink, or updating a flag on each call to Bundler.
Whether or not any given approach is better than others will depend on your workflow. No technical solution will fit all circumstances, so your mileage may legitimately vary.
I'm building a gem that provides a binary. This binary uses the gem's library.
I would like to make it possible to extend the features of the gem by installing other gems.
Let's say the gem is called xyz. I would like to be able to create the xyz-nice_feature that adds a nice feature.
When using a project like Rails it's ok to add dependencies in a Gemfile but in my case it's for end-users so no Gemfile.
What would be your way of doing this ?
And another question would be how to handle different versions of the extensions.
For the moment I have the following code to load the gems, but that's it:
Gem::Specification.find_all { |s| s.name =~ /xyz-.*/ }.map(&:name).uniq.each do |gem_name|
gem gem_name, '>= 0'
end
Thank you :)
I am working on a gem that has needs to set dependencies conditionally when the gem is installed. I've done some digging around
and it looks like i'm not alone in this need.
Rubygems: How do I add platform-specific dependency?
this is a long thread
http://www.ruby-forum.com/topic/957999
The only way I can see to add dependencies to a gem is to use add_dependency method within a Gem::Specifiction block in a .gemspec file
Gem::Specification.new do |s|
# ... standard setup stuff
# conditionally set dependencies
s.add_dependency "rb-inotify", "~> 0.8.8" if RUBY_PLATFORM =~ /linux/i
s.add_dependency "rb-fsevent", "~> 0.4.3.1" if RUBY_PLATFORM =~ /darwin/i
s.add_dependency "rb-fchange", "~> 0.0.5" if RUBY_PLATFORM =~ /mswin|mingw/i
end
Based on all of the docs and threads I found on the net, I would have expected that if you install the gem on
Linux, then, rb-inotify would be a dependency and auto-installed
Mac - rb-fsevent would be installed
Windows - rb-fchange would be installed
However, it seems that is not the case. The "if" statements within the block are evaluated at the time the gem is built and packaged. Therefore,
if you build and package the gem on Linux, then, rb-inotify is added as a dependency, Mac, then, rb-fsevent, Windows - rb-fchange.
Still needing a solution, I dug around in the rubygems code and it seems the following is a broad stoke of what happens.
build all of your code for your gem: foo.gem
create a foo.gemspec file
build, package, and release the gem to a gem server such as rubygems.org
let everyone know
developers install it locally via: gem install foo
the foo.gem file is downloaded, unpacked, and installed. all dependencies are installed as well.
everything should be set and we can beging using the gem.
It seems that when the gem is built and released the foo.gemspec file is loaded and the Gem::Specification block is evaluated and converted to YAML, compressed as
metadata.gz, and included in foo.gem. The ruby code is compressed into data.tar.gz and included as well. When the gem is installed on the local developer machine,
the YAML is extracted from metadata.gz and converted back into a Gem::Specification block, however, it is not converted back to the original block.
instead, you will see something like the following:
Gem::Specification.new do |s|
if s.respond_to? :specification_version then
s.specification_version = 3
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
s.add_runtime_dependency(%q<rb-inotify>, ["~> 0.8.8"])
else
s.add_dependency(%q<rb-inotify>, ["~> 0.8.8"])
end
else
s.add_dependency(%q<rb-inotify>, ["~> 0.8.8"])
end
end
Ok. So, I have a bird's eye view of the process, however, that does not change my desire to build a single gem and conditionally specify dependencies for a range of OS targets.
If anyone has a solution other than building multiple .gemspec files for each target OS... I'm all ears!!
I have also stumbled upon this problem in the past. The only workaround I could find was to create a Rake task for installing the dependencies. Of course, at that stage, you might just want to let the user figure out on his own which gem he is missing based on the error message he is receiving. In my case, there were several platform-dependent dependencies to be installed, so that wasn't an option.
Rakefile:
task :install do |t|
require './lib/library/installer'
Library::Installer.install
end
Installer:
module Library::Installer
require 'rubygems/dependency_installer'
def self.install
installer = Gem::DependencyInstaller.new
dependency = case RUBY_PLATFORM
when /darwin/i then ["rb-fsevent", "~> 0.4.3.1"]
when /linux/i then ["rb-inotify", "~> 0.8.8"]
when /mswin|mingw/i then ["rb-fchange", "~> 0.0.5"]
end
installer.install(*dependency)
end
Then, the user can use rake install to get install appropriate dependencies.
Conditional dependency install (not just based on platform, but based on user input, for example) is cruelly missing to RubyGems. Let's hope it'll get implemented in the future!
i have never done this myself, but there are some gems that are available in platform specific versions: http://rubygems.org/gems/libv8/versions
from what i understand it's just a naming thing, which can be configured by setting the platform option of your gemspec. have a look at the doc: http://guides.rubygems.org/specification-reference/#platform=
I have looked into this as well and have come to the conclusion that is not possible by design. Having a single 'mega gem' for all platforms causes the problem of not knowing if a platform is supported until the gem is downloaded and installed. A Gem would have to be smart enough to determine what is correct way to install depending on the platform. If a platform is not supported at all, the gem may fail horribly, opening a big can of worms. There use to be a callback after a gem was installed that was removed for the same reason, no magic to get a gem to install correctly. Some people have hacked around this using mkmf, but I suggest following the worn path of a gem per platform as the better solution.
Based on this, in a project that builds a gem for ruby and jruby, I have to manually create each gem and upload them to RubyGem. Using Jeweler this is as simple as specifing the Gemfile, but I have to rebuild the gem spec each time I package a gem. Fairly trivial when supporting only 2 platforms, but the build process is straight forward enough that it could be automated to provide support multiple platform gems.
I am building a framework that loads user provided ruby code. It is basically a plugin mechanism. I want the user provide ruby code to be able to require gems of its own. I intend to have the "plugin" package include a vendor directory with the gems.
How can I load gems that are required by the plugin without having them conflict with my framework's gems? For example, if my framework uses treetop version 1.3.0, and a plugin uses treetop 1.4.2 I want each to work with their specified version.
Likewise, is there a way to prevent plugins from conflicting with each other?
I have looked at gem_plugin, _why's sandbox, and some other tools. But I don't see any library that specifically handles this case - I assume its been done before.
I have also looked at the internals of Bundler to see how it manages gem versions. I am prepared to do some pretty complex stuff if need be. But I am still uncertain of how to go about it.
I also have a lot of freedom in how I implement this. So if you think I am barking up the wrong tree, please say so.
Thanks for any advice.
SIDE NOTE: It occurred to me while writing this that I need something similar to the Classloaders in a Java servlet container. A WAR file can include jar files, and the web application's class loader will prefer those over the jars that are on the global classpath. Is there any way in ruby to segment the ruby "classpath" (i.e. load_path, require, etc)?
To be blunt, you can't have two versions of the same gem loaded at the same time.
Bundler does a good (ish) job of looking through all of the required gems and finding a solution to the various overlapping dependencies, but even so it is limited to only loading one version of a gem at a time.
This leads to plugin developers constantly having to update to support any changes that are made in dependent gems in order to avoid just the situation you describe.
(Don't get me started on the screw up that results from the various competing JSON implementations and the pain you have to go through when you have several gem dependencies all requiring different ones.)
Respectfully disagree with the answer above. Here is how I do it:
ruby -S gem list my_gem
`*** LOCAL GEMS ***
my_gem (1.0.1, 1.0.0, 0.0.2)
`
ruby -S gem lock my_gem-1.0.0 > locklist.rb
which generates list of dependencies for a specific version into locklist
require 'rubygems'
gem 'my_gem', '= 1.0.0'
gem 'gem_base', '= 1.0.0'
gem 'rest-client', '= 1.7.2'
gem 'savon', '= 1.1.0'
gem 'addressable', '= 2.3.6'
gem 'mime-types', '= 1.25.1'
gem 'netrc', '= 0.11.0'
now you can do load('locklist.rb') which will load a specific version of a gem along with its dependencies. Look ma, no Bundler.
I have a Rakefile that includes this:
Jeweler::Tasks.new do |gem|
# ...
gem.add_dependency('json')
end
The gemspec that this generates builds a gem that can't be installed on jruby because the 'json' gem is native.
For jruby, this would have to be:
Jeweler::Tasks.new do |gem|
# ...
gem.add_dependency('json-jruby')
end
How do I conditionally add the dependency for 'json-jruby' when RUBY_ENGINE == 'java'?
It seems like my only option is to manually edit the gemspec file that jeweler generates to add the RUBY_ENGINE check. But I'd rather avoid this, since it kind of defeats the purpose of using jeweler in the first place.
Any ideas?
After some more digging around, I've come to the conclusion that the problem here isn't so much the limitation in Jeweler as it is a problem with how the json-jruby gem is named.
In a perfect world, the json-jruby gem should be named just json. That way when the gem file is built under JRuby (i.e. jgem build json-jruby.gem), the resulting gem file would be named json-1.4.3-universal-java-1.6.gem (RubyGems names the file based on the 'platform' property in the gemspec). When installed with the appropriate platform in the gem name, everything works transparently. There's no need to explicitly require json-jruby. RubyGems uses the appropriate version automatically.
I've contacted the maintainer of the json gem to see if he'd be willing to get json-1.4.3-universal-java-1.6.gem added to his gemcutter repo, but in the meantime you can download the drop-in jruby json gem here: http://roughest.net/json-1.4.3-universal-java-1.6.gem