Bundler - Resolving Dependencies from different sources - ruby

My company has a private Gem-in-a-box server where multiple teams can share internally created gems. Recently, a gem has been added to this server which I want to use. It turns out, this gem has a dependency on net-ssh and net-scp, which are available from ruby-gems.org, and are not stored on the Gem-in-a-box server. When I add the new gem to my Bundler Gemfile and run an install, I get the following error:
C:\jruby-1.7.18\bin\jruby.exe --1.9 C:\jruby-1.7.18\bin/bundle install
Fetching source index from http://my.server.org/geminabox/
Fetching gem metadata from http://rubygems.org/.
Fetching source index from http://my.server.org/geminabox/
Fetching gem metadata from http://rubygems.org/.
Fetching additional metadata from http://rubygems.org/........
Resolving dependencies...
Could not find gem '["net-ssh", "net-scp"] (>= 0) java', which is required by
gem 'gem_dependent_on_ssh (>= 0) java', in any of the sources.
Process finished with exit code 6
Here is a snippet from my Gemfile:
source 'http://rubygems.org' do
gem 'net-scp'
gem 'net-sftp'
end
source 'http://my.server.org/geminabox/' do
gem 'gem_dependent_on_ssh'
end
It looks like it is only looking for gem dependencies on the same server as the gem being loaded in... Is there something I can add to my Gemfile to get around this? Or, can I go to the team that created the gem and have them add something to tell it where to look for dependencies? Or, is the only solution to add the net-ssh and net-scp gems to the Gem-in-a-box server so it can find them as a local dependency?
Thanks in advance!

It looks like the docs say no (very bottom):
SOURCE PRIORITY
When attempting to locate a gem to satisfy a gem
requirement, bundler uses the following priority order:
The source explicitly attached to the gem (using :source, :path, or
:git)
For implicit gems (dependencies of explicit gems), any source,
git, or path repository declared on the parent. This results in
bundler prioritizing the ActiveSupport gem from the Rails git
repository over ones from rubygems.org
The sources specified via
global source lines, searching each source in your Gemfile from last
added to first added.
The relevant one being #2 here. It's a little vague but I think this other part of the docs helps clarify:
Bundler will search for child dependencies of this gem by first looking in the source selected for the parent, but if they are not found there, it will fall back on global sources using the ordering described in SOURCE PRIORITY.
So it sounds like you need to provide rubygems.org as the fallback global source if you want that to be the case, i.e. using the source 'http://rubygems.org' line on its own as you have done in your answer. Otherwise when looking for child gems (dependencies) it will look only on the source provided for the parent.
That said, I also have a company Gem-in-a-box server and tried to reproduce your issue but could not, so I'm not positive I'm interpreting those docs correctly.

After consulting with a few other sources, it appears that if you remove the blocks and simply declare the sources, it manages to install all gems:
source 'http://rubygems.org'
source 'http://my.server.org/geminabox/'
gem 'net-scp'
gem 'net-sftp'
gem 'gem_dependent_on_ssh'
While this does fix this specific issue, I can still imagine a scenario where you would want to specify the source for a specific gem, and yet use it to resolve a dependency in another source. So while our issue is resolved, the question still stands.

Related

Bundler: using shared gem when it exists, rather than downloading from gem server

Say, I have Gemfile like following.
source "GEM_REPOSITORY"
gem 'gem_A'
# gem_A has no additional dependency
gem 'gem_B'
# gem_B depends on gem_B_1 and gem_B_2
When I run bundle install, I want Bundler to do the following.
If a gem already exists in "local system-wide gems", it copies the gem from local.
If a gem doesn't exist in local, it looks for GEM_REPOSITORY.
I looked for some related articles, and found some of likely-answers like
Ruby Bundler multiple sources in Gemfile
SOURCE PRIORITY
But none of the above looks like the answer for me.
Using source repository priority does't work. Because in the example above, if dependent gem (say, gem_B_1) exits in local but the target gem (gem_B) doesn't exist in local, it'll download both of above from the remote repository.
Are there any work around for doing this?
If not, don't you guys think it's necessary considering the cost of the implementation and the effect?
This is the current behavior.
When running gem install, directly or via bundle install, gem will first build a dependency graph with all the needed gems. If the gem is found locally it will use it, otherwise it will try to download it from the specified source.
If you want, try it yourself.
bundle gem gem_a
bundle gem gem_b
cd gem_a
vim gem_a.gemspec
add
spec.add_dependency 'multi_json', '~> 1.10.1'
or any dependency you want to the gem and run bundle install.
cd ../gem_b
vim Gemfile
and add
gem 'gem_a', path: '../gem_a'
then run
bundle install --verbose
you will see that the multi_json or whatever dependency of gem_a uses the local version and does not download anything.
This is of course also true for gems from remote sources.

Force Bundler to Use my Gem Source

I am running my own gem server with geminabox. This gem server hosts private forks of several public gems, as well as my own gems. I have a Gemfile with the sources listed like so:
source https://rubygems.org
source http://my/gem/server
When Bundler installs a bundle, I would like it to always use the version available on my gem server if it exists and satisfies the version requirements. I can't get this to work. Instead, bundler seems to pull from rubygems unless it can't find a gem with the same name there. This may be because the versions on rubygems are higher. However, bundler's documentation makes no mention of version numbers. This is how it describes source priority:
SOURCE PRIORITY
When attempting to locate a gem to satisfy a gem requirement, bundler uses the following priority order:
The source explicitly attached to the gem (using :path or :git)
For implicit gems (dependencies of explicit gems), any git or path repository otherwise declared. This results in bundler prioritizing the ActiveSupport gem from the Rails git repository over ones from rubygems.org
The sources specified via source, searching each source in your Gemfile from last added to first added.
Also, I know it is possible to explicitly list a source for each gem in the Gemfile. I would prefer to avoid this, since I have many of my own gems, and it is annoying to make explicit entries in the Gemfile for dependencies of dependencies (which you have to do, because bundler will not read the Gemfile of dependencies).
How can I achieve my goal?

Ruby Bundler multiple sources in Gemfile

I need to ensure some of my gems are installed from our own gem repository rather than rubygems, while the rest are installed from rubygems. Can I set this up in the Gemfile without worrying about a naming conflict with a identically named gem in Rubygems? How Can I determine where the gem is downloaded from?
eg
Gemfile:
source :rubygems
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
source "http://our.own.gem.repo.com/the/path/to/it"
gem 'gemfromourrepo'
Bundler 1.7 has a new feature that allows you to select the source for specific gems by nesting them in a block:
source "https://rubygems.org"
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
source "http://our.own.gem.repo.com/the/path/to/it" do
gem 'gemfromourrepo'
end
or specifying it as an option:
source "https://rubygems.org"
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
gem 'gemfromourrepo', source: "http://our.own.gem.repo.com/the/path/to/it"
See http://bundler.io/v1.7/gemfile.html for details.
According to the Source Priority section in the Gemfile manpage sources are searched from last entered to first entered.
Based on what you said, it sounds like you want to always prefer your gem over rubygems.org. As long as you don't need to vary your preference (ie. some dups from rubygems.org and some dups from your private repo) then your problem is solved simply with the following Gemfile:
source 'https://rubygems.org'
source 'http://our.own.gem.repo.com/the/path/to/it'
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
gem 'gemfromourrepo'
The only way I found seems like a horrible hack.
Bundler will search for the best version of your gem starting at the source listed last and then searching all the sources listed previously. It doesn't matter where the source lines are relative to the gem lines, only relative to each other.
I tried to make it work using :git and :path, but neither of those work for gemservers. That leaves matching the best version.
If you set the version of your gem to something like 2.mine.1 and push that to your server, you can constrain the version in your Gemfile.
source :rubygems
source 'http://myrepo'
gem 'gemfromourrepo', '~> 2.ourrepo'
Then the best matching version should be one from your server. There's a chance someone could push their own gem of the same name with 2.ourrepo.2 to rubygems, but that is unlikely if it is unique.
The path command might be able to help. It allows you to set gem specific sources
gem "foo", "1.0", :path => "bar"
Source:http://bundler.io/v1.3/man/gemfile.5.html

Bundler can't find Ruby gem that appears on website

I installed Bundler on a pre-Rails 3 application and am trying to use it to install gems. My Gemfile contains the following lines:
source :rubygems
[...]
gem "RubyInline", "3.8.1"
However, when I run bundle install I get this error:
Fetching source index for http://rubygems.org/
Could not find gem 'RubyInline', required by 'memcache-client (= 1.6.3)', in any of the sources
The gem appears on the rubygems website:
http://rubygems.org/gems/RubyInline
Why is it giving me an error then?
I'm afraid this resolved itself after changes to fix other issues, and I'm not sure what the fix was. The source of several other issues was wrong permissions on various gems/binaries.
This kind of issue for me seems to be resolved on occasion by applying
bundle update
before
bundle install
The effect is to resolve old dependencies from when the bundle was originally produced and hence a gem that has been superseded (or whatever) will no longer be in the Gemfile.

Questions about building a new gem using Wycats template

I'm writing a new gem I'm basing off of Yehuda's new gem template and I'm slightly confused. Having a Gemfile with dependencies, and also specifying a gemspec with dependencies seems redundant to me. Can someone explain why this is desirable and if it's truly necessary?
Note, this is the first gem I've ever written so I'm new to all of this.
The .gemspec dependencies tell rubygems what it needs to grab to resolve the dependencies when a user installs your gem. The Gemfile is to manage the dependencies while you develop the gem. Rubygems and Bundler aren't connected, at least not yet.
The gemspec is required to build the gem. The Gemfile is more of a convenience, so that people who are working on your gem can easily pull in all the dependencies via Bundler. In the case that you're developing multiple related gems at once, you may also want to put git sources in the Gemfile, so all of the HEAD versions can be tested against.

Resources