I am running my own gem server with geminabox. This gem server hosts private forks of several public gems, as well as my own gems. I have a Gemfile with the sources listed like so:
source https://rubygems.org
source http://my/gem/server
When Bundler installs a bundle, I would like it to always use the version available on my gem server if it exists and satisfies the version requirements. I can't get this to work. Instead, bundler seems to pull from rubygems unless it can't find a gem with the same name there. This may be because the versions on rubygems are higher. However, bundler's documentation makes no mention of version numbers. This is how it describes source priority:
SOURCE PRIORITY
When attempting to locate a gem to satisfy a gem requirement, bundler uses the following priority order:
The source explicitly attached to the gem (using :path or :git)
For implicit gems (dependencies of explicit gems), any git or path repository otherwise declared. This results in bundler prioritizing the ActiveSupport gem from the Rails git repository over ones from rubygems.org
The sources specified via source, searching each source in your Gemfile from last added to first added.
Also, I know it is possible to explicitly list a source for each gem in the Gemfile. I would prefer to avoid this, since I have many of my own gems, and it is annoying to make explicit entries in the Gemfile for dependencies of dependencies (which you have to do, because bundler will not read the Gemfile of dependencies).
How can I achieve my goal?
Related
My company has a private Gem-in-a-box server where multiple teams can share internally created gems. Recently, a gem has been added to this server which I want to use. It turns out, this gem has a dependency on net-ssh and net-scp, which are available from ruby-gems.org, and are not stored on the Gem-in-a-box server. When I add the new gem to my Bundler Gemfile and run an install, I get the following error:
C:\jruby-1.7.18\bin\jruby.exe --1.9 C:\jruby-1.7.18\bin/bundle install
Fetching source index from http://my.server.org/geminabox/
Fetching gem metadata from http://rubygems.org/.
Fetching source index from http://my.server.org/geminabox/
Fetching gem metadata from http://rubygems.org/.
Fetching additional metadata from http://rubygems.org/........
Resolving dependencies...
Could not find gem '["net-ssh", "net-scp"] (>= 0) java', which is required by
gem 'gem_dependent_on_ssh (>= 0) java', in any of the sources.
Process finished with exit code 6
Here is a snippet from my Gemfile:
source 'http://rubygems.org' do
gem 'net-scp'
gem 'net-sftp'
end
source 'http://my.server.org/geminabox/' do
gem 'gem_dependent_on_ssh'
end
It looks like it is only looking for gem dependencies on the same server as the gem being loaded in... Is there something I can add to my Gemfile to get around this? Or, can I go to the team that created the gem and have them add something to tell it where to look for dependencies? Or, is the only solution to add the net-ssh and net-scp gems to the Gem-in-a-box server so it can find them as a local dependency?
Thanks in advance!
It looks like the docs say no (very bottom):
SOURCE PRIORITY
When attempting to locate a gem to satisfy a gem
requirement, bundler uses the following priority order:
The source explicitly attached to the gem (using :source, :path, or
:git)
For implicit gems (dependencies of explicit gems), any source,
git, or path repository declared on the parent. This results in
bundler prioritizing the ActiveSupport gem from the Rails git
repository over ones from rubygems.org
The sources specified via
global source lines, searching each source in your Gemfile from last
added to first added.
The relevant one being #2 here. It's a little vague but I think this other part of the docs helps clarify:
Bundler will search for child dependencies of this gem by first looking in the source selected for the parent, but if they are not found there, it will fall back on global sources using the ordering described in SOURCE PRIORITY.
So it sounds like you need to provide rubygems.org as the fallback global source if you want that to be the case, i.e. using the source 'http://rubygems.org' line on its own as you have done in your answer. Otherwise when looking for child gems (dependencies) it will look only on the source provided for the parent.
That said, I also have a company Gem-in-a-box server and tried to reproduce your issue but could not, so I'm not positive I'm interpreting those docs correctly.
After consulting with a few other sources, it appears that if you remove the blocks and simply declare the sources, it manages to install all gems:
source 'http://rubygems.org'
source 'http://my.server.org/geminabox/'
gem 'net-scp'
gem 'net-sftp'
gem 'gem_dependent_on_ssh'
While this does fix this specific issue, I can still imagine a scenario where you would want to specify the source for a specific gem, and yet use it to resolve a dependency in another source. So while our issue is resolved, the question still stands.
Say, I have Gemfile like following.
source "GEM_REPOSITORY"
gem 'gem_A'
# gem_A has no additional dependency
gem 'gem_B'
# gem_B depends on gem_B_1 and gem_B_2
When I run bundle install, I want Bundler to do the following.
If a gem already exists in "local system-wide gems", it copies the gem from local.
If a gem doesn't exist in local, it looks for GEM_REPOSITORY.
I looked for some related articles, and found some of likely-answers like
Ruby Bundler multiple sources in Gemfile
SOURCE PRIORITY
But none of the above looks like the answer for me.
Using source repository priority does't work. Because in the example above, if dependent gem (say, gem_B_1) exits in local but the target gem (gem_B) doesn't exist in local, it'll download both of above from the remote repository.
Are there any work around for doing this?
If not, don't you guys think it's necessary considering the cost of the implementation and the effect?
This is the current behavior.
When running gem install, directly or via bundle install, gem will first build a dependency graph with all the needed gems. If the gem is found locally it will use it, otherwise it will try to download it from the specified source.
If you want, try it yourself.
bundle gem gem_a
bundle gem gem_b
cd gem_a
vim gem_a.gemspec
add
spec.add_dependency 'multi_json', '~> 1.10.1'
or any dependency you want to the gem and run bundle install.
cd ../gem_b
vim Gemfile
and add
gem 'gem_a', path: '../gem_a'
then run
bundle install --verbose
you will see that the multi_json or whatever dependency of gem_a uses the local version and does not download anything.
This is of course also true for gems from remote sources.
I'm working on a command line tool (a ruby gem) to quickly and easily generate custom jekyll repositories from some basic user input for my company. I would like to add a feature where the gem can automatically build the repo and output the finished directory, but the jekyll gem doesn't seem to provide any internal ruby interface to do that, at least not one that would be easy to implement.
Is there a way I could have my gem call their gem as a program, i.e. system 'jekyll build', but without having to worry about whether or not the user has the jekyll gem installed?
You can add jekyll to your gems list of dependencies, by adding the following to your gem's .gemspec file. That means whenever someone installs your gem, jekyll would also be installed.
spec.add_dependency 'jekyll'
I am working on a new Ruby Gem. I am familiar with using Bundler to manage gems:
source "https://rubygems.org"
gemspec
gem 'rspec-rails'
I am familiar with specifying dependencies in a gemspec file:
Gem::Specification.new do |s|
# ...
s.add_dependency "rails", "~> 4.1.5"
end
The Gemfile that was generated mentions that I should move my dependency declarations from my Gemfile to my gemspec when I'm ready to release.
# Declare any dependencies that are still in development here instead of in
# your gemspec. These might include edge Rails or gems from your path or
# Git. Remember to move these dependencies to your gemspec before releasing
# your gem to rubygems.org.
Why would I want to do this? Why should my gemspec care what gems I'm using in development? What purpose does development_dependency serve that Bundler doesn't already do for me?
To best answer your questions, we should first untangle the concepts of Bundler and Rubygems. I think a great explanation can be found here.
Why Would I Want to [move dependencies from Gemfile to .gemspec]?
Gemfile allows you to specify not only dependencies, but where the dependencies come from. This is useful when you are also working on the dependencies themselves and need to point to a Git repo (or something).
Once work on these dependencies is complete, Rubygem conventions dictate that you move the dependency declaration on these released gems into your .gemspec file. Adding a line gemspec tells Bundler to read from this conventional Rubygems location. If you are working on a gem, and you are not actively developing the gem's dependencies, then all dependencies should be declared in your .gemspec
Why should my gemspec care what gems I'm using in development?
From the docs for add_development_dependency:
Development dependencies aren't installed by default and aren't activated when a gem is required.
A popular example for this Rspec. You should generally declare Rspec as a development dependency for yourself, but not force everyone else to download it when they grab your gem.
Note that the comment you cited does not say "development dependencies", it says "dependencies that are in development". Dependencies that are in development commonly include cutting-edge versions of gems that you are installing directly from a Git repo. RubyGems cannot install gems from a Git repo; however, Bundler can. If you are installing cutting-edge versions of gems installed from a source that RubyGems cannot handle (such as a VCS repo), you should list them in your Gemfile instead of the .gemspec file.
I need to ensure some of my gems are installed from our own gem repository rather than rubygems, while the rest are installed from rubygems. Can I set this up in the Gemfile without worrying about a naming conflict with a identically named gem in Rubygems? How Can I determine where the gem is downloaded from?
eg
Gemfile:
source :rubygems
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
source "http://our.own.gem.repo.com/the/path/to/it"
gem 'gemfromourrepo'
Bundler 1.7 has a new feature that allows you to select the source for specific gems by nesting them in a block:
source "https://rubygems.org"
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
source "http://our.own.gem.repo.com/the/path/to/it" do
gem 'gemfromourrepo'
end
or specifying it as an option:
source "https://rubygems.org"
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
gem 'gemfromourrepo', source: "http://our.own.gem.repo.com/the/path/to/it"
See http://bundler.io/v1.7/gemfile.html for details.
According to the Source Priority section in the Gemfile manpage sources are searched from last entered to first entered.
Based on what you said, it sounds like you want to always prefer your gem over rubygems.org. As long as you don't need to vary your preference (ie. some dups from rubygems.org and some dups from your private repo) then your problem is solved simply with the following Gemfile:
source 'https://rubygems.org'
source 'http://our.own.gem.repo.com/the/path/to/it'
gem 'gemfromrubygems1'
gem 'gemfromrubygems2'
gem 'gemfromourrepo'
The only way I found seems like a horrible hack.
Bundler will search for the best version of your gem starting at the source listed last and then searching all the sources listed previously. It doesn't matter where the source lines are relative to the gem lines, only relative to each other.
I tried to make it work using :git and :path, but neither of those work for gemservers. That leaves matching the best version.
If you set the version of your gem to something like 2.mine.1 and push that to your server, you can constrain the version in your Gemfile.
source :rubygems
source 'http://myrepo'
gem 'gemfromourrepo', '~> 2.ourrepo'
Then the best matching version should be one from your server. There's a chance someone could push their own gem of the same name with 2.ourrepo.2 to rubygems, but that is unlikely if it is unique.
The path command might be able to help. It allows you to set gem specific sources
gem "foo", "1.0", :path => "bar"
Source:http://bundler.io/v1.3/man/gemfile.5.html