I'm running a project that uses pip and a requirements.txt file to install and keep track of some dependencies. I want to write some sh scripts to run, build and test the application. For starters I would like a way to check if the current folder is in a pyenv and, if not, create one to enclose the application and not mess around other people's dependencies. Also, I would like an opinion of the best way to keep track of this kind of dependencies, if the requirements.txt is a good approach and if there's a way to keep track of installed versions just like happens with node packages.
Use Pipenv. It's a better way of tracking your depencies than requirements.txt and it uses Pyenv to automatically install your project's required Python version.
From the website:
The problems that Pipenv seeks to solve are multi-faceted:
You no longer need to use pip and virtualenv separately. They work together.
Managing a requirements.txt file can be problematic, so Pipenv uses Pipfile and Pipfile.lock to separate abstract dependency
declarations from the last tested combination.
Hashes are used everywhere, always. Security. Automatically expose security vulnerabilities.
Strongly encourage the use of the latest versions of dependencies to minimize security risks arising from outdated components.
Give you insight into your dependency graph (e.g. $ pipenv graph).
Streamline development workflow by loading .env files.
[...]
Pipenv Features
Enables truly deterministic builds, while easily specifying only what you want.
Generates and checks file hashes for locked dependencies.
Automatically install required Pythons, if pyenv is available.
Automatically finds your project home, recursively, by looking for a Pipfile.
Automatically generates a Pipfile, if one doesn’t exist.
Automatically creates a virtualenv in a standard location.
Automatically adds/removes packages to a Pipfile when they are installed or uninstalled.
Automatically loads .env files, if they exist.
Background: Not the Question
I have a project that is running in both development and production in a chruby environment, where ruby-install was installed as root and rubies are stored in /opt/rubies. I have a really hard time (as many people do) getting nokogiri to compile its native extensions as part of a bundle, but it compiles fine as any chruby user outside of the project directory and so long as there are no binstubs. That's just background context; it's not really what my question is about.
My Current Hack
I find that when I have committed binstubs to my project, all kinds of badness happens. So far, my best solution seems to be:
# skip any bundled/binstub version of bundler
rm -rf ./bin
`which bundle` install --binstubs --without="development test"
which seems to work just fine. Nokogiri is apparently not a production dependency (yay!) and the ./bin directory gets trimmed down to just the gems needed for production.
Question: Would the Bundler Deployment Flag Fix Anything?
So, here's the question: What's the difference between what I'm doing here, and bundle install --deploy with or without binstubs? I know it points to vendor/bundle instead of bin by default, but the docs don't really explain the pros and cons of this approach (at least not in a way that I'm understanding).
Aside from knowledge, which is valuable for its own sake, I'd really like to vendor in production gems in a way that it's likely to work across systems (e.g. RHEL6 and RHEL7) without having to rebuild native extensions or strip out development/test gems on production machines.
Is the deployment flag the answer? Or is there a better way to vendor gems with native extensions for cross-distro projects?
I'm writing a gem which needs a directory to store some config. I want to do this upon installation but I'm not sure how best to insert code into the installation process.
By adding the code to the Rakefile and adding spec.extensions = ["Rakefile"] to my gemspec, I've achieved what I want to do but this method means the installation process tells me I'm building native extensions; this is misleading.
Is there a built in way of doing this or via bundler? I'm assuming Bundler::GemHelper.install_tasks is close to what I'm after but I can't find helpful documentation.
EDIT: My current Rakefile.
It seems like the RubyGems designers intentionally left the option to run a script during installation... so I would say that the accepted way is to do the directory creation in the application.
The unix way is for the application to create the directory (vi and git do this).
FileUtils.mkdir_p is handy... it will save you from having to check if the directory is already there.
I've just updated RVM, and in place of the old .rvmrc, it auto-created .ruby-gemset and .ruby-version.
I've always had .rvmrc files with contents like rvm use --create default#project_name. However, .ruby-version contains the specific Ruby version I'm running rather than default. I'm hesitant to check this in.
Also, I heard someone say on a podcast that one shouldn't check in .ruby-gemset because others may have their own preferences about how to name gemsets.
When should or shouldn't I check in .ruby-gemset and/or .ruby-version?
Specifically:
What are some of the tradeoffs?
How does the type of project affect the decision (for example, applications vs gems)?
If they should be checked in, how does the type of project affect what should go in these files?
Citations from from the creators of tools like rvm, rbenv, etc would be appreciated in an answer.
For standard projects
Check in .ruby-version if your project depends on a ruby like ruby-2.0.0.
Check in .ruby-gemset only if your team agreed on it.
Add .rvmrc to .gitignore so anyone can override .ruby-* files settings with .rvmrc.
For gems
Check in .ruby-version with ruby-1.8.7 only if your project still targets ruby 1.8.7, otherwise check it in only if your gem requires it.
Do not check in .ruby-gemset.
Checking in .rvmrc, .ruby-version or .ruby-gemset?
FOR:
Your project has different branches (say a RubyGems project supporting Ruby 1.8, 1.9 and 2.0 versions). Its better to check in this file, so that your developers don't have to keep on editing these files when they switch branches. The same doesn't apply for an application though, where you'll mostly be working on only one Ruby version.
Same case as above, but say you are running a CI server (say TeamCity/Jenkins/...) which automatically just runs rake spec for every check-in. You don't want to create separate build pipelines for each branch, just for the sake of having a separate rvm use ... for each branch. You just want the Ruby version selected automatically depending on the branch
You have tight control over the environment and all the developers. You either don't need or dictate that they use the same ruby and gemset
You are using Phusion Passenger or Capistrano, which automatically read .rvmrc files and chooses the right ruby for deployment/hosting
Also refer RVM Best Practices
AGAINST:
You can compile your own Ruby in RVM, with some experimental patches, and give it a custom name.
e.g. rvm install 1.9.3 --patch railsexpress,falcon --name ruby-1.9.3-perf
In the above example, I've installed Ruby 1.9.3 with some great speed up patches (btw they are awesome), but rather than calling it 1.9.3, I'm calling it my own name. I would say rvm use ruby-1.9.3-perf whenever I need this. In this case, if the project has its own .ruby-version, then it messes up my environment. In my project, these patches are standard and we actively recommend those. But how developers name the resultant compiled Ruby is up to them
Similarly, people use different gemsets. Some don't use gemsets at all. Some share the same gemset with different (but similar) ruby projects. Given this, again a single .ruby-gemset also doesn't work for everybody
Your project has an obscure ruby version which just says 1.9.3. Your developers first installed the latest Ruby 1.9.3-p329. But they later just update RVM/Rbenv (since they're working on other projects). Their .rvmrc or .ruby-version just breaks, since the latest version of Ruby registered in RVM/Rbenv just changed from ruby-1.9.3-p329 to .ruby-1.9.3-p362, and it will say ruby-1.9.3-p362 not installed. This scenario tends to happen often.
As long as you specify a proper full name for your Ruby version (including patch level), you should be OK. Let's say your project's .ruby-version says ruby-1.9.3-p329. Its easy to compile your own Ruby with all these patches, and still just custom name it ruby-1.9.3-329 just so that the config files will pick up this ruby instead of the standard ruby.
I would include .ruby-version - you and anyone else working on the project, along with your servers, should be using the same version of Ruby.
.ruby-gemset.... up to you, I think.
I'm sort of new to bundler and the files it generates. I have a copy of a git repo from GitHub that is being contributed to by many people so I was surprised to find that bundler created a file that didn't exist in the repo and wasn't in the .gitignore list.
Since I have forked it, I know adding it to the repo won't break anything for the main repo, but if I do a pull request, will it cause a problem?
Should Gemfile.lock be included in the repository?
Update for 2022 from TrinitronX
Fast-forward to 2021 and now Bundler docs [web archive] now say to commit the Gemfile.lock inside a gem... ¯_(ツ)_/¯ I guess it makes sense for developers and ease of use when starting on a project. However, now CI jobs need to be sure to remove any stray Gemfile.lock files to test against other versions.
Legacy answer ~2010
Assuming you're not writing a rubygem, Gemfile.lock should be in your repository. It's used as a snapshot of all your required gems and their dependencies. This way bundler doesn't have to recalculate all the gem dependencies each time you deploy, etc.
From cowboycoded's comment below:
If you are working on a gem, then DO NOT check in your Gemfile.lock. If you are working on a Rails app, then DO check in your Gemfile.lock.
Here's a nice article explaining what the lock file is.
The real problem happens when you are working on an open-source Rails app that needs to have a configurable database adapter. I'm developing the Rails 3 branch of Fat Free CRM.
My preference is postgres, but we want the default database to be mysql2.
In this case, Gemfile.lock still needs be checked in with the default set of gems, but I need to ignore changes that I have made to it on my machine. To accomplish this, I run:
git update-index --assume-unchanged Gemfile.lock
and to reverse:
git update-index --no-assume-unchanged Gemfile.lock
It is also useful to include something like the following code in your Gemfile. This loads the appropriate database adapter gem, based on your database.yml.
# Loads the database adapter gem based on config/database.yml (Default: mysql2)
# -----------------------------------------------------------------------------
db_gems = {"mysql2" => ["mysql2", ">= 0.2.6"],
"postgresql" => ["pg", ">= 0.9.0"],
"sqlite3" => ["sqlite3"]}
adapter = if File.exists?(db_config = File.join(File.dirname(__FILE__),"config","database.yml"))
db = YAML.load_file(db_config)
# Fetch the first configured adapter from config/database.yml
(db["production"] || db["development"] || db["test"])["adapter"]
else
"mysql2"
end
gem *db_gems[adapter]
# -----------------------------------------------------------------------------
I can't say if this is an established best practice or not, but it works well for me.
My workmates and I have different Gemfile.lock, because we use different platforms, windows and mac, and our server is linux.
We decide to remove Gemfile.lock in repo and create Gemfile.lock.server in git repo, just like database.yml. Then before deploy it on server, we copy Gemfile.lock.server to Gemfile.lock on server using cap deploy hook
Agreeing with r-dub, keep it in source control, but to me, the real benefit is this:
collaboration in identical environments (disregarding the windohs and linux/mac stuff). Before Gemfile.lock, the next dude to install the project might see all kinds of confusing errors, blaming himself, but he was just that lucky guy getting the next version of super gem, breaking existing dependencies.
Worse, this happened on the servers, getting untested version unless being disciplined and install exact version. Gemfile.lock makes this explicit, and it will explicitly tell you that your versions are different.
Note: remember to group stuff, as :development and :test
Simple answer in the year 2021:
Gemfile.lock should be in the version control also for Rubygems. The accepted answer is now 11 years old.
Some reasoning here (cherry-picked from comments):
#josevalim https://github.com/heartcombo/devise/pull/3147#issuecomment-52193788
The Gemfile.lock should stay in the repository because contributors and developers should be able to fork the project and run it using versions that are guaranteed to work.
#rafaelfranca https://github.com/rails/rails/pull/18951#issuecomment-74888396
I don't think it is a good idea to ignore the lock file even for plugins.
This mean that a "git clone; bundle; rake test" sequence is not guarantee to be passing because one of yours dozens of dependencies were upgraded and made your code break. Also, as #chancancode said, it make a lot harder to bisect.
Also Rails has Gemfile.lock in git:
https://github.com/rails/rails/commit/0ad6d27643057f2eccfe8351409a75a6d1bbb9d0
The Bundler docs address this question as well:
ORIGINAL: http://gembundler.com/v1.3/rationale.html
EDIT: http://web.archive.org/web/20160309170442/http://bundler.io/v1.3/rationale.html
See the section called "Checking Your Code into Version Control":
After developing your application for a while, check in the
application together with the Gemfile and Gemfile.lock snapshot. Now,
your repository has a record of the exact versions of all of the gems
that you used the last time you know for sure that the application
worked. Keep in mind that while your Gemfile lists only three gems
(with varying degrees of version strictness), your application depends
on dozens of gems, once you take into consideration all of the
implicit requirements of the gems you depend on.
This is important: the Gemfile.lock makes your application a single
package of both your own code and the third-party code it ran the last
time you know for sure that everything worked. Specifying exact
versions of the third-party code you depend on in your Gemfile would
not provide the same guarantee, because gems usually declare a range
of versions for their dependencies.
The next time you run bundle install on the same machine, bundler will
see that it already has all of the dependencies you need, and skip the
installation process.
Do not check in the .bundle directory, or any of the files inside it.
Those files are specific to each particular machine, and are used to
persist installation options between runs of the bundle install
command.
If you have run bundle pack, the gems (although not the git gems)
required by your bundle will be downloaded into vendor/cache. Bundler
can run without connecting to the internet (or the RubyGems server) if
all the gems you need are present in that folder and checked in to
your source control. This is an optional step, and not recommended,
due to the increase in size of your source control repository.
No Gemfile.lock means:
new contributors cannot run tests because weird things fail, so they won't contribute or get failing PRs ... bad first experience.
you cannot go back to a x year old project and fix a bug without having to update/rewrite the project if you lost your local Gemfile.lock
-> Always check in Gemfile.lock, make travis delete it if you want to be extra thorough https://grosser.it/2015/08/14/check-in-your-gemfile-lock/
A little late to the party, but answers still took me time and foreign reads to understand this problem. So I want to summarize what I have find out about the Gemfile.lock.
When you are building a Rails App, you are using certain versions of gems in your local machine. If you want to avoid errors in the production mode and other branches, you have to use that one Gemfile.lock file everywhere and tell bundler to bundle for rebuilding gems every time it changes.
If Gemfile.lock has changed on your production machine and Git doesn't let you git pull, you should write git reset --hard to avoid that file change and write git pull again.
The other answers here are correct: Yes, your Ruby app (not your Ruby gem) should include Gemfile.lock in the repo. To expand on why it should do this, read on:
I was under the mistaken notion that each env (development, test, staging, prod...) each did a bundle install to build their own Gemfile.lock. My assumption was based on the fact that Gemfile.lock does not contain any grouping data, such as :test, :prod, etc. This assumption was wrong, as I found out in a painful local problem.
Upon closer investigation, I was confused why my Jenkins build showed fetching a particular gem (ffaker, FWIW) successfully, but when the app loaded and required ffaker, it said file not found. WTF?
A little more investigation and experimenting showed what the two files do:
First it uses Gemfile.lock to go fetch all the gems, even those that won't be used in this particular env. Then it uses Gemfile to choose which of those fetched gems to actually use in this env.
So, even though it fetched the gem in the first step based on Gemfile.lock, it did NOT include in my :test environment, based on the groups in Gemfile.
The fix (in my case) was to move gem 'ffaker' from the :development group to the main group, so all env's could use it. (Or, add it only to :development, :test, as appropriate)