I need to install a number of ruby gems (all with C extensions) on a production server which does not have any dev tools installed. I'd like to build gems on a dev server first and then repackage and install resulting native gems on production server.
However, there seems to be no standard methods to package gem with native extensions for redistribution. I am aware of rake-compiler, but none of the gems in concern works with it out of the box. Specifically, I am working with json-1.7.5, rb-inotify-0.8.8 and ffi-1.2.1 gems.
Any pointers on how to do this task or documentations on the subject are appreciated.
Using Jordan Sissel's fpm you can take various input archives (including gems) and compile and package them as (among others) DEBs or RPMs.
An example to compile the json gem into a deb package follows:
cd /tmp
fpm -s gem -t deb json
This will download the latest version of the json gem and create a rubygem-json-1.5.7-1.amd64.deb archive in /tmp which you can install on your server. Note that the compile box and the final server need to be rather identical. At least the distribution and bitness, the ruby version and its file layout, and the available loadable libraries should be the same. Basically all the constraints your upstream distribution deals with...
That said, in the long term I found it much easier to just install a compiler on the target servers and use rbenv or rvm on the server. For most small and mid-size installations, it's much easier to handle as you don't need to pre-compile and ship everything to your servers.
Hi You could do it with: gem-compiler
You need to tell RubyGems the filename of the gem you want to compile:
$ gem compile yajl-ruby-1.1.0.gem
The above command will unpack, compile any existing extensions found and repackage everything as a binary gem:
Unpacking gem: 'yajl-ruby-1.1.0' in temporary directory...
Building native extensions. This could take a while...
Successfully built RubyGem
Name: yajl-ruby
Version: 1.1.0
File: yajl-ruby-1.1.0-x86-mingw32.gem
It have well written documentations here:
https://github.com/luislavena/gem-compiler.
I am using it as well as we have own gem server. Just have to be careful with distribution because some gems compiled on wheezy won't work on jessie and so on.
You're going to have to build them on a system that's almost exactly the same for this to work. If you're linking against shared libraries that are in a different location or have slightly different versions it may not work at all. Sometimes you have some slack, where it will work with a range of versions, but this cannot be assured.
There's no way to package with native extensions for this very reason, there's just too many possible combinations of libraries.
You'll also need to make sure you're using the same architecture, including 32-bit or 64-bit as required.
Sometimes you'll get lucky and there's a package for your OS that can install these for you, but these won't work with rvm or rbenv.
I ran:
bundle install --path vendor as the first Bundler install. I then ran:
bundle package, which creates a vendor/cache directory and puts in the gems. But, there is also a vendor/ruby/1.9.1/cache/ directory too. It has the same contents (I md5'ed them).
So, what's the point of bundle package then?
Any insight is much appreciated.
bundle package stores the .gem source files in vendor/cache, which guarantees smooth deploys and allows reviving old projects easily. This is especially useful when using a fork of a gem because the author can close their repo at any time, leaving your app incomplete.
The vendor/ruby directory contains the installed gems, which in some cases will be the same. But some gems build with native extensions so these can't be used in deployment due to different architecture. You should add vendor/ruby to your .gitignore file.
How would one install a Ruby VM (JRuby, Ruby MRI, etc.) on a machine that doesn't have internet access?
I'd like to just drop the tar.gz file somewhere that RVM is able to see (or checks before it goes out and tries to retrieve the package itself). Any ideas?
This blog post explains that you can drop the source archive in your $HOME/.rvm/archives directory.
cp ruby-enterprise-1.8.7-2010.02.tar.gz /home/minhajuddin/.rvm/archives
rvm install ree-1.8.7
Or the docs say that you can specify the archives directory from the command line:
--archives - directory for downladed files (~/.rvm/archives/)
I have installed MacRuby 0.6 and then the hotcocoa gem from Github. However, when I try to create a hotcocoa application with eg.
hotcocoa demoapp
I get the error
-bash: /usr/local/bin/hotcocoa: No such file or directory
Typing 'which hotcocoa' results in
/usr/bin/hotcocoa
Other online documentation seems to suggest that this ought to have been installed to /usr/local/bin so I am confused (a) how it ended up here and (b) why it can't be found even though both locations are in my PATH
I installed hotcocoa with:
sudo macgem install hotcocoa
I had previously done a gem install by mistake but I uninstalled that when I realised I should be using macgem instead.
echo $PATH
/opt/local/bin:/opt/local/sbin:/usr/local/git/bin:/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec
Any thoughts?
Addendum
As an experiment, I also changed the order of the items in the PATH and it can now find hotocoa. My PATH is now...
/usr/local/git/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/local/bin:/bin:/sbin:/usr/sbin:/usr/libexec
..with /usr/bin listed earlier than /usr/local/bin
Of course this still doesn't explain why it couldn't find it before (all the path elements seem to exist) or why it was installed there.
Thank you for sharing your experience! I've been struggling with the same set of issues and outcomes for a while. Path order was a non-issue in my case.
It appears that for MacRuby 0.7.1 (installed via RVM) with hotcocoa (installed via sudo macgem), the use of 'sudo' is required to avoid potential load errors.
For example:
(1) keying in 'hotcocoa mydemo', displays the 'Could not find RubyGem hotcocoa (>= 0) (Gem::LoadError)' message on my box. Whereas, if I key in 'sudo hotcocoa mydemo', no load error occurs and, Rakefile, config, lib, and resources items can be listed in the 'mydemo' directory.
(2) keying in 'macrake' results in rake aborted: no such file to load -- hotcocoa/application_builder. Whereas, keying in 'sudo macrake' displays the Mydemo window and its 'Hello from HotCocoa' message.
I actually found that installing hotcocoa without the sudo command worked.
I'm trying to install some Ruby Gems so I can use Ruby to notify me when I get twitter messages. However, after doing a gem update --system, I now get a zlib error every time I try and do a gem install of anything. below is the console output I get when trying to install ruby gems. (along with the output from gem environment).
C:\data\ruby>gem install twitter
ERROR: While executing gem ... (Zlib::BufError)
buffer error
C:\data\ruby>gem update --system
Updating RubyGems
ERROR: While executing gem ... (Zlib::BufError)
buffer error
C:\data\ruby>gem environment
RubyGems Environment:
- RUBYGEMS VERSION: 1.2.0
- RUBY VERSION: 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]
- INSTALLATION DIRECTORY: c:/ruby/lib/ruby/gems/1.8
- RUBY EXECUTABLE: c:/ruby/bin/ruby.exe
- EXECUTABLE DIRECTORY: c:/ruby/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86-mswin32-60
- GEM PATHS:
- c:/ruby/lib/ruby/gems/1.8
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- http://gems.rubyforge.org/
Found it! I had the same problem on windows (it appeared suddenly without me doing an update, but whatever):
It has something to do with multiple conflicting zlib versions (I think).
In ruby/lib/ruby/1.8/i386-msvcrt, make sure that there exists a zlib.so file. In my case, it was already there. If not, you may try to install ruby-zlib.
Then go to ruby/lib/ruby/site_ruby/1.8./i386-msvcrt and delete the zlib.so file there.
In ruby/bin, there should be a zlib1.dll. For some reason my Ruby version did not use this dll. I downloaded the most recent version (1.2.3) and installed it there. I had to rename it to zlib.dll for it to be used.
And tada! Rubygems worked again.
Hope this helps.
Firstly, I thank the person, who came up with the solution to the missing zlib problem. (It wasn't me. :-)
Unfortunately I lost the link to the original posting, but the essence of the solution on Linux is to compile the Ruby while zlib header files are available to the Ruby configure script. On Debian it means that zlib development packages have to be installed before one starts to compile the Ruby.
The rest of my text here does not contain anything new and it is encouraged to omit it, if You feel comfortable at customizing Your execution environment at UNIX-like operating systems. The following is a combination of a brief intro to some basics and step by step instructions.
------The-start-of-the-HOW-TO-------------------------
If one wants to execute a program, let's say, irb, from a console, then the file named irb is searched from folders in an order that is described by an environment variable called PATH. It's possible to see the value of the PATH by typing to a bash shell (and pressing Enter key):
echo $PATH
For example, if there are 2 versions of irb in the system, one installed by the "official" package management system, let's say, yum or apt-get, to /usr/bin/irb and the other one that is compiled by the user named scoobydoo and resides in /home/scoobydoo/ourcompiledruby/bin then the question arises, which one of the two irb-s gets executed.
If one writes to the
/home/scoobydoo/.bashrc
a line like:
export PATH="/home/scoobydoo/ourcompiledruby/bin:/usr/bin"
and restarts the bash shell by closing the terminal window and opening a new one, then by typing irb to the console, the
/home/scoobydoo/ourcompiledruby/bin/irb gets executed. If one wrote
export PATH="/usr/bin:/home/scoobydoo/ourcompiledruby/bin"
to the
/home/scoobydoo/.bashrc
,then the /usr/bin/irb would get executed.
In practice one wants to write
export PATH="/home/scoobydoo/ourcompiledruby/bin:$PATH"
because this prepends all of the values that the PATH had prior to this assignment to the /home/scoobydoo/ourcompiledruby/bin. Otherwise there will be problems, because not all common tools reside in the /usr/bin and one probably wants to have multiple custom-built applications in use.
The same logic applies to libraries, except that the name of the environment variable is LD_LIBRARY_PATH
The use of the LD_LIBRARY_PATH and PATH allow ordinary users, who do not have root access or who want to experiment with not-that-trusted software, to build them and use them without needing any root privileges.
The rest of this mini-how-to assumes that we'll be building our own version of ruby and use our own version of it almost regardless of what is installed on the system by the distribution's official package management software.
1)=============================
First, one creates a few folders and set the environment variables, so that the folders are "useful".
mkdir /home/scoobydoo/ourcompiledruby
mkdir -p /home/scoobydoo/lib/our_gems
One adds the following 2 lines to the
/home/scoobydoo/.bashrc
export PATH="/home/scoobydoo/ourcompiledruby/bin:$PATH"
export GEM_HOME="/home/scoobydoo/lib/our_gems"
Restart the bash shell by closing the current terminal window and opening a new one or by typing
bash
on the command line of the currently open window.
The changes to the /home/scoobydoo/.bashrc do not have any effect on terminal windows/sessions that were started prior to the saving of the modified version of the /home/scoobydoo/.bashrc
The idea is that the /home/scoobydoo/.bashrc is executed automatically at the start of a session, even if one logs on over ssh.
2)=============================
Now one makes sure that the zlib development packages are available on the system. As of April 2011 I haven't sorted the details of it out, but
apt-get install zlibc zlib1g-dev zlib1g
seems to be sufficient on a Debian system. The idea is that both, the library file and header files, are available in the system's "official" search path. Usually apt-get and alike place the header files to the /usr/include and library files to the /usr/lib
3)=============================
Download and unpack the source tar.gz from the http://www.ruby-lang.org
./configure --prefix=/home/scoobydoo/ourcompiledruby
make
make install
4)=============================
If a console command like
which ruby
prints to the console
/home/scoobydoo/ourcompiledruby/bin/ruby
then the newly compiled version is the one that gets executed on the command
ruby --help
5)=============================
The rest of the programs, gem, irb, etc., can be properly executed by using commands like:
ruby `which gem` install rake
ruby `which irb`
It shouldn't be like that but as of April 2011 I haven't figured out any more elegant ways of doing it. If the
ruby `which gem` install rake
gives the zlib missing error again, then one should just try to figure out, how to make the zlib include files and library available to the Ruby configure script and recompile. (Sorry, currently I don't have a better solution to offer.)
May be a dirty solution might be to add the following lines to the
/home/scoobydoo/.bashrc
alias gem="`which ruby` `which gem` "
alias irb="`which ruby` `which irb` "
Actually, I usually use
alias irb="`which ruby` -KU "
but the gem should be executed without giving the ruby the "-KU" args, because otherwise there will be errors.
------The-end-of-the-HOW-TO------------------------
I just started getting this tonight as well. Googling turned up a bunch of suggestions that didn't deliver results
gem update --system
and some paste in code from jamis that is supposed to replace a function in package.rb but the original it is supposed to replace is nowhere to be found.
Reinstalling rubygems didn't help. I'm reinstalling ruby right now.........and it is fixed. Pain though.
How about cd into rubysrc/ext/zlib, then ruby extendconf.rb, then make, make install.
After do that, reinstall ruby.
I did this on ubuntu 10.04 and was successful.
A reinstall of Ruby sorted this issue out. It's not what I wanted; I wanted to know why I was getting the issue, but it's all sorted out.
It most often shows up when your download failed -- i.e. you have a corrupt gem, due to network timeout, faulty manual download, or whatever. Just try again, or download gems manually and point gem at the files.
if gem update --system not works and rename ruby/bin/zlib1.dll to zlib.dll not helps try:
Open file RUBY_DIR\lib\ruby\site_ruby\1.8\rubygems.rb
And replace existed def self.gunzip(data) by this:
def self.gunzip(data)
require 'stringio'
require 'zlib'
data = StringIO.new data
# Zlib::GzipReader.new(data).read
data.read(10) # skip the gzip header
zis = Zlib::Inflate.new(-Zlib::MAX_WBITS)
is = StringIO.new(zis.inflate(data.read))
end
Try updating ZLib before you do anything else. I had a similar problem on OS X and updating Compress::Zlib (a Perl interface to ZLib) cured it - so I think an old version of ZLib (is now 1.2.3) may be where your problem lies...
install pure ruby zlib if all else fails