I am building a Ruby 2.7 Lambda application.
My application depends on a gem that exists in github.
gem 'my-gem', git: 'https://github.com/my-org/my-gem', branch: 'main'
I want to write a script that can build a deployment zip file containing this dependency.
When I run bundle install, my gem is installed to vendor/bundle/ruby/2.7.0/bundler/gems/my-gem-GITHASH.
For Lambda packaging, I believe that I need construct the following
vendor/bundle/ruby/2.7.0/gems/my-gem-1.0.0/* (ruby code)
vendor/bundle/ruby/2.7.0/specifications/my-gem-1.0.0.gemspec
The following scripted actions can assemble this structure, but I wish I had a simpler approach.
cd vendor/bundle/ruby/2.7.0/bundler/gems/my-gem-*
# build the git gem (*.gem)
gem build
# copy the gem and the gemspec to the vendor/bundle/ruby/2.7.0 directories
cp *.gem ../../../gems
cp *.gemspec ../../../specifications/my-gem-1.0.0.gemspec
# upack the .gem file in the proper directory
cd ../../../gems
gem unpack *.gem
# return to the working directory
cd ../../../../..
# Zip the dependencies
zip -r deploy.zip \
vendor/bundle/ruby/2.7.0/gems \
vendor/bundle/ruby/2.7.0/specifications \
vendor/bundle/ruby/2.7.0/extensions \
lib
I would be curious to find a simpler approach to this problem.
Running bundle install --deployment should do what you want (documentation page).
Update: It seems that the AWS docs use bundle install --path vendor/bundle.
An alternative solution is to add the bundler/gems path to the $LOAD_PATH, like so:
load_paths = Dir['./vendor/bundle/ruby/2.7.0/bundler/gems/**/lib']
$LOAD_PATH.unshift(*load_paths)
(at the very top of lambda_function.rb)
Related
I have to create a JRuby jar file for my project. Below I provided details about my directory structure and files.
Top level directory - Project1
Under Project1 – I have bin, lib, src folders
Under Project1/bin – I have wrapper shell script from where I am calling jruby jar.
Under Project1/lib – I have jruby-complete-1.6.7.2.jar and ojdbc6.jar
Under Project1/src – I have lib and tool folders
Under Project1/src/lib – I have main.rb file and utilfolder
Under Project1/src/lib/util - I have 2-ruby scripts which are getting called in main.rb.
Under Project1/src/tool- I have Tool.java from where I call main.rb.
Now I have couple of questions -
Do I need to bundle all the gems which I used in my ruby scripts (for example: colorize, socket, net/ssh, etc)?
How do I create a JRuby jar? I saw the following posts on stackoverflow before posting my question but I got confused and kind of not able to figure out from where to start. Please provide some guidance on this.
Below are the steps which worked for me to bundle all the gems into jruby-complete.jar.
Download the jruby-complete-latest_version.jar from http://jruby.org/download.
Verify which gems are included in the downloaded jar, say java -jar jruby-complete-latest_version.jar -S gem list.
To push the gems into jruby-complete-latest_version.jar you need to check for all the required runtime dependecies for that gem. Example: for net-scp you need to download net-ssh gem before.
Download the gems under the same directory where you have jruby-complete-latest_version.jar by using the following command:
java -jar jruby-complete-latest_version.jar -S gem install -i ./net-ssh net-ssh --no-rdoc --no-ri
java -jar jruby-completelatest_version.jar -S gem install -i ./net-scp net-scp --no-rdoc --no-ri
Now add the gems inside ruby-complete-latest_version.jar by using update file (uf) option for the jar file. Example:
jar uf jruby-complete-latest_version.jar -C net-ssh .
jar uf jruby-complete-latest_version.jar -C net-scp .
Check the gem list for the jar file to make sure all the gems are added successfully
java -jar jruby-complete-latest_version.jar -S gem list
Last check to make sure gems are loading successfully, run require statements on irb.
java -jar jruby-complete-latest_version.jar -S irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'net/scp'
=> true
After I have the jruby-complete_latest_version.jar file under lib I used ANT to build the jar for my project.
Again this solution will work for the smaller projects. For big projects Warbler will the best choice as suggested by #joelparkerhenderson.
In Python I can install a package from source in "editable" mode using pip install -e. Then I can carry on editing the code, and any changes will be automatically picked by other Python scripts that import library
Is there a comparable workflow for developing Ruby gems? What is the "Ruby way" of using libs as they are being developed rather than, for example, compiling and installing a gem every time I make a change to the source?
There are two common approaches one might use with bundler:
one executes bundle install --path vendor/bundle and does not run bundle update unless everything is tested.
one tells a bundler to use a local version of the gem:
in Gemfile (this is not supported in mymaingem.gemspec due to rubygems maintainence issues): gem 'mycutegem', :git => 'git://github.com/myname/mycutegem', :branch => 'master';
in command line: bundle config local.mycutegem /path_to_local_git/mycutegem.
The first approach will download everything into subfolder of your current project (here it’d be vendor/bundle.) Feel free to modify everything there, it’ll be reflected.
The second approach is likely better. You are to clone the gem from github and instruct bundle to use your local clone of the respective git repository. This approach provides you with an ability to publish the changes to your main gem into the repository. As soon as dependent repo is published as well, the up-to-date version will be retrieven by your gem subscribers, assuming they have not instructed their bundlers to use their locals.
Hope this helps.
Let's assume you have your gem code residing in a folder (say my_project/mygem/lib). You have some Ruby code in my_project that you want using the mygem code.
What I'd do is add mygem/lib to the $LOAD_PATH global variable in the beginning of said files. Kinda like this:
$LOAD_PATH << File.expand_path('lib', './mygem') # Resolve global paths
require 'a_file' # Would require "mygem/lib/a_file.rb"
I am not sure if this is exactly what you want to achieve, but from the description I infer that you want to have a local copy of some gem and reference that gem in your project.
If this is the case, you can (usually) achieve it in two steps:
Clone the gem from VCS (in most cases: git), e.g.
git clone url-of-the-gem-repo
Reference the local copy using Bundler :path, e.g.
gem "some-gem", :path => "/path/to/local/copy"
If the gem is stored at github, an even better way is to first fork it at github and then clone your own copy. Then, if you provide any improvements to the code in the local repo, you can easily share it with the world using a pull request.
I realize this is a 5 year old question, but I found all of these answers unsatisfying. Since I use ruby to develop CLI tools, using bundler for testing is not ideal. I need to be able to execute my test commands anywhere, and get the actual equivalent of pip install --editable.
Here's my solution.
Use RVM to setup a new gemset for development
rvm gemset use tools-dev --create
Install your gems
gem install mygem.gem
Locate the installation directory
gem list tool -d
In the output, it will say installed at ${rvm_gemset_path}. Copy that path. It will also be an environment variable GEM_HOME, but I include this step for completeness and clarity.
Delete the installed copy
rm -Rf ${rvm_gemset_path}/gems/mygem-${version}
Create a symlink to the git repository working directory.
ln -s ${PWD} ${rvm_gemset_path}/gems/mygem-${version}
This bash script will do it in a more automated fashion assuming that you are in the git directory and you've set GEM_NAME as your gem's name.
rvm gemset use --create ${GEM_NAME}-dev
gem build ${GEM_NAME}.gemspec
gem install ${GEM_NAME}*.gem
gem_install_path=$(ruby -e "puts Gem::Specification.find_by_name('${GEM_NAME}').full_gem_path")
mv $gem_install_path "${gem_install_path}.bak"
ln -s $PWD $gem_install_path
I have a Ruby IronWorker which depends on a private gem that isn't published to RubyGems.
Is there a way to merge this local mygemname-0.0.1.gem into my IronWorker in the .worker file?
I'm hoping to be able to specify something the following in myruby.worker:
gem 'mygemname', '>=0.0.1', :path=> 'vendor/bundle'
Currently this give the following error
.rvm/gems/ruby-1.9.3-p0/gems/iron_worker_ng-0.12.2/lib/iron_worker_ng/code/base.rb:79 :in `eval':
wrong number of arguments (3 for 2) (ArgumentError)
Hoping for defaults gives:
gem 'mygemname', '>=0.0.1'
Gives the following error
Could not find gem 'mygemname (>= 0.0.1) ruby' in the gems available on this machine.
Am I on the right track trying to get this to work via the .worker file? Or should I be looking into specifying a custom build step?
If your unpublished gem itself has dependancies, you need to do a little massaging to get things going. Here is a technique that works for me:
mygem.worker
runtime "ruby"
#Merge in an unpublished local gem
dir '../opensource-cli-tools/facebook_exporter', '__gems__/gems'
file '../opensource-cli-tools/facebook_exporter/mygem.gemspec', '__gems__/specifications'
#Merge in a custom build script to fetch the unpublished gem's dependancies
file "Gemfile"
file "install_dependancies.sh"
remote_build_command 'chmod +x install_dependancies.sh && ./install_dependancies.sh'
#Run the puppy!
exec "run.rb"
install_dependancies.sh
echo "Installing dependancies to __gems__/"
gem install bundler --install-dir ./__gems__ --no-ri --no-rdoc
bundle install --standalone --path ./__gems__
cp -R ./__gems__/ruby/*/* ./__gems__
rm -rf ./__gems__/ruby
echo "Fixing install location of mygem"
mv ./__gems__/gems/mygem ./__gems__/gems/mygem-0.0.1
As far as i know, git and local paths unsupported right now.
Here is way to manually include local gem:
Add these lines to .worker file:
dir '../vendor/bundle/mygemname', '__gems__/gems'
file '../vendor/bundle/mygemname/mygemname.gemspec', '__gems__/specifications'
I have bunch of gems on my computer that I want to use in a chef recipe.
I know it is possible to put them in a directory like /tmp/gems and just:
cd /tmp/gems
gem install *.gem
Is it possible to put all gems in one directory where I can install them with bundler without downloading them again?
cd /somedir/my_rails_project
bundle
I want to save bandwidth.
bundle install --local should be what you want. From the bundle-install manpage:
--local
Do not attempt to connect to rubygems.org, instead using just the
gems located in vendor/cache. Note that if a more appropriate
platform-specific gem exists on rubygems.org, this will bypass
the normal lookup.
You can add local directories to your Gemfile (example from the docs):
gem "nokogiri", :path => "~/sw/gems/nokogiri"
Alternatively, you can set up a local Git repository with the gems in it and write a Gemfile like this:
gem "gem1", :git => "file:///tmp/gems",
:branch => "gem1"
Use
bundle package
Locks and then caches the gems into ./vendor/cache.
The package command will copy the .gem files for your gems in the
bundle into ./vendor/cache. Afterward, when you run bundle install,
Bundler will use the gems in the cache in preference to the ones on
rubygems.org.
http://bundler.io/v1.6/bundle_package.html
You can use the BUNDLE_CACHE_PATH configuration key:
cache_path (BUNDLE_CACHE_PATH): The directory that bundler will place cached gems in when running bundle package, and that bundler will look in when installing gems. Defaults to vendor/bundle.
Source: https://bundler.io/v1.16/bundle_config.html#LIST-OF-AVAILABLE-KEYS
In GitLab CI, I defined this value in the environment of runners: "BUNDLE_CACHE_PATH=/cache-ci/bundle", this directory is mounted automatically in CI runners.
Then bundle install will install gems from the cache directory (once cache will be populated).
If you want to use a local cache for the purpose of speeding up bundle install on CI, for example when a docker container is used to run the tests, you could use --path. This will use gems in the given path unless they are not present, otherwise it will download them to that location.
This assumes the CI build can mount a persistent volume inside the docker container. So for example if the CI machine has a directory /var/cache/drone which can be mounted in the docker container as ./cache then you can do:
bundle install --without=development --quiet --path=cache
I'm trying to develop a gem locally, and have installed it with Bundler.
My Gemfile looks like this:
source "http://rubygems.org"
gemspec
And my gemspec is a standard gemspec file.
I can install the gem with 'bundle install' in the directory, and i see the local gem and all it's dependencies install:
bundle install
Using rack (1.3.4)
Using tilt (1.3.3)
Using sinatra (1.3.1)
Using {my gem} (0.0.2) from source at .
Using bundler (1.0.21)
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
However, when I do a 'gem list', my gem is not included in the list of gems - which is my guess as to why my bin directory does not appear in the path. Is there a way to test a local gem and include it in the list of installed gems using bundler, so that the bin directory properly works?
Easiest way to get rid of bundler: command not found: {your bin executable}:
git add bin/* # git-ls-files will now list your bin executables.
bundle install
# No git-commit necessary.
bundle exec <MY_BIN_EXECUTABLE>
gem list shows your system installed gems, not the gems in your Bundle (this are often the same but not always--as in this case). When you're using Bundler, you should always execute gem executables with bundle exec so that Bundler can set up the environment for you. So, if you have a binary called, for example, mygem, you should use bundle exec mygem.
See more info at Bundler's site or in the manpage.
[Edit]
Also be sure that your gemspec includes a bin directory! Common convention is to create a directory called bin at the same level as your lib directory, put your binaries in there, and then add this as the directory in your gemspec. If you don't do this, Bundler won't expose your binaries!
I had this problem too.
Make sure the executables and default_executable lines don't contain 'bin/'. Then:
git add add . # You can be more precice if you want.
git commit -m "My lousy commit message."
bundle install
bundle exec <binaryname>