I have a pretty basic "app" which i'm trying to dockerize using some examples found on the internet but, for some reason, it seems that bundler can't see the gems installed in my env.
Gemfile
source 'https://rubygems.org'
gem 'streamio-ffmpeg'
Dockerfile
FROM dpsxp/ffmpeg-with-ruby:2.2.1
MAINTAINER mike
RUN apt-get update && apt-get install -y \
build-essential \
cmake
ENV APP_HOME=/app
RUN mkdir -p $APP_HOME
WORKDIR $APP_HOME
COPY Gemfile* ./
ENV BUNDLE_GEMFILE=$APP_HOME/Gemfile \
BUNDLE_PATH=/bundle
RUN gem install bundler
RUN bundle install --jobs 20 --retry 5 --path=/bundle
COPY . ./
CMD ["rake"]
docker-compose.yml
version: '2'
services:
app:
build:
context: .
volumes:
- '.:/app'
depends_on:
- bundle
bundle:
image: busybox
volumes:
- /bundle
And now, when i'm running $ docker-compose run app gem list, the streamio-ffmpeg gem is not on the list for some reason. Also, when i'm trying to require it inside my ruby app, i'm getting LoadError: cannot load such file
Build logs: https://gist.github.com/mbajur/569b4f221cadb8a85ecc5dae8a595401
When you deal with gems by bundler, all related commands should be run with bundle exec, so in your case, it should work with below command:
docker-compose run app bundle exec gem list
Bundler provides a consistent environment for Ruby projects by tracking and installing the exact gems and versions that are needed.
The command you run directly with gem list is to list the system gems, of course they are not in list.
Related
I try to build a simple ruby script that checks the balance on the Kraken crypto exchange using the custom jonatack/kraken_ruby_client gem from github which is not available on ruby gems.
AWS Lambda throws the following error:
"errorMessage": "https://github.com/jonatack/kraken_ruby_client.git (at master#cda6c28) is not yet checked out. Run `bundle install` first.",
"errorType": "Init<Bundler::GitError>",
"stackTrace": ["/var/runtime/gems/bundler-2.1.4/lib/bundler/source/git.rb:204:in `rescue in load_spec_files'",
...
What I'm doing:
1. Gemfile
I've added the gem to the Gemfile as usual:
gem 'kraken_ruby_client', git: 'https://github.com/jonatack/kraken_ruby_client.git'
Everything works perfectly on my local machine.
2. Custom Dockerfile
Since the gem requires curb I created a Dockerfile, based on the AWS Lambda image for Ruby 2.7, to install libcurl:
FROM lambci/lambda:build-ruby2.7
RUN yum install -y curl-devel
RUN gem install curb
3. Deployment
I then "deploy" the script:
docker run -v `pwd`:`pwd` -w `pwd` -i -t kraken-app:latest bundle install --deployment
zip build.zip * -r -x .git/\* \*.sh specs/\* tests/\* \*.zip config/application.yml
aws lambda update-function-code --function-name kraken-app --zip-file fileb://build.zip --region=eu-west-1 --publish --profile XXXX
And it fails with the above error.
What I've tried with multiple StackOverflow answers:
1. Do what the error says - bundle install
I "bundle installed" and "bundled" as much as I could
2. Paths
All gems from Rubygems are installed into vendor/bundle/ruby/2.7.0/gems/, while the gem from github is installed into /vendor/bundle/ruby/2.7.0/bundler/gems/
So I've tried to add the path to $LOADPATH:
load_paths = Dir["./vendor/bundle/ruby/2.7.0/bundler/gems/**/lib"]
$LOAD_PATH.unshift(*load_paths)
As that didn't help, I even manually moved the folder from /bundler/gems to /gems before deploy, but no avail.
3. Local install in Docker image
I amended the Dockerfile to install the gem "locally" in the image by adding:
RUN gem install specific_install
RUN gem specific_install https://github.com/jonatack/kraken_ruby_client.git
I can run the script just fine in Docker in bash:
docker run -it --rm -v "$PWD":/var/task kraken-app:latest bash
Everything works in the Docker container on my local machine but it doesn't work when deployed to Lambda for some reason? Other Lambda functions work perfectly, I just can't get this custom gem to work...
Do you have any idea what else to try?
Thanks in advance!
Update: I've narrowed the [or a?] problem down to the line - groupy-gemcache:/usr/local/bundle in my services: app: volumes dictionary. If I remove it, the container runs fine [i think] but presumably i lose my local gem cacheing.
tldr: After running docker-compose build, things seem ok, but I cannot run any gem or bundle inside my running docker container if I add something to my gemfile. for example, after docker-compose build && docker-compose run app bash:
root#2ea58aff612e:/src# bundle check
The Gemfile's dependencies are satisfied
root#2ea58aff612e:/src# echo 'gem "hello-world"' >> Gemfile
root#2ea58aff612e:/src# bundle
Could not find gem 'hello-world' in any of the gem sources listed in your Gemfile.
Run `bundle install` to install missing gems.
root#2ea58aff612e:/src# bundle install
Could not find gem 'hello-world' in any of the gem sources listed in your Gemfile.
Run `bundle install` to install missing gems.
root#2ea58aff612e:/src# gem
Could not find gem 'hello-world' in any of the gem sources listed in your Gemfile.
Run `bundle install` to install missing gems.
root#2ea58aff612e:/src# gem env
Could not find gem 'hello-world' in any of the gem sources listed in your Gemfile.
Run `bundle install` to install missing gems.
I'm still pretty novice with docker, but i've been trying to configure a perfect dockerfile that can build, cache gems and update while still committing its Gemfile.lock to git [maybe this isn't actually perfect and I'm open to suggestions there]. In my use case, I'm using a docker compose file with images for a rails app and sidekiq worker as well as postgres and redis images- very simliar to the setups described here and here.
My Dockerfile [some things commented out that I cobbled from the tutorials above:
FROM ruby:2.3
ENTRYPOINT ["bundle", "exec"]
ARG bundle_path
# throw errors if Gemfile has been modified since Gemfile.lock
# RUN bundle config --global frozen 1
ENV INSTALL_PATH /src
RUN mkdir -p $INSTALL_PATH
WORKDIR $INSTALL_PATH
# Install dependencies:
# - build-essential: To ensure certain gems can be compiled
# - nodejs: Compile assets
# - npm: Install node modules
# - libpq-dev: Communicate with postgres through the postgres gem
# - postgresql-client-9.4: In case you want to talk directly to postgres
RUN apt-get update && apt-get install -qq -y build-essential nodejs npm libpq-dev postgresql-client-9.4 --fix-missing --no-install-recommends && \
rm -rf /var/lib/apt/lists/*
COPY app/Gemfile app/Gemfile.lock ./
# Bundle and then save the updated Gemfile.lock in our volume since it will be clobbered in the next step
RUN echo $bundle_path && bundle install --path=$bundle_path && \
cp Gemfile.lock $bundle_path
# ./app contains the rails app on host
COPY app .
# Unclobber the updated gemfile.lock
RUN mv $bundle_path/Gemfile.lock ./
CMD ["./script/start.sh"]
docker-compose.yml:
version: '2'
volumes:
groupy-redis:
groupy-postgres:
groupy-gemcache:
services:
app:
build:
args:
bundle_path: /usr/local/bundle
context: .
dockerfile: Dockerfile
command: "./script/start.sh"
links:
- postgres
- redis
volumes:
- ./app:/src
- groupy-gemcache:/usr/local/bundle
ports:
- '3000:3000'
env_file:
- .docker.env
stdin_open: true
tty: true
Wow, I figured it out. The problem was coming from my ENTRYPOINT dockerfile command, as explained at the end of this article on CMD vs ENTRYPOINT.
I believe the result of ENTRYPOINT ["bundle", "exec"] with CMD ["./script/start.sh"] OR docker-compose run app bash was to run a command like bundle exec 'bash'. I confirmed this by removing the entrypoint from the dockerfile, entering the shell as above and manually running bundle exec 'bash' and sure enough i landed in a subshell where I couldn't run any bundle or gem commands- had to exit twice to leave.
We have a rails app developed with a few rails engine so it looks like
app/ <-- main app
app.json
Gemfile -> config/Gemfile
Gemfile.lock
api/ <-- this is an engine app
integrations/ <-- this is an engine app
other_engine_app/
everything works fine but now I am looking to deployed on docker using docker-compose
I created a simple Dockerfile
FROM ruby:2.3.1
RUN apt-get update -qq && apt-get install -y apt-utils build-essential nodejs
ENV APP_HOME /myapp
RUN mkdir -p $APP_HOME
WORKDIR $APP_HOME
ADD Gemfile Gemfile.lock $APP_HOME/
RUN gem install bundler
RUN bundle install
ADD . $APP_HOME
The docker-compose.yml file is looking like
app:
build: .
volumes:
- .:/myapp
ports:
- "3000:3000"
links:
- db
- redis
db:
image: mysql:5.6
redis:
image: redis
At this stage when I run docker-compose up I get
$ docker-compose up
Building app
Step 1/10 : FROM ruby:2.3.1
.....
Step 9/10 : RUN bundle install
---> Running in 3748ba73eb9b
The path `/myapp/api` does not exist.
ERROR: Service 'app' failed to build: The command '/bin/sh -c bundle install' returned a non-zero code: 13
I can bypass this error by adding the Gemfile into each engine app from my Dockerfile as
FROM ruby:2.3.1
RUN apt-get update -qq && apt-get install -y apt-utils build-essential nodejs
ENV APP_HOME /myapp
RUN mkdir -p $APP_HOME
WORKDIR $APP_HOME
ADD Gemfile Gemfile.lock $APP_HOME/
ADD Gemfile Gemfile.lock $APP_HOME/api/
ADD Gemfile Gemfile.lock $APP_HOME/integrations/
ADD Gemfile Gemfile.lock $APP_HOME/other_engine_app/
RUN gem install bundler
RUN bundle install
ADD . $APP_HOME
but now I get the following error
$ docker-compose up
Building app
Step 1/10 : FROM ruby:2.3.1
.......
Step 12/13 : RUN bundle install
---> Running in 3928848ed6e5
Fetching https://github.com/rails-api/active_model_serializers
Fetching https://github.com/activeadmin/activeadmin
Fetching gem metadata from https://rails-assets.org/..
Fetching version metadata from https://rails-assets.org/.
Fetching gem metadata from https://rails-assets.org/..
Fetching gem metadata from http://rubygems.org/..........
Fetching version metadata from https://rails-assets.org/..
Fetching version metadata from http://rubygems.org/...
Fetching dependency metadata from https://rails-assets.org/..
Fetching dependency metadata from http://rubygems.org/..
Could not find gem 'api' in source at `api`.
Source does not contain any versions of 'api'
ERROR: Service 'app' failed to build: The command '/bin/sh -c bundle install' returned a non-zero code: 7
EDIT :
Thanks to gwcodes, I can move forward but still hit an issue
The main Gemfile has dependency defined on the rails engine as
source 'http://rubygems.org'
ruby '2.3.1'
gem 'dotenv-rails', require: 'dotenv/rails-now'
# Engines.
gem 'api', path: 'api/'
gem 'integrations', path: 'integrations'
gem 'other_engine_app', path: 'other_engine_app'
....
in api/ folder we have gemspec defined as `api.gemspec``
$:.push File.expand_path('../lib', __FILE__)
require 'api/version'
Gem::Specification.new do |s|
s.name = 'api'
s.version = Api::VERSION
s.authors = ...
s.email = ...
s.homepage = ...
s.summary = ...
s.files = Dir['{app,config,lib}/**/*'] + ['Rakefile']
s.add_dependency 'rails'
s.add_dependency 'workflow'
s.add_dependency 'will_paginate'
s.add_dependency 'analytics-ruby'
end
I have added the gemspec file from the Dockerfile as
FROM ruby:2.3.1
RUN apt-get update -qq && apt-get install -y apt-utils build-essential nodejs
ENV APP_HOME /myapp
RUN mkdir -p $APP_HOME
WORKDIR $APP_HOME
ADD Gemfile Gemfile.lock $APP_HOME/
ADD Gemfile Gemfile.lock $APP_HOME/api/
ADD api/api.gemspec $APP_HOME/api/
ADD Gemfile Gemfile.lock $APP_HOME/integrations/
ADD Gemfile Gemfile.lock $APP_HOME/other_engine_app/
RUN gem install bundler
RUN bundle install
ADD . $APP_HOME
and so I get the following error
Step 13/14 : RUN bundle install
---> Running in c7fce6cd25d8
[!] There was an error while loading `api.gemspec`: cannot load such file -- api/version
Does it try to require a relative path? That's been removed in Ruby 1.9. Bundler cannot continue.
# from /myapp/api/api.gemspec:3
# -------------------------------------------
#
> require 'api/version'
#
# -------------------------------------------
ERROR: Service 'app' failed to build: The command '/bin/sh -c bundle install' returned a non-zero code: 14
How are the engines loaded? Do you have a line that looks like this in the Gemfile of the main application:
gem 'api', path: '/api'
If so, you may also need to copy across the .gemspec files.
I've tried two forms of Dockerfile to get a simple Ruby/Sinatra app running, and in both scenarios it fails for different reasons (I'll explain both in a moment).
Effectively I want to access the Sinatra web server from my host (Mac OS X using Boot2Docker).
The app structure is:
.
├── Dockerfile
├── Gemfile
├── app.rb
├── config.ru
The contents of the files are:
Dockerfile
Version 1...
FROM ruby
RUN mkdir -p /app
WORKDIR /app
COPY Gemfile /app/
RUN bundle install --quiet
COPY . /app
EXPOSE 5000
ENTRYPOINT ["bash"]
CMD ["bundle", "exec", "rackup", "-p", "5000"]
Version 2...
FROM ubuntu:latest
RUN apt-get -qq update
RUN apt-get -qqy install ruby ruby-dev
RUN apt-get -qqy install libreadline-dev libssl-dev zlib1g-dev build-essential bison openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev
RUN gem install bundler
RUN mkdir -p /app
WORKDIR /app
COPY Gemfile /app/
RUN bundle install --quiet
COPY . /app
EXPOSE 5000
CMD ["bundle", "exec", "rackup", "-p", "5000"]
Gemfile
source "https://rubygems.org/"
gem "puma"
gem "sinatra"
app.rb
require "sinatra/base"
class App < Sinatra::Base
set :bind, "0.0.0.0"
get "/" do
"<p>hello world</p>"
end
end
config.ru
require "sinatra"
require "./app.rb"
run App
I build the docker image like so:
docker build --rm -t ruby_app .
I run the container like so:
docker run -d -p 7080:5000 ruby_app
I then try to verify I can connect to the running service (on my Mac using Boot2Docker) like so:
curl $(boot2docker ip):7080
With version 1 of the Dockerfile I get the following error before being able to run the curl command:
/usr/local/bundle/bin/rackup: line 9: require: command not found
/usr/local/bundle/bin/rackup: rackup: line 10: syntax error near unexpected token `('
/usr/local/bundle/bin/rackup: rackup: line 10: `ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../../../../app/Gemfile",'
With version 2 of the Dockerfile it seems to run the rack server fine from inside the container but I'm unable to connect via the host environment and so when running the curl command I get the error:
curl: (7) Failed to connect to 192.168.59.103 port 7080: Connection refused
Does anyone have any idea as to what I'm missing? Seems it shouldn't be this hard to get a very simple Ruby/Sinatra app running inside a Docker container from which I can access via my host (Mac OS X via Boot2Docker).
Change the dockerfile to use this instead:
["bundle", "exec", "rackup", "--host", "0.0.0.0", "-p", "5000"]
So I'm experimenting with Docker, and I set up a Rails App talking to postgres. Everything works, I'm able to access the app, migrate the database, etc.
Dockerfile
FROM ruby:2.2.1
RUN apt-get update -qq && apt-get install -y build-essential
RUN apt-get install -y libxml2-dev libxslt1-dev
RUN apt-get install -y libqt4-webkit libqt4-dev xvfb
RUN apt-get install -y nodejs
ENV APP_HOME /code
RUN mkdir $APP_HOME
WORKDIR $APP_HOME
ADD . $APP_HOME
RUN bundle install
docker-compose.yml
web:
build: .
command: bin/rails server --port 3000 --binding 0.0.0.0
ports:
- "3000:3000"
volumes:
- .:/code
links:
- db
db:
image: postgres
ports:
- "5432:5432"
After making a couple models and a controller, I decided to use Slim templates for the project and added it to my Gemfile
gem 'slim'
I would have expected to be able to update this dependency on the web container by running bundle install on it:
docker-compose run web bundle install
Which appears to work at first, it installs the gem and it's dependencies. However these are just on a throw-away copy of the built container. IT doesn't stick around when I run docker-compose up the next time.
The only way to get the gem in permanently is to build web from scratch again, which can take 3-4 minutes to install all the gems from scratch again, mostly due to nokigiri.
Is this the only way to "update" a base image? I would expect I would only need to rebuild if I'm modifying the Dockerfile itself, when it would make sense to rebuild from scratch.
If you want, you can also update an existing image by "committing" the changes you've made to a container. The syntax is docker commit <container ID> <image>.
Alternatively you can take advantage of the build cache, by separating your bundle install command into individual gem installs. Then you can add your new gem to the end and it'll reuse all the previous layers when you rebuild the image. This may not be ideal considering that there may be overlapping dependencies.