prevent rebuilding whole docker container everytime? improving speed - performance

Dockerizing a Rails application taking ages to rebuild the container.
I tried to ADD as far at the end but not possible I think more.
Any suggestions on how to improve the rebuild speed of my docker container?
Or general suggestions on how to improve the docker file, it takes very long to rebuild every time.
Also are there smart ways to check if for example a directory already exist without throwing an error and not be able to complete the build?
FROM ruby:2.2.0
EXPOSE 80
EXPOSE 22
ENV RAILS_ENV production
RUN apt-get update -qq && apt-get install -y build-essential
# --------------------------------------
# GEM PRE-REQ
# --------------------------------------
#RUN apt-get install -y libpq-dev
#RUN apt-get install -y libxml2-dev libxslt1-dev #nokigiri
#RUN apt-get install -y libqt4-webkit libqt4-dev xvfb
RUN cd /tmp && git clone https://github.com/maxmind/geoipupdate && cd geoipupdate && ./bootstrap
# --------------------------------------
# HOME FOLDER
# --------------------------------------
WORKDIR /srv/my
ADD . /srv/my
ADD ./Gemfile /srv/my/Gemfile
ADD ./Gemfile.lock /srv/my/Gemfile.lock
#RUN mkdir /srv/my
RUN bundle install --without development test
#RUN bundle install foreman
RUN bundle exec rake assets:precompile --trace
# --------------------------------------
# UNICORN AND NGINX
# --------------------------------------
ADD ./config/_server/unicorn_my /etc/init.d/unicorn_my
RUN chmod 755 /etc/init.d/unicorn_my
RUN update-rc.d unicorn_my defaults
ADD ./config/_server/nginx.conf /etc/nginx/sites-available/default
RUN apt-get install -y nginx
RUN echo "\ndaemon off;" >> /etc/nginx/nginx.conf
#RUN chown -R www-data:www-data /var/lib/nginx ??
ADD ./config/_server/nginx.conf /etc/nginx/my.conf
ADD ./config/_server/my.conf /etc/nginx/sites-enabled/my.conf
ADD ./config/_server/unicorn.rb /srv/my/config/unicorn.rb
ADD ./config/_server/Procfile /srv/my/Procfile
#RUN service unicorn_my start
#RUN foreman start -f ./Procfile

You can improve your build speed by:
Install all of your requirement as early as possible.
Combine all apt-get/yum into a single command, after that clean up the apt/yum cache. It can decrease your image size.
Sample:
RUN \
apt-get -y update && \
apt-get -y install curl build-essential nginx && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
Put ADD/COPY as late as possible, because it will invalidate the Docker image cache.
Avoid put long-running task (e.g: apt-get, download large file, etc.) after ADD/COPY file or directory that is often changed.
Docker take a "snapshot" for each your command. So, when you build a new image from same state (no Dockerfile/file/directory change), it should be fast.
Comment/uncomment Dockerfile in order to reduce apt-get install time might not help you, because it will invalidate your Docker cache.

Related

gem command not found when install rbenv in debian

I'm creating a Dockerfile to run truffleruby. I'm getting an error when trying to install bundler and foreman. The error is /bin/sh: 1: gem: not found
Dockerfile
FROM debian:buster-slim
# Install packages for building ruby
RUN apt update -y && apt install -y git curl libssl-dev libpq-dev libreadline-dev zlib1g-dev \
autoconf bison build-essential libyaml-dev \
libreadline-dev libncurses5-dev libffi-dev libgdbm-dev
RUN apt clean
# Install rbenv and ruby-build
RUN git clone https://github.com/sstephenson/rbenv.git /root/.rbenv
RUN git clone https://github.com/sstephenson/ruby-build.git /root/.rbenv/plugins/ruby-build
RUN /root/.rbenv/plugins/ruby-build/install.sh
ENV PATH /root/.rbenv/bin:$PATH
RUN echo 'eval "$(rbenv init -)"' >> /etc/profile.d/rbenv.sh # or /etc/profile
RUN echo 'eval "$(rbenv init -)"' >> .bashrc
RUN . ~/.bashrc
RUN rbenv install truffleruby-20.3.0
RUN rbenv global truffleruby-20.3.0
RUN rbenv rehash
ENV BUNDLER_VERSION=2.2.4 NODE_ENV=production RAILS_ENV=production RAILS_SERVE_STATIC_FILES=true RAILS_LOG_TO_STDOUT=true PORT=3000
ENV CONFIGURE_OPTS --disable-install-doc
RUN apt-get install -y curl -sL https://deb.nodesource.com/setup_14.x | bash - && \
apt-get update && apt-get install -y nodejs && \
apt-get clean
RUN rbenv versions
RUN gem install bundler:2.2.4 foreman
RUN mkdir /app
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN bundle config set --local deployment 'true'
RUN bundle config set --local without 'development test'
RUN bundle install
COPY . .
EXPOSE 3000
CMD ["foreman", "start"]
tail of build
Removing intermediate container 1a445fde7fc0
---> 43c3d72b7eb6
Step 17/27 : RUN rbenv versions
---> Running in feb5bb9361cc
* truffleruby-20.3.0 (set by /root/.rbenv/version)
Removing intermediate container feb5bb9361cc
---> c7d1a5826af5
Step 18/27 : RUN gem install bundler:2.2.4 foreman
---> Running in 998461afc89c
/bin/sh: 1: gem: not found
The command '/bin/sh -c gem install bundler:2.2.4 foreman' returned a non-zero code: 127
You don't generally use version managers like rbenv in Docker. There are a couple of reasons for this. One is that an image usually only contains a single application and its single runtime, so you'd never have more than one Ruby in an image and therefore there's no need to switch. A second is that most common paths of running containers (including docker run and the Dockerfile RUN directive) don't look at shell dotfiles like .bashrc or /etc/profile, so the version manager setup will never get run.
TruffleRuby is distributed (among other ways) as a standalone tar file so you can just install that in your Dockerfile. I'd make the Dockerfile look roughly like:
FROM debian:buster-slim
# Install the specific dependency packages TruffleRuby recommends
# (build-essential is much larger but might actually be necessary)
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
curl \
gcc \
libssl-dev \
libz-dev \
make
# Download and unpack TruffleRuby
ARG TRUFFLERUBY_VERSION=20.3.0
ENV PATH /opt/truffleruby-$TRUFFLERUBY_VERSION-linux-amd64/bin:$PATH
RUN cd /opt \
&& curl -L https://github.com/oracle/truffleruby/releases/download/vm-$TRUFFLERUBY_VERSION/truffleruby-$TRUFFLERUBY_VERSION-linux-amd64.tar.gz | tar xz \
&& /opt/truffleruby-$TRUFFLERUBY_VERSION-linux-amd64/lib/truffle/post_install_hook.sh
# Now build and install your application
RUN gem install bundler:2.2.4
WORKDIR /app
COPY Gemfile Gemfile.lock ./
RUN bundle config set --local deployment 'true'
RUN bundle config set --local without 'development test'
RUN bundle install
COPY . .
ENTRYPOINT ["bundle", "exec"]
EXPOSE 3000
CMD ["rails", "start"]
You can reasonably split this into two separate Dockerfiles. End the first one before the "build and install your application" comment, and build it with docker build -t myname/truffleruby:20.3.0 -f Dockerfile.truffleruby .. Then the second one can begin with FROM myname/truffleruby:20.3.0 in the same way as the standard Docker Hub ruby image.
Does the same work with CRuby?
I'd suspect RUN doesn't read shell files, so PATH needs to be modified.
RUN git clone https://github.com/sstephenson/rbenv.git /root/.rbenv
...
RUN /root/.rbenv/plugins/ruby-build/install.sh
ENV PATH /root/.rbenv/bin:$PATH
seems a bit weird, is rbenv cloned and installed in the same place?
I would skip rbenv in Docker, and instead just:
RUN ruby-build truffleruby ~/.rubies/truffleruby
ENV PATH $HOME/.rubies/truffleruby/bin:$PATH

Why dotnet command is not found in dockerfile?

I tried to run a dotnet command located in a shell file which will be called by dockerfile during the docker build process.
Here is the dockerfile snippet:
FROM ubuntu:16.04
FROM microsoft/dotnet:2.2-sdk as build-env
# .net core
RUN apt-get update -y && apt-get install -y wget apt-transport-https
RUN wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb && dpkg -i packages-microsoft-prod.deb
RUN apt-get update -y && apt-get install -y aspnetcore-runtime-2.2=2.2.1-1
# dotnet tool command
RUN apt-get update -y && apt-get install dotnet-sdk-2.2 -y
# for dot net tool #https://stackoverflow.com/questions/51977474/install-dotnet-core-tool-dockerfile
ENV PATH="${PATH}:/root/.dotnet/tools"
# Supervisor
RUN apt-get update -y && apt-get install -y supervisor && mkdir -p /etc/supervisor
# main script as defacult command when docker container runs
# Run the main sh script to run script in each xxx/*/db-migrate.sh.
CMD ["/xxx/main-migrate.sh"]
# Microservice files
ADD xxx /xxx
# install the xxx deploy tool
WORKDIR /xxx
RUN for d in /xxx/*/ ; do cd "$d"; if [ -f "./install.sh" ]; then sh ./install.sh; fi; done
In the install.sh, here is the code:
dotnet tool install -g xxx.DEPLOY --version [$(cat version)] --add-source /xxx/
When I run docker build -t xxx:v0 ., I get an error message saying:
./install.sh: 1: ./install.sh: dotnet: not found
I have added FROM microsoft/dotnet:2.2-sdk as build-env & RUN apt-get update -y && apt-get install dotnet-sdk-2.2 -y, but why Docker could not find the dotnet command during build?
How do I call the dotnet command located in the shell script file during the docker build process?
Thank you
FROM ubuntu:16.04
FROM microsoft/dotnet:2.2-sdk as build-env
In the above lines FROM ubuntu:16.04 will be totally ignored as there should be only one base image, so the last FROM will be considered as a base image which is FROM microsoft/dotnet:2.2-sdk not the ubuntu.
So if your base image is FROM microsoft/dotnet:2.2-sdk as build-env then why to bother to run these complex script to install dotnet?
You are good to go to check version of dotnet.
FROM microsoft/dotnet:2.2-sdk as build-env
RUN dotnet --version
output
Step 1/6 : FROM microsoft/dotnet:2.2-sdk as build-env
---> f13ac9d68148
Step 2/6 : RUN dotnet --version
---> Running in f1d34507c7f2
> 2.2.402
Removing intermediate container f1d34507c7f2
---> 7fde8596c331

E: Package 'mysql-client' has no installation candidate in php-fpm image build using docker compose

Im fairly new to docker and so im trying to learn more about it using a laravel project, im following this tutorial:
https://www.digitalocean.com/community/tutorials/how-to-set-up-laravel-nginx-and-mysql-with-docker-compose
Ive adjusted the Dockerfile a bit from what the tutorial has but even the tutorial file causes the same result.
FROM php:7.3-fpm
# Copy composer.lock and composer.json
COPY composer.lock composer.json /var/www/
# Install dependencies
RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - && \
apt-get update && apt-get install -y mysql-client \
RUN npm install -g npm
# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
# Install extensions
RUN docker-php-ext-install pdo pdo_mysql
# Install composer
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
# Add user for laravel application
RUN groupadd -g 1000 www
RUN useradd -u 1000 -ms /bin/bash -g www www
# Copy existing application directory contents
COPY . /var/www
# Copy existing application directory permissions
COPY --chown=www:www . /var/www
# Change current user to www
USER www
# Set working directory
WORKDIR /var/www
# Expose port 9000 and start php-fpm server
EXPOSE 9000
CMD ["php-fpm"]
But i keep getting the following error when i run docker-compose up -d:
E: Package 'mysql-client' has no installation candidate
ERROR: Service 'app' failed to build: The command '/bin/sh -c curl -sL https://deb.nodesource.com/setup_10.x | bash - && apt-get update && apt-get install -y mysql-client nodejs build-essential vim git curl' returned a non-zero code: 100
Am i missing something?
I expected this to work since i am running apt-get update before installing mysql-client.
Thanks.
php:7.3-fpm now use Debian 10 (Buster) as its base image and Buster ships with MariaDB, so just replace mysql-client with mariadb-client should fix it.
If you still want to use the mysql client, it's called default-mysql-client now.
php:7.2-apache triggers the error as well, but I resolve it using php:7.2.18-apache
it worked for me: sudo apt-get update && apt-get install -y git curl libmcrypt-dev default-mysql-client
or alternatively apt-cache search mysql-server
find out your servers then sudo apt-get install default-mysql-server default-mysql-server-core mariadb-server-10.6 mariadb-server-core-10.6
in my case it was the above codes

Bundler GemspecError when building inside Docker container

People were reporting this error and solved it by deleting a folder holding cached gems. Since I'm working inside a Docker container, this didn't seem like helpful advice for me. Indeed, the cache folder was not empty but did contain two items (I guess they are there from gem install bundle).
This is my (faulty) dockerfile
FROM nginx
RUN apt-get update -qqy && \
apt-get install -qqy \
build-essential \
ruby-full \
ruby-dev
RUN gem install bundle
RUN useradd -ms /bin/bash udo
USER udo
WORKDIR /home/udo
COPY . .
RUN bundle install
RUN bundle exec -- jekyll build
USER root
RUN cp -r _site/* /usr/share/nginx/html
I could fix the error by replacing RUN bundle install with
RUN bundle install --path=tmp
at the expense of having this local tmp directory that I then had to add to the exclude list in the _config.yml for Jekyll.
I'd prefer to know what was actually going on. I suspect there's a problem with the way I use the non-root user.
Note: bundle install --no-cache did not help.

ADD command not working Docker

FROM phusion/baseimage:0.9.16
MAINTAINER Raheel <raheelwp#gmail.com>
RUN apt-get update
RUN apt-get -y install apache2 libapache2-mod-php5 curl git
RUN add-apt-repository ppa:ondrej/php5-5.6
RUN apt-get update
RUN apt-get -y install python-software-properties
RUN apt-get update
RUN apt-get -y --force-yes install php5 php5-cli php5-mcrypt php5-curl php5-xdebug php5-json
RUN apt-get clean
RUN php5enmod mcrypt
RUN rm -rf /var/www/html
RUN curl -sS https://getcomposer.org/installer | php
RUN mv composer.phar /usr/local/bin/composer
COPY 000-default.conf /etc/apache2/sites-available/000-default.conf
ADD /home/raheel/code/laravel-app /var/www/laravel-app
EXPOSE 80
COPY setup-laravel.sh /var/www/laravel-app/setup-laravel.sh
RUN ["chmod", "+x", "/var/www/laravel-app/setup-laravel.sh"]
ENTRYPOINT ["/var/www/laravel-app/setup-laravel.sh"]
I created the above Dockerfile to run laravel application.
Then i ran docker build -t raheelwp/laravel-app . command and it created the image without name. Then i ran docker tag imageid raheelwp/laravel-app . After this when i check docker images it shows my image with name.
So far so good. But when i run docker run -t raheelwp/laravel-app and then login into container by docker exec -it containerid bash and do the ls on /var/www directory there is no laravel-app folder present there.
I am new to docker and this is my first ever docker file. I would be very thankful to you to please guide me why i am having this problem and how to fix this.
Thanks

Resources