How to rebuild dockerfile quick by using cache? - caching

Question

I want to optimize my Dockerfile. And I wish to keep cache file in disk.
But, I found when I run docker build . It always try to get every file from network.
I wish to share My cached directory during build (eg. /var/cache/yum/x86_64/6).
But, it works only on docker run -v ....
Any suggestion?(In this example, only 1 rpm installed, in real case, I require to install hundreds rpms)
My draft Dockerfile
FROM centos:6.4
RUN yum update -y
RUN yum install -y openssh-server
RUN sed -i -e 's:keepcache=0:keepcache=1:' /etc/yum.conf
VOLUME ["/var/cache/yum/x86_64/6"]
EXPOSE 22
At second time, I want to build a similar image
FROM centos:6.4
RUN yum update -y
RUN yum install -y openssh-server vim
I don't want the fetch openssh-server from internat again(It is slow). In my real case, it is not one package, it is about 100 packages.

Answer 1

An update to previous answers, current docker build
accepts --build-arg that pass environment variables like http_proxy
without saving it in the resulting image.
Example:
# get squid
docker run --name squid -d --restart=always \
--publish 3128:3128 \
--volume /var/spool/squid3 \
sameersbn/squid:3.3.8-11
# optionally in another terminal run tail on logs
docker exec -it squid tail -f /var/log/squid3/access.log
# get squid ip to use in docker build
SQUID_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' squid)
# build your instance
docker build --build-arg http_proxy=http://$SQUID_IP:3128 .

Answer 2

Just use an intermediate/base image:
Base Dockerfile, build it with docker build -t custom-base or something:
FROM centos:6.4
RUN yum update -y
RUN yum install -y openssh-server vim
RUN sed -i -e 's:keepcache=0:keepcache=1:' /etc/yum.conf
Application Dockerfile:
FROM custom-base
VOLUME ["/var/cache/yum/x86_64/6"]
EXPOSE 22

Answer 3

You should use a caching proxy (f.e Http Replicator, squid-deb-proxy ...) or apt-cacher-ng for Ubuntu to cache installation packages. I think, you can install this software to the host machine.
EDIT:
Option 1 - caching http proxy - easier method with modified Dockerfile:
> cd ~/your-project
> git clone https://github.com/gertjanvanzwieten/replicator.git
> mkdir cache
> replicator/http-replicator -r ./cache -p 8080 --daemon ./cache/replicator.log --static
add to your Dockerfile (before first RUN line):
ENV http_proxy http://172.17.42.1:8080/
You should optionally clear the cache from time to time.
Option 2 - caching transparent proxy, no modification to Dockerfile:
> cd ~/your-project
> curl -o r.zip https://codeload.github.com/zahradil/replicator/zip/transparent-requests
> unzip r.zip
> rm r.zip
> mv replicator-transparent-requests replicator
> mkdir cache
> replicator/http-replicator -r ./cache -p 8080 --daemon ./cache/replicator.log --static
You need to start the replicator as some user (non root!).
Set up the transparent redirect:
> iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner <replicator-user> --dport 80 -j REDIRECT --to-port 8080
Disable redirect:
> iptables -t nat -D OUTPUT -p tcp -m owner ! --uid-owner <replicator-user> --dport 80 -j REDIRECT --to-port 8080
This method is the most transparent and general and your Dockerfile does not need to be modified. You should optionally clear the cache from time to time.