how to write the Dockerfile for a Ruby Capybara scraper? - ruby

Question

I am trying to write a Dockerfile to run a Ruby Capybara scraper on a docker container. I tested the following code on my host OS. But it is making an error on a docker container.
Dockerfile
FROM ruby:2.6.6
RUN apt-get update -y && \
apt-get install -y xvfb
RUN wget https://ftp.mozilla.org/pub/firefox/releases/80.0.1/linux-x86_64/en-US/firefox-80.0.1.tar.bz2
RUN tar -xjf firefox-80.0.1.tar.bz2
RUN mv firefox /opt/firefox80
RUN ln -s /opt/firefox80/firefox /usr/bin/firefox
RUN ls /opt/firefox80
RUN wget -N https://github.com/mozilla/geckodriver/releases/download/v0.27.0/geckodriver-v0.27.0-linux64.tar.gz
RUN tar -xvzf geckodriver-v0.27.0-linux64.tar.gz
RUN chmod +x geckodriver
RUN mv -f geckodriver /usr/local/share/geckodriver
RUN ln -s /usr/local/share/geckodriver /usr/local/bin/geckodriver
RUN ln -s /usr/local/share/geckodriver /usr/bin/geckodriver
RUN mkdir capybara
WORKDIR /capybara/
COPY . /capybara
RUN bundle install
main.rb
require 'capybara'
require 'capybara/dsl'
require 'selenium-webdriver'
include Capybara::DSL
Capybara.register_driver :selenium_headless_firefox do |app|
browser_options = ::Selenium::WebDriver::Firefox::Options.new()
browser_options.args << '--headless'
Capybara::Selenium::Driver.new(
app,
browser: :firefox,
options: browser_options
)
end
target = "https://maps.google.com/?cid=13666314335012854449"
session = Capybara::Session.new(:selenium_headless_firefox)
session.visit(target)
Gemfile
source 'https://rubygems.org'
gem 'selenium-webdriver'
gem 'capybara', '~>3.30'
gem 'geckodriver-helper'
Error Message on Docker
/usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:72:in `assert_ok': invalid argument: can't kill an exited process (Selenium::WebDriver::Error::UnknownError)
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:34:in `initialize'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `new'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `create_response'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/default.rb:114:in `request'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:64:in `call'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:167:in `execute'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:102:in `create_session'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/firefox/marionette/driver.rb:44:in `initialize'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/firefox/driver.rb:33:in `new'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/firefox/driver.rb:33:in `new'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/driver.rb:54:in `for'
from /usr/local/bundle/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver.rb:88:in `for'
from /usr/local/bundle/gems/capybara-3.33.0/lib/capybara/selenium/driver.rb:52:in `browser'
from /usr/local/bundle/gems/capybara-3.33.0/lib/capybara/selenium/driver.rb:71:in `visit'
from /usr/local/bundle/gems/capybara-3.33.0/lib/capybara/session.rb:278:in `visit'
This is what I get when I run the main.rb file on a docker container. I am looking forward to any help from the developer community.
I ran the main.rb file by docker run [docker_image] ruby main.rb

Answer 1

The issue is not with Capybara, but with Firefox - the tar.bz2 file you download does not contain its dependencies, which causes it to crash. The easiest solution is to install it via apt. Assuming all your files are in the same directory your Dockerfile should look like:
FROM ruby:2.6.6
WORKDIR /app
COPY . .
RUN apt-get update -y && \
apt-get install -y xvfb firefox-esr && \
wget -N https://github.com/mozilla/geckodriver/releases/download/v0.27.0/geckodriver-v0.27.0-linux64.tar.gz && \
tar -xvzf geckodriver-v0.27.0-linux64.tar.gz && \
chmod +x geckodriver && \
mv -f geckodriver /usr/local/share/geckodriver && \
ln -s /usr/local/share/geckodriver /usr/local/bin/geckodriver && \
ln -s /usr/local/share/geckodriver /usr/bin/geckodriver && \
bundle install && \
apt-get clean && \
rm geckodriver-v0.27.0-linux64.tar.gz && \
rm -rf /var/lib/apt/lists/*
CMD [ "ruby", "/app/main.rb" ]
Then you can run:
docker build -t capybara:latest . # Build image
docker run -it --rm --env DISPLAY=$DISPLAY --volume="$HOME/.Xauthority:/root/.Xauthority:rw" --net=host capybara:latest firefox # Verify Firefox works
docker run -it --rm capybara:latest # Run your script
Note: The second command will only work on Linux, running dockerized Linux GUI applications on Windows is a bit more difficult and requires some additional setup.
Edit:
There is no such thing as installing something
"on Docker". Docker is not an OS. It's an application containerization framework. It can run various operating systems inside the containers (or no OS at all - see base image). Thhat means method of installing something inside a Docker image (or container - not recommended) depends on what's already installed.
In this case your base image ruby:2.6.6 is based on a Debian Buster image (see Dockerfile), so you need to install the browser you need the way you would on regular desktop or server install of the system.
Debian Buster does not come with Chrome, because it's not open source. To install it's open source equivalent - Chromium - modify your Dockerfile as follows:
FROM ruby:2.6.6
WORKDIR /app
COPY . .
RUN apt-get update -y && \
apt-get install -y xvfb chromium && \
wget -N https://github.com/mozilla/geckodriver/releases/download/v0.27.0/geckodriver-v0.27.0-linux64.tar.gz && \
tar -xvzf geckodriver-v0.27.0-linux64.tar.gz && \
chmod +x geckodriver && \
mv -f geckodriver /usr/local/share/geckodriver && \
ln -s /usr/local/share/geckodriver /usr/local/bin/geckodriver && \
ln -s /usr/local/share/geckodriver /usr/bin/geckodriver && \
bundle install && \
apt-get clean && \
rm geckodriver-v0.27.0-linux64.tar.gz && \
rm -rf /var/lib/apt/lists/*
CMD [ "ruby", "/app/main.rb" ]
If you really need Chrome, follow the official documentation (keeping in mind you need to remove archive files after installation). With that said the Dockerfile for Chrome would be:
FROM ruby:2.6.6
WORKDIR /app
COPY . .
RUN apt-get update -y && \
apt-get install -y xvfb && \
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb && \
apt install -y ./google-chrome-stable_current_amd64.deb && \
wget -N https://github.com/mozilla/geckodriver/releases/download/v0.27.0/geckodriver-v0.27.0-linux64.tar.gz && \
tar -xvzf geckodriver-v0.27.0-linux64.tar.gz && \
chmod +x geckodriver && \
mv -f geckodriver /usr/local/share/geckodriver && \
ln -s /usr/local/share/geckodriver /usr/local/bin/geckodriver && \
ln -s /usr/local/share/geckodriver /usr/bin/geckodriver && \
bundle install && \
apt-get clean && \
rm google-chrome-stable_current_amd64.deb && \
rm geckodriver-v0.27.0-linux64.tar.gz && \
rm -rf /var/lib/apt/lists/*
CMD [ "ruby", "/app/main.rb" ]