Trying to build a docker image with the execution of a pre-requisites installation script inside the Dockerfile fails for fetching packages via apt-get from archive.ubuntu.com.
Using the apt-get command inside the Dockerfile works flawless, despite being behind a corporate proxy, which is setup via the ENV command in the Dockerfile.
Anyway, executing the apt-get command from a bash-script in a terminal inside the resulting docker container or as "postCreateCommand" in a devcontainer.json of Visual Studio Code does work as expected too. But it won't work in my case for the invocation of a bash script from inside a Dockerfile.
It simply will tell:
Starting installation of package iproute2
Reading package lists...
Building dependency tree...
The following additional packages will be installed:
libatm1 libcap2 libcap2-bin libmnl0 libpam-cap libxtables12
Suggested packages:
iproute2-doc
The following NEW packages will be installed:
iproute2 libatm1 libcap2 libcap2-bin libmnl0 libpam-cap libxtables12
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 971 kB of archives.
After this operation, 3,287 kB of additional disk space will be used.
Err:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libcap2 amd64 1:2.32-1
Could not resolve 'archive.ubuntu.com'
... more output ...
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libc/libcap2/libcap2_2.32-1_amd64.deb Could not resolve 'archive.ubuntu.com'
... more output ...
Just for example a snippet of the Dockerfile looks like this:
FROM ubuntu:20.04 as builderImage
USER root
ARG HTTP_PROXY_HOST_IP='http://172.17.0.1'
ARG HTTP_PROXY_HOST_PORT='3128'
ARG HTTP_PROXY_HOST_ADDR=$HTTP_PROXY_HOST_IP':'$HTTP_PROXY_HOST_PORT
ENV http_proxy=$HTTP_PROXY_HOST_ADDR
ENV https_proxy=$http_proxy
ENV HTTP_PROXY=$http_proxy
ENV HTTPS_PROXY=$http_proxy
ENV ftp_proxy=$http_proxy
ENV FTP_PROXY=$http_proxy
# it is always helpful sorting packages alpha-numerically to keep the overview ;)
RUN apt-get update && \
apt-get -y upgrade && \
apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
&& \
apt-get -y install \
default-jdk \
git \
python3 python3-pip
SHELL ["/bin/bash", "-c"]
ADD ./env-setup.sh .
RUN chmod +x env-setup.sh && ./env-setup.sh
CMD ["bash"]
The minimal version of the environment script env-setup.sh, which is supposed to be invoked by the Dockerfile, would look like this:
#!/bin/bash
packageCommand="apt-get";
sudo $packageCommand update;
packageInstallCommand="$packageCommand install";
package="iproute2"
packageInstallCommand+=" -y";
sudo $packageInstallCommand $package;
Of course the usage of variables is down to making use of a list for the packages to be installed and other aspects.
Hopefully that has covered everything essential to the question:
Why is the execution of apt-get working with a RUN and as well running the bash script inside the container after creating, but not from the very same bash script while building the image from a Dockerfile?
I was hoping to find the answer with the help of an extensive web-search, but unfortunately I was only able to find anything but an answer to this case.
As pointed out in the comment section underneath the question:
using sudo to launch the command, wiping out all the current vars set in the current environment, more specifically your proxy settings
So that is the case.
The solution is either to remove sudo from the bash script and invoke the script as root inside the Dockerfile.
Or, using sudo will work with ENV variables, just apply sudo -E.
I am entirely new to the concept of dockers. I am creating the following Dockerfile as an exercise.
FROM ubuntu:latest
MAINTAINER kesarling
RUN apt update && apt upgrade -y
RUN apt install nginx curl zip unzip -y
RUN apt install openjdk-14-jdk python3 python3-doc clang golang-go gcc g++ -y
RUN curl -s "https://get.sdkman.io" | bash
RUN bash /root/.sdkman/bin/sdkman-init.sh
RUN sdk version
RUN yes | bash -c 'sdk install kotlin'
CMD [ "echo","The development environment has now been fully setup with C, C++, JAVA, Python3, Go and Kotlin" ]
I am using SDKMAN! to install Kotlin. The problem initially was that instead of using RUN bash /root/.sdkman/bin/sdkman-init.sh, I was using RUN source /root/.sdkman/bin/sdkman-init.sh. However, it gave the error saying source not found. So, I tried using RUN . /root/.sdkman/bin/sdkman-init.sh, and it did not work. However, RUN bash /root/.sdkman/bin/sdkman-init.sh seems to work, as in does not give any error and tries to run the next command. However, the docker then gives error saying sdk: not found
Where am I going wrong?
It should be noted that these steps worked like charm for my host distribution (The one on which I'm running docker) which is Pop!_OS 20.04
Actually the script /root/.sdkman/bin/sdkman-init.sh sources the sdk
source is a built-in to bash rather than a binary somewhere on the filesystem.
source command executes the file in the current shell.
Each RUN instruction will execute any commands in a new layer on top of the current image and commit the results.
The resulting committed image will be used for the next step in the Dockerfile.
Try this:
FROM ubuntu:latest
MAINTAINER kesarling
RUN apt update && apt upgrade -y
RUN apt install nginx curl zip unzip -y
RUN apt install openjdk-14-jdk python3 python3-doc clang golang-go gcc g++ -y
RUN curl -s "https://get.sdkman.io" | bash
RUN /bin/bash -c "source /root/.sdkman/bin/sdkman-init.sh; sdk version; sdk install kotlin"
CMD [ "echo","The development environment has now been fully setup with C, C++, JAVA, Python3, Go and Kotlin" ]
SDKMAN in Ubuntu Dockerfile
tl;dr
the sdk command is not a binary but a bash script loaded into memory
Shell sessions are a "process", which means environment variables and declared shell function only exist for the duration that shell session exists; which lasts only as long as the RUN command.
Manually tweak your PATH
RUN apt-get update && apt-get install curl bash unzip zip -y
RUN curl -s "https://get.sdkman.io" | bash
RUN source "$HOME/.sdkman/bin/sdkman-init.sh" \
&& sdk install java 8.0.275-amzn \
&& sdk install sbt 1.4.2 \
&& sdk install scala 2.12.12
ENV PATH=/root/.sdkman/candidates/java/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/scala/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/sbt/current/bin:$PATH
Full Version
Oh wow this was a journey to figure out. Below each line is commented as to why certain commands are run.
I learnt a lot about how unix works and how sdkman works and how docker works and why the intersection of the three give very unusual behaviour.
# I am using a multi-stage build so I am just copying the built artifacts
# from this stage to keep final image small.
FROM ubuntu:latest as ScalaBuild
# Switch from `sh -c` to `bash -c` as the shell behind a `RUN` command.
SHELL ["/bin/bash", "-c"]
# Usual updates
RUN apt-get update && apt-get upgrade -y
# Dependencies for sdkman installation
RUN apt-get install curl bash unzip zip -y
#Install sdkman
RUN curl -s "https://get.sdkman.io" | bash
# FUN FACTS:
# 1) the `sdk` command is not a binary but a bash script loaded into memory
# 2) Shell sessions are a "process", which means environment variables
# and declared shell function only exist for
# the duration that shell session exists
RUN source "$HOME/.sdkman/bin/sdkman-init.sh" \
&& sdk install java 8.0.275-amzn \
&& sdk install sbt 1.4.2 \
&& sdk install scala 2.12.12
# Once the real binaries exist these are
# the symlinked paths that need to exist on PATH
ENV PATH=/root/.sdkman/candidates/java/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/scala/current/bin:$PATH
ENV PATH=/root/.sdkman/candidates/sbt/current/bin:$PATH
# This is specific to running a minimal empty Scala project and packaging it
RUN touch build.sbt
RUN sbt compile
RUN sbt package
FROM alpine AS production
# setup production environment image here
COPY --from=ScalaBuild /root/target/scala-2.12/ $INSTALL_PATH
ENTRYPOINT ["java", "-cp", "$INSTALL_PATH", "your.main.classfile"]
Generally you want to avoid using "version manager" type tools in Docker; it's better to install a specific version of the compiler or runtime you need.
In the case of Kotlin, it's a JVM application distributed as a zip file so it should be fairly easy to install:
FROM openjdk:15-slim
ARG KOTLIN_VERSION=1.3.72
# Get OS-level updates:
RUN apt-get update \
&& apt-get install --no-install-recommends --assume-yes \
curl \
unzip
# and if you need C/Python dependencies, those too
# Download and unpack Kotlin
RUN cd /opt \
&& curl -LO https://github.com/JetBrains/kotlin/releases/download/v${KOTLIN_VERSION}/kotlin-compiler-${KOTLIN_VERSION}.zip \
&& unzip kotlin-compiler-${KOTLIN_VERSION}.zip \
&& rm kotlin-compiler-${KOTLIN_VERSION}.zip
# Add its directory to $PATH
ENV PATH=/opt/kotlinc/bin:$PATH
The real problem with version managers is that they heavily depend on the tool setting environment variables. As #JeevanRao notes in their answer, each Dockerfile RUN command runs in a separate shell in a separate container, and any environment variable settings within that command get lost for the next command.
# Does absolutely nothing: environment variables do not stay set
RUN . /root/.sdkman/bin/sdkman-init.sh
Since an image generally contains only one application and its runtime, you don't need the ability to change which version of the runtime or compiler you're using. My Dockerfile example passes it as an ARG, so you can change it in the Dockerfile or pass a docker build --build-arg KOTLIN_VERSION=... option to use a different version.
I'm writing a basic script in bash to automatically install a few packages on new machines (basically, the first script to run on a new machine before doing anything else). I'm downloading openssh-server and python for example.
My goal is just to launch the script in sudo mode, wait a few minutes and get started. The only problem I'm facing is the machine requires the user to input "y" when needed and I can't find a way to bypass this.
This is just a sample of my script :
#!/bin/bash
sudo apt update
sudo apt install openssh-server
sudo apt-get install python3
what I expect is just run this command and let it roll without the user to have and put "y" when needed
sudo bash start-script.sh
Use the -y option of apt-get. From the apt-get man page:
-y, --yes, --assume-yes
Automatic yes to prompts; assume "yes" as answer to all prompts and
run non-interactively. If an undesirable situation, such as
changing a held package, trying to install a unauthenticated
package or removing an essential package occurs then apt-get will
abort. Configuration Item: APT::Get::Assume-Yes.
After I finally got lxml installed on my computer by downloading the .whl file from http://www.lfd.uci.edu/~gohlke/pythonlibs/, I thought my lxml installation problems were behind me. But, clearly I was very wrong. It does not seem to be any easier on the EC2 platform. I have tried just about every command I can think of—sudo pip3/pip3.4/pip-3.4/easy_install-3.4 lxml/lxml-3.5.0-cp34-none-win32.whl/lxml-3.5.0-cp34-none-win32.whl, STATIC_DEPS=true sudo pip3/pip3.4/pip-3.4/easyinstall-3.4 install lxml. I keep getting the Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed? error message.
I have tried to install these libraries too, but so far not successfully. Any help with installing either lxml or the dependent libraries would be greatly appreciated.
I was eventually able to install libxml2 by executing the following procedure:
Download the tar.gz. file from http://www.linuxfromscratch.org/blfs/view/7.7/general/libxml2.html, transfer it to the EC2 instance, and run tar -zxvf libxml2-2.9.2.tar.gz from the directory in which the file is located.
Run sudo yum install libxml2-devel libxslt-devel python34-devel. (The reason why I was unable to install these packages at first was that I used the omnipresent “-dev” wording, which did not work in this particular setup.)
From the libxml2-2.9.2 folder (as per the instructions at http://www.linuxfromscratch.org/blfs/view/7.7/general/libxml2.html) run
sed \
-e /xmlInitializeCatalog/d \
-e 's/((ent->checked =.*&&/(((ent->checked == 0) ||\
((ent->children == NULL) \&\& (ctxt->options \& XML_PARSE_NOENT))) \&\&/' \
-i parser.c
,
./configure --prefix=/usr --disable-static --with-history --with-python=/usr/bin/python3 && make
, and
make install
Then . . . , when I finally ran sudo easy_install-3.4 lxml, guess what . . . ; yes, the instance ran out of memory. So, I had to apply the following fix from How do you add swap to an EC2 instance?:
sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
sudo /sbin/mkswap /var/swap.1
sudo /sbin/swapon /var/swap.1
After that, it took quite a while to install using sudo easy_install-3.4 lxml, but it worked.
While running
./configure --prefix=/mingw
on a MinGW/MSYS system for a library I had previously run
'./configure --prefix=/mingw && make && make install'
I came across this message:
WARNING: A version of the Vamp plugin SDK is already installed. Expect worries and sorrows if you install a new version without removing the old one first. (Continuing)
This had me worried. What's the opposite of 'make install', i.e. how is a library uninstalled in Linux? Will 'make clean' do the job, or are there other steps involved?
make clean removes any intermediate or output files from your source / build tree. However, it only affects the source / build tree; it does not touch the rest of the filesystem and so will not remove previously installed software.
If you're lucky, running make uninstall will work. It's up to the library's authors to provide that, however; some authors provide an uninstall target, others don't.
If you're not lucky, you'll have to manually uninstall it. Running make -n install can be helpful, since it will show the steps that the software would take to install itself but won't actually do anything. You can then manually reverse those steps.
If sudo make uninstall is unavailable:
In a Debian based system, instead of (or after*) doing make install you can run sudo checkinstall to make a .deb file that gets automatically installed. You can then remove it using the system package manager (e.g. apt/synaptic/aptitude/dpkg). Checkinstall also supports creating other types of package, e.g. RPM.
See also http://community.linuxmint.com/tutorial/view/162 and some basic checkinstall usage and debian checkinstall package.
*: If you're reading this after having installed with make install you can still follow the above instructions and do a dpkg -r $PACKAGE_NAME_YOU_CHOSEN afterwards.
If you have a manifest file which lists all the files that were installed with make install you can run this command which I have from another answer:
cat install_manifest.txt | xargs echo rm | sh
If you have sudo make install you will need to add a sudo to your uninstall:
cat install_manifest.txt | xargs echo sudo rm | sh
How to uninstall after "make install"
Method #1 (make uninstall)
Step 1: You only need to follow this step if you've deleted/altered the build directory in any way: Download and make/make install using the exact same procedure as you did before.
Step 2: try make uninstall.
cd $SOURCE_DIR
sudo make uninstall
If this succeeds you are done. If you're paranoid you may also try the steps of "Method #3" to make sure make uninstall didn't miss any files.
Method #2 (checkinstall -- only for debian based systems)
Overview of the process
In debian based systems (e.g. Ubuntu) you can create a .deb package very easily by using a tool named checkinstall. You then install the .deb package (this will make your debian system realize that the all parts of your package have been indeed installed) and finally uninstall it to let your package manager properly cleanup your system.
Step by step
sudo apt-get -y install checkinstall
cd $SOURCE_DIR
sudo checkinstall
At this point checkinstall will prompt for a package name. Enter something a bit descriptive and note it because you'll use it in a minute. It will also prompt for a few more data that you can ignore. If it complains about the version not been acceptable just enter something reasonable like 1.0. When it completes you can install and finally uninstall:
sudo dpkg -i $PACKAGE_NAME_YOU_ENTERED
sudo dpkg -r $PACKAGE_NAME_YOU_ENTERED
Method #3 (install_manifest.txt)
If a file install_manifest.txt exists in your source dir it should contain the filenames of every single file that the installation created.
So first check the list of files and their mod-time:
cd $SOURCE_DIR
sudo xargs -I{} stat -c "%z %n" "{}" < install_manifest.txt
You should get zero errors and the mod-times of the listed files should be on or after the installation time. If all is OK you can delete them in one go:
cd $SOURCE_DIR
mkdir deleted-by-uninstall
sudo xargs -I{} mv -t deleted-by-uninstall "{}" < install_manifest.txt
User Merlyn Morgan-Graham however has a serious notice regarding this method that you should keep in mind (copied here verbatim): "Watch out for files that might also have been installed by other packages. Simply deleting these files [...] could break the other packages.". That's the reason that we've created the deleted-by-uninstall dir and moved files there instead of deleting them.
99% of this post existed in other answers. I just collected everything useful in a (hopefully) easy to follow how-to and tried to give extra attention to important details (like quoting xarg arguments and keeping backups of deleted files).
Depending on how well the makefile/configure script/autofoo magic of the program in question is the following might solve your problem:
make uninstall
The problem is that you should execute this on the source tree of the version you've got installed and with exactly the same configuration that you used for installing.
make clean generally only cleans built files in the directory containing the source code itself, and rarely touches any installed software.
Makefiles generally don't contain a target for uninstallation -- you usually have to do that yourself, by removing the files from the directory into which they were installed. For example, if you built a program and installed it (using make install) into /usr/local, you'd want to look through /usr/local/bin, /usr/local/libexec, /usr/local/share/man, etc., and remove the unwanted files. Sometimes a Makefile includes an uninstall target, but not always.
Of course, typically on a Linux system you install software using a package manager, which is capable of uninstalling software "automagically".
The "stow" utility was designed to solve this problem: http://www.gnu.org/software/stow/
There is no standard unfortunately, this is one of the perils of installing from source. Some Makefiles will include an "uninstall", so
make uninstall
from the source directory may work. Otherwise, it may be a matter of manually undoing whatever the make install did.
make clean usually just cleans up the source directory - removing generated/compiled files and the like, probably not what you're after.
Make
Make is the program that’s used to install the program that’s compiled from the source code. It’s not the Linux package manager so it doesn’t keep track of the files it installs. This makes it difficult to uninstall the files afterward.
The Make Install command copies the built program and packages into the library directory and specified locations from the makefile. These locations
can vary based on the examination that’s performed by the configure script.
CheckInstall
CheckInstall is the program that’s used to install or uninstall programs that are compiled from the source code. It monitors and copies the files that are installed using the make program. It also installs the files using the Linux package manager which allows it to be uninstalled like any regular package.
The CheckInstall command is used to call the Make Install command. It monitors the files that are installed and creates a binary package from them. It also installs the binary package with the Linux package manager.
Replace "source_location.deb" and "name" with your information from the Screenshot.
Execute the following commands in the source package directory:
Install CheckInstall sudo apt-get install checkinstall
Run the Configure script sudo ./configure
Run the Make command sudo make
Run CheckInstall sudo checkinstall
Reinstall the package sudo dpkg --install --force-overwrite source_location.deb
Remove the package sudo apt remove name
Here's an article article I wrote that covers the whole process with explanations.
Method 1
From the source folder:
#make uninstall
Method 2
If there is no uninstall procedure:
open install_manifest.txt (created by #make install)
remove all the directories/files listed
remove any remaining files you missed:
#xargs rm < install_manifest.txt
remove any hidden directories/files:
$rm -rf ~/.packagename
Remove the source folder.
Method 3
If none of the above options work, view the install procedure:
#make -n install
and reverse the install procedure:
#rm -rf all directories/files created
Example
For example, this is how to uninstall nodejs, npm, and nvm from source:
How do I completely uninstall Node.js, and reinstall from beginning (Mac OS X)
which you can do using any of the above methods.
I know of few packages that support "make uninstall" but many more that support make install DESTDIR=xxx" for staged installs.
You can use this to create a package which you install instead of installing directly from the source. I had no luck with checkinstall but fpm works very well.
This can also help you remove a package previously installed using make install. You simply force install your built package over the make installed one and then uninstall it.
For example, I used this recently to deal with protobuf-3.3.0.
On RHEL7:
make install DESTDIR=dest
cd dest
fpm -f -s dir -t rpm -n protobuf -v 3.3.0 \
--vendor "You Not RedHat" \
--license "Google?" \
--description "protocol buffers" \
--rpm-dist el7 \
-m you#youraddress.com \
--url "http:/somewhere/where/you/get/the/package/oritssource" \
--rpm-autoreqprov \
usr
sudo rpm -i -f protobuf-3.3.0-1.el7.x86_64.rpm
sudo rpm -e protobuf-3.3.0
Prefer yum to rpm if you can.
On Debian9:
make install DESTDIR=dest
cd dest
fpm -f -s dir -t deb -n protobuf -v 3.3.0 \
-C `pwd` \
--prefix / \
--vendor "You Not Debian" \
--license "$(grep Copyright ../../LICENSE)" \
--description "$(cat README.adoc)" \
--deb-upstream-changelog ../../CHANGES.txt \
--url "http:/somewhere/where/you/get/the/package/oritssource" \
usr/local/bin \
usr/local/lib \
usr/local/include
sudo apt install -f *.deb
sudo apt-get remove protobuf
Prefer apt to dpkg where you can.
I've also posted answer this here
Make can tell you what it knows and what it will do.
Suppose you have an "install" target, which executes commands like:
cp <filelist> <destdir>/
In your generic rules, add:
uninstall :; MAKEFLAGS= ${MAKE} -j1 -spinf $(word 1,${MAKEFILE_LIST}) install \
| awk '/^cp /{dest=$NF; for (i=NF; --i>0;) {print dest"/"$i}}' \
| xargs rm -f
A similar trick can do a generic make clean.
Preamble
below may work or may not, this is all given as-is, you and only you are responsible person in case of some damage, data loss and so on. But I hope things go smooth!
To undo make install I would do (and I did) this:
Idea: check whatever script installs and undo this with simple bash script.
Reconfigure your build dir to install to some custom dir. I usually do this: --prefix=$PWD/install. For CMake, you can go to your build dir, open CMakeCache.txt, and fix CMAKE_INSTALL_PREFIX value.
Install project to custom directory (just run make install again).
Now we push from assumption, that make install script installs into custom dir just same contents you want to remove from somewhere else (usually /usr/local). So, we need a script.
3.1. Script should compare custom dir, with dir you want clean. I use this:
anti-install.sh
RM_DIR=$1
PRESENT_DIR=$2
echo "Remove files from $RM_DIR, which are present in $PRESENT_DIR"
pushd $RM_DIR
for fn in `find . -iname '*'`; do
# echo "Checking $PRESENT_DIR/$fn..."
if test -f "$PRESENT_DIR/$fn"; then
# First try this, and check whether things go plain
echo "rm $RM_DIR/$fn"
# Then uncomment this, (but, check twice it works good to you).
# rm $RM_DIR/$fn
fi
done
popd
3.2. Now just run this script (it will go dry-run)
bash anti-install.sh <dir you want to clean> <custom installation dir>
E.g. You wan't to clean /usr/local, and your custom installation dir is /user/me/llvm.build/install, then it would be
bash anti-install.sh /usr/local /user/me/llvm.build/install
3.3. Check log carefully, if commands are good to you, uncomment rm $RM_DIR/$fn and run it again. But stop! Did you really check carefully? May be check again?
Source to instructions:
https://dyatkovskiy.com/2019/11/26/anti-make-install/
Good luck!