I know questions with such an error have already been asked several times, however I couldn't find a solution so I'm assuming that my case is different.
I have a Dockerfile like this:
FROM alpine:latest as builder
RUN apk add --no-cache bash
COPY bin /main/bin
COPY conf /main/conf
RUN ./main/bin/my-script 1
FROM mysql:5.7
COPY --from=builder /main/dist/sql /sql
ADD generate-databases.sh /docker-entrypoint-initdb.d
and this is working absolutely fine both on Linux, and on macOS(using Docker Desktop).
The problem is that on Windows using Docker Desktop I get the following error:
Service 'db' failed to build: The command '/bin/sh -c ./main/bin/my-script 1' returned a non-zero code: 127
why does this happen only on Windows, while working fine in other platforms?
The problem was actually not strictly related with Docker but with the line-ending format of my bash scripts, which on Windows were CRLF by default while they should have been LF.
To solve the issue I forced LF format in my .gitattributes.
I have following Dockerfile:
FROM ubuntu:16.04
ARG path1=def_path1
RUN mkdir ${path1}
When I build this Dockerfile using following command:
docker build --build-arg path1=/home/dragan -t build_arg_ex .
I get following error when I execute it in MINGW bash on Windows 10:
$ ./build.sh --no-cache
Sending build context to Docker daemon 6.144kB
Step 1/3 : FROM ubuntu:16.04
---> 2a4cca5ac898
Step 2/3 : ARG path1=def_path1
---> Running in a35241ebdef3
Removing intermediate container a35241ebdef3
---> 01475e50af4c
Step 3/3 : RUN mkdir ${path1}
---> Running in 2759e683cbb1
mkdir: cannot create directory 'C:/Program': No such file or directory
mkdir: cannot create directory 'Files/Git/home/dragan': No such file or
directory
The command '/bin/sh -c mkdir ${path1}' returned a non-zero code: 1
Building same Dockerfile in Windows Command Prompt or on Linux or Mac is ok. The problem is only in MINGW bash terminal on Windows because it adds 'C:/Program Files/Git' before the path that is passed as argument.
Is there a way to execute this in MINGW bash so it does not add the 'C:/Program Files/Git' prefix?
Thanks
This is actually a bug/limitation of Git for Windows as described in the Release Notes under Known issues:
If you specify command-line options starting with a slash, POSIX-to-Windows path conversion will kick in converting e.g. "/usr/bin/bash.exe" to "C:\Program Files\Git\usr\bin\bash.exe". When that is not desired -- e.g. "--upload-pack=/opt/git/bin/git-upload-pack" or "-L/regex/" -- you need to set the environment variable MSYS_NO_PATHCONV temporarily, like so:
MSYS_NO_PATHCONV=1 git blame -L/pathconv/ msys2_path_conv.cc
Alternatively, you can double the first slash to avoid POSIX-to-Windows path conversion, e.g. "//usr/bin/bash.exe".
Further to #mat007's answer:
This bash function solved the problem more permanently for docker, without enabling MSYS_NO_PATHCONV globally, which causes another world of pain.
.bashrc
# See https://github.com/docker/toolbox/issues/673#issuecomment-355275054
# Workaround for Docker for Windows in Git Bash.
docker()
{
(export MSYS_NO_PATHCONV=1; "docker.exe" "$#")
}
You may need to do the same for docker-compose
I have created a docker image which contains the following CMD:
CMD ["sh", "start.sh"]
When I run the docker image I use the following command inside a Makefile
docker run --rm -v ${PWD}:/selenium $(DOCKER_IMAGE)
which copies the files from the current (host-)directory to the docker's /selenium folder. The files include files for selenium tests, as well as the file start.sh. But after the container has started, I get immediately the error
"sh: 0: Can't open start.sh"
Maybe the host volume is mounted inside docker after the command has been run? Anything else that can explain this error, and how to fix it?
Maybe there is a way to run more than one command inside docker to see whats going on? Like
CMD ["ls", ";", "pwd", ";", "sh", "start.sh"]
Update
when I use the following command i the Dockerfile
CMD ["ls"]
I get the error
ls: cannot open directory '.': Permission denied
Extra information
Docker version 1.12.6
Entrypoint: WORKDIR /work
Your mounting your volume to the /selenium folder in your container. Therefor the start.sh file isn't going to be in your working directory its going to be in /selenium. You want to mount your volume to a selenium folder inside your working directory then make sure the command references this new path.
If you use docker-compose the YAML-file to run the container would look something like this:
version: '3'
services:
start:
image: ${DOCKER_IMAGE}
command: sh selenium/start.sh
volumes:
- .:/work/selenium
If you try and perform each step manually, using docker run with bash,
docker exec -it (container name) /bin/bash
It will be more easier and quicker to look at the errors, and you can change the permissions, view where the file is located, before running the .sh file and try again.
Check the permission using ls -l.
Give the permission 777 using sudo chmod 777 file_name.
Repeat for other files you might find.
Running Docker Toolbox on Windows 10 host.
There is a Dockerfile:
FROM 16.04
...
RUN if [ some_condition ]; then ./foo.sh; fi
...
There is a foo.sh:
#!/bin/bash
...
echo 'Me working'
Now when trying to build the Docker image:
docker build -t name_of_the_image .
Getting error:
Step 7/12 : RUN ./foo.sh
---> Running in e7e0703d3f8f
/bin/sh: 1: ./foo.sh: not found
The command '/bin/sh -c ./foo.sh' returned a non-zero code: 127
I would assume error 127 would be the Docker doesn't see the bash. Any suggestion how to fix this?
Edit: already copying all files into the Docker, Dockerfile:
FROM ubuntu:16.04
MAINTAINER Mr Anderson "mr#anderson.com"
# set workdir
COPY . /app
WORKDIR /app
# Run scripts
RUN ./foo.sh
You'll need to copy/COPY the file into the container before you can execute/RUN the script.
Also since you're using a relative path when you call the script be sure to set a WORKDIR.
COPY ./foo.sh /app/foo.sh
WORKDIR /dir
RUN chmod +x /app/foo.sh
RUN if [ some_condition ]; then ./foo.sh; fi
Also make sure the script is executable.
After some further investigation:
Using CMD over RUN is not a perfect solution because of the way those commands work. RUN can be used any amount of times, to build Docker image layer by layer, while CMD can be executed only once when the image has been build.
In my case the solution was to:
Open ./foo.sh file with VIM and run: :set fileformat=unix and save the file.
Long story short: the line ending in the shell-script were incorrect and had to be converted to the Unix ones.
I'm trying to build a new Docker image for our development process, using cpanm to install a bunch of Perl modules as a base image for various projects.
While developing the Dockerfile, cpanm returns a failure code because some of the modules did not install cleanly.
I'm fairly sure I need to get apt to install some more things.
Where can I find the /.cpanm/work directory quoted in the output, in order to inspect the logs? In the general case, how can I inspect the file system of a failed docker build command?
After running a find I discovered
/var/lib/docker/aufs/diff/3afa404e[...]/.cpanm
Is this reliable, or am I better off building a "bare" container and running stuff manually until I have all the things I need?
Everytime docker successfully executes a RUN command from a Dockerfile, a new layer in the image filesystem is committed. Conveniently you can use those layers ids as images to start a new container.
Take the following Dockerfile:
FROM busybox
RUN echo 'foo' > /tmp/foo.txt
RUN echo 'bar' >> /tmp/foo.txt
and build it:
$ docker build -t so-26220957 .
Sending build context to Docker daemon 47.62 kB
Step 1/3 : FROM busybox
---> 00f017a8c2a6
Step 2/3 : RUN echo 'foo' > /tmp/foo.txt
---> Running in 4dbd01ebf27f
---> 044e1532c690
Removing intermediate container 4dbd01ebf27f
Step 3/3 : RUN echo 'bar' >> /tmp/foo.txt
---> Running in 74d81cb9d2b1
---> 5bd8172529c1
Removing intermediate container 74d81cb9d2b1
Successfully built 5bd8172529c1
You can now start a new container from 00f017a8c2a6, 044e1532c690 and 5bd8172529c1:
$ docker run --rm 00f017a8c2a6 cat /tmp/foo.txt
cat: /tmp/foo.txt: No such file or directory
$ docker run --rm 044e1532c690 cat /tmp/foo.txt
foo
$ docker run --rm 5bd8172529c1 cat /tmp/foo.txt
foo
bar
of course you might want to start a shell to explore the filesystem and try out commands:
$ docker run --rm -it 044e1532c690 sh
/ # ls -l /tmp
total 4
-rw-r--r-- 1 root root 4 Mar 9 19:09 foo.txt
/ # cat /tmp/foo.txt
foo
When one of the Dockerfile command fails, what you need to do is to look for the id of the preceding layer and run a shell in a container created from that id:
docker run --rm -it <id_last_working_layer> bash -il
Once in the container:
try the command that failed, and reproduce the issue
then fix the command and test it
finally update your Dockerfile with the fixed command
If you really need to experiment in the actual layer that failed instead of working from the last working layer, see Drew's answer.
The top answer works in the case that you want to examine the state immediately prior to the failed command.
However, the question asks how to examine the state of the failed container itself. In my situation, the failed command is a build that takes several hours, so rewinding prior to the failed command and running it again takes a long time and is not very helpful.
The solution here is to find the container that failed:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6934ada98de6 42e0228751b3 "/bin/sh -c './utils/" 24 minutes ago Exited (1) About a minute ago sleepy_bell
Commit it to an image:
$ docker commit 6934ada98de6
sha256:7015687976a478e0e94b60fa496d319cdf4ec847bcd612aecf869a72336e6b83
And then run the image [if necessary, running bash]:
$ docker run -it 7015687976a4 [bash -il]
Now you are actually looking at the state of the build at the time that it failed, instead of at the time before running the command that caused the failure.
Update for newer docker versions 20.10 onwards
Linux or macOS
DOCKER_BUILDKIT=0 docker build ...
Windows
# Command line
set DOCKER_BUILDKIT=0 docker build ...
# PowerShell
$env:DOCKER_BUILDKIT=0
Use
DOCKER_BUILDKIT=0 docker build ...
to get the intermediate container hashes as known from older versions.
On newer versions, Buildkit is activated per default. It is recommended to only use it for debugging purposes. Build Kit can make your build faster.
For reference:
Buildkit doesn't support intermediate container hashes: https://github.com/moby/buildkit/issues/1053
Thanks to #David Callanan and #MegaCookie for their inputs.
Docker caches the entire filesystem state after each successful RUN line.
Knowing that:
to examine the latest state before your failing RUN command, comment it out in the Dockerfile (as well as any and all subsequent RUN commands), then run docker build and docker run again.
to examine the state after the failing RUN command, simply add || true to it to force it to succeed; then proceed like above (keep any and all subsequent RUN commands commented out, run docker build and docker run)
Tada, no need to mess with Docker internals or layer IDs, and as a bonus Docker automatically minimizes the amount of work that needs to be re-done.
Currently with the latest docker-desktop, there isn't a way to opt out
of the new Buildkit, which doesn't support debugging yet (follow the
latest updates on this on this GitHub Thread:
https://github.com/moby/buildkit/issues/1472).
Find out at which line in your Dockerfile it is failing.
Add to the top of your Dockerfile: FROM xxx as debug
Add an additional target: FROM xxx as next just one line before the failing command (as you don't want to build that part). Example:
FROM xxx as debug
RUN echo "working command"
FROM xxx as next
RUN echoo "failing command"
Run docker build -f Dockerfile --target debug --tag debug .
Then you can debug the container with: docker run -it debug /bin/sh
You can quit the shell by pressing CTRL P + CTRL Q
If you want to use docker compose build instead of docker build it's possible by adding target: debug in your docker-compose.yml under build.
Then start the container by docker compose run xxxYourServiceNamexxx and use either:
The second top answer to find out how to run a shell inside the container.
Or add ENTRYPOINT /bin/sh before the FROM xxx as next line in your Dockerfile.
Debugging build step failures is indeed very annoying.
The best solution I have found is to make sure that each step that does real work succeeds, and adding a check after those that fails. That way you get a committed layer that contains the outputs of the failed step that you can inspect.
A Dockerfile, with an example after the # Run DB2 silent installer line:
#
# DB2 10.5 Client Dockerfile (Part 1)
#
# Requires
# - DB2 10.5 Client for 64bit Linux ibm_data_server_runtime_client_linuxx64_v10.5.tar.gz
# - Response file for DB2 10.5 Client for 64bit Linux db2rtcl_nr.rsp
#
#
# Using Ubuntu 14.04 base image as the starting point.
FROM ubuntu:14.04
MAINTAINER David Carew <carew#us.ibm.com>
# DB2 prereqs (also installing sharutils package as we use the utility uuencode to generate password - all others are required for the DB2 Client)
RUN dpkg --add-architecture i386 && apt-get update && apt-get install -y sharutils binutils libstdc++6:i386 libpam0g:i386 && ln -s /lib/i386-linux-gnu/libpam.so.0 /lib/libpam.so.0
RUN apt-get install -y libxml2
# Create user db2clnt
# Generate strong random password and allow sudo to root w/o password
#
RUN \
adduser --quiet --disabled-password -shell /bin/bash -home /home/db2clnt --gecos "DB2 Client" db2clnt && \
echo db2clnt:`dd if=/dev/urandom bs=16 count=1 2>/dev/null | uuencode -| head -n 2 | grep -v begin | cut -b 2-10` | chgpasswd && \
adduser db2clnt sudo && \
echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
# Install DB2
RUN mkdir /install
# Copy DB2 tarball - ADD command will expand it automatically
ADD v10.5fp9_linuxx64_rtcl.tar.gz /install/
# Copy response file
COPY db2rtcl_nr.rsp /install/
# Run DB2 silent installer
RUN mkdir /logs
RUN (/install/rtcl/db2setup -t /logs/trace -l /logs/log -u /install/db2rtcl_nr.rsp && touch /install/done) || /bin/true
RUN test -f /install/done || (echo ERROR-------; echo install failed, see files in container /logs directory of the last container layer; echo run docker run '<last image id>' /bin/cat /logs/trace; echo ----------)
RUN test -f /install/done
# Clean up unwanted files
RUN rm -fr /install/rtcl
# Login as db2clnt user
CMD su - db2clnt
In my case, I have to have:
DOCKER_BUILDKIT=1 docker build ...
and as mentioned by Jannis Schönleber in his answer, there is currently no debug available in this case (i.e. no intermediate images/containers get created).
What I've found I could do is use the following option:
... --progress=plain ...
and then add various RUN ... or additional lines on existing RUN ... to debug specific commands. This gives you what to me feels like full access (at least if your build is relatively fast).
For example, you could check a variable like so:
RUN echo "Variable NAME = [$NAME]"
If you're wondering whether a file is installed properly, you do:
RUN find /
etc.
In my situation, I had to debug a docker build of a Go application with a private repository and it was quite difficult to do that debugging. I've other details on that here.
If you are using docker-compose to build docker images try to add DOCKER_BUILDKIT=0 before the command to see the last successful layer id
DOCKER_BUILDKIT=0 docker-compose ...
This will temporarily disable DOCKER_BUILDKIT for the command only.
Having the last layer id you can connect to it using the command from the top answer
docker run --rm -it LAST_LAYER_ID sh
my solution would be to see what step failed in the docker file, RUN bundle install in my case,
and change it to
RUN bundle install || cat <path to the file containing the error>
This has the double effect of printing out the reason for the failure, AND this intermediate step is not figured as a failed one by docker build. so it's not deleted, and can be inspected via:
docker run --rm -it <id_last_working_layer> bash -il
in there you can even re run your failed command and test it live.
What I would do is comment out the Dockerfile below and including the offending line. Then you can run the container and run the docker commands by hand, and look at the logs in the usual way. E.g. if the Dockerfile is
RUN foo
RUN bar
RUN baz
and it's dying at bar I would do
RUN foo
# RUN bar
# RUN baz
Then
$ docker build -t foo .
$ docker run -it foo bash
container# bar
...grep logs...
Still using BuildKit, as in Alexis Wilke's answer, you can use ktock/buildg.
See "Interactive debugger for Dockerfile" from Kohei Tokunaga
buildg is a tool to interactively debug Dockerfile based on BuildKit.
Source-level inspection
Breakpoints and step execution
Interactive shell on a step with your own debugigng tools
Based on BuildKit (needs unmerged patches)
Supports rootless
Example:
$ buildg.sh debug --image=ubuntu:22.04 /tmp/ctx
WARN[2022-05-09T01:40:21Z] using host network as the default
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 195B done
#2 DONE 0.1s
#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 3.0s
#4 [build1 1/2] FROM docker.io/library/busybox#sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8
#4 resolve docker.io/library/busybox#sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8 0.0s done
#4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 772.81kB / 772.81kB 0.2s done
Filename: "Dockerfile"
2| RUN echo hello > /hello
3|
4| FROM busybox AS build2
=> 5| RUN echo hi > /hi
6|
7| FROM scratch
8| COPY --from=build1 /hello /
>>> break 2
>>> breakpoints
[0]: line 2
>>> continue
#4 extracting sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0.0s done
#4 DONE 0.3s
...