How to run sql scripts against mariaDB docker container - bash

I am trying to create a mariadb instance in docker and then run all the files in a directory against it. I know my script works when I execute it after my dockerfile runs, but when I put the script into the docker file it reports that mariadb got a 127 error. I have tried putting the call to mysqld inside of the script, but that has not fixed the issue.
Dockerfile
ENV MYSQL_ROOT_PASSWORD test
ENV MYSQL_DATABASE mydatabase
COPY . /usr/src
WORKDIR /usr/src
RUN script_runner.sh test
EXPOSE 3306
CMD ["mysqld"]
script_runner.sh
files=`ls start_script | grep ^'do'`
for script in $files
do
mysql -u root --password=$1 < `pwd`/start_script/$script
done
docker-compose.yml
...
mariadb:
build:
context: ./mariaDB
restart: always
ports:
- "3306:3306"
volumes:
- "/var/lib/mysql:/var/lib/mysql"
- "/srv/docker/sockets/mariadb.container.sock:/var/run/mysqld/mysqld"
file system
-repo/
--docker-compose.yml
--mariadb/
---Dockerfile
---script_runner.sh
----start_script/
----do-release.sql
error
Building mariadb
Step 1/8 : FROM mariadb:10.4.11-bionic
---> bc20d5f8d0fe
Step 2/8 : ENV MYSQL_ROOT_PASSWORD test
---> Running in 5987d662632b
Removing intermediate container 5987d662632b
---> e40256430e39
Step 3/8 : ENV MYSQL_DATABASE mydatabase
---> Running in a865ef21cdcc
Removing intermediate container a865ef21cdcc
---> dc5997996fef
Step 4/8 : COPY . /usr/src
---> 5314d67545bb
Step 5/8 : WORKDIR /usr/src
---> Running in 4643fe58e44e
Removing intermediate container 4643fe58e44e
---> 88e7901d501a
Step 6/8 : RUN script_runner.sh test
---> Running in 502ab4fddbb8
/bin/sh: 1: script_runner.sh: not found
ERROR: Service 'mariadb' failed to build: The command '/bin/sh -c script_runner.sh test' returned a non-zero code: 127

Your script_runner.sh file needs executable permissions, otherwise it will not be found as executable. Try setting the attribute in your local folder (chmod 7xx) or inside the Dockerfile
COPY . /usr/src
WORKDIR /usr/src
RUN chmod 711 script_runner.sh
RUN script_runner.sh test

I agree with the comment that says you need to make the script_runner.sh script executable and that would likely be the cause of the 127 return code. However I think you'll find it still won't work even after making it executable.
At the time the RUN script_runner.sh test gets executed, the MySQL server won't actually running yet inside the Docker container during the build process. So if you made the script executable I think you'll see that it'll return a "cannot connect to mysql sock" type error. You'd have to start mysqld as part of your Docker build. Also, remember that a docker build will execute each RUN statement by itself and make a layer of the disk changes after that command finishes. If you simply try to issue a RUN mysqld (or similar), either the mysqld process will start and block forever (assuming it doesn't daemonize) or (if you tell it to daemonize) it will start and as soon as it moves to the background, docker will make a layer of the disk and then execute your script_runner.sh but mysql will have exited.
If you want to do this you have two options:
Combine the two into a single RUN statement such as RUN mysqld_safe && script_runner.sh
Have your script_runner.sh start mysqld before it tries to execute the mysql client.
Either way should work and would (I think) end up with the same single layer of disk changes. The documents touch on this in their best practices recommendations (although here they're trying to minimize the number of layers for performance reasons):
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#minimize-the-number-of-layers

Related

Docker build command creates large amount of folders in windowsfilter folder: what's the purpose and how to control

I'm building a new image like so docker build . -t test and copy contents from host OS folder into it:
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir test
COPY test /root/test
#USING 'COPY' here instead of 'ADD' because of https://stackoverflow.com/questions/24958140/what-is-the-difference-between-the-copy-and-add-commands-in-a-dockerfile
I get this output in Powershell:
PS D:\Programs> docker build . -t test
Sending build context to Docker daemon 1.644GB
Step 1/7 : FROM mcr.microsoft.com/windows/servercore:ltsc2019
ltsc2019: Pulling from windows/servercore
4612f6d0b889: Pull complete
c67ded6868b6: Pull complete
Digest: sha256:1be9c8378b8c32d31538253a4b96169f5138a5f6d4c0a04e8d8f9a80b9ac8097
Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:ltsc2019
---> d1724c2d9a84
Step 2/7 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
---> Running in 1bfefefbe433
Removing intermediate container 1bfefefbe433
---> 37de702deb33
Step 3/7 : RUN mkdir root
---> Running in e26d6b49ced7
Directory: C:\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 9/8/2021 1:07 PM root
Removing intermediate container e26d6b49ced7
---> 451c4d3f3ea1
Step 4/7 : RUN cd root
---> Running in 74a228f8118f
Removing intermediate container 74a228f8118f
---> 3f175ac67f1d
Step 5/7 : WORKDIR /root
---> Running in 5f783d5b2332
Removing intermediate container 5f783d5b2332
---> 68b24e033f87
Step 6/7 : RUN mkdir test
---> Running in 5771bb7a593a
Directory: C:\root
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 9/8/2021 1:08 PM test
Removing intermediate container 5771bb7a593a
---> 35fa0b2af157
Step 7/7 : COPY test /root/test
---> 60eab8242865
Successfully built 60eab8242865
Successfully tagged test:latest
It creates EIGHT(!) folders on the host OS in folder "\docker\windowsfilter"
This seems excessive, it looks like a folder for each step in my Dockerfile, so what are all these folders for and could I prevent creation/autodelete the unneeded ones to keep everything tidy?
Host OS: Windows Server 2019 standard Version 1809 17763.2114
Docker: version 20.10.4, build 110e091
Please note that I do NOT use the Docker Desktop application, see here how I installed Docker.
Update 1
I updated my Dockerfile based on #Noam's suggestion like so:
#RUN mkdir root
#RUN cd root
#WORKDIR /root
WORKDIR /root #creates root directory if not exists, then enters it
After this change, 6 folders (instead of 8) are created in windowsfilter, so my hunch that Docker creates a folder for each command in Dockerfile seems to be correct.
Hallo #Flo this issue seems to be a long time issue specifically with docker desktop,
Causes:
One option that some times seems to explain it is upgrading from older versions of docker desktop and the software not cleaning up old directories.
Another I that docker saves built image layers there so if you have many versions of the same image your dangling ones could reside there requesting a run of docker image prune or maybe even docker system prune
Solutions:
An opensource cleanup script has been published and it seems to help a great deal, its called docker-ci-zap and for the most part it appears to handle it quite well.
if the issue still persists there is a fix suggested in the docker for desktop github issue page is adding -removing suffix to the folder names in the windowsfilter directory followed by a restart to allow docker to cleanup the unneeded folders
if it's still causing issues try returning to factory deafults and if that doesn't do it either maybe a fresh docker for desktop install might seal the deal though even then some experienced some difficulties so eventually if it really bothers and you need to use it, reinstalling windows usually looks like the final blow.
suggestion for Dockerfile improvments
in your Dockerfile the lines describing the folders might cause problems up a head du to unabsolute paths and improper use of the Dockerfile command.
replace:
RUN mkdir root
RUN cd root
WORKDIR /root
with:
WORKDIR /root
it achives the same purpose (creating and entering the new (if didn't exist yet) directory)
what does docker-ci-zap do?
You can get a rough sense by looking at the code used to compile the exe file that resides in the GitHub repo:
it uses the hcsshim driver and its DestroyLayer function that deletes the layer files from the host disk, or as the docs put it:
DestroyLayer will remove the on-disk files representing the layer
with the given path, including that layer's containing folder, if
any.

The docker image build is stuck

I need to build a windows docker image for a window app on windows 10 with docker desktop installed on vm, but when the build process is stuck when running the installation of this app. the log looks like something bellow,
PS C:\docker> docker image build . -t app
Sending build context to Docker daemon 144.1MB
Step 1/6 : FROM mcr.microsoft.com/dotnet/framework/aspnet:4.8
---> 9b87edf03093
Step 2/6 : COPY . /app
---> ac4b1124d856
Step 3/6 : WORKDIR /app
---> Running in cf4bd2345d26
Removing intermediate container cf4bd2345d26
---> d4f28097afd9
Step 4/6 : RUN .\ELSA1.0_2.6.6.243.exe
---> Running in b9356f975aa6
(And the process is stuck for several hours here and is terminated by me)
the docker file is
FROM mcr.microsoft.com/dotnet/framework/aspnet:4.8 # this is a base image because asp.net 3.5 is a prerequisite for the app
COPY . /app
WORKDIR /app
RUN .\ELSA1.0_2.6.6.243.exe # I am stuck here!
I have tried to do something in the base image like so, but it seems that I cannot do anything
PS C:\docker> docker container run -it
mcr.microsoft.com/dotnet/framework/aspnet:4.8
Service 'w3svc' has been stopped
Service 'w3svc' started
Are there any good ideas to debug this issue? By the way the installer can work normally on windows 10.
You should never execute the command in the RUN statement that not terminate. I see in Docker build logs that you started exe file in RUN command. This will keep stuck your docker build process and will wait for SIGINT. The same will happen like if you execure RUN npm start so it will hang the build process.
Add your executable at entrypoint or CMD.
Another thing that can be the issue in such cased
Considerations for using CMD with Windows
On Windows, file paths specified in the CMD instruction must use
forward slashes or have escaped backslashes \. The following are
valid CMD instructions:
dockerfile
# exec form
CMD ["c:\\Apache24\\bin\\httpd.exe", "-w"]
# shell form
CMD c:\\Apache24\\bin\\httpd.exe -w
You can further read about Window CMD here
However, the following format without the proper slashes will not work:
dockerfile
CMD c:\Apache24\bin\httpd.exe -w

Running a Docker image on Windows 10 using a Makefile

I apologize if this a silly question but when I type inside PowerShell of Window 10:
docker run -it -v C:\Users\Bob\Documents\test:/usr/python -w /usr/python
bob/python
It works just fine and I receive the following prompt:
root#63eef6ac2b96:/usr/python#
To avoid repeating the command over and over, I build a makefile that has the following command
docker:
docker run -it -v C:\Users\Bob\Documents\test:/usr/python -w /usr/python bob/python
when I try to execute
make docker
I receive the following error
PS C:\Users\Bob\documents\test> make docker
docker run -it -v C:\Users\Bob\Documents\test:/usr/python -w /usr/python
bob/python
c:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from
daemon: the working directory 'C:/MinGW/msys/1.0/python' i
s invalid, it needs to be an absolute path.
See 'c:\Program Files\Docker\Docker\Resources\bin\docker.exe run --help'.
make.exe": *** [docker] Error 125
Any suggestion is greatly appreciated.
You do not have to use a Makefile. Docker compose is what you are looking for.
In brief, you need to create a docker-compose.yml file and inside it describe all the desired steps. I am not aware of your full setup but I will try to provide a skeleton for your docker-compose file.
version: '3.7'(depends on your docker engine version)
services:
python_service(add a name of your choice):
build: build/ (The path of image's Dockerfile)
volumes:
- C:\Users\Bob\Documents\test:/usr/python
working_dir: /usr/python
In the snippet above:
-v flag replaced with volumes section
-w flag replced with working_dir section
How to use:
Now that your docker-compose file is ready, you need to use it. So you do not need to remember/repeat the docker run command, you will simple execute docker-compose up in the directory where your compose file is located and you will have your container up and running.
Note that this is a simple example on how to use docker-compose. It is a powerful feature allowing to start containers from multiple images, creating networks and much more. I would recommend you to read the official documentation for additional information.

Cannot write into ~/.m2 in docker maven container

Why aren't files written in /root/.m2 in the maven3 docker image persistent during the build?
A simple dockerfile:
FROM maven:3-jdk-8
RUN touch /root/.m2/testfilehere && ls -a /root/.m2
RUN ls -a /root/.m2
CMD ["bash"]
Produces the following output.
$ docker build -t test --no-cache .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM maven:3-jdk-8
---> 42e3884987fb
Step 2 : RUN touch /root/.m2/testfilehere && ls -a /root/.m2
---> Running in 1c1dc5e9f082
.
..
testfilehere
---> 3da352119c4d
Removing intermediate container 1c1dc5e9f082
Step 3 : RUN ls -a /root/.m2
---> Running in df506db8c1dd
.
..
---> d07cc155b20e
Removing intermediate container df506db8c1dd
Step 4 : RUN stat /root/.m2/testfilehere
---> Running in af44f30aafe5
stat: cannot stat ‘/root/.m2/testfilehere’: No such file or directory
The command '/bin/sh -c stat /root/.m2/testfilehere' returned a non-zero code: 1
The file created at the first command is gone when the intermmediate container exists.
Also, this does not happen in the ubuntu image, just maven.
edit: using ADD hostfile /root/.m2/containerfile does work as a workaround, but is not what I want.
the maven docker image has an entrypoint
ENTRYPOINT ["/usr/local/bin/mvn-entrypoint.sh"]
On container started, entrypoint copy files from /usr/share/maven/ref into ${MAVEN_CONFIG} and erase your file
You can see script executed on startup by following this link
https://github.com/carlossg/docker-maven/blob/33eeccbb0ce15440f5ccebcd87040c6be2bf9e91/jdk-8/mvn-entrypoint.sh
That's because /root/.m2 is defined as a VOLUME in the image. When a container runs with a volume, the volume storage is not part of the UnionFS - so its data is not stored in the container's writable layer:
A data volume is a specially-designated directory within one or more containers that bypasses the Union File System.
The first RUN command creates a file in the volume, but that's in an intermediary container with its own volume. The file isn't saved to the image layer because it's in a volume.
The second RUN command is running in a new intermediary container which has its own volume. There's no content in the volume from the base image, so the volume is empty in the new container.
If you want to pre-populate the volume with data, you need to do it in the Dockerfile as you've seen.
There is a documentation for this in Docker Maven:
https://github.com/carlossg/docker-maven#packaging-a-local-repository-with-the-image
COPY settings.xml /usr/share/maven/ref/
After you run your Docker image, the settings.xml will appear in /root/.m2.

Why this docker file is not running with a shell script?

Now, I made a docker file named as (Dockerfile) as follows:
When I build this using the following command:
docker build -f Dockerfile .
I get the following output:
Step 1 : FROM ******.dkr.ecr.us-east-1.amazonaws.com/centos-base:7
---> 9ab68a0dd16a
Step 2 : COPY echo_hello.sh /echo_hello.sh
---> Using cache
---> e7d541f5cf53
Step 3 : RUN bash /echo_hello.sh
---> Running in 4b5518faab28
hello world
hello world
.......
But, when I then begin to run it using the following command:
docker run -it d2cc33b16e8f
This doesn't happen and instead it shows me an error:
the command to run to start the application
Where am I going wrong in this?
You will need to change your Dockerfile to the following:
FROM *******.dkr.ecr.us-east-1.amazonaws.com/centos-base:7
COPY echo_hello.sh /echo_hello.sh
RUN chmod u+x /echo_hello.sh
CMD /echo_hello.sh
This page explains the Dockerfile instructions pretty well:
http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/
RUN commands are executed at build time (when you do a docker build). What you need is CMD or ENTRYPOINT. Either of these sets the executable that is run when the container starts (via docker run)
Your Dockerfile should look similar to what #Rezney has in his answer.

Resources