I would to start a container with specific shell script using docker compose.
For example, a tomcat container starts initweb.sh creating the empty file /tmp/testweb
$ ls -lR
.:
total 8
-rw-rw-r-- 1 xxxxxx xxxxxx 223 déc. 4 19:45 docker-compose.yml
drwxrwxr-x 2 xxxxxx xxxxxx 4096 déc. 4 19:37 web
./web:
total 4
-rwxrwxr-x 1 xxxxxx xxxxxx 31 déc. 4 19:37 initweb.sh
$ cat docker-compose.yml
version: '3'
services:
web:
container_name: web
hostname: web
image: "tomcat:7.0-jdk8"
ports:
- 8080:8080
volumes:
- "./web/:/usr/local/bin/"
command: sh -c "/usr/local/bin/initweb.sh"
$ cat web/initweb.sh
#!/bin/bash
touch /tmp/testweb
When I execute docker-compose up
$ docker-compose up -d
Creating network "tomcat_default" with the default driver
Creating web ... done
$ docker-compose run web ls -l /usr/local/bin/
total 4
-rwxrwxr-x 1 1000 1000 31 Dec 4 18:37 initweb.sh
$ docker-compose run web ls -l /tmp
total 4
drwxr-xr-x 1 root root 4096 Nov 24 01:29 hsperfdata_root
The owner of my script initweb.sh is not root, so maybe that's why it is not executed but I don't know how to resolve this issue.
You need to make initweb.sh to behave like a server process :
#!/bin/bash
touch /tmp/testweb
sleep infinity
then
~$ docker-compose up -d
Creating network "tmp_default" with the default driver
Creating web ... done
~$ docker-compose exec web ls -l /tmp
total 4
drwxr-xr-x 1 root root 4096 Nov 24 01:29 hsperfdata_root
-rw-r--r-- 1 root root 0 Dec 4 22:39 testweb
Related
Dockerfile:
FROM ubuntu:latest
RUN apt install -y bash
CMD []
build and run:
docker build -t test .
docker run -it test bash
minimal reproduction:
root#8807902e27b4:/# mkdir parent
root#8807902e27b4:/# cd parent
root#8807902e27b4:/parent# mkdir example
root#8807902e27b4:/parent# chmod 000 example
root#8807902e27b4:/parent# ls -la
total 12
drwxr-xr-x 3 root root 4096 Apr 28 19:33 .
drwxr-xr-x 1 root root 4096 Apr 28 19:32 ..
d--------- 2 root root 4096 Apr 28 19:33 example
root#8807902e27b4:/parent# cd example
root#8807902e27b4:/parent/example# echo "test" > test.txt
root#8807902e27b4:/parent/example# chmod 100 test.txt
root#8807902e27b4:/parent/example# cat test.txt
test
root#8807902e27b4:/parent/example# ls -la
total 12
d--------- 2 root root 4096 Apr 28 19:33 .
drwxr-xr-x 3 root root 4096 Apr 28 19:33 ..
---x------ 1 root root 5 Apr 28 19:33 test.txt
In the above example, the cd example command should fail, and even if it doesn't, running cat test.txt should fail. Anyone know what's up?
Here are the same (working) commands run in osx:
beaushinkle#Beaus-MBP ~/p/example-docker> mkdir parent
beaushinkle#Beaus-MBP ~/p/example-docker> cd parent
beaushinkle#Beaus-MBP ~/p/e/parent> mkdir example
beaushinkle#Beaus-MBP ~/p/e/parent> chmod 000 example
beaushinkle#Beaus-MBP ~/p/e/parent> cd example
cd: Permission denied: 'example'
beaushinkle#Beaus-MBP ~/p/e/parent [1]> chmod 777 example
beaushinkle#Beaus-MBP ~/p/e/parent> cd example
beaushinkle#Beaus-MBP ~/p/e/p/example> echo "test" > test.txt
beaushinkle#Beaus-MBP ~/p/e/p/example> chmod 100 test.txt
beaushinkle#Beaus-MBP ~/p/e/p/example> cat test.txt
cat: test.txt: Permission denied
If the prompt is anything to go by, we are logged in as root in the minimal reproduction. Thus, we have root privileges and can read and write all files (external link).
I'm trying to iterate over url entries in a file and use each file as an input for a crawler tool. It's result should be written to a file.
here is the gitlab-ci.yml file:
stages:
- test
test:
stage: test
tags:
- shell-docker
script:
- wget https://github.com/FaKeller/sireg/releases/download/v0.3.1/sireg-linux
- chmod 775 sireg-linux
- mkdir output
- ls -alF
- while read line; do
echo $line;
./sireg-linux exec --loader-sitemap-sitemap \"$line\" >> ./output/${line##*/}_out.txt;
done < sitemap-index
- ls -alF output
artifacts:
paths:
- output/*
expire_in: 1 hrs
and here is the sitemap-index file (only one entry):
http://example.com/sitemap.xml
both files are in the same directory. I expect a file sitemap.xml_out.txt to be written into the output folder(also the same directory). I am pretty sure the ./sireg-linux script does not execute because it usually takes few minutes to complete (tested locally).
the output of the stage looks like this:
2020-04-02 18:22:21 (4,26 MB/s) - »sireg-linux« saved [62566347/62566347]
$ chmod 775 sireg-linux
$ mkdir output
$ ls -alF
total 61128
drwxrwxr-x 4 gitlab-runner gitlab-runner 4096 Apr 2 18:22 ./
drwxrwxr-x 10 gitlab-runner gitlab-runner 4096 Apr 2 15:46 ../
drwxrwxr-x 5 gitlab-runner gitlab-runner 4096 Apr 2 18:22 .git/
-rw-rw-r-- 1 gitlab-runner gitlab-runner 512 Apr 2 18:22 .gitlab-ci.yml
drwxrwxr-x 2 gitlab-runner gitlab-runner 4096 Apr 2 18:22 output/
-rw-rw-r-- 1 gitlab-runner gitlab-runner 30 Apr 2 15:46 README.md
-rwxrwxr-x 1 gitlab-runner gitlab-runner 62566347 Nov 11 2017 sireg-linux*
-rw-rw-r-- 1 gitlab-runner gitlab-runner 55 Apr 2 18:08 sitemap-index
$ while read line; do echo $line; ./sireg-linux **exec** --loader-sitemap-sitemap \"$line\" >>
./output/${line##*/}_out.txt; done < sitemap-index
$ ls -alF output
total 8
drwxrwxr-x 2 gitlab-runner gitlab-runner 4096 Apr 2 18:22 ./
drwxrwxr-x 4 gitlab-runner gitlab-runner 4096 Apr 2 18:22 ../
Uploading artifacts...
Runtime platform arch=amd64 os=linux pid=23813 revision=1f513601 version=11.10.1
WARNING: output/*: no matching files
ERROR: No files to upload
Job succeeded
update
tried to move all steps into a separate script but that did not work either.
update 2
forgot to add exec in the command:
./sireg-linux exec --loader-sitemap-sitemap \"$line\" >>
./output/${line##*/}_out.txt;
unfortunately it didn't help.
what can I do to make it working?
Try changing ./sireg-linux --loader-sitemap-sitemap \"$line\" to ./sireg-linux exec --loader-sitemap-sitemap "$line". Hope this helps!
EDIT: Also, it looks like the script doesn't enter the while loop at all. Maybe the file sitemap-index is empty or it has only one line without a newline at the end?
EDIT 2: The back-slashes in the command line are wrong. corrected my answer
You can of course painfully debug multi-line commands in YAML.
You can even use YAML multi-line strings:
How do I break a string over multiple lines?
https://gitlab.com/snippets/1717579
But I would just wrap code into a shell script, store it in the same GitLab repo, and just call it in .gitlab-ci.yml.
This way you can run this script exactly the same way both locally and in CI, which is a best practice in Continuous Delivery.
- ./script.sh
I have an issue that I'm not able to solve. I want to run a bash script that is inside my Docker CLI container, and I want to execute it, passing parameters. Usually, I run scripts using a notation like this:
docker exec -i $CLI_ID bash "./script.sh"
But I don't know how to pass parameters to the script. I tried to execute it with:
docker exec -i $CLI_ID bash "./script.sh PARAM"
But it doesn't work. How can I do it?
Thanks
Try with bash -c option
docker exec -i $CLI_ID bash -c "./script.sh PARAM"
Hope this helps.
Make sure the script is executable , then you dont need the bash and double quotes etc. Just run it striaght away by path/name and provide the options.
[root#ap-p1m-ff ~]# docker exec -i 0c cat /tmp/test.sh
#!bin/bash
ls $1
[root#ap-p1m-ff ~]# docker exec -i 0c /tmp/test.sh -l
total 4
-rw-r--r-- 1 root root 159 Jun 4 18:32 RELEASE
drwxr-xr-x 2 root root 120 Jun 4 18:33 assets
drwxr-xr-x 1 root root 31 Jun 4 18:33 bin
drwxr-xr-x 2 root root 6 Apr 12 2016 boot
drwxr-xr-x 5 root root 340 Jun 7 19:50 dev
drwxr-xr-x 1 root root 22 Jun 7 19:50 etc
but if the script is not already executble the bash should work as well:
[root#ap-p1m-ff ~]# docker exec -i 0c bash /tmp/test.sh -l
total 4
-rw-r--r-- 1 root root 159 Jun 4 18:32 RELEASE
drwxr-xr-x 2 root root 120 Jun 4 18:33 assets
drwxr-xr-x 1 root root 31 Jun 4 18:33 bin
drwxr-xr-x 2 root root 6 Apr 12 2016 boot
drwxr-xr-x 5 root root 340 Jun 7 19:50 dev
drwxr-xr-x 1 root root 22 Jun 7 19:50 etc
drwxr-xr-x 2 root root 6 Apr 12 2016 home
drwxr-xr-x 1 root root 45 Sep 13 2015 lib
drwxr-xr-x 2 root root 34 May 15 14:22 lib64
drwxr-xr-x 2 root root 6 May 15 14:22 media
Add your parameter as a separate argument rather than as part of the filename:
docker exec -i "$CLI_ID" bash "./script.sh" PARAM
This way, you don't have to add another level of escaping to the parameter.
My goal is to have Jenkins 2 execute alpha integration tests between an express js app and a postgres db. I am to spin up containerized resources locally and test successfully with bash scripts that employ docker-compose. The relevant bash script is scripts/docker/dockerRunTest.sh.
However, when I try to do the same thing via Jenkins, Jenkins claims that the initiating script is not found.
Jenkinsfile
stage('Alpha Integration Tests') {
agent {
docker {
image 'tmaier/docker-compose'
args '-u root -v /var/run/docker.sock:/var/run/docker.sock --network host'
}
}
steps {
sh 'ls -lah ./scripts/docker/'
sh './scripts/docker/dockerRunTest.sh'
}
}
Output
+ ls -lah ./scripts/docker/
total 36
drwxr-xr-x 2 root root 4.0K Jan 26 21:31 .
drwxr-xr-x 6 root root 4.0K Jan 26 20:54 ..
-rwxr-xr-x 1 root root 2.2K Jan 26 21:31 docker.lib.sh
-rwxr-xr-x 1 root root 282 Jan 26 21:31 dockerBuildApp.sh
-rwxr-xr-x 1 root root 289 Jan 26 21:31 dockerBuildTestRunner.sh
-rwxr-xr-x 1 root root 322 Jan 26 21:31 dockerDown.sh
-rw-r--r-- 1 root root 288 Jan 26 21:31 dockerRestart.sh
-rwxr-xr-x 1 root root 482 Jan 26 21:31 dockerRunTest.sh
-rwxr-xr-x 1 root root 284 Jan 26 21:31 dockerUp.sh
+ ./scripts/docker/dockerRunTest.sh
/var/jenkins_home/workspace/project-name#2#tmp/durable-9ac0d23a/script.sh: line 1: ./scripts/docker/dockerRunTest.sh: not found
ERROR: script returned exit code 127
The file clearly exists per the ls output. I have some hazy idea that there may be some conflict between how shell scripts and bash scripts work, but I cannot quite grasp the nuance in how Jenkins is not able to execute a script that clearly exists.
edit (including script contents):
dockerRunTest.sh
#!/bin/bash
MY_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd -P )"
MY_DIR="${MY_DIR:?}"
SCRIPTS_DIR="$(realpath "${MY_DIR}/..")"
ROOT_DIR="$(realpath "${SCRIPTS_DIR}/..")"
TEST_DIR="${ROOT_DIR}/test/integration"
SRC_DIR="${ROOT_DIR}/src"
REPORTS_DIR="${ROOT_DIR}/reports"
. "${SCRIPTS_DIR}/docker/docker.lib.sh"
dockerComposeUp
dockerExecuteTestRunner
dockerComposeDown
docker.lib.sh
#!/bin/bash
CURRENT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd -P )"
CURRENT_DIR="${CURRENT_DIR:?}"
SCRIPTS_DIR="$(realpath "${CURRENT_DIR}/..")"
ROOT_DIR="$(realpath "${SCRIPTS_DIR}/..")"
. "${SCRIPTS_DIR}/lib.sh"
dockerComposeUp() {
docker-compose build --no-cache
docker-compose up --detach --force-recreate
DC_CODE=$?
if [ ${DC_CODE} -ne 0 ]; then
# Introspection
docker-compose logs
docker-compose ps
exit ${DC_CODE}
fi
}
dockerComposeDown() {
# docker-compose rm: Removes stopped service containers.
# -f, --force - Don't ask to confirm removal.
# -s, --stop - Stop the containers, if required, before removing.
# -v - Remove any anonymous volumes attached to containers.
docker-compose rm --force --stop -v
}
dockerComposeRestart() {
dockerComposeDown
dockerComposeUp
}
dockerBuildTestRunner() {
docker build -f test/Dockerfile -t kwhitejr/botw-test-runner .
}
dockerExecuteTestRunner() {
IMAGE_NAME="kwhitejr/botw-test-runner"
echo "Build new ${IMAGE_NAME} image..."
dockerBuildTestRunner
echo "Run ${IMAGE_NAME} executable test container..."
docker run -it --rm --network container:api_of_the_wild_app_1 kwhitejr/botw-test-runner
}
tmaier/docker-compose image doesn't have /bin/bash interpreter installed by default since latest tag is an alpine image [1, 2]. This can be confirmed by running:
$ docker run -it --rm tmaier/docker-compose bash
/usr/local/bin/docker-entrypoint.sh: exec: line 35: bash: not found
To get the script working, either install bash in the docker image using apk add bash or change the shebang to #!/bin/sh if the script can be run using ash shell (the default shell in busybox).
[1] https://github.com/tmaier/docker-compose/blob/b740feb61fb25030101638800a609605cfd5e96a/Dockerfile#L2
[2] https://github.com/docker-library/docker/blob/d94b9832f55143f49e47d00de63589ed41f288e7/18.09/Dockerfile#L1
I have the similar issue but in my case, it is because the shell script file has EOL in Windows format (if you open the file in the terminal using vi, you will see each line ends with ^M)
I can fix this using Notepad++ Edit -> EOL Conversion -> Unix (LF)
I´m using lemonlatte / docker-webvirtmgr as base file, but the problem is that there are no ssh keys configured for the user www-data, so I wrote the following shell script:
#!/bin/sh
if [ ! -d "/var/local/webvirtmgr/nginxhome" ]; then
mkdir /var/local/webvirtmgr/nginxhome
chown -R www-data:www-data /var/local/webvirtmgr/nginxhome
usermod -d /var/local/webvirtmgr/nginxhome www-data
su - www-data -s /bin/bash -c "ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N ''"
su - www-data -s /bin/bash -c "touch /var/local/webvirtmgr/nginxhome/.ssh/config && echo -e 'StrictHostKeyChecking=no\nUserKnownHostsFile=/dev/null' >> /var/local/webvirtmgr/nginxhome/.ssh/config"
su - www-data -s /bin/bash -c "chmod 0600 ~/.ssh/config"
fi
After that I added the two statements to the dockerfile:
ADD setupssh.sh /webvirtmgr/setupssh.sh
RUN /bin/sh -c "/webvirtmgr/setupssh.sh"
I already tried CMD /webvirtmgr/setupssh.sh, RUN /webvirtmgr/setupssh.sh but with no success...
When I run the script inside the container by hand it is working fine.
What is wrong here?
greetings
UPDATE:
Here is the link to the repo of the maintainer: link
UPDATE 2:
The build of the dockerfile was successful and I put the statement between:
RUN apt-get -ys clean
<statements were here>
WORKDIR /
The directory /var/local/webvirtmgr is defined as a volume.
VOLUME /var/local/webvirtmgr
Therefore this directory is a mountpoint in the running container and what you have added to it gets overwritten.
You will have to use a different directory, then your script will work.
Here´s a Dockerfile to test it:
FROM lemonlatte/docker-webvirtmgr
RUN mkdir /var/local/webvirtmgr2
RUN touch /var/local/webvirtmgr2/t && touch /var/local/webvirtmgr/t
RUN ls -la //var/local/webvirtmgr
RUN ls -la /var/local/webvirtmgr2
Output:
Sending build context to Docker daemon 4.608 kB
Sending build context to Docker daemon
Step 0 : FROM lemonlatte/docker-webvirtmgr
---> 18e2839dffea
Step 1 : RUN mkdir /var/local/webvirtmgr2
---> Running in d7a1e897108e
---> cc029293525e
Removing intermediate container d7a1e897108e
Step 2 : RUN touch /var/local/webvirtmgr2/t && touch /var/local/webvirtmgr/t
---> Running in 1a1375651fa7
---> e314c2529d90
Removing intermediate container 1a1375651fa7
Step 3 : RUN ls -la //var/local/webvirtmgr
---> Running in 5228691c84f5
total 8
drwxr-xr-x 2 www-data www-data 4096 Jun 6 09:22 .
drwxr-xr-x 6 root root 4096 Jun 6 09:22 ..
---> ec4113936961
Removing intermediate container 5228691c84f5
Step 4 : RUN ls -la /var/local/webvirtmgr2
---> Running in a6d2a683391a
total 8
drwxr-xr-x 2 root root 4096 Jun 6 09:22 .
drwxr-xr-x 6 root root 4096 Jun 6 09:22 ..
-rw-r--r-- 1 root root 0 Jun 6 09:22 t
---> 3cb98c5c1baf
Removing intermediate container a6d2a683391a
Successfully built 3cb98c5c1baf