I'm new to Makefiles, I'm trying to use one to help me automate the build of my docker images.
I'm struggling to write, what feels like, a simple recipe that does the following:
Check to see if there are running containers (docker ps -a -q)
If running containers exist remove them (docker rm $(docker ps -a -q))
This is what I've manage to cobble together so far through various Google searches:
DOCKER_CONTAINER_LIST := $(shell docker ps -a -q)
clean-docker:
if [[ ! $(DOCKER_CONTAINER_LIST) == "" ]];then \
docker rm $$(docker ps -a -q) \
fi
Which yields the following output:
if [[ de6e694da4f9 == "" ]];then \
/bin/sh: -c: line 0: conditional binary operator expected
/bin/sh: -c: line 0: syntax error near 'de6e694da4f9'
/bin/sh: -c: line 0: 'if [[ de6e694da4f9 == "" ]];then \ '
make: *** [clean-docker] Error 2
Would be great if someone could show me how to achieve this please
UPDATE
Thank #I0b0 for your answer, I think I'm almost there now. The Makefile now looks like this:
DOCKER_CONTAINER_LIST := $(shell docker ps -a -q)
clean-docker:
if [ -n "$(DOCKER_CONTAINER_LIST)" ] \
then \
docker rm "$(DOCKER_CONTAINER_LIST)" \
fi
But I get the following error when running it:
if [ -n "6778d35e5c95" ] \
then \
docker rm "6778d35e5c95" \
fi
/usr/bin/sh: -c: line 4: syntax error: unexpected end of file
make: *** [clean-docker] Error 1
I had a look at this answer regarding line endings, but that made no difference. I am using Makefile on both Mac and Windows.
It can be done with:
# for images
docker rmi $(shell docker images -a -q)
# for containers
docker rm $(shell docker ps -a -q)
# for volumes
docker volume rm $(shell docker volume ls -q)
PS. macOS
if [ -n "$(DOCKER_CONTAINER_LIST)" ] should do it:
-n is the standard test for a non-empty string
Use More Quotes™ never gets old
[[ is a bashism, but sh is not bash
Bonus points for reusing the variable with docker rm "$(DOCKER_CONTAINER_LIST)".
After the update the problem is related to a fundamental difference between scripts and makefiles: the latter cannot be broken into lines in the exact same way as scripts by just adding backslashes before the end of line. What you've ended up with is a single line if [ -n "$(DOCKER_CONTAINER_LIST)" ] then docker rm "$(DOCKER_CONTAINER_LIST)" fi. Because newlines in a makefile don't work like ; in a shell script you have to add two:
if [ -n "$(DOCKER_CONTAINER_LIST)" ]; \
then \
docker rm "$(DOCKER_CONTAINER_LIST)"; \
fi
I couldn't figure out how to divide those commands into multiple lines either, but ugly oneliners work for me.
So what you were asking about would be something like this
DOCKER_CONTAINER_LIST := $(shell docker ps -a -q)
clean_docker:
#if [ -n "$(DOCKER_CONTAINER_LIST)" ]; then echo "Removing docker containers" && docker rm "$(DOCKER_CONTAINER_LIST)"; else echo "No containers found"; fi;
Related
I need to split a comma-separated string into array and run k6 for each of the array values parallely. Since shell script doesn't support arrays, I m using bash command in the script. I am not able to run it as bash using Dockerfile in TeamCity.
Dockerfile:
FROM loadimpact/k6:0.34.1
COPY ./src/lib /lib
COPY ./src/scenarios /scenarios
COPY ./src/k6-run-all.sh /k6-run-all.sh
WORKDIR /
ENTRYPOINT []
RUN bash -c "./k6-run-all.sh"
Shell script:
#!/bin/bash
K6_RUN_OPTIONS=${K6_RUN_OPTIONS}
ENV_NAME=${ENV_NAME:-qa}
IS_TEST_RUN=${IS_TEST_RUN:-true}
SCENARIO_NAME=${SCENARIO_NAME:-"full-card-visa"}
GWC_PC_ID=${GWC_PC_ID}
IFS=',' read -r -a PCIds <<< "$GWC_PC_ID"
echo "Number of PC ids provided in environment variables=" ${#PCIds[#]}
if [[ ${#PCIds[#]} > 0 ]]; then
for pcId in "$#"
do
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT GWC_PC_ID=$pcId k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js'' &
done
fi
existCode=$?
if [ $existCode -ne 0 ]; then
echo "Scenario $SCENARIO_NAME completed with the error"
exit $existCode
fi
Error:
#9 [5/6] RUN bash -c "./k6-run-all.sh"
17:02:02 #9 0.356 /bin/sh: bash: not found
17:02:02 #9 ERROR: executor failed running [/bin/sh -c bash -c "./k6-run-all.sh"]: exit code: 127
17:02:02 ------
17:02:02 > [5/6] RUN bash -c "./k6-run-all.sh":
17:02:02 #9 0.356 /bin/sh: bash: not found
17:02:02 ------
17:02:02 failed to solve: executor failed running [/bin/sh -c bash -c "./k6-run-all.sh"]: exit code: 127
How to modify Dockerfile or shell script to run this shell script as bash?
Previously to run it as bash script the last line of Dockerfile used to be:
CMD ["sh", "-c", "./k6-run-all.sh"]
******* Edit: **********
Updated full script after knittl's answer (current issue is after adding & for parallel runs it is not working, it is not running anything inside the for loop and it is not giving any extra error or information in logs, it is like it is skipping it):
K6_RUN_OPTIONS=${K6_RUN_OPTIONS}
ENV_NAME=${ENV_NAME:-qa}
IS_TEST_RUN=${IS_TEST_RUN:-true}
SCENARIO_NAME=${SCENARIO_NAME:-"full-card-visa"}
GWC_PC_ID=${GWC_PC_ID}
OPTIONS_VARIANT=""
if $IS_TEST_RUN; then
OPTIONS_VARIANT="-test"
fi
SCENARIO_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
SCENARIO_PATH="${SCENARIO_DIR}/scenarios"
SCENARIO="${SCENARIO_PATH}/${SCENARIO_NAME}"
echo "Executing scenario path $SCENARIO"
SCENARIO_NAME=${SCENARIO:${#SCENARIO_PATH}+1:${#SCENARIO}}
echo "Scenario Name: $SCENARIO_NAME"
echo "Run option: $SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT"
echo "pc ids provided in environment variable=" $GWC_PC_ID
if [ -z "$GWC_PC_ID" ]
then
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js''
else
for pcId in $(printf '%s' "$GWC_PC_ID" | tr , ' ');
do
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT GWC_PC_ID=$pcId k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js'' &
done
fi
existCode=$?
if [ $existCode -ne 0 ]; then
echo "Scenario $SCENARIO_NAME completed with the error"
exit $existCode
fi
k6 Docker containers do not come with bash preinstalled, but with busybox. I see two options:
Create your own Docker image based off grafana/k6 and manually install bash in your image.
Rewrite your script to not rely on bashisms. Should be fairly easy: split your list of tests to run into one path per line and while read -r path; do …; done them.
Or if support for whitespace in filenames is not required, then for path in $(printf '%s' "$GWC_PC_ID" | tr , ' '); do …; done
Note that your current script will return with the exit code of your last k6 process, meaning that if any other test failed but the last one was successfull, that will mask the error.
PS. Time to upgrade your base Docker image too. loadimpact/k6:0.34.1 is really old (exactly 1 year). It's better to switch to grafana/k6:0.40.0, which was released a week ago.
So my main goal is to access Tshock console so i could run some commands on the server directly.
From what i found as soon as the command to run the server is executed there is no option to get back to the console so:
I'd like to run the server in screen mode.
Dockerfile is basicly running some bash script but i'm getting errors when i'm trying to add "screen" to it. I'm getting "bootstrap.sh: 33: bootstrap.sh: Syntax error: "fi" unexpected (expecting "then")
Entering script"
I've tried everything i could found on google but nothing works :( This is my first time when i'm doing any scripting in bash so i would be grateful for understanding :)
Here is the link to original repo: https://github.com/ryansheehan/terraria/blob/master/tshock/bootstrap.sh
I would be glad for any hints on how to make this script working or if there is any other simplier option to access the console of the server :)
I've added extra line in dockerfile to download screen so it looks like that now:
FROM alpine:3.11.6 AS base
RUN apk add --update-cache \
unzip
# add the bootstrap file
COPY bootstrap.sh /tshock/bootstrap.sh
ENV TSHOCKVERSION=v4.4.0-pre12
ENV TSHOCKZIP=TShock4.4.0_Pre12_Terraria1.4.0.5.zip
# Download and unpack TShock
ADD https://github.com/Pryaxis/TShock/releases/download/$TSHOCKVERSION/$TSHOCKZIP /
RUN unzip $TSHOCKZIP -d /tshock && \
rm $TSHOCKZIP && \
chmod +x /tshock/TerrariaServer.exe && \
# add executable perm to bootstrap
chmod +x /tshock/bootstrap.sh
FROM mono:6.8.0.96-slim
LABEL maintainer="Ryan Sheehan <rsheehan#gmail.com>"
# documenting ports
EXPOSE 7777 7878
# env used in the bootstrap
ENV CONFIGPATH=/root/.local/share/Terraria/Worlds
ENV LOGPATH=/tshock/logs
ENV WORLD_FILENAME=""
# Allow for external data
VOLUME ["/root/.local/share/Terraria/Worlds", "/tshock/logs", "/plugins"]
# install nuget to grab tshock dependencies
RUN apt-get update -y && \
apt-get install -y nuget && \
apt-get install -y screen
# rm -rf /var/lib/apt/lists/* /tmp/*
# copy game files
COPY --from=base /tshock/ /tshock/
# Set working directory to server
WORKDIR /tshock
# run the bootstrap, which will copy the TShockAPI.dll before starting the server
ENTRYPOINT [ "/bin/sh", "bootstrap.sh" ]
And here is my modified code for bootstrap.sh :
#!/bin/sh
echo "Entering script"
if [ -z "$STY" ];then
echo "Opening screen mode ..."
exec screen -dm -S terraria bin/bash "$0"
else
echo "Continuing with script in screen mode"
echo "\nBootstrap:\nworld_file_name=$WORLD_FILENAME\nconfigpath=$CONFIGPATH\nlogpath=$LOGPATH\n"
echo "Copying plugins..."
cp -Rfv /plugins/* ./ServerPlugins
WORLD_PATH="/root/.local/share/Terraria/Worlds/$WORLD_FILENAME"
if [ -z "$WORLD_FILENAME" ]; then
echo "No world file specified in environment WORLD_FILENAME."
if [ -z "$#" ]; then
echo "Running server setup..."
else
echo "Running server with command flags: $#"
fi
mono --server --gc=sgen -O=all TerrariaServer.exe -configpath "$CONFIGPATH" -logpath "$LOGPATH" "$#"
else
echo "Environment WORLD_FILENAME specified"
if [ -f "$WORLD_PATH" ]; then
echo "Loading to world $WORLD_FILENAME..."
mono --server --gc=sgen -O=all TerrariaServer.exe -configpath "$CONFIGPATH" -logpath "$LOGPATH" -world "$WORLD_PATH" "$#"
else
echo "Unable to locate $WORLD_PATH.\nPlease make sure your world file is volumed into docker: -v <path_to_world_file>:/root/.local/share/Terraria/Worlds"
exit 1
fi
fi
fi
After removing empty spaces as #KamilCuk suggested the script is running but the screen doesn't seem to be working.
Here is the output from console:
PS D:\TerrariaServer\Source\terraria\tshock> docker run --rm -p 7777:7777 -v D:/TerrariaServer/World:/root/.local/share/Terraria/Worlds --name="terraria" terraria-image -world /root/.local/share/Terraria/Worlds/TestWorld.wld
Entering screen mode
+ [ -z ]
+ echo Entering screen mode
+ screen -d -m -S terraria bin/bash bootstrap.sh
+ echo Screen mode activated
+ echo Continuing with script in screen mode
+ echo \nBootstrap:\nworld_file_name=\nconfigpath=/root/.local/share/Terraria/Worlds\nlogpath=/tshock/logs\n
+ echo Copying plugins...
+ cp -Rfv /plugins/* ./ServerPlugins
Screen mode activated
Continuing with script in screen mode
Bootstrap:
world_file_name=
configpath=/root/.local/share/Terraria/Worlds
logpath=/tshock/logs
Copying plugins...
cp: cannot stat '/plugins/*': No such file or directory
+ WORLD_PATH=/root/.local/share/Terraria/Worlds/
+ [ -z ]
+ echo No world file specified in environment WORLD_FILENAME.
+ [ -z -world /root/.local/share/Terraria/Worlds/TestWorld.wld ]
+ echo Running server with command flags: -world /root/.local/share/Terraria/Worlds/TestWorld.wld
+ mono --server --gc=sgen -O=all TerrariaServer.exe -configpath /root/.local/share/Terraria/Worlds -logpath /tshock/logs -world /root/.local/share/Terraria/Worlds/TestWorld.wld
No world file specified in environment WORLD_FILENAME.
Running server with command flags: -world /root/.local/share/Terraria/Worlds/TestWorld.wld
Error Logging Enabled.
TerrariaAPI Version: 2.1.0.0 (Protocol v1.4.0.5 (230), OTAPI 1.4.0.5)
[TShock] Info Config path has been set to /root/.local/share/Terraria/Worlds
[TShock] Info Log path has been set to /tshock/logs
TShock was improperly shut down. Please use the exit command in the future to prevent this.
TShock 4.4.0.0 (Go to sleep Patrikkk, Icy, Chris, Death, Axeel, Zaicon, hakusaro, Zack, and Yoraiz0r <3) now running.
AutoSave Enabled
Backups Enabled
Welcome to TShock for Terraria!
TShock comes with no warranty & is free software.
You can modify & distribute it under the terms of the GNU GPLv3.
[Server API] Info Plugin TShock v4.4.0.0 (by The TShock Team) initiated.
Terraria Server v1.4.0.5
Resetting game objects 1%
Resetting game objects 2%
Resetting game objects 3%
...
I'm not directly seenig major issues with your entrypoint script. But I want provide some steps I would do to hopefully resolve these kind of issues:
Avoid the line f [ -z "$#" ].
See here why.
Suggestion: f [ -z "$*" ]
Also this line is problematic too: echo "Running server with command flags: $#"
See here why.
Suggestion: echo "Running server with command flags: $*"
Sometimes it can help, to enter the container and "play around". This allows you to work in your docker container like you would do in any normal Linux shell.
Have a look at the -interactive, -i option for docker run.
Example
If you want to start your container, you can run e. g. run a bash (...or sh, or whatever shell you are using):
docker run -it <img_name> <arguments> bash
This also works with exec when your container is already running and you want to enter it.
docker exec -it <container_id> bash
Try the bash -x option when invoking your entrypoint script to get a verbose output. Sometimes it helps finding the error.
ENTRYPOINT [ "/bin/bash", "-x" "bootstrap.sh" ]
I have the following Makefile:
build: clean
${GOPATH}/bin/dep ensure
env GOOS=linux go build -o ./bin/status ./lib/status/main.go
elm-app build
init:
${GOPATH}/bin/dep init -v
test:
env GOOS=linux go test -v ./lib/status
strip:
strip ./bin/status
clean:
if [ -f ./bin/status ]; then
rm -f ./bin/status
fi
but I get
if [ -f ./bin/status ]; then
/bin/sh: 1: Syntax error: end of file unexpected
Makefile:16: recipe for target 'clean' failed
make: *** [clean] Error 2
what am i missing?
any advice is much appreciated
Each line of a makefile is run in a separate shell. This means your rule here:
clean:
if [ -f ./bin/status ]; then
rm -f ./bin/status
fi
actually runs the following commands:
/bin/sh -c "if [ -f ./bin/status ]; then"
/bin/sh -c "rm -f ./bin/status"
/bin/sh -c "fi"
You can see why you get this message. To ensure that all lines are send into a single shell you need to use backslashes to continue the lines like this:
clean:
if [ -f ./bin/status ]; then \
rm -f ./bin/status; \
fi
Note this means you also need a semicolon after the rm command so separate it from the ending fi.
Now you get a shell invocation like this:
/bin/sh -c "if [ -f ./bin/status ]; then \
rm -f ./bin/status; \
fi"
I am going through a setup script that I am attempting to understand; how the sed line works, in this instance. From my understanding, it is editing the src/conf-cc inline at the first line and appending -include /usr/include/errno.h/ to the last line of input? I have been referencing the sed manual to help me break this sed command down.
#!/usr/bin/env bash
# A script which installs daemontools
#
# Run as root!
#
if [ "$(id -u)" != "0" ]; then
echo "You must be root!" 1>&2
exit 1
fi
mkdir /package
chmod 1755 /package
cd /package
wget http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
tar -xpf daemontools-0.76.tar.gz
rm -f daemontools-0.76.tar.gz
cd admin/daemontools-0.76
sed -i '1s/$/ -include \/usr\/include\/errno.h/' src/conf-cc
package/install
echo -e "start on runlevel [3] \nrespawn \nexec /command/svscanboot" >> /etc/init/svscan.conf
initctl reload-configuration
initctl start svscan
mkdir /var/svc.d
No, it just appends something to the first line. It's a substitution command:
addr s/pattern/replacement/
where addr is 1 (first line), pattern is $ (regex: end of line) and the replacement is the -include ... string. It's not really "replacing" anything as $ has zero width anyway.
Your misunderstanding is interpreting $ as an address instead of a regular expression.
Ive recently been introduced to bash scripting... So, Ive used my advanced theft course to throw together the attached script. it runs... and exits with "/xxx/ not mounted. You are not root! I have rdiff-backup and sshfs installed and working. The commands work fine on their own on the commandline, but in the script, well... Can you guys take a look and lemme know? PS I copied a LOT of this from scripts I found here and a few other places.
<code>
#!/bin/bash
# Version 1.5
# Prior to running this make sure you have ssh-keygen -t rsa to generate a key, then
# ssh username#target "mkdir .ssh/;chmod 700 .ssh"
# scp .ssh/id_rsa.pub username#target:.ssh/authorized_keys
#
# then check you can login and accept the ssh key
# ssh username#target "ls -la"
#
# Key things to remember, no spaces in pathnames, and try to use full paths (beginning with / )
#
# variables determine backup criteria
DATESTAMP=`date +%d%m%y`
USERNAME=username #remote site username here
TARGET=remote.ip.here #add the ip v4 address of the target
INCLUDES=/path/to/file/includes.txt #this is a txt file containing a list of directories you want backed up
EXCLUDES="**" #this is a list of files etc you want to skip
BACKUPLOG=/path/to/logfile/in/home/backuplog${DATESTAMP}.txt
OLDERTHAN=20W #change 20 to reflect how far back you want backups to exist
# to activate old backup expiry, uncomment the line below
#RMARGS=" --force --remove-older-than ${OLDERTHAN}"
TARGETMAIL="yourmailaddress#your.domain"
HOSTNAME=`hostname` #Dont change this!
TMPDIR=/backups Change this to the source folder
TARGETFOLDER=/backups change this to the TARGET folder
ARGS=" -v0 --terminal-verbosity 0 --exclude-special-files --exclude-other-filesystems --no-compression -v6"
# detecting distro and setting the correct path
if [ -e /etc/debian_version ];then
NICE=/usr/bin/nice
elif [ -e /etc/redhat-release ];then
NICE=/bin/nice
fi
if [ -e /tmp/backup.lock ];then
exit 0
fi
touch /tmp/backup.lock
touch -a ${BACKUPLOG}
cd /
/bin/mkdir -p ${TMPDIR}
/usr/bin/sshfs -o idmap=user -o ${USERNAME}#${TARGET}:/${TARGETFOLDER} ${TMPDIR} &>${BACKUPLOG}
# if you get errors mounting this then try
# mknod /dev/fuse -m 0666 c 10 229
for ITEMI in ${INCLUDES} ; do
ARGS="${ARGS} --include ${ITEMI} "
done
for ITEME in ${EXCLUDES} ; do
ARGS="${ARGS} --exclude-regexp '${ITEME}' "
done
# the --exclude ** / is a hack because it wont by default do multiple dirs, so use --include for all dirs then exclude everything else and sync / - if you dont understand dont worry
# ref: http://www.mail-archive.com/rdiff-backup-users#nongnu.org/msg00311.html
#echo /usr/bin/rdiff-backup ${ARGS} --exclude \'**\' / ${TMPDIR}/ &&
cat ${INCLUDES} | while read DIR; do
${NICE} -19 /usr/bin/rdiff-backup --exclude '**' ${DIR} ${TMPDIR}/ &>${BACKUPLOG}
if [ $? != 0 ]; then
echo "System Backup Failed" | mutt -s "Backup Log: System Backup Failed, Log attached!" -a ${BACKUPLOG} ${TARGETMAIL}
exit 1;
fi
done
#${NICE} -19 /usr/bin/rdiff-backup ${ARGS} --exclude '**' / ${TMPDIR}/ &>${BACKUPLOG} &&
echo Removing backups older than ${RMARGS}
${NICE} -19 /usr/bin/rdiff-backup -v0 --terminal-verbosity 0 ${RMARGS} ${TMPDIR}/ &>${BACKUPLOG}
/bin/umount ${TMPDIR} && /bin/rm -rf ${TMPDIR}/ &>${BACKUPLOG}
echo "System Backup Run" | mutt -s "Backup Log: System Backup Done!" -a ${BACKUPLOG} ${TARGETMAIL}
rm /tmp/backup.lock
rm ${BACKUPLOG}
Sorry, cannot paste, couldnt attach... BLIKSEM!
Thanks for ANY input... One HELL of a learning curve!!!
Regards,
B.