Way to pass a script as a docker build argument? - shell

I want to pass a multi-line script as a argument to docker build command, something like that:
docker build -t tertparam --build-arg load_cat_agent=true --build-arg deploy_cat_script='
echo "aaa";
echo "bbb"
' --no-cache .
and execute it during build, my Dockerfile is like
FROM python:3-alpine
ARG load_cat_agent
ARG deploy_cat_script
ADD . /root/
WORKDIR /root/
RUN if [ $load_cat_agent == "true" ]; then \
$deploy_cat_script;\
fi
CMD /root/start.sh && /root/wait.sh
but i found that it always just print
Step 6/7 : RUN if [ $load_cat_agent == "true" ]; then $deploy_cat_script; fi
---> Running in 7868c310e8e5
"aaa" echo "bbb"
how can i do that?

One way is to write the build args to shell script and then run the shell script.
FROM python:3-alpine
ARG load_cat_agent
ARG deploy_cat_script
ADD . /root/
WORKDIR /root/
RUN echo $deploy_cat_script > ./deploy_cat_script.sh
RUN chmod +x ./deploy_cat_script.sh
RUN if [ $load_cat_agent == "true" ]; then \
./deploy_cat_script.sh;\
fi
CMD /root/start.sh && /root/wait.sh
output:
Step 8/9 : RUN if [ $load_cat_agent == "true" ]; then ./deploy_cat_script.sh; fi
---> Running in 08a2f528a14d
aaa
bbb

If you have two images that are so different that the commands you need to build them are different, it's better to just have two separate Dockerfiles. The docker build -f command can specify which Dockerfile to use, and the Docker Compose build: block has a similar dockerfile: option.
# Dockerfile
FROM python:3-alpine
WORKDIR /root/
ADD . ./
CMD ["/root/start.sh"]
# Dockerfile.deploy
FROM python:3-alpine
WORKDIR /root/
ADD . ./
RUN echo "aaa" \
&& echo "bbb"
CMD ["/root/start.sh"]
If you don't mind needing to run docker build multiple times, you can have one image be built FROM the other. It will inherit its filesystem and metadata settings like the default CMD.
# Dockerfile.deploy, version 2
FROM tertparam
RUN echo "aaa" \
&& echo "bbb"
docker build -t tertparam .
docker build -t tertparam-deploy -f Dockerfile.deploy .
In your original example you might be able to get away with eval the string, but that setup is complex enough that you'll need to script it anyways, so the Dockerfile-based approach probably isn't any more difficult.

The problem with $deploy_cat_script is that shell expansions are detecting command separators before variable expansions. One solution is to use eval. Make sure to learn about [eval command and security issues associated with it](Eval command and security issues).
Dockerfile:
FROM python:3-alpine
ARG load_cat_agent
ARG deploy_cat_script
RUN set -x && if [ "$load_cat_agent" == "true" ]; then \
eval "$deploy_cat_script"; \
fi
With something more complicated such as deploy_cat_script='for i in a b c; do echo $i | sed "s/^/test: /"; done' executes like so:
$ docker build -t tertparam --build-arg load_cat_agent=true --build-arg deploy_cat_script='for i in a b c; do echo $i | sed "s/^/test: /"; done' .
Sending build context to Docker daemon 7.168kB
Step 1/4 : FROM python:3-alpine
---> 59acf2b3028c
Step 2/4 : ARG load_cat_agent
---> Using cache
---> 6e383d31f589
Step 3/4 : ARG deploy_cat_script
---> Using cache
---> 04fc43723e0f
Step 4/4 : RUN set -x && if [ "$load_cat_agent" == "true" ]; then eval "$deploy_cat_script"; fi
---> Running in 72e46c08072e
+ '[' true '==' true ]
+ eval 'for i in a b c; do echo $i | sed "s/^/test: /"; done'
test: a
+ echo a
+ sed 's/^/test: /'
+ sed 's/^/test: /'
+ echo b
test: b
+ sed 's/^/test: /'
+ echo c
test: c
Removing intermediate container 72e46c08072e
---> 765de7cf22a1
Successfully built 765de7cf22a1
Successfully tagged tertparam:latest

Related

Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value

Did you notice that set-threads do not work with a recent version of snakemake? It looks long but you just have to copy/paste. Here is a MRE:
mkdir snakemake-test && cd snakemake-test
touch snakeFile
mkdir profile && touch profile/config.yaml && touch profile/status-sacct.sh && chmod +x profile/status-sacct.sh
mkdir envs && touch envs/environment1.yaml && touch envs/environment2.yaml
In envs/environment1.yaml:
channels:
- bioconda
- conda-forge
dependencies:
- snakemake-minimal=7.3.8
- pandas=1.4.2
- peppy=0.31.2
- eido=0.1.4
In envs/environment2.yaml:
channels:
- bioconda
- conda-forge
dependencies:
- snakemake-minimal=6.15.1
- pandas=1.4.2
- peppy=0.31.2
- eido=0.1.4
In snakeFile:
onstart:
print("\t Creating jobs output subfolders...\n")
shell("mkdir -p jobs/downloadgenome")
GENOME = "mm39"
PREFIX = "Mus_musculus.GRCm39"
rule all:
input:
expand("data/fasta/{genome}/{prefix}.dna.chromosome.1.fa", genome=GENOME, prefix=PREFIX)
rule downloadgenome:
output:
"data/fasta/{genome}/{prefix}.dna.chromosome.1.fa"
params:
genomeLinks = "http://ftp.ensembl.org/pub/release-106/fasta/mus_musculus/dna/Mus_musculus.GRCm39.dna.chromosome.1.fa.gz"
threads: 4
shell:
"""
wget {params.genomeLinks}
gunzip {wildcards.prefix}.dna.chromosome.1.fa.gz
mkdir -p data/fasta/{wildcards.genome}
mv {wildcards.prefix}.dna.chromosome.1.fa data/fasta/{wildcards.genome}
"""
In profile/config.yaml:
snakefile: snakeFile
latency-wait: 60
printshellcmds: True
max-jobs-per-second: 1
max-status-checks-per-second: 10
jobs: 400
jobname: "{rule}.{jobid}"
cluster: "sbatch --output=\"jobs/{rule}/slurm_%x_%j.out\" --error=\"jobs/{rule}/slurm_%x_%j.log\" --cpus-per-task={threads} --ntasks=1 --parsable" # --parsable added for handling the timeout exception
cluster-status: "./profile/status-sacct.sh" # Use to handle timeout exception, do not forget to chmod +x
set-threads:
- downloadgenome=2
In profile/status-sacct.sh:
#!/usr/bin/env bash
# Check status of Slurm job
jobid="$1"
if [[ "$jobid" == Submitted ]]
then
echo smk-simple-slurm: Invalid job ID: "$jobid" >&2
echo smk-simple-slurm: Did you remember to add the flag --parsable to your sbatch call? >&2
exit 1
fi
output=`sacct -j "$jobid" --format State --noheader | head -n 1 | awk '{print $1}'`
if [[ $output =~ ^(COMPLETED).* ]]
then
echo success
elif [[ $output =~ ^(RUNNING|PENDING|COMPLETING|CONFIGURING|SUSPENDED).* ]]
then
echo running
else
echo failed
fi
Now build the conda environments:
cd envs
conda env create -p ./smake --file environment1.yaml
conda env create -p ./smake2 --file environment2.yaml
cd ..
If you run the whole thing with smake2 (snakemake snakemake-minimal=6.15.1) it indeeds run the job with 2 CPUs:
conda activate envs/smake2
snakemake --profile profile/
conda deactivate
rm -r data
rm -r jobs
If you do the same thing with smake (snakemake-minimal=7.3.8), it will crash with the error: Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value: '{downloadgenome :'.
conda activate envs/smake
snakemake --profile profile/
more jobs/downloadgenome/*log
I tried many things without success to solve the problem...
This was indeed a bug and has been fixed in PR 1615.

Unable to execute command inside double square brackets inside my Makefile

clean:
#for container_name in ${NEW_DJANGO_IMAGE_NAME} \
${NEW_MSQL_IMAGE_NAME} \
${NEW_NGINX_IMAGE_NAME} \
${NEW_REDIS_IMAGE_NAME}; \
do if [[ 'a' == 'a' ]]; then echo 'fdfdf'; fi; done;
If I do something like this it works. Now instead of this silly line
do if [[ 'a' == 'a' ]]; then echo 'fdfdf'; fi; done;
I want to write the following:
do if [[ docker ps --filter "name=^/$$container_name$$" --format '{{.Names}}' == $$container_name ]]; then echo 'fdfdf'; fi; done;
The idea is that I iterate over a number of docker containers and if it happens that some of them are running I want to stop them. So in the place of echo 'fdfdf' I want to see this line: docker container stop <CONTAINER_NAME>;
Looks as simple as hell but I can't get it to work in the Makefile...What am I doing wrong?
You apparently think that [[ cmd == "string" ]] executes cmd before performing the test. This is not the case. Use:
[[ `cmd` == "string" ]]
instead. In your case it would look like this:
do if [[ `docker ps --filter "name=^/$$container_name$$" --format '{{.Names}}'` == $$container_name ]]; then docker container stop $$container_name; fi; done;
Or, a bit more readable, maybe:
IMAGES := $(NEW_DJANGO_IMAGE_NAME) $(NEW_MSQL_IMAGE_NAME) \
$(NEW_NGINX_IMAGE_NAME) $(NEW_REDIS_IMAGE_NAME)
clean:
#for cn in $(IMAGES); do \
tmp=`docker ps --filter "name=^/$$cn$$" --format '{{.Names}}'`; \
if [ -n "$$tmp" ]; then \
docker container stop $$cn; \
fi; \
done
Note that, in this last version, we use the bourne shell test commmand ([) instead of the bash-only conditional expression ([[...]]).

I'm not able to run the MVN goals through shell script

I just used the below shell script in Post Build activity in JENKINS "Execute Shell" cmd
#!/bin/bash
mvn --version
export M2_HOME=/opt/maven/maven-3.3.3 # your Mavan home path
export PATH=$PATH:$M2_HOME/bin
mvn --version
echo $HOME
echo $WORKSPACE
file=$WORKSPACE/XXXX/XXXX-reports
cd $file
cp XXXX-1.html test.html
fail=`grep "test-method.*FAIL" results.xml | sed -e 's/^.*test-
instance-name="\(.*\) ' | tr '\n' ','`
echo $fail
count=0
while [ $count -lt 5 ]
do
if [ ! -z "$fail" -a "$fail" != " " ];
then
echo `$M2_HOME/bin/mvn clean test -f ../../pom.xml -DInclude=${fail}`
cp XXXX-report.html ReReport_$count.html
retry=`expr $count + 1`
fi
done
In the above shell script I'm trying to run the mvn command and continuing to run 5 times to increase pass count in UNIT script execution.
But It's not working. It's failed to execute the maven goals. So, I just comment the executable and added echo to debug.
But no luck.
Any leads
I suggest that you use this following script that modifies yours. Add your parameters there.
The product log file: mvn_jenkins.log will allow you to easily find problems as suggested by mjuarez:
#!/bin/bash
MVNCMD=$(command -v mvn)
MVNPARAMS=" -version" # replace with your appropriate params
LOGFILE="mvn_jenkins.log"
COUNT=0
while [ $COUNT -lt 5 ]; do
echo "Execute: $MVNCMD $MVNPARAMS" >> $LOGFILE
echo "-------" >> $LOGFILE
$MVNCMD $MVNPARAMS 2>&1>> $LOGFILE
echo "" >> $LOGFILE
COUNT=$((COUNT + 1))
done
Explanation:
MVNCMD=$(command -v mvn): put the full qualified name of maven execute file in MVNCMD var
$MVNCMD $MVNPARAMS 2>&1>> $LOGFILE: Execute mvn command with parameters and redirect error output messages (2) and regular output messages (1) to the log file.
About your maven parameters I don't understand why you use -DskipTests=true.

Bash: How to check if dynamic env variable is set?

I'm writing a script where I want to dynamically create an environment variable name, and check if it has been set.
#!/bin/bash
#######################################
# Builds options string
# Globals:
# JVM_OPTS_DIR
# Arguments:
# 1. Options env variable name
# 2. File containing options defaults
# Returns:
# Options
#######################################
function buildOpts() {
declare -n opts=$1
declare -n excludeOpts="EXCLUDE_$1"
local -r optsFile=$2
local x=
if [ -z ${excludeOpts+x} ]; then
while read -r o; do
if [ -n "${o// }" ]; then
x+=" $o"
fi
done <"$JVM_OPTS_DIR/$optsFile"
fi
if [ -n "$opts" ]; then
for o in $opts; do
x+=" $o"
done
fi
printf '%s' "$x"
}
# https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html
# Standard options
stdOpts=$(buildOpts STD_OPTS std.opts)
...
javaCmd="java $stdOpts $nonStdOpts $advRtOpts $advCompOpts $advServOpts $advGcOpts -jar $APP_DIR/app.jar"
printf '%s' "$javaCmd"
# eval $javaCmd "$#"
The above script serves as a Docker entrypoint
docker build -t jdk . && docker run --rm -it jdk -e APP_NAME=test -e APP_LOG_DIR=test -e APP_DIR=test -e STD_OPTS='a b' -e EXCLUDE_STD_OPTS=true
However, I don't see a and b included in the javaCmd, neither do I see the EXCLUDE working. Basically, none of the if conditions in function buildOpts are working.
I'm a backend programmer, and not a Bash wizard. Help.
You need to put the envs flags before the image name.
Syntax of docker run is:
docker run [OPTIONS] IMAGE[:TAG|#DIGEST] [COMMAND] [ARG...]
Anything after the image name will be treated as [COMMAND][ARG...]

Conditional in Docker argument in Bash

In my bash file I've something like this
docker run -d \
--network=host \
--name my-service \
--log-driver="$LOGGING" \
if [[ "$LOGGING" == 'splunk' ]]; then
echo "--log-opt tag={{.ImageName}}/{{.Name}}/{{.ID}} \\";
echo "--log-opt env=NODE_ENV \\";
fi
But shellcheck complains by showing the following result. Any idea?
https://github.com/koalaman/shellcheck/wiki/SC1089
Build the argument list first (in an array), then call docker. This has the additional benefit of getting rid of the ugly line continuation characters.
docker_opts=(
-d
--network=host
--name my-service
--log-driver="$LOGGING"
--log-opt="$log_opt"
)
if [[ $LOGGING == splunk ]]; then
docker_opts+=(
--log-opt "tag={{.ImageName}}/{{.Name}}/{{.ID}} \\"
--log-opt "env=NODE_ENV \\"
)
fi
docker run "${docker_opts[#]}"
The main idea, though, is to keep the conditional code as small as possible and keep it separate from the unconditional code.
I suggest to use $(if ..; then ...; fi):
docker run -d \
--network=host \
--name my-service \
--log-driver="$LOGGING" \
$(if [[ "$LOGGING" == 'splunk' ]]; then
echo "--log-opt tag={{.ImageName}}/{{.Name}}/{{.ID}}"
echo "--log-opt env=NODE_ENV"
fi)

Resources