Code
Consider the following makefile snippet:
COMMIT := $(shell git rev-parse HEAD)
build:
docker build -f Dockerfile --no-cache=false -t $(COMMIT) .
rebuild:
docker build -f Dockerfile --no-cache=true -t $(COMMIT) .
The problem
The only difference between build and rebuild is the value of the --no-cache parameter. Obviously, rewriting the same command with a slight change is a bad practice; it breaks the DRY principle, and if I ever need to change something else in the command - for example, the value of -t - I would have to change it across all relevant targets.
I had something like this in mind:
COMMIT := $(shell git rev-parse HEAD)
NO_CACHE := false
build:
docker build -f Dockerfile --no-cache=$(NO_CACHE) -t $(COMMIT) .
rebuild:
NO-CACHE = true
make build
I tried playing with the variables, with no luck.
My question
What would be an elegant way to write the docker build command once, and have each target alter its parameter?
You can use constructed variable names:
COMMIT := $(shell git rev-parse HEAD)
build_NOCACHE = false
rebuild_NOCACHE = true
build rebuild:
docker build -f Dockerfile --no-cache=$($#_NOCACHE) -t $(COMMIT) .
Or you can use target-specific variables:
COMMIT := $(shell git rev-parse HEAD)
build: NOCACHE = false
rebuild: NOCACHE = true
build rebuild:
docker build -f Dockerfile --no-cache=$(NOCACHE) -t $(COMMIT) .
Use the call function
Positional arguments are specified from 1 to n and used in the command definition as $(1), $(2), $(n).
COMMIT := $(shell git rev-parse HEAD)
DOCKER_BUILD_CMD = docker build -f Dockerfile --no-cache=$(1) -t $(COMMIT) .
build:
$(call DOCKER_BUILD_CMD, false)
rebuild:
$(call DOCKER_BUILD_CMD, true)
Related
Thank you for visiting here.
First of all, I apologize for my bad English, maybe a little wrong, hope you can help me.
Then I had a little problem when deploying a new CI/CD system on k8s platform (v1.23.5+1) with Gitlab runner (14.9.0) and dind (docker:dind)
When deploying CI to Golang apps with private repositories at https://gitlab.domain.com, (I did the go env -w GOPRIVATE configuration), I had a problem with the go mod tidy command. Specifically getting the unexpected EOF error. I've tried go mod tidy -v but it doesn't seem to give any more info.
I did a lot of work to figure out the problem. Specifically, I have done wget and git clone commands with my private repository and they are still able to download successfully. I tried adding a private repository at https://gitlab.com in go.mod, they can still be retrieved without any errors.
And actually, without using my new runner, I can still git clone and go mod tidy in another vps.
All of this leaves me wondering where am I actually getting the error? Is it my gitlab or my k8s gitlab runner
This is runner output
go: downloading gitlab.domain.com/nood/fountain v0.0.12
unexpected EOF
Cleaning up project directory and file based variables
ERROR: Job failed: command terminated with exit code 1
This is my .gitlab-ci.yml
image: docker:latest
stages:
- build
- deploy
variables:
GTV_ECR_REPOSITORY_URL: repo.domain.com
PROJECT: nood
APP_NAME: backend-super-system
APP_NAME_ECR: backend-super-system
IMAGE_TAG: $GTV_ECR_REPOSITORY_URL/$PROJECT/$APP_NAME_ECR
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
services:
- name: docker:dind
entrypoint: ["env", "-u", "DOCKER_HOST"]
command: ["dockerd-entrypoint.sh", "--tls=false"]
build:
stage: build
allow_failure: false
script:
- echo "Building image."
- docker pull $IMAGE_TAG || echo "Building runtime from scratch"
- >
docker build
--cache-from $IMAGE_TAG
-t $IMAGE_TAG --network host .
- docker push $IMAGE_TAG
Dockerfile
FROM golang:alpine3.15
LABEL maintainer="NoodExe <nood.pr#gmail.com>"
WORKDIR /app
ENV BIN_DIR=/app/bin
RUN apk add --no-cache gcc build-base git
ADD . .
RUN chmod +x scripts/env.sh scripts/build.sh \
&& ./scripts/env.sh \
&& ./scripts/build.sh
# stage 2
FROM alpine:latest
WORKDIR /app
ENV BIN_DIR=/app/bin
ENV SCRIPTS_DIR=/app/scripts
ENV DATA_DIR=/app/data
# Build Args
ARG LOG_DIR=/var/log/nood
# Create log directory
RUN mkdir -p ${BIN_DIR} \
mkdir -p ${SCRIPTS_DIR} \
mkdir -p ${DATA_DIR} \
mkdir -p ${LOG_DIR} \
&& apk update \
&& addgroup -S nood \
&& adduser -S nood -G nood \
&& chown nood:nood /app \
&& chown nood:nood ${LOG_DIR}
USER nood
COPY --chown=nood:nood --from=0 ${BIN_DIR} /app
COPY --chown=nood:nood --from=0 ${DATA_DIR} ${DATA_DIR}
COPY --chown=nood:nood --from=0 ${SCRIPTS_DIR} ${SCRIPTS_DIR}
RUN chmod +x ${SCRIPTS_DIR}/startup.sh
ENTRYPOINT ["/app/scripts/startup.sh"]
scripts/env.sh
#!/bin/sh
go env -w GOPRIVATE=gitlab.domain.com/*
git config --global --add url."https://nood_deploy:rvbsosecret_Hizt97zQSn#gitlab.domain.com".insteadOf "https://gitlab.domain.com"
scripts/build.sh
#!/bin/sh
grep -v "replace\s.*=>.*" go.mod > tmpfile && mv tmpfile go.mod
go mod tidy
set -e
BIN_DIR=${BIN_DIR:-/app/bin}
mkdir -p "$BIN_DIR"
files=`ls *.go`
echo "****************************************"
echo "******** building applications **********"
echo "****************************************"
for file in $files; do
echo building $file
go build -o "$BIN_DIR"/${file%.go} $file
done
Thank you for still being here :3
This is a known issue with installing go modules from gitlab in nested locations. The issue describes several workarounds/solutions. One solution is described as follows:
create a gitlab Personal Access Token with at least read_api and read_repository scopes.
create a .netrc file:
machine gitlab.com
login yourname#gitlab.com
password yourpersonalaccesstoken
use go get --insecure to get your module
do not use the .gitconfig insteadOf workaround
For self-hosted instances of GitLab, there is also the additional option of using the go proxy, which is what I do to resolve this problem.
For additional context, see this answer to What's the proper way to "go get" a private repository?
What is the meaning of this make target:
.PHONY: docker.%
docker.%:
go mod tidy
docker build -t $* .
docker run -p 5751:5751 -v $$(pwd)/:/work $* /work/config.yaml
In particular, what is the meaning of % in docker.% and how is it being utilized here?
I wrote this simple Makefile to illustrate my problem.
$make target
calls the dep as a dependency and pulls the image
But the subsequent check for docker image list -q $(IMG) does not find my image.
What is happening here and how should I fix this?
IMG := hello-world
.PHONY: target
target: dep
ifeq ($(shell docker image list -q $(IMG)),)
echo "docker image list did not recognize the pull"
endif
.PHONY: dep
dep:
#docker pull $(IMG)
That test isn't subsequent. It's substituted into the Makefile when it's read, before any rules are executed.
You probably want to perform that test in the commands of the target rule:
target: dep
if test -z "$$(docker image list -q $(IMG))"; then \
echo "docker image list did not recognize the pull" >&2; \
false; \
fi
We could change the command to just run docker image inspect - that will return a true status if the image exists, and false otherwise:
target: dep
if ! docker image inspect "$(IMG))" >/dev/null 2>&1; then \
echo "docker image list did not recognize the pull" >&2; \
false; \
fi
So, I have a Docker build command that I have tested which works great
docker build \
-t app \
--no-cache --network host \
--build argssh_private_key="$(cat ~/.ssh/id_rsa)"\
--build-arg python_version="3.6.8" -f Dockerfile .
To ease the pain of the team learning Docker I encapsulated a few of the commands - build, start, stop - within a Makefile. However, within the Makefile I need to change the command slightly by modifying
$(cat ~/.ssh/id_rsa)
to
$(shell cat ~/.ssh/id_rsa)
When I execute the following:
make build
I receive the following message:
Step 13/20 : RUN git clone --depth 1 "${git_user}#${git_host}:${git_repo}" app
---> Running in d2eb41a71315
Cloning into 'app'...
Warning: Permanently added the ECDSA host key for IP address [ip_address] to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
However, I do not have the same issue when executing from the command-line. I I think it has something to do with the way the call the "cat" command but, I do not know a way to resolve.
Any ideas ?
Makefile:
APP_NAME=ccs_data_pipeline
DATA?="${HOME}/data"
DOCKER_FILE=Dockerfile
PYTHON_VERSION?=3.6.8
SRC?=$(shell dirname `pwd`)
PRIVATE_KEY?=$(shell echo $(shell cat ~/.ssh/id_rsa))
build: ## Build container for ccs data pipeline
docker build \
-t $(APP_NAME) \
--no-cache --network host \
--build-arg ssh_private_key="$(PRIVATE_KEY)" \
--build-arg python_version="$(PYTHON_VERSION)" \
-f $(DOCKER_FILE) .
start: ## Start the docker container
docker run \
-it -v $(DATA):/data \
--network host \
--rm \
--name="$(APP_NAME)" $(APP_NAME)
stop: ## Stop the docker container
docker stop $(APP_NAME); \
docker rm $(APP_NAME)
Please show your actual makefile, or at least the entire rule that is having the error. The single command you provided, with no context, is not enough to understand what you're doing or what might be wrong.
Note that it is often not correct to replace a shell operation like $(...) with a make shell command $(shell ...). However, sometimes it will work "by accident", where the real differences between those commands don't happen to matter.
In general you should never use $(shell ...) inside a recipe (I have no idea if this command appears in a recipe). Instead, you should escape all the dollar signs that you want to be passed verbatim to the shell when it runs your recipe:
$$(cat ~/.ssh/id_rsa)
I have several AWS lambda functions which i want to build and deploy and have the following Makefile
#!/bin/bash
#SHELL:=/bin/bash --login
functions = $(sort $(dir $(wildcard */)))
define deploy
cd $1 && npm deploy-dev
endef
deploy:
$(foreach fn,$(functions),$(eval $(call deploy,$(fn))))
Basically each function is a node.js project that has the following entry in the package.json
"scripts": {
"deploy-dev": "npm build && rm -rf node_modules && npm i --production --legacy-bundling && AWS_PROFILE=dev SLS_DEBUG=* sls deploy --stage dev --region us-east-1"
cd into dir and running npm deploy-dev works as expected, but how to automate this?
i get an error:
➜ new_ingest git:(feature/organise) ✗ make deploy (git)-[feature/organise]
Makefile:11: *** missing separator. Stop.
For purposes of finding the problem, we can reduce it to one function, changing this:
deploy:
$(foreach fn,$(functions),$(eval $(call deploy,$(fn))))
to this:
deploy:
$(eval $(call deploy,foo))
(where foo/ is one of your directories.) Now we can see the problem. The $(call ...) part expands to cd foo && npm deploy-dev. It's a command to be passed to the shell, mot to be interpreted by Make. So when you tell Make to eval it, Make says "this is gibberish". If you remove the eval, it works.
Restoring the foreach causes a problem: all of the commands are concatenated into a single line (cd foo && npm deploy-dev cd bar && npm deploy-dev ...). To solve this you could hack deploy:
define deploy
cd $(1) && pwd ; cd - ;
endef
or you could revise the scheme to give each function its own rule.