I want to write statements about Podman in Makefile. A UID mapping is used here. But I found that I was always unable to do numerical calculations.
Below is my Makefile. But here ${uid}+1 and similar operations will become empty strings. How should I solve this problem?
Thanks!
HOST_GEM5 := /mnt/disk/cuiyujie/workspace/workGem5/gem5
SIM := ${HOST_GEM5}/X86/gem5.opt
SHELL := /bin/bash
DOCKER_GEM5 := /usr/local/src/gem5
subuidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.UIDMap }}+{{.Size }}{{end }}" ) - 1 )))
subgidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.GIDMap }}+{{.Size }}{{end }}" ) - 1 )))
uid := $(shell id -u)
gid := $(shell id -g)
DOCKER_DIRS_MAP := \
-v ${HOST_GEM5}/runScripts:${DOCKER_GEM5}/runScripts
MAP_CMD := \
--user ${uid}:${gid} \
--uidmap ${uid}:0:1 \
--uidmap 0:1:${uid} \
--uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
--gidmap ${gid}:0:1 \
--gidmap 0:1:${gid} \
--gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid))
.PHONY: default clean run build
default: build
build:
podman run -it --rm ${DOCKER_DIRS_MAP} --security-opt seccomp=unconfined \
${MAP_CMD} \
gerrie/gem5:v1 "/bin/bash"
clean:
rm -rf ${HOST_GEM5}/build/*
Make always uses /bin/sh as the shell it invokes, both for recipes and for $(shell ...) functions. /bin/sh is a POSIX-conforming shell. The syntax you're using is not POSIX shell syntax: it's special enhanced syntax that is only available in the bash shell.
You can either rewrite your scripting to work in POSIX shells (probably by using the expr program to do the math), or add this to your makefile to tell it you want to use bash instead of /bin/sh:
SHELL := /bin/bash
Note, of course, that your makefile will now no longer work on any system that doesn't have a /bin/bash shell.
Related
So I found a link that shows I should use the following, but perhaps my logic is wrong within the Makefile. I need to use the Makefile for testing purposes to work on both Mac and Windows. The image is fine and the docker container works, I am just trying to make use of the fact that in Linux/Mac \ can be used to shorten the commands, whereas in Windows you have to use the backtick (`).
Example:
.PHONY: validate-lookml
validate-lookml:
UNAME_S=$(shell uname -s)
ifeq ($(UNAME), Linux)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
-e LOOKER_CLIENTID=${LOOKER_CLIENTID} \
-e LOOKER_CLIENT_SECRET=${LOOKER_CLIENTSECRET} mirantis/mirantis_spectacles \
lookml \
--base-url ${LOOKER_BASE_URL} \
--client-id ${LOOKER_CLIENTID} \
--client-secret ${LOOKER_CLIENTSECRET} \
--project ${PROJECT} \
--branch ${BRANCH}
endif
ifeq ($(UNAME), Darwin)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
-e LOOKER_CLIENTID=${LOOKER_CLIENTID} \
-e LOOKER_CLIENT_SECRET=${LOOKER_CLIENTSECRET} mirantis/mirantis_spectacles \
lookml \
--base-url ${LOOKER_BASE_URL} \
--client-id ${LOOKER_CLIENTID} \
--client-secret ${LOOKER_CLIENTSECRET} \
--project ${PROJECT} \
--branch ${BRANCH}
endif
ifeq ($(UNAME), Windows_NT)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} `
-e LOOKER_CLIENTID=${LOOKER_CLIENTID} `
-e LOOKER_CLIENT_SECRET=${LOOKER_CLIENTSECRET} mirantis/mirantis_spectacles `
lookml `
--base-url ${LOOKER_BASE_URL} `
--client-id ${LOOKER_CLIENTID} `
--client-secret ${LOOKER_CLIENTSECRET} `
--project ${PROJECT} `
--branch ${BRANCH}
endif
Unfortunately, it doesn't work on Windows, and I need my Makefile to support analysts on Windows Laptops:
Error:
C:\Users\richa\Git\Mirantis\dataops-looker [main ≡ +0 ~1 -0 !]> make -s validate-lookml
process_begin: CreateProcess(NULL, uname -s, ...) failed.
Makefile:9: pipe: No error
process_begin: CreateProcess(NULL, uname -s, ...) failed.
Makefile:12: pipe: No error
process_begin: CreateProcess(NULL, uname -s, ...) failed.
Makefile:15: pipe: No error
usage: spectacles lookml [-h] [--config-file CONFIG_FILE] --base-url BASE_URL
--client-id CLIENT_ID --client-secret CLIENT_SECRET
[--port PORT] [--api-version API_VERSION] [-v]
[--log-dir LOG_DIR] [--do-not-track]
[--severity {success,info,warning,error,fatal}]
--project PROJECT [--branch BRANCH]
[--remote-reset | --commit-ref COMMIT_REF | --pin-imports PIN_IMPORTS [PIN_IMPORTS ...]]
spectacles lookml: error: argument --base-url: expected one argument
make: *** [Makefile:40: validate-lookml] Error 2
Much confusion here.
A makefile consists of lines written in two completely different languages: one is the make language, and the other is the shell. You cannot send make operations to the shell, and you cannot run (directly) shell commands in make.
Make tells the difference between these two by use of the TAB character. Lines that are not indented with TAB are parsed by make, and lines that are indented with TAB are given to the shell. So, in your makefile:
validate-lookml:
UNAME_S=$(shell uname -s)
ifeq ($(UNAME), Linux)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
this is not right because the first two indented lines here are make commands, and the third is a shell command. You should write this like:
UNAME_S := $(shell uname -s)
validate-lookml:
ifeq ($(UNAME), Linux)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
...
endif
ifeq ($(UNAME), Darwin)
...
etc.
But, there is no uname command on Windows so when you run this it won't work, that's why you're getting the error process_begin: CreateProcess(NULL, uname -s, ...) failed. If you have GNU make 4.0 or better I recommend that you look at the MAKE_HOST variable and use that instead of trying to run uname.
Finally, you don't have to worry about the backslash difference, because make will parse all the backslashes and remove them on its own BEFORE it starts the shell. So just use backslashes to continue all the lines in your recipe and it will work the same way on all different platforms.
I'm using a Makefile to run various docker-compose commands and I'm trying to capture the output of a script run on my local machine and pass that value to a Docker image.
start-service:
VERSION=$(shell aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$(VERSION) \
connect make run-service
When I run this I can see the variable being assigned but it still errors. Why is the value not getting passed into the -e argument:
VERSION=1.2.3-build342 && \
docker-compose -f ./compose/docker-compose.yml run --rm \
-e VERSION?=$(VERSION) \
connect make run-connect
/bin/sh: VERSION: command not found
You're mixing several different Bourne shell and Make syntaxes here. The Make $$(VERSION) translates to shell $(VERSION), which is command-substitution syntax; GNU Make $(shell ...) generally expands at the wrong time and isn't what you want here.
If you were writing this as an ordinary shell command it would look like
# Set VERSION using $(...) substitution syntax
# Refer to just plain $VERSION
VERSION=$(aws s3 ls s3://redact/downloads/1.2.3/) && ... \
-e VERSION=$VERSION ... \
So when you use this in a Make context, if none of the variables are Make variables (they get set and used in the same command), just double the $ to $$ not escape them.
start-service:
VERSION=$$(aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$VERSION \
connect make run-service
I have a docker image that I want to run locally and to make my life easier I am using make file to pass AWS environment variable.
aws_access_key_id := $(shell aws configure get aws_access_key_id)
aws_secret_access_key := $(shell aws configure get aws_secret_access_key)
aws_region := $(shell aws configure get region)
docker-run:
docker run -e AWS_ACCESS_KEY_ID="$(aws_access_key_id)" -e AWS_SECRET_ACCESS_KEY="$(aws_secret_access_key)" -e AWS_DEFAULT_REGION="$(aws_region)" --rm mydocker-image
And I need to find a way to do something like this in my terminal
make docker-run -d my_db -s dev -t my_table -u my_user -i URI://redshift
make docker-run --pre-actions "delete from dev.my_table where first_name = 'John'" -s dev -t my_table
make docker-run -s3 s3://temp-parquet/avro/ -s dev -t my_table -u myuser -i URI://redshift
These are the arguments that my docker (python application with argparse) will accept.
You can't do that, directly. The command line arguments to make are parsed by make, and must be valid make program command line arguments. Makefiles are not shell scripts and make is not a general interpreter: there's no facility for passing arbitrary options to it.
You can do this by putting them into a variable, like this:
make docker-run DOCKER_ARGS="-d my_db -s dev -t my_table -u my_user -i URI://redshift"
make docker-run DOCKER_ARGS="-d my_db -s dev -t my_table"
then use $(DOCKER_ARGS) in your makefile. But that's the only way.
If you want to do argument parsing yourself, you probably don't want a Makefile! You should probably write a Bash script instead.
Example:
#!/usr/bin/env bash
set -euo pipefail
aws_access_key_id="$(aws configure get aws_access_key_id)"
aws_secret_access_key="$(aws configure get aws_secret_access_key)"
aws_region="$(aws configure get region)"
docker run -e AWS_ACCESS_KEY_ID="$aws_access_key_id" -e AWS_SECRET_ACCESS_KEY="$aws_secret_access_key" -e AWS_DEFAULT_REGION="$aws_region" --rm mydocker-imagedocker "$#"
Note the $# at the end, which passes the arguments from Bash to the docker command.
You might want to try someting like:
$ cat Makefile
all:
#echo make docker-run -d my_db -s dev -t my_table $${MYUSER+-u "$(MYUSER)"} $${URI+-i "URI://$(URI)"}
$ make
make docker-run -d my_db -s dev -t my_table
$ make MYUSER=myuser URI=redshift
make docker-run -d my_db -s dev -t my_table -u myuser -i URI://redshift
I'm trying to create a Python virtual environment with a Makefile and also activate it once the make command finishes to ease things for the user. Apparently, this is not possible because "a child process can not alter the parent's environment." I was wondering if there's any workaround for this. This is part of my Makefile so far:
.PHONY: create-venv venv
.DEFAULT_GOAL := all
SHELL=/bin/bash
CPUTYPE = $(shell uname -m | sed "s/\\ /_/g")
SYSTYPE = $(shell uname -s)
BUILDDIR = build/$(SYSTYPE)-$(CPUTYPE)
VENV_NAME?=venv
VENV_DIR=$(BUILDDIR)/${VENV_NAME}
VENV_BIN=$(shell pwd)/${VENV_DIR}/bin
VENV_ACTIVATE=. ${VENV_BIN}/activate
PYTHON=${VENV_BIN}/python3
create-venv:
test -d $(BUILDDIR) || mkdir -p $(BUILDDIR)
which python3 || apt install -y python3 python3-pip
test -d $(VENV_DIR) || python3 -m venv $(VENV_DIR)
venv: ${VENV_BIN}/activate
${VENV_BIN}/activate: setup.py
test -d $(VENV_DIR) || make create-venv
${PYTHON} -m pip install -r requirements.txt
touch $(VENV_BIN)/activate
source ${VENV_BIN}/activate # <- doesn't work
. ${VENV_BIN}/activate # <- doesn't work either
You can activate the environment and run a shell in the activated env:
. ${VENV_BIN}/activate && exec bash
(Please note it must be in one line to be run in one shell; exec is used to replace the shell with a new one.)
When you finish working with the environment you exit and then the Makefile is finished.
You could do something like this.
It relies on your looking at the activate script and seeing what env vars it sets, so it is totally ugly.
$(eval $(shell source $(PYTHON3_VENV)/bin/activate && echo "export PATH := $$PATH; export PYTHONHOME := $$PYTHONHOME; export VIRTUAL_ENV := $$VIRTUAL_ENV" ))
I have the following code in a Makefile, which execute one sequence of command or another based on a environmental variable.
generate :
if test -z "$$VIRTUAL_ENV"; then \
$(PYTHON) -m fades -V &>/dev/null || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt script.py;"; \
else \
python -m pip install -r requirements.txt && python script.py; \
fi
It works as expected, but I would like to do the same thing on multiple targets, to use it on other files, without having to copy this snippet of code multiple times.
My idea would be to set a variable dynamically (based on the condition that has been evaluated), containing the one command or the other, to be used over and over again, like alias in Bash.
Is that a good idea? Is it possible to set a global alias in the Makefile so it can choose between two Python interpreters based on an environmental variable?
Assuming you're using GNU make, you can do it like this:
ifdef VIRTUAL_ENV
PYCMD = python -m pip install -r requirements.txt && python
else
PYCMD = $(PYTHON) -m fades -V >/dev/null 2>&1 || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt
endif
generate:
$(PYCMD) script.py
Note I changed &>/dev/null to >/dev/null 2>&1 because the former is a bash-only feature and is not valid in POSIX sh, and make (by default) always runs /bin/sh which is (on many systems) a POSIX sh.
I don't know why you're using python in one section and $(PYTHON) in the other; it seems like you'd want to use the same in both but anyway.