How To Make Makefile with .PHONY Targets that work on Windows (Windows_NT) and Mac (Darwin) - makefile

So I found a link that shows I should use the following, but perhaps my logic is wrong within the Makefile. I need to use the Makefile for testing purposes to work on both Mac and Windows. The image is fine and the docker container works, I am just trying to make use of the fact that in Linux/Mac \ can be used to shorten the commands, whereas in Windows you have to use the backtick (`).
Example:
.PHONY: validate-lookml
validate-lookml:
UNAME_S=$(shell uname -s)
ifeq ($(UNAME), Linux)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
-e LOOKER_CLIENTID=${LOOKER_CLIENTID} \
-e LOOKER_CLIENT_SECRET=${LOOKER_CLIENTSECRET} mirantis/mirantis_spectacles \
lookml \
--base-url ${LOOKER_BASE_URL} \
--client-id ${LOOKER_CLIENTID} \
--client-secret ${LOOKER_CLIENTSECRET} \
--project ${PROJECT} \
--branch ${BRANCH}
endif
ifeq ($(UNAME), Darwin)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
-e LOOKER_CLIENTID=${LOOKER_CLIENTID} \
-e LOOKER_CLIENT_SECRET=${LOOKER_CLIENTSECRET} mirantis/mirantis_spectacles \
lookml \
--base-url ${LOOKER_BASE_URL} \
--client-id ${LOOKER_CLIENTID} \
--client-secret ${LOOKER_CLIENTSECRET} \
--project ${PROJECT} \
--branch ${BRANCH}
endif
ifeq ($(UNAME), Windows_NT)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} `
-e LOOKER_CLIENTID=${LOOKER_CLIENTID} `
-e LOOKER_CLIENT_SECRET=${LOOKER_CLIENTSECRET} mirantis/mirantis_spectacles `
lookml `
--base-url ${LOOKER_BASE_URL} `
--client-id ${LOOKER_CLIENTID} `
--client-secret ${LOOKER_CLIENTSECRET} `
--project ${PROJECT} `
--branch ${BRANCH}
endif
Unfortunately, it doesn't work on Windows, and I need my Makefile to support analysts on Windows Laptops:
Error:
C:\Users\richa\Git\Mirantis\dataops-looker [main ≡ +0 ~1 -0 !]> make -s validate-lookml
process_begin: CreateProcess(NULL, uname -s, ...) failed.
Makefile:9: pipe: No error
process_begin: CreateProcess(NULL, uname -s, ...) failed.
Makefile:12: pipe: No error
process_begin: CreateProcess(NULL, uname -s, ...) failed.
Makefile:15: pipe: No error
usage: spectacles lookml [-h] [--config-file CONFIG_FILE] --base-url BASE_URL
--client-id CLIENT_ID --client-secret CLIENT_SECRET
[--port PORT] [--api-version API_VERSION] [-v]
[--log-dir LOG_DIR] [--do-not-track]
[--severity {success,info,warning,error,fatal}]
--project PROJECT [--branch BRANCH]
[--remote-reset | --commit-ref COMMIT_REF | --pin-imports PIN_IMPORTS [PIN_IMPORTS ...]]
spectacles lookml: error: argument --base-url: expected one argument
make: *** [Makefile:40: validate-lookml] Error 2

Much confusion here.
A makefile consists of lines written in two completely different languages: one is the make language, and the other is the shell. You cannot send make operations to the shell, and you cannot run (directly) shell commands in make.
Make tells the difference between these two by use of the TAB character. Lines that are not indented with TAB are parsed by make, and lines that are indented with TAB are given to the shell. So, in your makefile:
validate-lookml:
UNAME_S=$(shell uname -s)
ifeq ($(UNAME), Linux)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
this is not right because the first two indented lines here are make commands, and the third is a shell command. You should write this like:
UNAME_S := $(shell uname -s)
validate-lookml:
ifeq ($(UNAME), Linux)
docker run --rm -it -e LOOKER_BASE_URL=${LOOKER_BASE_URL} \
...
endif
ifeq ($(UNAME), Darwin)
...
etc.
But, there is no uname command on Windows so when you run this it won't work, that's why you're getting the error process_begin: CreateProcess(NULL, uname -s, ...) failed. If you have GNU make 4.0 or better I recommend that you look at the MAKE_HOST variable and use that instead of trying to run uname.
Finally, you don't have to worry about the backslash difference, because make will parse all the backslashes and remove them on its own BEFORE it starts the shell. So just use backslashes to continue all the lines in your recipe and it will work the same way on all different platforms.

Related

How to write numeric expressions in Makefile?

I want to write statements about Podman in Makefile. A UID mapping is used here. But I found that I was always unable to do numerical calculations.
Below is my Makefile. But here ${uid}+1 and similar operations will become empty strings. How should I solve this problem?
Thanks!
HOST_GEM5 := /mnt/disk/cuiyujie/workspace/workGem5/gem5
SIM := ${HOST_GEM5}/X86/gem5.opt
SHELL := /bin/bash
DOCKER_GEM5 := /usr/local/src/gem5
subuidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.UIDMap }}+{{.Size }}{{end }}" ) - 1 )))
subgidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.GIDMap }}+{{.Size }}{{end }}" ) - 1 )))
uid := $(shell id -u)
gid := $(shell id -g)
DOCKER_DIRS_MAP := \
-v ${HOST_GEM5}/runScripts:${DOCKER_GEM5}/runScripts
MAP_CMD := \
--user ${uid}:${gid} \
--uidmap ${uid}:0:1 \
--uidmap 0:1:${uid} \
--uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
--gidmap ${gid}:0:1 \
--gidmap 0:1:${gid} \
--gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid))
.PHONY: default clean run build
default: build
build:
podman run -it --rm ${DOCKER_DIRS_MAP} --security-opt seccomp=unconfined \
${MAP_CMD} \
gerrie/gem5:v1 "/bin/bash"
clean:
rm -rf ${HOST_GEM5}/build/*
Make always uses /bin/sh as the shell it invokes, both for recipes and for $(shell ...) functions. /bin/sh is a POSIX-conforming shell. The syntax you're using is not POSIX shell syntax: it's special enhanced syntax that is only available in the bash shell.
You can either rewrite your scripting to work in POSIX shells (probably by using the expr program to do the math), or add this to your makefile to tell it you want to use bash instead of /bin/sh:
SHELL := /bin/bash
Note, of course, that your makefile will now no longer work on any system that doesn't have a /bin/bash shell.

Passing Variables in Makefile

I'm using a Makefile to run various docker-compose commands and I'm trying to capture the output of a script run on my local machine and pass that value to a Docker image.
start-service:
VERSION=$(shell aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$(VERSION) \
connect make run-service
When I run this I can see the variable being assigned but it still errors. Why is the value not getting passed into the -e argument:
VERSION=1.2.3-build342 && \
docker-compose -f ./compose/docker-compose.yml run --rm \
-e VERSION?=$(VERSION) \
connect make run-connect
/bin/sh: VERSION: command not found
You're mixing several different Bourne shell and Make syntaxes here. The Make $$(VERSION) translates to shell $(VERSION), which is command-substitution syntax; GNU Make $(shell ...) generally expands at the wrong time and isn't what you want here.
If you were writing this as an ordinary shell command it would look like
# Set VERSION using $(...) substitution syntax
# Refer to just plain $VERSION
VERSION=$(aws s3 ls s3://redact/downloads/1.2.3/) && ... \
-e VERSION=$VERSION ... \
So when you use this in a Make context, if none of the variables are Make variables (they get set and used in the same command), just double the $ to $$ not escape them.
start-service:
VERSION=$$(aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$VERSION \
connect make run-service

Docker build fails when executed within Makefile

So, I have a Docker build command that I have tested which works great
docker build \
-t app \
--no-cache --network host \
--build argssh_private_key="$(cat ~/.ssh/id_rsa)"\
--build-arg python_version="3.6.8" -f Dockerfile .
To ease the pain of the team learning Docker I encapsulated a few of the commands - build, start, stop - within a Makefile. However, within the Makefile I need to change the command slightly by modifying
$(cat ~/.ssh/id_rsa)
to
$(shell cat ~/.ssh/id_rsa)
When I execute the following:
make build
I receive the following message:
Step 13/20 : RUN git clone --depth 1 "${git_user}#${git_host}:${git_repo}" app
---> Running in d2eb41a71315
Cloning into 'app'...
Warning: Permanently added the ECDSA host key for IP address [ip_address] to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
However, I do not have the same issue when executing from the command-line. I I think it has something to do with the way the call the "cat" command but, I do not know a way to resolve.
Any ideas ?
Makefile:
APP_NAME=ccs_data_pipeline
DATA?="${HOME}/data"
DOCKER_FILE=Dockerfile
PYTHON_VERSION?=3.6.8
SRC?=$(shell dirname `pwd`)
PRIVATE_KEY?=$(shell echo $(shell cat ~/.ssh/id_rsa))
build: ## Build container for ccs data pipeline
docker build \
-t $(APP_NAME) \
--no-cache --network host \
--build-arg ssh_private_key="$(PRIVATE_KEY)" \
--build-arg python_version="$(PYTHON_VERSION)" \
-f $(DOCKER_FILE) .
start: ## Start the docker container
docker run \
-it -v $(DATA):/data \
--network host \
--rm \
--name="$(APP_NAME)" $(APP_NAME)
stop: ## Stop the docker container
docker stop $(APP_NAME); \
docker rm $(APP_NAME)
Please show your actual makefile, or at least the entire rule that is having the error. The single command you provided, with no context, is not enough to understand what you're doing or what might be wrong.
Note that it is often not correct to replace a shell operation like $(...) with a make shell command $(shell ...). However, sometimes it will work "by accident", where the real differences between those commands don't happen to matter.
In general you should never use $(shell ...) inside a recipe (I have no idea if this command appears in a recipe). Instead, you should escape all the dollar signs that you want to be passed verbatim to the shell when it runs your recipe:
$$(cat ~/.ssh/id_rsa)

How to break up command in CircleCI yml to multiple lines? [duplicate]

This question already has answers here:
How do I break a string in YAML over multiple lines?
(9 answers)
Closed 5 years ago.
I have a CircleCI configuration file that looks like so:
# Customize test commands
test:
override:
- docker run -e VAR1=$VAR! -e VAR2=$VAR2 -e $VAR3-$VAR3 --entrypoint python my_image:latest -m unittest discover -v -s test
How can I break up the docker run command into multiple lines like:
docker run \
-e VAR1=$VAR! \
-e VAR2=$VAR2 \
-e $VAR3-$VAR3 \
--entrypoint python my_image:latest \
-m unittest discover -v -s test
I've tried using the | operator for yaml, but CircleCI was unable to parse because it expects override to be a list.
# Customize test commands
test:
override: |
docker run \
-e VAR1=$VAR! \
-e VAR2=$VAR2 \
-e $VAR3-$VAR3 \
--entrypoint python my_image:latest \
-m unittest discover -v -s test
Using this answer which details the various ways to break up a string over multiple lines in yaml, I was able to deduce a solution which works nicely.
Note the use of the >- operator in the override section.
test:
override:
- >-
docker run
-e VAR1=$VAR!
-e VAR2=$VAR2
-e $VAR3-$VAR3
--entrypoint python my_image:latest
-m unittest discover -v -s test
This generates a nice single-line command of:
docker run -e VAR1=$VAR! -e VAR2=$VAR2 -e $VAR3-$VAR3 --entrypoint python my_image:latest -m unittest discover -v -s test

Make 2 volumes in docker gcc and java

I have a docker volume set up in a gcc container. I need to compile the code and give it some stdin through a text file. I'm able to do that using the following command.
docker run \
-v /home/usr/workspace/proj/WebContent/files:/mycode \
gcc:4.9 \
sh -c 'cd mycode; gcc -o myapp ./mycode.c; ./myapp < ./test.txt'
now my question is, I need to make a separate folder for each of my users with their username, but the text.txt stays in the same folder as above. How do I give them their own paths. because right now i get an error that test.txt is not found, and of course it wouldn't. I tried making a separate volume for the test.txt but I guess making two volumes in one container isn't possible or I'm doing it wrong.
What I've tried (please don't judge, i'm just learning :P)
docker run \
-v /home/usr/workspace/proj/WebContent/file/username:/mycode \
-v /home/usr/workspace/proj/WebContent/file/:/tst \
gcc:4.9 \
sh -c 'cd mycode; gcc -o myapp ./mycode.c; cd tst; ./myapp < ./test.txt'
Have a close look at this command:
sh -c 'cd mycode; gcc -o myapp ./mycode.c; cd tst; ./myapp < ./test.txt'
Because you first cd mycode and then cd tst inside the same shell, you're trying to cd mycode/tst, whereas your mount is at /tst.
Similarly, myapp is in /mycode, not in tst, so you can't run ./myapp inside the tst directory and expect it to work.
Instead:
sh -xc 'cd /mycode && gcc -o myapp ./mycode.c && cd /tst && /mycode/myapp < ./test.txt'

Resources