I'm trying to create a Python virtual environment with a Makefile and also activate it once the make command finishes to ease things for the user. Apparently, this is not possible because "a child process can not alter the parent's environment." I was wondering if there's any workaround for this. This is part of my Makefile so far:
.PHONY: create-venv venv
.DEFAULT_GOAL := all
SHELL=/bin/bash
CPUTYPE = $(shell uname -m | sed "s/\\ /_/g")
SYSTYPE = $(shell uname -s)
BUILDDIR = build/$(SYSTYPE)-$(CPUTYPE)
VENV_NAME?=venv
VENV_DIR=$(BUILDDIR)/${VENV_NAME}
VENV_BIN=$(shell pwd)/${VENV_DIR}/bin
VENV_ACTIVATE=. ${VENV_BIN}/activate
PYTHON=${VENV_BIN}/python3
create-venv:
test -d $(BUILDDIR) || mkdir -p $(BUILDDIR)
which python3 || apt install -y python3 python3-pip
test -d $(VENV_DIR) || python3 -m venv $(VENV_DIR)
venv: ${VENV_BIN}/activate
${VENV_BIN}/activate: setup.py
test -d $(VENV_DIR) || make create-venv
${PYTHON} -m pip install -r requirements.txt
touch $(VENV_BIN)/activate
source ${VENV_BIN}/activate # <- doesn't work
. ${VENV_BIN}/activate # <- doesn't work either
You can activate the environment and run a shell in the activated env:
. ${VENV_BIN}/activate && exec bash
(Please note it must be in one line to be run in one shell; exec is used to replace the shell with a new one.)
When you finish working with the environment you exit and then the Makefile is finished.
You could do something like this.
It relies on your looking at the activate script and seeing what env vars it sets, so it is totally ugly.
$(eval $(shell source $(PYTHON3_VENV)/bin/activate && echo "export PATH := $$PATH; export PYTHONHOME := $$PYTHONHOME; export VIRTUAL_ENV := $$VIRTUAL_ENV" ))
Related
I want to write statements about Podman in Makefile. A UID mapping is used here. But I found that I was always unable to do numerical calculations.
Below is my Makefile. But here ${uid}+1 and similar operations will become empty strings. How should I solve this problem?
Thanks!
HOST_GEM5 := /mnt/disk/cuiyujie/workspace/workGem5/gem5
SIM := ${HOST_GEM5}/X86/gem5.opt
SHELL := /bin/bash
DOCKER_GEM5 := /usr/local/src/gem5
subuidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.UIDMap }}+{{.Size }}{{end }}" ) - 1 )))
subgidSize=$(shell $(( $(podman info --format "{{ range .Host.IDMappings.GIDMap }}+{{.Size }}{{end }}" ) - 1 )))
uid := $(shell id -u)
gid := $(shell id -g)
DOCKER_DIRS_MAP := \
-v ${HOST_GEM5}/runScripts:${DOCKER_GEM5}/runScripts
MAP_CMD := \
--user ${uid}:${gid} \
--uidmap ${uid}:0:1 \
--uidmap 0:1:${uid} \
--uidmap $(($uid+1)):$(($uid+1)):$(($subuidSize-$uid)) \
--gidmap ${gid}:0:1 \
--gidmap 0:1:${gid} \
--gidmap $(($gid+1)):$(($gid+1)):$(($subgidSize-$gid))
.PHONY: default clean run build
default: build
build:
podman run -it --rm ${DOCKER_DIRS_MAP} --security-opt seccomp=unconfined \
${MAP_CMD} \
gerrie/gem5:v1 "/bin/bash"
clean:
rm -rf ${HOST_GEM5}/build/*
Make always uses /bin/sh as the shell it invokes, both for recipes and for $(shell ...) functions. /bin/sh is a POSIX-conforming shell. The syntax you're using is not POSIX shell syntax: it's special enhanced syntax that is only available in the bash shell.
You can either rewrite your scripting to work in POSIX shells (probably by using the expr program to do the math), or add this to your makefile to tell it you want to use bash instead of /bin/sh:
SHELL := /bin/bash
Note, of course, that your makefile will now no longer work on any system that doesn't have a /bin/bash shell.
I have the following simple statement that works without problems in the shell:
if [ -z "$(command -v brew)" ]; then
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
fi
in short: if the program is not found, then install it.
The problem is that I can't convert this construction into a Makefile. So far I understand that the construction itself should look like this:
if [ -z "$(shell command -v brew)" ]; then \
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" \
fi
But how to correctly convert /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" string I do not quite understand. Could you give some tips in this matter?
The dollar signs need to be doubled so make doesn't interpret them. Also there needs to be a semicolon at the end of the bash command to separate it from fi since the backslash will eat the usual newline command separator.
some_target:
if [ -z "$$(command -v brew)" ]; then \
/bin/bash -c "$$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"; \
fi
You need to break it into a couple of items.
Firstly, check for brew:
BREW := $(shell command -v brew)
Then check if the variable is set. If it's not set, then run the installer:
ifeq ($(BREW),)
$(shell bash -c "$$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)")
BREW := $(shell command -v brew)
endif
It's doable in an entire single shell command, but this breaks it into a couple of shell pieces.
the use of := means that the evaluation happens at the time the line is parsed, which is how you get to check and override the value.
In a small, self-contained makefile:
BREW := $(shell command -v brew)
ifeq ($(BREW),)
$(shell bash -c "$$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)")
BREW := $(shell command -v brew)
endif
all:
echo $(BREW)
In a directory I have a config file with my db variables.
This file (db/database.ini) looks like this:
[PostgreSQL]
host=localhost
database=...
user=postgres
password=...
I have another file (db/create_stmts.sql) where I have all my raw create table statements, and i am trying to experiment the use of a Makefile to have a command like this:
make create-db from_file=db/create_stmts.sql
In order not to repeat myself, I thought of tailing the variables of db/database.ini to a file which I would then source, creating shell variables to pass to psql in the make file.
Here's my plan:
make-db:
# from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=$(PWD)/file.sh
tail -n4 db/database.ini > file.sh && . $(file_path)
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U $(user) -d $(database) -q -f $(from_file) && rm file.sh
Which I run by: make create-db from_file=db/create_stmts.sql
Which gives me this message - from which i kindof understand that the sourcing just did not work.
#from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=/home/gabriele/Desktop/TIUK/companies-house/file.sh
tail -n4 db/database.ini > file.sh && .
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U -d -q -f db/schema_tables.sql && rm file.sh
psql: FATAL: Peer authentication failed for user "-d"
Makefile:3: recipe for target 'create-db' failed
make: *** [create-db] Error 2
Any help?
Another solution, perhaps simpler to understand:
make-db:
file_path=$$PWD/file.sh; \
tail -n4 db/database.ini > file.sh && . $$file_path; \
psql -U $$user -d $$database -q -f $$from_file && rm file.sh
Note using ; and \ to convince make to run all commands in a single shell, and using $$ to escape the $ and use shell variable references.
The error is in the text, namely
psql -U -d -q -f db/schema_tables.sql && rm file.sh
This happens because the variables $(user) and $(database) aren't set. Every line within a target is executed in a sub shell. There is now way to use source like you would in a regular script.
You could create a file named database.mk in which you define these variables and use include database.mk at the top of your makefile to include them:
Makefile
CONFILE ?= database
include $(CONFILE).mk
test:
#echo $(user)
#echo $(database)
database.mk
user := user
database := data
If you want to parse the ini file you could do that as such
CONFILE := db/database.ini
make-db: _setup_con
echo $(user) $(database)
# your target
_setup_con:
$(eval user=$(shell grep "user=" $(CONFILE) | grep -Eo "[^=]*$$"))
$(eval database=$(shell grep "database=" $(CONFILE) | grep -Eo "[^=]*$$"))
# and so forward
I would make it more Make-way by using feature of automatic Makefile generation. Given that a configuration file is a simple properties file, its syntax is easily parseable by Make, it's sufficient to just get the lines with variables, i.e.:
include database.mk
database.mk: db/database.ini
grep -E '^\w+=\w+$$' $< > $#
.PHONY: create-db
create-db: $(from_file)
psql -U $(user) -d $(database) -q -f $<
Some additional notes:
create-db should be made .PHONY to avoid situation when nothing is done due to somebody creating (accidentally or not) a file named create-db,
by making create-db depending on from_file one can get a clean and readable error from make that a file does not exist instead of possibly cryptic error later.
I have the following code in a Makefile, which execute one sequence of command or another based on a environmental variable.
generate :
if test -z "$$VIRTUAL_ENV"; then \
$(PYTHON) -m fades -V &>/dev/null || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt script.py;"; \
else \
python -m pip install -r requirements.txt && python script.py; \
fi
It works as expected, but I would like to do the same thing on multiple targets, to use it on other files, without having to copy this snippet of code multiple times.
My idea would be to set a variable dynamically (based on the condition that has been evaluated), containing the one command or the other, to be used over and over again, like alias in Bash.
Is that a good idea? Is it possible to set a global alias in the Makefile so it can choose between two Python interpreters based on an environmental variable?
Assuming you're using GNU make, you can do it like this:
ifdef VIRTUAL_ENV
PYCMD = python -m pip install -r requirements.txt && python
else
PYCMD = $(PYTHON) -m fades -V >/dev/null 2>&1 || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt
endif
generate:
$(PYCMD) script.py
Note I changed &>/dev/null to >/dev/null 2>&1 because the former is a bash-only feature and is not valid in POSIX sh, and make (by default) always runs /bin/sh which is (on many systems) a POSIX sh.
I don't know why you're using python in one section and $(PYTHON) in the other; it seems like you'd want to use the same in both but anyway.
This is the install part of my Makefile:
install:
for e in $(EXEC); do \
sudo cp --remove-destination ${CURDIR}/$$e /usr/local/bin; done
sudo cp ${CURDIR}/bin/stage2.d /etc/bash_completion.d/stage2
. /etc/bash_completion
Where "stage2" is the name of my executable.
The last line is what provides the issue. After adding a file into bash_completion.d directory I want to source the bash_completion. But calling source or . yields:
. /etc/bash_completion
/etc/bash_completion: 32: [[: not found
/etc/bash_completion: 38: [[: not found
/etc/bash_completion: 50: Bad substitution
make: *** [install] Error 2
I see two issues:
Using sudo in a Makefile rule is a bad idea. Avoid that, and call sudo make install instead.
Why would you want to source the bash_completion file in the non-interactive shell which is the make rule? It makes no sense.
As to solving them:
install:
$(INSTALL) -m 0755 -d $(DESTDIR)$(bindir)
$(INSTALL) -m 0755 -p $(EXEC) $(DESTDIR)$(bindir)/
$(INSTALL) -m 0755 -d $(DESTDIR)$(sysconfdir)/bash_completion.d
$(INSTALL) -m 0644 -p ${CURDIR}/bin/stage2.d $(DESTDIR)$(sysconfdir)/bash_completion.d/stage2
for the make rule part and
sudo make install && . /etc/bash_completion.d/stage2
for the actual running of that command. You will want to document the latter in a README, or in a line which the install: target prints when finished.
Make uses /bin/sh by default. you have to force it to use bash since [[ is not supported by normal sh.
gnu make lets you set SHELL variable [in the makefile] to force it to use bash. So you would need to add a line
SHELL=/bin/bash
at the top of your makefile