Makefile path substitution with multiple wildcards - makefile

I have a target that is a path to a test.mk file which needs to be generated (modified by a script based on the configuration) and I want to extract a part of the path that is the component to be tested.
The path is in the following format, where [component] and [test_name] are unknown.
TEST=absolute_path/[component]/unit_tests/[test_name]/test.mk
I want to reformat this path as relative_path/[component] and I have tried this line which doesn't substitute anything.
$(TEST):
#echo $(patsubst absolute_path/%/unit_tests/%, relative_path/%, $#)
The GNU make manual has this to say about patsubst so I think it should work with multiple wildcards in a single patsubst invocation:
Here pattern may contain a ‘%’ which acts as a wildcard, matching any number of any characters within a word. If replacement also contains a ‘%’, the ‘%’ is replaced by the text that matched the ‘%’ in pattern. Only the first ‘%’ in the pattern and replacement is treated this way; any subsequent ‘%’ is unchanged.

I presume that you know the relative path. I also assume that the name of the component and the name of the test are variable (they can change). So I call them 'component_X' and 'test_name_Y'. Assuming all that, the following should work:
# Definitions for the space-string and the newline-string
# ---------------------------------------------------------
nullstring :=
space := $(nullstring) #End
TEST=some/abs/path/component_X/unit_tests/test_name_Y/test.mk
# Some manipulations of the path:
# ---------------------------------
TEMP1=$(subst /,$(space),$(TEST))
TEMP2=$(filter-out $(lastword $(TEMP1)),$(TEMP1))
TEMP3=$(filter-out $(lastword $(TEMP2)),$(TEMP2))
TEMP4=$(filter-out $(lastword $(TEMP3)),$(TEMP3))
ABS_PATH=$(subst $(space),/,$(filter-out $(lastword $(TEMP4)),$(TEMP4)))
REL_PATH=some/rel/path
REL_FILE_PATH=$(patsubst $(ABS_PATH)/%,$(REL_PATH)/%,$(TEST))
COMP_NAME=$(firstword $(subst /,$(space),$(patsubst $(ABS_PATH)/%,$(nullstring)%,$(TEST))))
TEST_NAME=$(firstword $(subst /,$(space),$(patsubst $(ABS_PATH)/$(COMP_NAME)/unit_tests/%,$(nullstring)%,$(TEST))))
# The test target
# ----------------
$(TEST): FORCE
#echo.
#echo The calculated ABS_PATH:
#echo $(ABS_PATH)
#echo.
#echo The chosen REL_PATH:
#echo $(REL_PATH)
#echo.
#echo The relative file path:
#echo $(REL_FILE_PATH)
#echo.
#echo The component name:
#echo $(COMP_NAME)
#echo.
#echo The test name:
#echo $(TEST_NAME)
FORCE:
Don't forget that copy-pasting can lead to errors in make. You should replace all 4-spaces by a tab-character in the recipe lines.
Please let me know if this solution worked for you.
I tested it on Windows. But it should work on Linux as well.
EDIT :
I made a small error in the definition for ABS_PATH=.... Now it should be correct :-)

Related

Why does make select pattern rule if only a substring matches?

I am writing a makefile that is supposed to create multiple executables from the code for different architectures. The idea is that the suffix of each executable indicates the architecture for which it was built. This is a simplified excerpt:
TARGET_NAME = foo
TARGET_AMD64 = $(TARGET_NAME)_amd64
TARGET_ARMHF = $(TARGET_NAME)_armhf
.PHONY:
all: $(TARGET_AMD64) $(TARGET_ARMHF) ;
$(TARGET_NAME)_%: lib/mylib.a
#echo "$# built."
lib/mylib.a: bin/mylib/src/mac/foo_ct.o
#echo "$# built."
When typing make I receive a circular dependency:
make: Circular bin/mylib/src/mac/foo_ct.o <- lib/mylib.a dependency dropped.
bin/mylib/src/mac/foo_ct.o built.
lib/mylib.a built.
foo_amd64 built.
foo_armhf built.
Apparently, make is looking for a rule that matches bin/mylib/src/mac/foo_ct.o. For reasons beyond my understanding, make takes $(TARGET_NAME)_% (the second rule).
Why is that the case? Only the filename foo_ct.o would match the pattern rule $(TARGET_NAME)_%, but why is the preceding path bin/mylib/src/mac/ ignored?
This is actually the defined behaviour:
When the target pattern does not contain a slash (and it usually does not), directory names in the file names are removed from the file name before it is compared with the target prefix and suffix. After the comparison of the file name to the target pattern, the directory names, along with the slash that ends them, are added on to the prerequisite file names generated from the pattern rule’s prerequisite patterns and the file name. The directories are ignored [...]
You will therefore have to specify your target with its absolute path:
TARGET_NAME = $(CURDIR)/foo
Note however, that $(CURDIR) is a GNU make extension.
Alternatively you could also specify a more specific rule for `bin/mylib/src/mac/foo_ct.o` to break the cycle:
TARGET_NAME = foo
TARGET_AMD64 = $(TARGET_NAME)_amd64
TARGET_ARMHF = $(TARGET_NAME)_armhf
.PHONY:
all: $(TARGET_AMD64) $(TARGET_ARMHF;
$(TARGET_NAME)_%: lib/mylib.a
#echo "$# built."
lib/mylib.a: bin/mylib/src/mac/foo_ct.o
#echo "$# built."
bin/mylib/src/mac/foo_ct.o:
true

How to perform a string replacement on a path in a Makefile?

I am trying to remove the path prefix. Here is a small example showing just the issue.
Makefile
dist_directory = ./dist
default: build
build: $(patsubst %.md, $(dist_directory)/%.html, $(wildcard *.md))
$(dist_directory)/%.html: %.md
#echo start
#echo $#
#echo ${$#//$(dist_directory)/}
#echo end
Create a file: touch stuff.md
Then build: make
The output is:
start
dist/stuff.html
end
The expected output is:
start
dist/stuff.html
/stuff.html
end
There are similar posts on Stack Exchange. However, they have not worked for me in a Makefile for some reason. I'm probably doing something wrong.
https://unix.stackexchange.com/questions/311758/remove-specific-word-in-variable
Remove a fixed prefix/suffix from a string in Bash
Remove substring matching pattern both in the beginning and the end of the variable
You have many issues here. The most fundamental one is that if you want to use shell variables you have to escape the dollar sign so that make doesn't interpret it. And, you can only use shell variable substitutions on shell variables, while $# is a make variable, so you need:
#foo='$#' ; echo $${foo//$(dist_directory)/}
The more subtle one is that make always uses /bin/sh (POSIX standard shell) when it invokes recipes, and the above syntax is specific to bash. One way around that would be to explicitly set SHELL := /bin/bash in your makefile to force make to use bash. Luckily that is not necessary because POSIX sh can also do this, as mentioned by Reda in another answer:
#foo='$#' ; echo $${###*/}
But even more, you don't need any of this because make sets the automatic variable $* to the part of the target that matches the stem (the %):
#echo $*.html
It also sets $(#F) to the filename part of the $# variable:
#echo $(#F)
ETA
If you want to do something very similar to your shell variable expansion using GNU make you can use:
#echo $(patsubst $(dist_directory)/%,%,$#)

GNU make - transform every prerequisite into target (implicitly)

I have another make-like tool that produces an XML as an artifact after parsing my makefile which I'll then further process with Python.
It'd simplify things for me - a lot - if I could have make consider every single prerequisite to be an actual target because then this other tool
will classify each and every file as a "job".
This is a fragment of my makefile:
.obj/eventlookupmodel.o: C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h \
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h \
...
I'd want for make to think I have a dummy rule for each prerequisite such as below:
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qvariant.h:
#echo target pre= $#
C:/Users/User1/Desktop/A/PROJ/qt5binaries/include/QtCore/qabstractitemmodel.h:
#echo target pre=$#
C:/Users/User1/Desktop/A/PROJ/src/AL2HMIBridge/LookupModels/eventlookupmodel.cpp :
#echo target pre=$#
C:\Users\User1\Desktop\A\PROJ\src\AL2HMIBridge\LookupModels\eventlookupmodel.h:
#echo target pre=$#
I don't care about the exact form of the rule just that each file is considered an actual target.
My method of passing in this rule would be by setting the MAKEFILES variable like so
make all MAKEFILES=Dummy.mk
with Dummy.mk containing this rule so that I do not modify the makefiles.
I've tried the following so far.
Dummy.mk:
%.h:
#echo header xyz = $#
%:
#echo other xyz= $#
This partially works.
I run make all --trace --print-data-base MAKEFILES=Dummy.mk and I can see that
make does "bind" the %.h: rule to the header files. In the --print-data-base section, I see that rule being assigned to the header files.
C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef.h:
# Implicit rule search has been done.
# Implicit/static pattern stem: 'C:/Users/User1/Desktop/A/QNX_SDK/target/qnx6/usr/include/stddef'
# Last modified 2016-05-27 12:39:16
# File has been updated.
# Successfully updated.
# recipe to execute (from '#$(QMAKE) top_builddir=C:/Users/User1/Desktop/A/HMI_FORGF/src/../lib/armle-v7/release/ top_srcdir=C:/Users/User1/Desktop/A/HMI_FORGF/ -Wall CONFIG+=release CONFIG+=qnx_build_release_with_symbols CONFIG+=rtc_build -o Makefile C:/Users/User1/Desktop/A/HMI_FORGF/src/HmiLogging/HmiLogging.pro
', line 2):
#echo header xyz = $#
However, I do NOT see the "echo header xyz $#"-rule being executed.
Regarding the %: rule, it is neither executed for the .cpp files nor "bound" to them in the --print-data-base section.
However, it is bound and executed for existing targets which have no suffix i.e.
all: library binary
binary: | library
ifs: | library
For the %: rule, the reason for this behavior is because of 10.5.5 Match-Anything Pattern Rules: If you do not mark the match-anything rule as terminal, then it is non-terminal. A non-terminal match-anything rule cannot apply to a file name that indicates a specific type of data. A file name indicates a specific type of data if some non-match-anything implicit rule target matches it.
If I make it non-terminal - no double colon - then the rule doesn't apply to built-in types like .cppunless I un-define the built-in rules that negate my intended %: rule.
If I make it terminal, "it does not apply unless its prerequisites actually exist". But a .h or .cpp doesn't technically have prerequisites; can I just create a dummy file and have that as its prerequisite?
NOTE: This has NOTHING to do with gcc -M generation. Yes the -M option would help in the specific case of header and source files but this question is for more generic targets and prerequisites that already exist in the makefile when make is launched.
This may take a few iterations. Try:
%.h: null
#echo header xyz = $#
%: null
#echo other xyz= $#
null:
#:
Try generating static pattern rules for the header files. See one of the answers to Make ignoring Prerequisite that doesn't exist.
Static pattern rules only apply to an explicit list of target files like this:
$(OBJECTS): %.o: %.c
*recipe here*
where the variable OBJECTS is defined earlier in the makefile to be a list of target files (separated by spaces), for example:
OBJECTS := src/fileA.c src/fileB.c src/fileC.c
Note that you can use the various make utility functions to build that list of target files. For example, $(wildcard pattern), $(addsuffix), etc.
You should also ensure that the recipe "touches" the header file to change the timestamp.
I've found that using static pattern rules instead of pattern rules fixes problems where make doesn’t build prerequisites that don’t exist, or deletes files that you want.
Here is an example of using wildcard to copy files from one directory to another.
# Copy images to build/images
img_files := $(wildcard src/images/*.png src/images/*.gif src/images/*.jpg \
src/images/*.mp3)
build_images := $(subst src/,$(BUILD_DIR)/,$(img_files))
$(build_images): $(BUILD_DIR)/images/% : src/images/%
mkdir -p $(dir $#)
cp -v -a $< $#
There are other make functions like addprefix that could be used to generate a more complex file specification.

Double slashes in paths in Makefile

Here is a Makefile example that uses double slashes in paths to targets:
out/file.txt:
#mkdir -p $(dir $#)
#echo aaa > $#
out//file.txt:
#mkdir -p $(dir $#)
#echo bbb > $#
make interprets this as two different targets. If you run make out/file.txt, the first rule will be executed. If you run make out//file.txt, the second rule will be executed.
Also, if you run make out///file.txt and file.txt does not exist, you'll get the error:
make: *** No rule to make target `out///file.txt'
However if the file exists, it just says:
make: Nothing to be done for `out///file.txt'
Also make builds targets successfully if you run make .///out/file.txt or even make .////././././././////.///.////out/file.txt
So, is there any defined behavior how make works with paths that are literally different, but point to the same file in the filesystem?
I got from this answer, that operating system itself doesn't differ such paths. But for make they are different.
The problem originates from the such usage:
my_target: $(SOME_DIR)/some_file
If SOME_DIR already has trailing slash, the code above doesn't work. It expands to something like this: some_dir//some_file and the rule for the specific case with double slashes doesn't exist.
How can such problems be avoided? Is there any path canonization means in make?
Here is direct workaround for my problem - creating a macro that trims trailing slashes:
trslashes = $(if $(filter %/,$(1)),$(call trslashes,$(patsubst %/,%,$(1))),$(1))
This macro must be used in every place where double slashes can cause a problem:
my_target: $(call trslashes,$(SOME_DIR))/some_file
If $(SOME_DIR) is empty the file from root directory will be used:
/some_file. If empty variable should mean current directory then another macro should be used:
trslashes_cur = $(if $(1),$(call trslashes,$(1)),.)
Then $(call trslashes,$(SOME_DIR))/some_file will expand to ./somefile.

How to get absolute paths after vpath matching in make?

I have a makefile that depending on some properties sets vpath and generates a list of source files into one variable. I need to run the makefile without compiling anything (the compilation is actually handled by a different makefile) and just see to which real files the filenames get matched depending on the vpath settings.
Option 1: Let make do its path search:
.PHONY: whichfiles
whichfiles: $(LIST_OF_SOURCE_FILES)
#echo $+
Option 2: Simulate the path search using $(wildcard):
.PHONY: whichfiles
whichfiles:
#echo $(foreach f,$(LIST_OF_SOURCE_FILES),$(firstword $(wildcard $(VPATH:%=%/$f)) not-found:$f))
Either way, "make whichfiles" will print the list of matched files.
If some of the files can't be found, option 1 will fail with "no rule to make" reporting the first file that could not be found. Option 2 will print "not-found:" for each missing file.

Resources