Matching different/multiple parts of a Makefile target - makefile

Here is a Makefile that I currently use to make targets with different configurations, i.e., I am building different software packages with the same target, either all at once or individually.
.PHONY: build test %.build %.test build-all test-all
%.build %.test: PACKAGE = $*
%.build:
#echo build $(PACKAGE)
%.test:
#echo test $(PACKAGE)
build-all: a.build b.build
test-all: a.test b.test
build: $(PACKAGE).build
test: $(PACKAGE).test
I can now build all packages with make build-all or individual packages with, e.g., make build PACKAGE=a. However I would like to switch the body of the %.build and build, etc. targets as in the following:
.PHONY: build test %.build %.test build-all test-all
build:
#echo build $(PACKAGE)
test:
#echo test $(PACKAGE)
build-all: a.build b.build
test-all: a.test b.test
%.build %.test: PACKAGE = $*
$(PACKAGE).%: $*
This way, the pattern matching logic is fully separated from the "main" targets build and test that should contain the actual build commands; making the important parts of the Makefile more readable. However, the last line does not work as intended, i.e., running make a.build and thus make build-all should trigger the target build with PACKAGE=a. The variable assignment in second-last line works, the target matching in the last line does not.
Question: Is there a way to express a matching target like $(PACKAGE).%: $* or to match separate parts of a target like %.%: $2?

As MadScientist explained, the problem cannot be solved easily in GNU make. For completeness, I would like to add and explain my final and more comprehensive solution:
.PHONY: all build test clean %.build %.test build-all test-all
PACKAGES = a b c e f g
PACKAGE = a
all: clean build-all test-all
%.build %.test: PACKAGE = $*
%.build:
#echo build $(PACKAGE)
%.test:
#echo test $(PACKAGE)
clean:
#echo remove build dir
build-all: $(addsuffix .build, $(PACKAGES))
test-all: $(addsuffix .test, $(PACKAGES))
build: $(PACKAGE).build
test: $(PACKAGE).test
This solution avoids eval and foreach and is based on my initial working solution, where the dynamic %.build and %.test targets contain the actual build commands. I added the PACKAGES variable to facilitate easy addition of new packages, a default PACKAGE to prevent execution of misconfigured build commands, and the common targets all and clean as complements.
From the command line, you just call make all, clean, build PACKAGE=x, build-all, etc., i.e., only the static targets, which will then trigger the build commands in the dynamic targets. The static targets and the two variables are also visible in the Bash/Zsh auto-completion.
I think this is the most flexible and yet readable way to build multiple dynamic targets.

First you probably want:
$(PACKAGE).% : %
not using $*, which is an automatic variable and so it has no value except inside the recipe; you can't use it like that in the prerequisite lists.
Second, you can't do this in GNU make. A pattern rule with no recipe doesn't just a create prerequisite relationship, like an explicit rule would do; instead it deletes the pattern rule. Since you didn't have a pattern rule for $(PACKAGE).% yet, this is basically a no-op. Also, target-specific variables are only available inside the recipe, so trying to use $(PACKAGE) in the target definition and expecting it to take the value from some previously set target-specific variable cannot work.
You could do something like this, but it's not fully dynamic (you still need the list of packages and types):
PACKAGES = a b
TYPES = build test
$(foreach T,$(TYPES),$(eval $(addsuffix .$T,$(PACKAGES)): $T))

Related

Makefile run command for each matching directory but exclude specific pattern

QUESTION
Using Make how do I run a command for every directory that contains a file matching *.csproj but does not including a file matching *.Test.csproj using pure make.
SCENARIO
I have previously used Fake and Rake extensively but this is my first time using Make to do anything over and above the simple use of dumb targets.
I have a simple makefile that compiles a .net core solution, runs some tests and then packages up a nuget package. Here is a simplified example.
build:
dotnet build ./src \
...
test:
dotnet test ./src/TestProject \
...
package:
dotnet pack ./src/PackageProject \
...
I now want to introduce additional projects which are also packaged but I do not want to specify each project to package individually. I want the make file to automagically pick up each project that can be packaged. This method has been tried in tested in various Fake builds.
I have been able to implement the following matching on the projects csproj extension and works fine but I have not been able to filter out the test project which also gets packaged.
package: ./src/**/%.csproj
%.csproj:
dotnet pack $(#D) \
...
I have been trying to understand the Make pattern rules and how to apply the filter-out function but have sadly failed. $(filter-out src/**/*.Test.csproj, src/**/%.csproj)
Would appreciate any help on figuring this one out.
EDIT
Based on the question from MadScientist if I run the following using make package using this dumbed down example:
package: ./src/*/%.csproj
%.csproj :
echo $(#)
I get the following output:
echo src/Namespace.Project1/%.csproj
src/Namespace.Project1/%.csproj
echo src/Namespace.Project2/%.csproj
src/Namespace.Project2/%.csproj
echo src/Namespace.Test/%.csproj
src/Namespace.Test/%.csproj
Additionally based on MadScientist's comments I have also been able to create a list of the directories I want to call the dotnet pack command against but I am now stuck on how to call the target for each match.
Note: I am trying to keep this pure Make and avoid using any bash specific syntax
projects := $(filter-out $(dir $(wildcard ./src/Bombora.Namespace*Test/.) ), $(dir $(wildcard ./src/Namespace.*/.) ) )
package:
echo $(projects)
Results in:
echo ./src/Namespace.Project1/ ./src/Namespace.Project2/
./src/Namespace.Project1/ ./src/Namespace.Project2/
EDIT
I have been able to make this work but I do not know if I have gone about it the correct way or if I am abusing something which will come back to bite me later.
This is what I am now doing which is working as expected:
PACKAGE_PROJECTS := $(filter-out $(wildcard ./src/Namespace*Test/*.csproj), $(wildcard ./src/Namespace*/*.csproj) )
package: $(PACKAGE_PROJECTS)
$(PACKAGE_PROJECTS): .
dotnet pack $(#D) \
...
Make implements standard globbing as defined by POSIX. It doesn't provide advanced globbing as implemented in some advanced shells like zsh (or bash if you enable it).
So, ** is identical *; there's no globbing character that means "search in all subdirectories". If you want to do that you need to use find.
Also, in make a pattern is a template that can match some target that you specifically want to build. It's not a way to find targets. And pattern rules only are pattern rules if the target contains the pattern character %; putting a % in a prerequisite of an explicit target doesn't do anything, make treats it as if it were just a % character.
so:
package: ./src/**/%.csproj
is identical to writing:
package: ./src/*/%.csproj
where it finds files matching the literal string %.csproj of which you probably don't have any.
I don't see how this package target does anything at all.
I don't understand what exactly you want to do: you need to make your question more explicit. Make works on targets and prerequisites. So, what is the target you want to build and what are the prerequisites used to create that target? What is the make command line you are invoking, what is the output you got, and what is the intended output you want?
ETA
You asked:
At the end of the day the question is about how do I run a command for every directory that contains a file matching *.csproj but not including *.Test.csproj using pure make.
This will get you that list:
TEST_PROJECTS := $(dir $(wildcard src/*/*.Test.csproj))
PROJECTS := $(filter-out $(TEST_PROJECTS),$(dir $(wildcard src/*/*.csproj)))
projects: $(PROJECTS)
$(PROJECTS):
...run commands...
.PHONY: $(PROJECTS)
You combine setting PROJECTS into one line if you prefer.
Please see below changes which will help you understand applying a filter with make targets and getting ahead.
Usage of filter and filter-out:
filter : Select words in text that match one of the pattern words.
Syntax : $(filter pattern...,text)
filter-out: Select words in text that do not match any of the pattern words.
Syntax : $(filter-out pattern...,text)
In the below example I will use filter to match pattern and if the conditions evaluates to true , then you can use your target with whatever execution you would like to do.
# List the project extensions that you would like to support
MYPROJECTS=.csproj .xyzproj
# MY_FILES would contain all files eg: abc.csproj def.csproj abc.xyzproj
# Below GETPATTERN will extract the suffix: eg. it would produce result .csproj .csproj .xyzproj
GETPATTERN:= $(suffix $(MY_FILES))
ifeq ($(filter $(GETPATTERN),$(MYPROJECTS)),)
package:
dotnet pack ./src/PackageProject
else
package: <default or any other dependency you want>
dotnet pack ./src/PackageDefaultProject
endif

Re-evaluating GNU make makefile variable

I have inherited a large branched project? that requires a volatile set of .a archives $(LIB_FILES) to be included into link target, located in some directories $(LIB_DIRS). I can write an expression like this:
LIBDEP = $(foreach ldir, $(LIB_DIRS), \
$(filter $(addprefix %/, $(LIB_FILES)), $(wildcard $(ldir)/* )))
The problem is that they might not exist at moment of make's invocation and would be built by invoking $(MAKE) inside of another target's rule, which is a prerequisite to the link step.
The problem is actual list of files that should be created varies on external factors determined at their build steps, that I can't hard-code it properly, without turning makefile into a spaghetti mess and said variable is not re-evaluated at the moment of link command invocation.
I have suspicion that $(eval ) function can be used somehow, but manual is not very forthcoming as well as I didn't found examples of its use in this way.
Toolchain: GCC and binutils, make 3.81
Another solution is to create an explicit dependency of your make script on the output of the step which currently creates the variable $(LIB_FILES). This is what the manual is dealing with in the chapter How makefiles are remade and it aims at the technique which make is best at, namely deriving dependencies from the existence and timestamp of files (instead of variables). The following hopefully depicts your situation with the process of deducing a new set of libraries simulated by the two variables $(LIBS_THIS_TIME) and $(LIB_CONFIG_SET).
LIBS_THIS_TIME = foo.a:baz.a:bar.a
LIB_CONFIG_SET = $(subst :,_,$(LIBS_THIS_TIME))
include libdeps.d
linkstep:
#echo I am linking $^ now
touch $#
libdeps.d: $(LIB_CONFIG_SET)
-rm libdeps.d
$(foreach lib,$(subst :, ,$(LIBS_THIS_TIME)),echo linkstep: $(lib) >> libdeps.d;)
$(LIB_CONFIG_SET):
touch $#
If make finds that libdeps.d is not up to date to your current library configuration it is remade before make executes any other rule, although it is not the first target in the makefile. This way, if your build process creates a new or different set of libraries, libdeps.d would be remade first and only then make would carry on with the other targets in your top makefile, now with the correct dependecy information.
It sometimes happens that you need to invoke make several times in succession. One possibility to do this is to use conditionals:
ifeq ($(STEP),)
all:
<do-first-step>
$(MAKE) STEP=2 $#
else ifeq ($(STEP),2)
all:
<do-second-step>
$(MAKE) STEP=3 $#
else ifeq ($(STEP),3)
all:
<do-third-step>
endif
In each step you can generate new files and have them existing for the next step.

makefile: from 3 input generate one output

I have this version of makefile
[sbsuser#compute-00-01 415]$ make --version GNU Make 3.81
I have directory SOMATIC where I have 3 file . I want to produce a only one output. This is what I wrote.
`
OUTSOMATIC=SOMATIC
FINAL=FINAL
INPUT=$(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2= $(patsubst $(OUTSOMATIC)/%.vcf,$(FINAL)/%somatic.ensemble.gz,$(INPUT))
$(info lista $(OUTSORT2))
$(info lista $(INPUT))
.PHONY: all
all: $(INPUT) $(OUTSOMATIC) $(OUTSORT2) $(FINAL)
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $(FINAL)/somatic_ensemble.gz /illumina/software/database/database_2016/hg19_primary.fa $^
`
With this script make 3 time the same files. I don't understand how to create only one output from list of input to use in the same time.
What is the best way to do this?
If I change $(FINAL)/%somatic.ensemble.gz: in $(FINAL)/somatic.ensemble.gz I have this error:
make: *** No rule to make target FINAL/415_merge_mutect2.somaticsomatic.ensemble.gz', needed byall'. Stop`
You probably should review the GNU make manual introductory sections where they describe how make works.
Let's look at your makefile; first you define some variables. Let's assume that you have the files SOMATIC/foo.vcf, SOMATIC/bar.vcf, and SOMATIC/baz.vcf. Then the variables you created will have these values, after they are expanded:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf
Now your patsubst finds all words in INPUT that match the pattern SOMATIC/%.vcf and replace that with FINAL/%somatic.ensemble.gz, where the part that matches the % in the input is substituted into the output:
OUTSORT2 = FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz
Now, make sees that you've defined an all target. Since it's the first target in the makefile this is the target that will be run by default. After expansion, it will look like this:
all: SOMATIC/foo.vcf SOMATIC/bar.vcf SOMATIC/baz.vcf SOMATIC FINAL/foosomatic.ensemble.gz FINAL/barsomatic.ensemble.gz FINAL/bazsomatic.ensemble.gz FINAL
So, make will try to build every prerequisite of the all target to be sure it's up to date. First it tries to build the SOMATIC/*.vcf files. Those files already exist and make doesn't have any rules about how to rebuild them, so it assumes they're up to date.
Next it tries to build the SOMATIC file. This is a directory and it also has no rule to be built, so make assumes that's up to date as well.
Next make tries to build the target FINAL/foosomatic.ensemble.gz. Make does have a rule that can build it, you've created one:
$(FINAL)/%somatic.ensemble.gz: $(OUTSOMATIC)/%.vcf $(INPUT)
~/jdk1.8.0_121/bin/java ...
This matches the target you want to build, with a % value of foo, so then make substitutes the % in the prerequisite for foo and finds that SOMATIC/foo.vcf exists and doesn't need to be rebuilt, so it runs your recipe. However your recipe doesn't actually create the target FINAL/foosomatic.ensemble.gz; it creates the target FINAL/somatic_ensemble.gz. So this rule is broken because it tells make it will do one thing, but it does something else.
You should always ensure all your recipes build the file represented by the automatic variable $#; that will ensure that you and make agree on the meaning of your rule. If you want your recipe to build some other file, then your rule is written incorrectly.
Next make does the same thing with the next prerequisite of all: FINAL/barsomatic.ensemble.gz. Since that file doesn't exist, make tries to build it using the pattern rule, but again that creates the same output file.
And again for the third .gz file FINAL/bazsomatic.ensemble.gz. That's why things are run three times.
If you change the pattern rule to an explicit rule building FINAL/somatic.ensemble.gz, which is what you want, then make can't find any way to build the prerequisites of the all target so it gives this error.
Your problem is the creation of OUTSORT2. You want to create only one output file, but you've set OUTSORT2 to contain three different files, so make tries to create all three files. You want this:
OUTSOMATIC = SOMATIC
FINAL = FINAL
INPUT = $(wildcard $(OUTSOMATIC)/*.vcf)
OUTSORT2 = $(FINAL)/somatic.ensemble.gz
.PHONY: all
all: $(OUTSORT2)
$(OUTSORT2): $(INPUT)
~/jdk1.8.0_121/bin/java -XX:+UseSerialGC -Xms1g -Xmx10g -jar /illumina/software/PROG2/bcbio-variation-recall-0.1.7 ensemble -n 1 $# /illumina/software/database/database_2016/hg19_primary.fa $^

make: trigger without rebuilding dependency

Basically I have the usual Makefile construct:
target: dependency1 dependency2 dependency3
runtargetscript.sh
However in this case, the target only needs one of the dependencies and some dependencies may not be buildable. (so I cannot just build all dependencies)
Is it possible to tell make to trigger "target" when one of the dependencies changed/was created (i.e. normal behaviour) but NOT to try to rebuild any missing dependencies?
With GNU make you can use shell escapes to build the dependencies dynamically, adding them only if they already exist:
if_exist = $(shell if [ -e $(1) ]; then echo $(1); fi)
target: $(call if_exist,dependency1) $(call if_exist,dependency2) $(call if_exist,dependency3)
runtargetscript.sh
This will run the script if target does not exist, or if it is older than any of the dependencies that do exist at the time the makefile was read, but will not attempt to build them if they do not exist at that time.
Note the important caveat there -- if the file(s) do not exist, but some other unrelated rule runs an action that creates them, it won't rebuild target, unless you rerun make target again.
Assuming your rule body does not do anything special depending on which dependencies are newer than the target you can use the -W flag to make to instruct it to consider certain targets as always new (and thus not in need of building).
So for the given example assuming you can (and want) to build dependency2 but not dependency1 or dependency3 you would run:
make -W dependency1 -W dependency3 target
Edit: As pointed out in the comments this does not work correctly when dependency2 is not newer than target as target will still be built.
In that case I believe the only solution (given the comments below) is to use something like:
DEPENDENCY_BIN := $(or $(wildcard /path/to/mysql),$(wildcard /path/to/sqlite3),/path/that/does/not/exist)
target: $(DEPENDENCY_BIN) dependency2

Override target in makefile to add more commands?

At work we use a common makefile that other makefiles include (via the include statement) and it has a generic "clean" target that kills some common files. I want to add on to that target in my new makefile so I can delete some specific files, but if I add a clean target in my makefile, it just overrides the old one.
I know I can just make a new target with a new name and have it call clean, and then do other stuff, but for sake of consistency I'd like to be able to just call make clean and have it do everything.
Is that possible?
I've seen this done at several shops. The most common approach is to use double-colon rules, assuming you're using something like GNU make. In your common makefile you would have something like this:
clean::
# standard cleanup, like remove all .o's:
rm -f *.o
Note that there are two colons following clean, not just one!
In your other makefile you just declare clean again, as a double-colon rule:
clean::
# custom cleanup, like remove my special generated files:
rm -f *.h.gen
When you invoke make clean, GNU make will automagically run both of these "branches" of the clean rule:
% make clean
rm -f *.o
rm -f *.h.gen
It's simple to set up and it composes quite neatly I think. Note that specifically because it is a double-colon rule, you don't get the "overriding commands" errors you normally get when you define two rules for the same target. That's sort of the point of double-colon rules.
You can write your own clean and make it a preq of the common clean.
clean: myclean
myclean:
rm whatever
Yours will run first. If for some reason you want the common clean to run first then the solution will be more complicated.
EDIT:
Here is the best solution I can see which runs the common rule before the local one:
include Makefile.common
clean:
$(MAKE) -f Makefile.common $#
rm whatever additional things
The include directive is necessary because the local makefile relies on the common one for things other than clean. The local clean rule overrides the common clean rule, but invokes the common clean rule before doing the additional work. (This overriding will cause some warnings, which is a nuisance; I don't know a good way to silence them.)
Use implicit rules:
existing-target: my-extention
my-extention:
echo running command 1
echo running command 2
Very simple make tutorial to ramp up.
When using :: you can run into issues since make complains when you mix single colon : and double colon :: rules:
a:
echo a
a::
echo aa
will result in:
. . .
*** target file `a' has both : and :: entries. Stop.
It seems like the common makefile's rule should be called something like common-clean. Then each main makefile would declare their clean rule as
clean: common-clean
and you're set.
If that isn't an option, you could take a look at double colon rules, but those introduce a whole other set of issues to consider.
Adding another possible solution I've seen for posterity... I know the OP was wary about changing the common makefile, but something like this works and involves minimal changes.
local makefile 1:
CLEAN=MyExe1 MyExe2
....
include /my/common/makefile
local makefile 2:
CLEAN=MyExe3 MyExe4
....
include /my/common/makefile
common makefile:
clean:
rm -f *.dep *.o *.a $(CLEAN)
Basically the idea is to define some variable (in this case CLEAN) in each local makefile with all the specific items you want to delete. Then the common makefile runs rm -f on all the common file types to delete, plus whatever was specifically flagged for deletion in each local makefile via the CLEAN variable. If there's nothing specific to delete, simply omit the variable declaration or leave it empty (CLEAN=)
So now if we run make clean for local makefile 1, it executes
rm -f *.dep *.o *.a MyExe1 MyExe2
And if we run make clean for local makefile 2, it executes
rm -f *.dep *.o *.a MyExe3 MyExe4
I've found a better solution:
.PHONY: my-extra-clean
clean: my-extra-clean
my-extra-clean:
rm <whatever-you-want>
include Makefile.common
The key line is clean: my-extra-clean. Ie, you can add dependencies in separate stanzas in different makefiles to add behaviour. my-extra-clean is run as a dependency of the root clean target.
For ours, we define a variable, EXTRAFILESTOCLEAN, then when the clean rule runs, it has a step to remove anything specified in the EXTRAFILESTOCLEAN variable
clean:
rm -f *.o
ifdef $(EXTRAFILESTOCLEAN)
rm -f $(EXTRAFILESTOCLEAN)
endif
That can cause unexpected problems if you set that variable to weird values, but you could guard against those by adding prefixes or other tests.
It's in the docs: https://www.gnu.org/software/make/manual/html_node/Overriding-Makefiles.html
So instead of include Makefile you use a wildcard target and forward it to the base Makefile:
# -include base.Makefile <--- not this
%:
#$(MAKE) -f base.Makefile $#

Resources