Given a Makefile prerequisite
PY_FILES := $(shell find . -type f -name *.py)
my_target: $(PY_FILES)
do_something $(PY_FILES)
Is it possible to call do_something on a subset of it i.e. exclude all the python files that match a pattern e.g. */test/*?
My understanding is that $(filter-out pattern, text) can do something similar, but pattern has to be a list of files that I want to exclude.
In that case I could as well find with -not and save the result to another prerequisite which is what I am trying to avoid – I want to reuse the PY_FILES prereq.
There's not good built-in way to match a */test/* style pattern with make built-ins. The pattern matching just isn't that sophisticated.
You could try something like this but splitting the files at find time is likely going to perform better and be more useful in the end.
define testfilter
$(foreach f,$1,$(if $(findstring /test/,$f),,$f))
endef
my_target: $(PY_FILES)
do_something $(call testfilter,$(PY_FILES))
Related
Some code from Makefile:
tempDir := ...
javaSources := $(wildcard src/java/**/%.java)
javaClasses := $(subst src/java, $(tempDir)/java/classes, $(subst .java,.class, $(javaSources)))
$(javaClasses): $(javaSources)
mkdir -p $(tempDir)/java/classes || true
javac \
-d $(tempDir)/java/classes \
-cp $(tempDir)/java/classes \
$?
How to create a pattern rule (like here) to preserve in / out order?
#MadScientist
First, your wildcard won't work. GNU make uses only basic shell globbing, which means it can't understand advanced globbing like ** meaning "search all subdirectories". Second, % is not a shell globbing character at all so you're just looking for files that are literally named %.java.
Instead you probably want something like this:
javaSources := $(shell find src/java -name '*.java')
Next, to create the javaClasses content you really don't want to use subst because it substitutes everywhere which can give false matches (e.g., $(subst .x,.y,foo.xbar) will yield foo.ybar which is probably not what you want).
Something like this is simpler to understand:
javaClasses := $(patsubst src/java/%.java,$(tempdir)/java/classes/%.class,$(javaSources))
Finally, you are repeating exactly the same error you made in the previous question, where you tried to list all the targets and all the prerequisites in the same rule. Just as I said for that question, that is not right.
The answer is exactly the same as in the previous question: you should write a pattern rule that describes how to build one single target from one single source file.
And again you need an all target or similar which depends on all the outputs.
In complement to MadScientist answer, you should probably use a pattern rule like:
$(tempDir)/java/classes/%.class: src/java/%.java
mkdir -p $(dir $#)
javac -d $(dir $#) -cp $(dir $#) $<
(not sure what -cp should be in this case, it depends on your specific project). And as MadScientist also suggested, you will need:
.PHONY: all
all: $(javaClasses)
such that you can call make all to compile all the source files that need to be. Put it before any other explicit target if you want all to be the default goal (the goal make selects if you just call make), or use the .DEFAULT_GOAL special variable:
.DEFAULT_GOAL := all
I want to check if all the files that have a specific name have in them a string , if not to report them . I wrote this sequence and tried multiple some others, but I don't know how to access the contain of a file from list.
SOURCES := $(shell find $(SOURCEDIR) -name 'mod.mak')#here I take the list of targeted files (this works fine)
$(foreach File, Files,
$(if $(grep -q "aaabbb" "$File"),,#echo "WARNING Missing sequence")
)
You have multiple issues with your script.
First of all, you need some rule/target. For your example we can make a PHONY target test. Second, to iterate over values in SOURCES, you need to reference it as $(SOURCES). Similarly for $(file) in call to grep. Also, make's if is interpreting value, not exit code, so you shouldn't silence grep.
This will do it:
.PHONY: test
SOURCES := $(shell find "$(SOURCEDIR)" -name 'mod.mak')
test:
$(foreach file,$(SOURCES),$(if $(shell grep "aaabbb" "$(file)"),,#echo "WARNING Missing sequence in $(file)"))
So some anonymous developers have decided to use a ridiculous convention of using spaces in their folder names that contain their source files. I would change these folders not to use spaces but sadly I don't make the rules around here so that's not an option (though I wish it were).
LUAC = luac
SRC_DIR = .
SOURCE = \
stupid/naming\ convention/a.lua \
stupid/naming\ convention/very\ annoying/b.lua \
vpath .lua $(SRC_DIR)
OUT_DIR = ../out/
OUTPUT = $(patsubst %.lua, $(OUT_DIR)/%.luac, $(SOURCE))
all: $(OUTPUT)
$(OUT_DIR)/%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#"
.PHONY: all
Simple Makefile. All it's meant to do is compile all the Lua files that I have and put them into an output directory.
No matter I do it keeps wanting to split the SOURCE string on the spaces in the folder, so I end with a beautiful error like this:
make: *** No rule to make target `stupid/naming ', needed by `all'. Stop.
Is there a way to fix this without renaming the folders?
Thanks in advance.
The very short, but IMO ultimately correct, answer is that make (not just GNU make, but all POSIX-style make implementations) does not support pathnames containing whitespace. If you want to use make, your "anonymous developers" simply cannot use them. If they insist that this is an absolute requirement you should switch to a different build tool altogether, that does support whitespace in filenames.
Yes, it's barely possible to create a makefile that will work with filenames containing whitespace, but you will essentially have to rewrite all your makefiles from scratch, and you will not be able to use many of the features of GNU make so your makefiles will be long, difficult to read, and difficult to maintain.
Just tell them to get over themselves. Or if they really can't, try having them create their workspace in a pathname without any whitespace in the names, then create a symbolic link containing whitespace pointing to the real workspace (the other way around won't work in all situations).
Unfortunately, GNU Make's functions that deal with space-separated list do not
respect the escaping of the space. The only exception is wildcard.
Edit:
Here's my workaround:
LUAC = luac
SRC_DIR = .
SOURCE = \
stupid/naming\ convention/a.lua \
stupid/naming\ convention/very\ annoying/b.lua \
vpath .lua $(SRC_DIR)
OUT_DIR = ../out/
OUTPUT = $(patsubst %.lua,%.luac,$(SOURCE))
all: $(OUTPUT)
%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#""
.PHONY: all
I tried to output it first like that:
%.luac: %.lua
#echo "$<"
#echo "$#""
Output looks as follows:
stupid/naming convention/a.lua
../out/stupid/naming convention/a.luac
stupid/naming convention/very annoying/b.lua
../out/stupid/naming convention/very annoying/b.luac
If you look at this excellent write up: http://www.cmcrossroads.com/article/gnu-make-meets-file-names-spaces-them, the author suggests that this is mostly a difficult task. But his substitution functions could get you going in case you really can't avoid the spaces.
Putting this into your makefile would look like this (sorry if I changed some of your paths, but this works on my Cygwin installation):
LUAC = luac
s+ = $(subst \\ ,+,$1)
+s = $(subst +,\ ,$1)
SRC_DIR = .
SOURCE := stupid/naming\\ convention/a.lua
SOURCE := $(call s+,$(SOURCE))
vpath .lua $(SRC_DIR)
OUT_DIR = out/
OUTPUT = $(patsubst %.lua, $(OUT_DIR)/%.luac, $(SOURCE))
all: $(call +s,$(OUTPUT))
$(OUT_DIR)/%.luac: %.lua
$(LUAC) "$<"
mv luac.out "$#"
.PHONY: all
I know that's not a complete answer, but maybe an encouragement that it actually is possible. But I agree with the other posters that if you can actually avoid spaces altogether, you will have a much easier life!
Another strategy which works when you are generating your Makefile automatically is this one, also used in Perl's ExtUtils::MakeMaker: to separate the name formatted to be usable in recipes, versus it being usable as a dependency. The example here has a THISFILE and a THISFILEDEP.
AWKWARD_DIR = sub dir
AWKWARD_DIRDEP = sub\ dir
THISFILE = $(AWKWARD_DIR)/d1
THISFILEDEP = $(AWKWARD_DIRDEP)/d1
AWKWARD_DIR_EXISTS = $(AWKWARD_DIR)/.exists
AWKWARD_DIR_EXISTSDEP = $(AWKWARD_DIRDEP)/.exists
TARGET = $(AWKWARD_DIR)/t1
TARGETDEP = $(AWKWARD_DIRDEP)/t1
MAKEFILE = spacemake.mk
$(TARGETDEP): $(THISFILEDEP) $(AWKWARD_DIR_EXISTSDEP)
cat "$(THISFILE)" >"$(TARGET)"
$(THISFILEDEP): $(AWKWARD_DIR_EXISTSDEP)
echo "yo" >"$(THISFILE)"
$(AWKWARD_DIR_EXISTSDEP): $(MAKEFILE)
#echo MAKEFILE = $(MAKEFILE)
-mkdir "$(AWKWARD_DIR)"
touch "$(AWKWARD_DIR_EXISTS)"
You can try it by placing it in a file called e.g. spacemake.mk, then run it with gmake -f spacemake.mk.
I am writing a Makefile and I get stuck on a filter function limitation.
Indeed, filter takes only one wildcard.
What I would like to do is:
I have a list a files, some matching the regexp blabla, some not. But for this I need 2 wildcards, thus i cannot use filter function.
I would like to split my original list in 2 lists, one containing all the element containing the blabla string (filter equivalent) and the other one containing the not matching one (filter-out equivalent).
thanks for your help.
You can do this without running any external commands. Define the two macros
containing = $(foreach v,$2,$(if $(findstring $1,$v),$v))
not-containing = $(foreach v,$2,$(if $(findstring $1,$v),,$v))
Now you can do
LIST := a_old_tt x_old_da a_new_da q_ty_we
LIST_OLD := $(call containing,old,$(LIST))
LIST_NOT_OLD := $(call not-containing,old,$(LIST))
One of Make's greatest shortcomings is its poor ability to handle regular expressions. The functions filter and filter-out can't find "old" in the middle of a word. I'd suggest this hack:
NOT_OLD = $(shell echo $(LIST) | sed 's/[^ ]*old[^ ]* *//g')
OLD = $(filter-out $(NOT_OLD), $(LIST))
You could take advantage of your shell's more advanced string handling capabilities. Assuming that you have bash, you could use the following in your makefile:
LIST := a_old_tt x_old_da a_new_da q_ty_we
LIST_NOT_OLD := $(shell l=($(LIST)); echo $${l[#]//*old*})
LIST_OLD := $(filter-out $(LIST_NOT_OLD),$(LIST))
You can find an explanation of the bash string replacement mechanism in how to delete elements from an array based on a pattern. The double $ is required to keep the $ sign in the shell invocation.
I want to write something like regex:
SRC:="a.dat.1 a.dat.2"
$(SRC): %.dat.%: (\\1).rlt.(\\2)
dat2rlt $^ $#
so that a.dat.1 and a.dat.2 will give a.rlt.1 and a.rlt.2.
In GNU Make info page, it says "the % can be used only once".
Is there some trick to achieve this in GNU Make?
I'm afraid what you are trying to do is not possible the way you suggest to do it, since - as you already mention - (GNU) make only allows a single stem '%', see http://www.gnu.org/software/make/manual/make.html#Pattern-Rules:
A pattern rule looks like an ordinary rule, except that its target
contains the character ‘%’ (exactly one of them).
Without it, creating such 'multi-dimensional' targets is cumbersome.
One way around this is by rebuilding the name of the dependency in the command (rather than in the dependency list):
SRC := a.dat.1 a.dat.2
all : $(SRC:%=%.dat2rlt)
%.dat2rlt :
dat2rtl $(word 1,$(subst ., ,$*)).rlt.$(word 2,$(subst ., ,$*)) $*
Of course, however, this way you would lose the dependency, it will not rebuild once the rlt has been updated.
The only way I can see to address that is by generating the rules explicitly:
SRC := a.dat.1 a.dat.2
all : $(SRC)
define GEN_RULE
$1.dat.$2 : $1.rlt.$2
dat2rtl $$< $$#
endef
$(foreach src,$(SRC),$(eval $(call GEN_RULE,$(word 1,$(subst ., ,$(src))),$(word 3,$(subst ., ,$(src))))))
Using named variables, we can write more readable code (based on answer of Paljas):
letters:=a b c
numbers:=1 2 3 4
define GEN_RULE
$(letter).dat.$(number) : $(letter).rlt.$(number)
./rlt2dat $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(GEN_RULE)) \
) \
)
We can generate SRC in a similar way. Note that using that method SRC will contain all the combinations. That may or may not be beneficial.
Building on the answer of Erzsébet Frigó, you might additionally choose to:
in the inner loop, eval not the macro itself but the result of calling it
name the macro after program you're calling, dat2rtl
in combination, allowing you to
refer to the program name using make's ${0}
define a target, ${0}s (expanding to dat2rts - note the pluralization) with preconditions of all combinations of letters and numbers on which dat2r2 was called
Like this:
letters:=a b c
numbers:=1 2 3 4
define rlt2dat
${0}s::$(letter).dat.$(number)
$(letter).dat.$(number): $(letter).rlt.$(number)
./${0} $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(call rlt2dat))))
allowing you to build all rlt2dat targets as:
make rlt2dats
For the limited example you gave, you can use a pattern with one %.
SRC := a.dat.1 a.dat.2
${SRC}: a.dat.%: a.rlt.%
dat2rlt $^ $#
$* in the recipe will expand to whatever the % matched.
Note that the "s around your original macro are definitely wrong.
Have a look at .SECONDEXPANSION in the manual for more complicated stuff (or over here).