Suppose I have the following directory structure, with a root-node:
/root/
Makefile
branch1/
branch2/
.../
And I write the following minimal makefile:
branches=$(shell find * -maxdepth 1 -type d -printf "%f "}
%:
-${MAKE} ${MAKECMDGOALS} -C branch1
-${MAKE} ${MAKECMDGOALS} -C branch2
...
Having prepared to dynamically perform this relay with the branches variable and having tried several wild-card and descendant rule variations without success, my question amounts to: how do I capture any command goal from outer scope (like I am doing now), and perform the make relay for each of the files my find expression detects?
In pseudo code (did not work with my version of make, which is the latest greatest available on Cygwin):
branches=$(shell find * -maxdepth 1 -type d -printf "%f "}
branch-%:
-${MAKE} ${MAKECMDGOALS} -C $*
%: ${foreach branch,${branches}, branch-${branch}}
Unlike the original makefile, this does not work. However, it seems like it should. Is there a way to do this?
And there is a second issue
Make's parallelism will be broken by my pseudo-code method (if it worked) with an exponential fan out using the -j option, whereas the first method I used will not break parallelism.
However, ideally, this makefile should be able to dynamically execute one make relay for each file in the branches list. However, I don't currently see a way to implement this dynamically.
First, I'm not sure why you're using a complex shell function; why not just:
branches := $(wildcard */.)
Or, if you don't want the /. at the end:
branches := $(patsubst %/.,%,$(wildcard */.))
Second, the reason your second attempt doesn't work is that it's not valid to create a pattern rule with no recipe. See Canceling Pattern Rules.
Instead, you can use the .DEFAULT special target. It would look something like this:
branch-%:
-${MAKE} -C $* $(CMD)
.DEFAULT:
#$(MAKE) CMD=$# ${addprefix branch-,${branches}}
This does use recursive make and it behaves slightly differently than your original.
I'm not sure I understand your second point about -j. GNU make (unless you're using a truly old version) can communicate among all the submakes to ensure they are starting as many, but not more, jobs than you requested.
Oh, I forgot, there's another obvious way to do it if you don't want to use .DEFAULT and recursion:
$(MAKECMDGOALS): $(addprefix branch-,$branches))
Related
I have a folder of markdown files that are being converted to HTML files using a Makefile. To figure out the source and dest, I use the following Makefile construct
TARGETS_TO_BUILD := $(patsubst src/%.md, out/%.html, $(wildcard src/*.md src/**/*.md))
If you echo out $(TARGETS_TO_BUILD) you get a list of paths ./out/index.html ./out/folder1/somepage.html and so on. Works fine.
However, if I start putting my source files into deeper and deeper folders, such that I end up with a deep tree, things stop working. That wildcard (src/**/*.md) doesn't work. I have to start doing things like this:
TARGETS_TO_BUILD := $(patsubst src/%.md, out/%.html, $(wildcard src/*.md src/**/*.md src/**/**/*.md src/**/**/**/*.md))
I have to keep adding more and more of those.
That glob string isn't working as expected. I thought they would work with infinite depth.
You are mistaken. The docs for wildcard support are quite clear on the syntax, and ** is not listed. ** is an enhanced wildcard that is not supported by POSIX glob(3) and fnmatch(3) functions (which is what GNU make uses to implement globbing), and is not supported by POSIX sh. In fact that syntax is not even available (by default) in bash, the standard shell in most GNU/Linux and MacOS systems.
In standard glob syntax, there is no difference between * and **. In fact ***, ****, etc. all mean the same thing as *: there is no difference between matching zero or more characters once, twice, or 100 times.
If you want to recurse infinitely, the simplest thing to use is the find command via a $(shell ...) function:
TARGETS_TO_BUILD := $(patsubst src/%.md, out/%.html, $(shell find src -name '*.md' -print))
Is there a way to force a target-rule to run as part of setting a something in a variable?
For example let's say we have a target and rule:
all_mp3s:
find / -name "*.mp3" > all_mp3s
And then a variable:
MP3S := $(file < all_mp3s)
Is there a way to make sure all_mp3s file is getting created before evaluating the MP3S variable?
There is no simple straightforward way to force a rule to be evaluated before a variable gets assigned. There are more complex ways. The following is for GNU make.
Let's first assume that you want to run the (slow) find command only if the file all_mp3s does not exist, else use its content. You can use GNU make conditionals:
ifeq ($(wildcard all_mp3s),all_mp3s)
MP3S := $(shell cat all_mp3s)
else
MP3S := $(shell $(MAKE) all_mp3s ; cat all_mp3s)
endif
all_mp3s:
find / -name "*.mp3" > $#
But I if your Makefile is more complex than this, uses MP3S several times, and what you really want is:
avoid running your super-slow find several times,
run it only if needed (and only once),
get the result in a file (all_mp3s) plus a make variable (MP3S),
MadScientist has a wonderful GNU make trick that can be used here:
MP3S = $(eval MP3S := $$(shell find / -name "*.mp3"))$(MP3S)
all_mp3s:
printf '%s\n' '$(MP3S)' > all_mp3s
.PHONY: help clean
help:
printf 'MP3 finder\n'
clean:
rm -f all_mp3s
If the MP3S recursively expanded make variable is expanded because some part of your Makefile is evaluated and needs its value (e.g. if you run make all_mp3s while all_mp3s does not exist), the find command will be run, its result stored in the variable... and the variable will be turned into a simply expanded make variable, which further expansions, if any, will reuse the same, already computed, value.
Else, if your invocation of make (e.g. make cleanor make help) does not need MP3S value, the find command will not even be run.
The all_mp3s file is generated from the value of MP3S (instead of the opposite in the other solution).
However, there is another important thing to decide: do you want to declare all_mp3s as a phony target:
.PHONY: all_mp3s
or not?
If you declare it as phony, the find command will be run once and only once each time you invoke make all_mp3s (or another goal that depends on all_mp3s). But targets depending on all_mp3s will always be rebuilt too, which is not necessarily what you want.
If you don't declare it as phony, and the file exists already, the find command will not be run at all (unless the value of MP3S is needed elsewhere in your Makefile), and the content of all_mp3s will not be updated, which is not necessarily what you want.
As you do not give enough information in your question to decide, it is up to you.
I find myself a little torn between two possibilities to declare GNU make tragets phony in Makefiles.
One is declaring all phonies in one go:
.PHONY: targ1 targ2 targ3
targ1:
...
targ2:
...
targ3:
...
Which has the advantage of being more readable (to me) and more tidy. But one can't see quickly which targets are phony.
The other possibility is declaring the phoniness along with the rule (directly in front or behind):
.PHONY: targ1
targ1:
...
.PHONY: targ2
targ2:
...
targ3:
...
.PHONY: targ3
Which (to me) is harder to read. Also I don't like the duplication of the rule name. Is there a solution like function decorators in python? Something like this:
#pny
targ1:
...
#pny
targ2:
...
#pny
targ3:
...
I suspect something like this might be more useful for applications other than just making a rule phony, seeing it's just a matter of my personal tastes. Hence the broader title of my question.
I'm not aware of a Make built-in for that. But for a (not-too-portable, not advisable) solution you could do something like this:
MAKEFILE = $(lastword $(MAKEFILE_LIST))
.PHONY: $(shell grep -E -A1 "^\s*\#\s*phy" $(MAKEFILE) | \
grep -Pio "^[a-z][-_.a-z0-9]+\s*(?=:)")
#phy
targ1:
...
#phy
targ2:
...
#phy
targ3:
...
This solution searches the Makefile for the string #phy in the line above every rule. It extracts the rule-name. Both using the unix program grep and a shell call. The rule names are then taken as sources for the .PHONY.
To make a more general "decorator", you could combine it with techniques from this answer: https://stackoverflow.com/a/36941727/8655091, i.e. the part where define is used to build the text of a make rules, dependent on an input parameter. That define is then later call-ed in a foreach loop to acutally create the rule.
However, grep especially with the -P option might not be present on every system. Also, including sub-makefiles might make problems. Also, most people encountering this DIRTY HACK, will want to harm you.
I have a project that involves sub-directories with sub-makefiles. I'm aware that I can pass variables from a parent makefile to a sub-makefile through the environment using the export command. Is there a way to pass variables from a sub-makefile to its calling makefile? I.e. can export work in the reverse? I've attempted this with no success. I'm guessing once the sub-make finishes its shell is destroyed along with its environment variables. Is there another standard way of passing variables upward?
The short answer to your question is: no, you can't [directly] do what you want for a recursive build (see below for a non-recursive build).
Make executes a sub-make process as a recipe line like any other command. Its stdout/stderr get printed to the terminal like any other process. In general, a sub-process cannot affect the parent's environment (obviously we're not talking about environment here, but the same principle applies) -- unless you intentionally build something like that into the parent process, but then you'd be using IPC mechanisms to pull it off.
There are a number of ways I could imagine for pulling this off, all of which sound like an awful thing to do. For example you could write to a file and source it with an include directive (note: untested) inside an eval:
some_target:
${MAKE} ${MFLAGS} -f /path/to/makefile
some_other_target : some_target
$(eval include /path/to/new/file)
... though it has to be in a separate target as written above because all $(macro statements) are evaluated before the recipe begins execution, even if the macro is on a later line of the recipe.
gmake v4.x has a new feature that allows you to write out to a file directly from a makefile directive. An example from the documentation:
If the command required each argument to be on a separate line of the
input file, you might write your recipe like this:
program: $(OBJECTS)
$(file >$#.in) $(foreach O,$^,$(file >>$#.in,$O))
$(CMD) $(CMDFLAGS) #$#.in
#rm $#.in
(gnu.org)
... but you'd still need an $(eval include ...) macro in a separate recipe to consume the file contents.
I'm very leery of using $(eval include ...) in a recipe; in a parallel build, the included file can affect make variables and the timing for when the inclusion occurs could be non-deterministic w/respect to other targets being built in parallel.
You'd be much better off finding a more natural solution to your problem. I would start by taking a step back and asking yourself "what problem am I trying to solve, and how have other people solved that problem?" If you aren't finding people trying to solve that problem, there's a good chance it's because they didn't start down a path you're on.
edit You can do what you want for a non-recursive build. For example:
# makefile1
include makefile2
my_tool: ${OBJS}
# makefile2
OBJS := some.o list.o of.o objects.o
... though I caution you to be very careful with this. The build I maintain is extremely large (around 250 makefiles). Each level includes with a statement like the following:
include ${SOME_DIRECTORY}/*/makefile
The danger here is you don't want people in one tree depending on variables from another tree. There are a few spots where for the short term I've had to do something like what you want: sub-makefiles append to a variable, then that variable gets used in the parent makefile. In the long term that's going away because it's brittle/unsafe, but for the time being I've had to use it.
I suggest you read the paper Recursive Make Considered Harmful (if that link doesn't work, just google the name of the paper).
Your directory structure probably looks like this:
my_proj
|-- Makefile
|-- dir1
| `-- Makefile
`-- dir2
`-- Makefile
And what you are doing in your parent Makefile is probably this:
make -C ./dir1
make -C ./dir2
This actually spawns/forks a new child process for every make call.
You are asking for updating the environment of the parent process from its children, but that's not possible by design (1, 2).
You still could work around this by:
using a file as shared memory between two processes (see Brian's answer)
using the child's exit error code as a trigger for different actions [ugly trick]
I think the simplest solution is using standard out from a sub Makefile.
Parent Makefile
VAR := $(shell $(MAKE) -s -C child-directory)
all:
echo $(VAR)
Child Makefile
all:
#echo "MessageToTheParent"
Before I start, I'll mention that I'm not using GNU Make in this case for building a C/C++ project.
Makefile:
DEST_DIR = build/
SRC_DIR = src/
$(SRC_DIR)a/ : $(SOMETHING_ELSE)
$(DO_SOMETHING_TO_GENERATE_A_DIR)
$(DEST_DIR)% : $(SRC_DIR)%
cp -r $^ $#
ALL_DEPS += <SOMETHING>
... more code which appends to ALL_DEPS ...
.PHONY: all
all : $(ALL_DEPS)
I've got some files not generated via Make rules in $(SRC_DIR). (For the sake of this example, let's say there's a directory $(SRC_DIR)b/ and a file $(SRC_DIR)c .)
I want to append to ALL_DEPS all targets which represent files or directories in $(DEST_DIR) so that "make all" will run all of the available $(DEST_DIR)% rules.
I thought to do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(wildcard $(SRC_DIR)*)))
But of course, that doesn't catch anything that hasn't yet been made. (i.e. it doesn't append $(DEST_DIR)a/ to the list because $(SRC_DIR)a/ doesn't yet exist when the $(wildcard ...) invocation is evaluated and the shell doesn't include it in the results returned by the $(wildcard ...) invocation.)
So, rather than a function which finds all (currently-existing) files matching a pattern, I need one which finds all targets matching a pattern. Then, I could do something like this:
ALL_DEPS += $(addprefix $(DEST_DIR),$(notdir $(targetwildcard $(SRC_DIR)*)))
If it matters any, I've got much of the GNU Make code split across multiple files and included by a "master" Makefile. The ALL_DEPS variable is appended to in any of these files which has something to add to it. This is in an attempt to keep the build process modular as opposed to dropping it all in one monster Makefile.
I'm definitely still learning GNU Make, so it's not unlikely that I'm missing something fairly obvious. If I'm just going about this all wrong, please let me know.
Thanks!
It is simply not possible to do what you're trying to do; you're trying to get make to recognise something that doesn't exist.
This is part of the reason why, in general, wildcards are bad (the other being that you can end up including stuff you didn't mean to). The right thing to do here is to explicitly create a list of source files (ls -1 | sed -e 's/\(.*\)/sources+=\1/' > dir.mk) and perform the patsubst transformation on that list.
If you have additional files that are generate as part of the build, then you can append them to that list and their rules will be found as you'd expect.