I have a series (dozens) of projects that consist of large amounts of content in git repositories. Each repository has a git submodule of a common toolkit. The toolkit contains libraries and scripts needed to process the content repositories and build a publishable result. All the repositories are pushed to a host that runs CI and publishes the results. The idea is to keep the repeated code to an absolute minimum and mostly have content in the repositories and rely on and the toolkit to put it all together the same way for every project.
Each project has a top level Makefile that typically only has a couple lines, for example:
STAGE = stage
include toolkit/Makefile
The stage variable has some info about what stage this particular is in which determine which formats get built. Pretty much everything else is handled by the 600 line Makefile in the toolkit. Building some of the output formats can require a long chain of dependencies: The process of a source might trigger a target rule, but to get to the target there might be 8–10 intermediate dependencies where various files get generated before the final target can be made.
I've run across a couple situations where I want to completely replace (not just extend) a specific target rule in just one project. The target gets triggered in the middle of a chain of dependencies but I want to do something completely different for that one step.
I've tried just replacing the target in the top level Makefile:
STAGE = stage
%-target.fmt:
commands
include toolkit/Makefile
This is specifically documented not to be supported, but tantalizingly it works sometime of the time. I've tried changing the order of declaring the custom target and the include but that doesn't seem to significantly affect this. In case it matters, yes, the use of patterns in targets is important.
Sometimes it is useful to have a makefile that is mostly just like another makefile. You can often use the ‘include’ directive to include one in the other, and add more targets or variable definitions. However, it is invalid for two makefiles to give different recipes for the same target.
Interestingly if I put custom functions in the top level Makefile below the include I can override the functions from the toolkit such that $(call do_thing) will use my override:
STAGE = stage
include toolkit/Makefile
define do_thing
commands
endef
However the same does not seem to be true for targets. I am aware of the two colon syntax, but I do not want to just extend an existing target with more dependencies, I want to replace the target entirely with a different way of generating the same file.
I've thought about using recursive calls to make as suggested in the documentation, but then the environment including helper functions that are extensively setup in the toolkit Makefile would not be available to any targets in the top level Makefile. This would be a show stopper.
Is there any way to make make make an exception for me? Can it be coerced into overriding targets? I'm using exclusively recent versions of GNU-Make and am not too concerned about portability. Failing that is there another conceptual way to accomplish the same ends?
¹ My brain hasn't had enough coffee today. In trying to open Stack Overflow to ask this question I typed makeoverflow.com into my browser and was confused why auto-completion wasn't kicking in.
Updated answer:
If your recipe has dependencies, these cannot be overriden by default. Then $(eval) might save you like this:
In toolkit have a macro definition with your generic rule:
ifndef TARGET_FMT_COMMANDS
define TARGET_FMT_COMMANDS
command1 # note this these commands should be prefixed with TAB character
command2
endef
endif
define RULE_TEMPLATE
%-target.fmt: $(1)
$$(call TARGET_FMT_COMMANDS)
endef
# define the default dependencies, notice the ?= assignment
TARGET_DEPS?=list dependencies here
# instantiate the template for the default case
$(eval $(call RULE_TEMPLATE,$(TARGET_DEPS)))
Then into the calling code, just define TARGET_FMT_COMMANDS and TARGET_DEPS before including the toolkit and this should do the trick.
(please forgive the names of the defines/variables, they are only an example)
Initial answer:
Well, I'd write this in the toolkit:
define TARGET_FMT_COMMANDS
command1 # note this these commands should be prefixed with TAB character
command2
endef
%-target.fmt:
$(call TARGET_FMT_COMMANDS)
The you could simply redefine TARGET_FMT_COMMANDS after include toolkit/Makefile
The trick is to systematically have the TAB character preced the commands inside the definition if not you get weird errors.
You can also give parameters to the definition, just as usual.
I ran into the same issue, how I ended up working around the problem that overriding a target does not override the prerequisites was to override the pre-requisites' rules as well to force some of them to be empty commands.
in toolkit/Makefile:
test: depend depend1 depend2
#echo test toolkit
...
in Makefile:
include toolkit/Makefile
depend1 depend2: ;
test: depend
#echo test
Notice how depend1 and depend2 now have empty targets, so the test target's command is overridden and the dependencies are effectively overridden as well.
Related
I was recently debugging a vague problem which turned out to be caused by a misplaced sub-Makefile which was conditionally included into a main Makefile by the -include directive. Mind the starting minus sign. According to GNU Make manual:
If you want make to simply ignore a makefile which does not exist or
cannot be remade, with no error message, use the -include directive
instead of include, like this:
-include filenames…
This acts like include in every way except that there is no error (not
even a warning) if any of the filenames (or any prerequisites of any
of the filenames) do not exist or cannot be remade.
For compatibility with some other make implementations, sinclude is
another name for -include.
The nastiest problem with this directive is that no diagnostics whatsoever are given when the sub-Makefile cannot be found. Needless to say, this complicates debugging a lot.
In fact, there was no real need to use it there, a regular include worked just fine and is much more robust. I understand the original author's intention for using -include. That sub-Makefile contained some "secret" stuff that was not meant to be shared with 3rd party engineers. But this functionality was never used in the end, and it could have been implemented in a more transparent way.
I wonder if there are other practical cases when -include is useful. Maybe some cases when one or several makefiles are dynamically generated during the build process?
Surely, the most useful application of -include is when the include file is auto-generated by make itself.
Remember that all include files also become make's targets automatically. So -include generated_file does not make make to fail prematurely, but implies that generated_file will be (re-)built using other rules in the current Makefile. This can be exploited in auto-dependencies generation, for example.
BTW. Another trick with 'include' is that include $(empty_var) also works without errors (i.e. is no-op).
Ok so im getting in to Kernel Module Development and the guides all pretty much use the same basic make file that contains this line:
make -C /lib/modules/`uname -r`/build M=$(PWD) modules
So my questions are:
Why is a make file calling make? that seems recursive
what is the M for? i cant find a make -M flag in any of the man pages
Recursive use of make is a common technique for introducing modularity into your build process. For example, in your particular case, you could support a new architecture by putting the relevant component in a folder whose name matches the uname -r output for that architecture, and you wouldn't have to change the master makefile at all. Another example, if you make one component modular, it makes it much easier to reuse in another project without making large changes to the new project's master makefile.
Just like it can be helpful to separate your code into files, modules, and classes (the latter for languages other than C, obviously), it can be helpful to separate your build process into separate modules. It's just a form of organization to make managing your projects easier. You might group related functionality into separate libraries, or plugins, and build them separately. Different individuals or teams could work on the separate components without all of them needing write access to the master makefile. You might want to build your components separately so that you can test them separately.
It's not impossible to do all of these things without recursive use of make, of course, but it's one common way of organizing things. Even if you don't use make recursively, you're still going to end up with a bunch of different component "makefiles" on a large project - they'll just be imported or included into the master makefile, rather than standing alone by themselves and being run via separate invocations of make.
Creating and maintaining a single makefile for a very large project is not a trivial matter. However, as the article Recursive make considered harmful describes, recursive use of make is not without its own problems, either.
As for your M, that's just overriding a variable at the command line. Somewhere in the makefile(s) the variable M will be used, and if you specify its value at the command line in this way, then the value you specify will override any other assignments to that variable that may occur in the makefile(s).
I've encountered a little problem during my Makefile learning adventure.
I have prepared two scripts. The first one is the main Makefile:
all:
#echo $(COMPONENTNAME)
include ~/Projects/tests/mk
And here's the mk file
COMPONENTNAME:=ISeeYou
Now, when I run
make all
I get
ISeeYou
I am not sure why... I was convinced that a variable must be defined before it is used. And the include is after COMPONENTNAME is being used. My suspicion is that all of the includes are done first, and then the target creation is started. Is that the case?
My suspicion is that all of the includes are done first, and then the target creation is started.
Yes, you are right.
All global sections in Makefile should be completely parsed before make is able to decide, which targets(and in which order) should be built. Target's recipes are evaluated only when it is needed to build target. This is explicitely stated in make manual:
The other way in which make processes recipes is by expanding any variable references in them. This occurs after make has finished reading all the makefiles and the target is determined to be out of date; so, the recipes for targets which are not rebuilt are never expanded.
I am working on a Makefile written by someone else in my lab and I see the following:
include /path/to/Makefile.inc
TARGET_A: pre_req_1 pre_req_2
cmd_1;
cmd_2;
...
When I look at /path/to/Makefile.inc I see that it also includes a target TARGET_A
with other pre-requisites and recipes.
Is this a normal practice? (and would it work?) Would make treat the two rules separately? Can we safely make any assumptions about which one is considered first?
It is a dangerous practice since it is confusing to know which one is applied.
The "include" will act has if the included file content was in the Makefile, and the targets will be overrided while reading the Makefile. So, the last target will be honored, and the first one (in the included file) will be ignored.
I am looking for suggestions to properly handle separate debug and release build subdirectories, in a recursive makefile system that uses the $(SUBDIRS) target as documented in the gnumake manual to apply make targets to (source code) subdirectories.
Specifically, I'm interested in possible strategies to implement targets like 'all', 'clean', 'realclean' etc. that either assume one of the trees or should work on both trees are causing a problem.
Our current makefiles use a COMPILETYPE variable that gets set to Debug (default) or Release (the 'release' target), which properly does the builds, but cleaning up and make all only work on the default Debug tree. Passing down the COMPILETYPE variable gets clumsy, because whether and how to do this depends on the value of the actual target.
One option is to have specific targets in the subdirectories for each build type. So if you do a "make all" at the top level, it looks at COMPILETYPE and invokes "make all-debug" or "make all-release" as appropriate.
Alternatively, you could set a COMPILETYPE environment variable at the top level, and have each sub-Makefile deal with it.
The real solution is to not do a recursive make, but to include makefiles in subdirectories in the top level file. This will let you easily build in a different directory than the source lives in, so you can have build_debug and build_release directories. It also allows parallel make to work (make -j). See Recursive Make Considered Harmful for a full explanation.
If you are disciplined in your Makefiles about the use of your $(COMPILETYPE) variable to reference the appropriate build directory in all your rules, from rules that generate object files, to rules for clean/dist/etc, you should be fine.
In one project I've worked on, we had a $(BUILD) variable that was set to (the equivalent of) build-(COMPILETYPE) which made rules a little easier since all the rules could just refer to $(BUILD), e.g., clean would rm -rf $(BUILD).
As long as you are using $(MAKE) to invoke sub-makes (and using GNU make), you can automatically exporting the COMPILETYPE variable to all sub-makes without doing anything special. For more information, see the relevant section of the GNU make manual.
Some other options:
Force a re-build when compiler flags change, by adding a dependency for all objects on a meta-file that tracks the last used set of compiler flags. See, for example, how Git manages object files.
If you are using autoconf/automake, you can easily use a separate build out-of-place build directory for your different build types. e.g., cd /scratch/build/$COMPILETYPE && $srcdir/configure --mode=$COMPILETYPE && make which would take the build-type out of the Makefiles and into configure (where you'd have to add some support for specifying your desired build flags based on the value of --mode in your configure.ac)
If you give some more concrete examples of your actual rules, maybe you will get some more concrete suggestions.