Makefile foreach - makefile

I'm trying to create a makefile which downloads some pre-requisite files to a path.
But the foreach documentation is sadly lacking in detail and examples.
I want something like:
image_files = a b
image_versions = 701.2 802.1
image_path = images
images = $(foreach ...) ??
I'd like that to result in an expansion to:
images/701.2/a
images/701.2/b
images/802.1/a
images/802.1/b
And have a phony target to download them from a URL like:
mytarget: $(images)
wget somepath $<
How do I do that?
Ok I have gotten a little further with this. But I'm still a little perplexed as to how I can get this to work.
tag = my-registry:8443/boot-server-data
versions = 557.0.0 607.0.0
images_a = $(foreach ver, $(versions), images/$(ver)/coreos_production_pxe_image.cpio.gz)
images_b = $(foreach ver, $(versions), images/$(ver)/coreos_production_pxe.vmlinuz)
all: build
.PHONY: build $(images_a) $(images_b)
build:
./make-profiles
docker build -t $(tag) .
docker push $(tag)
$(images_a):
wget http://stable.release.core-os.net/amd64-usr/$(foreach version... but depends on each image)/coreos_production
How do you do this?
In fact I only want it to download the images if they aren't there. But for some reason it downloads it every time. It's literally been years since I used Make. I normally use another build tool, but that build tool needs to be modified to make it do what I want here. So I thought I'd just whip this up in the meantime. It's prooving to be a little harder than expected.

You are pretty close, but the problem does not lie with foreach. Let's have a look at just the bit that does the downloading. When make reads the makefile it ends up with something like (after shortening the names a bit for clarity):
images/1/file.cpio.gz images/2/file.cpio.gz:
<recipe>
If, for some reason, make decides to rebuild images/1/file.cpio.gz say, at this point it will expand the recipe, and pass each line of that expansion to a separate shell.
Your job is to write a recipe that does not care whether the target is images/1/file.cpio.gz or images/2/file.cpio.gz. That's another way of saying the recipe should use macros like $# (it will expand to the target).
A sketch:
${images_a}:
wget -O $# http://stable.release.core-os.net/amd64-usr/$#
You may have to munge $# so that wget gets the right url. Just one example:
${images_a}:
wget -O $# http://stable.release.core-os.net/$(dirname $#)/deeper/$(notdir $#)
One complaint about your original makefile: the dependencies are wrong. build needs the downloads to have completed before it runs.
.PHONY: build
build: $(images_a) $(images_b)
...
The images are not phony (just ensure you don't lie to make abut their filenames) either.
The massive advantage of writing your makefile in this way is that it's parallel safe (and that's the whole point of make). When -j is in force, both wgets can proceed at the same time, halving the download time.

Related

GNU Make: Batching multiple targets in single command invocation

Consider the following setup:
$ touch 1.src 2.src 3.src
$ cat Makefile
%.dst: %.src
#convert -o "$#" "$<"
We can compile our .src files into .dst files by running make 1.dst 2.dst 3.dst which calls the convert (just a placeholder) tool three times.
This setup is fine if there is little overhead in calling convert. However, in my case, it has a startup penalty of a few seconds for every single call. Luckily, the tool can convert multiple files in a single call while paying the startup penalty only once, i.e. convert -o '{}.dst' 1.src 2.src 3.src.
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
Edit: To be more precise, what feature I am looking for: Say that 1.dst is already newer than 1.src so it doesn't need to be recompiled. If I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
A quick and dirty way would be creating a .PHONY rule that simply converts all src files to dst files but that way I would convert every src file each and every time. Further more, specifying dst files as prerequisites in other rules would also no longer be possible.
Thanks in advance!
If you have GNU make 4.3 or above, you can use grouped targets like this:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(_DST_FILES:.dst=.src)
all: $(DST_FILES)
$(DST_FILES) &: $(SRC_FILES)
convert -o '{}.dst' $?
#touch $(DST_FILES)
If your convert is only updating some of the targets then you need the explicit touch to update the rest.
Here's a way to do it with passing a goal on the command line that might work; change DST_FILES to:
DST_FILES := $(or $(filter %.dst,$(MAKECMDGOALS)),1.dst 2.dst 3.dst)
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
It is possible, but messy, to write make rules for build steps that produce multiple targets with a single run of the recipe, such that the recipe is executed just once if any of the targets needs to be updated. However, you clarify that
[if] 1.dst is already newer than 1.src [, and] I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
. That's a slightly different problem. You can use the $? automatic variable in a recipe to get the prerequisites that are newer than the rule's target, but for that to serve the purpose, you need a rule with a single target.
Here's one slightly convoluted way to make it work:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(DST_FILES:.dst=.src)
$(DST_FILES): dst.a
ar x $< $#
dst.a: $(SRC_FILES)
convert -o '{}.dst' $?
x='$?'; ar cr $# $${x//src/dst}
The dst.a archive serves as the one target with all the .src files as prerequisites, so as to provide a basis for use of $?. Additionally, it provides a workaround for the problem that whenever that target is updated, it becomes newer than all the then-existing .dst files: .dst files that are out of date with respect to the archive but not with respect to the corresponding .src file are extracted from the archive instead of being rebuilt from scratch.

Number Files to get rebuilt by Make

Is there a way to let make determine the number of files to be recompiled before actually compiling? The problem is this: Consider having a quite big project with hundreds of source files. It would very convenient to have a rough idea of how long compilation will take, but to know that, one needs to know the number of files to be compiled.
The general answer is no, because your build could generate files which themselves are inputs to other rules which generate more files. And so on. However if a rough answer is good enough you can try the --dry-run flag. From the GNU make documentation...
“No-op”. Causes make to print the recipes that are needed to make the targets up to date, but not actually execute them. Note that some recipes are still executed, even with this flag (see How the MAKE Variable Works). Also any recipes needed to update included makefiles are still executed (see How Makefiles Are Remade).
As you can see, despite its name even the --dry-run flag will change the state of your build.
"make -n" will do the dry run. But you can't get the list of files to be rebuilt. May be you can write shell script to store the last modified time of files and get the list of files.
I think a found a decent solution for unix. Here SRC are your source files, HDR your headers and DEP the dependency files (something like DEP:=$(OBJ:.o=.d) )
isInDepFile+=$(shell grep -q $(modifiedFile) $(depFile) 1>&2 2> /dev/null && echo $(depFile))
COMPFILES=
checkDepFiles=$(foreach depFile,$(DEP), $(eval filesToCompile+=$(isInDepFile))) $(thinOutDepFiles)
thinOutDepFiles=$(foreach fileToCompile,$(filesToCompile),$(eval DEP=$(filter-out $(fileToCompile),$(DEP))))
countFilesToCompile: $(SRC) $(HDR)
$(eval modifiedFiles=$?)
$(foreach modifiedFile,$(modifiedFiles), $(call checkDepFiles))
$(eval numOfFilesToCompile = $(words $(filesToCompile)))
$(eval numDepFiles = $(words $(DEP)))
$(eval NumSRCFiles = $(words $(SRC)))
#echo $(NumSRCFiles) sources
#echo $(numDepFiles) files to leave
#echo $(numOfFilesToCompile) files to compile
#touch $#
This first generates a list of modified files within your source and header files lists. Then for each modified file it checks all dependency files for its filename. If a dependency file contains the current file we are looking at, it is added to the list of filesToCompile. It is also removed from the list of dependency files to avoid duplication.
This can be invoked in the main building rule of your project. The advantage of that over the dry run is that it gives you a simple number to work with.

Make target using a pattern

I'm doing something which it feels like should be pretty straightforward. I have source files in a directory called ./src which are transformed and saved to ./. For the sake of the question, I'll just say they get copied there.
Here's what the directory looks like before building:
/src/lib/foo.js
/src/lib/mod/bar.js
/src/bin/baz.js
/Makefile
Here's what should be there after building:
/src/lib/foo.js
/src/lib/mod/bar.js
/src/bin/baz.js
/lib/foo.js
/lib/mod/bar.js
/bin/baz.js
/Makefile
In my Makefile I have this:
SRC_FILES := src/lib/foo.js src/lib/mod/bar.js src/bin/baz.js
OUT_FILES := lib/foo.js lib/mod/bar.js bin/baz.js
These are generated from find and a pattern substitution, but are listed like this here for simplicity...
Now, what I was hoping would work would be:
%.js: src/%.js
cp $< $#
But when I run make lib/foo.js I get Nothing to be done for 'lib/foo.js'.
Looking at the output from debugging, it appears to be trying to find a dependency named lib/src/foo.js. This is kind of what is described in the manual.
This feels as though it should be a really easy thing! What am I doing wrong?
Additionally, I tried:
$(OUT_FILES): $(SRC_FILES)
cp $< $#
...which works, but it rebuilds every single file if any of the sources change. (Remember that my actual transformation isn't just cp, it's a relatively expensive operation, so this is undesirable)
Found something which works. Using secondary expansion, you have access to the target variable in the dependencies. I saw this earlier, but missed the all-important thing: adding an empty target called .SECONDEXPANSION
.SECONDEXPANSION:
all: $(OUT_FILES)
%.js: src/$$#
cp $< $#

How can I use a recipe in Make to run several distinct Makefiles?

I inherited a project that has multiple top level Makefiles, one for each very-similar platform that the firmware image runs on. They have names like:
Makefile.apple.mak
Makefile.banana.mak
...
So every time I want to build a specific image, I run the command
Make -f Makefile.apple.mak // default is release
Much of the code is shared between the different images that get built. A problem with doing this is that sometimes when a change is made that compiles and works for the apple platform, it breaks the banana platform. There is a whole mess of #ifdefs that conditionally include or exclude code based on the platform specified by the Makefile. I know, it's a mess and badly in need of a refactor (did I mention I inherited this?)
In order to easily build all of the firmware images after making a change and declaring it good on one platform, I want to have a wrapping Makefile that invokes a build of one or all of the available firmware images.
I am currently looking at this brute-force approach. I know there's got to be a more efficient way to do this. I also don't know offhand how to pass additional arguments to this outer Makefile (i.e. make apple debug) if that is possible. Any help is appreciated.
.PHONY: all
all: cleanall releaseall
.PHONY: cleanall
cleanall:
make -f Makefile.apple.mak clean # Can I instead call apple target with an arg?
make -f Makefile.banana.mak clean # Can I instead call banana target with an arg?
.PHONY: releaseall
releaseall:
make -f Makefile.apple.mak
make -f Makefile.banana.mak
.PHONY: apple
apple:
make -f Makefile.apple.mak # how do I pass in the <clean | debug | release> args?
.PHONY: banana
banana:
make -f Makefile.banana.mak # how do I pass in ...
(BTW, you never want to use make to invoke a recursive make. Always use $(MAKE))
The way to do this is construct target names encoding the image and the target for that image you want to build. Then you can use make prerequisites.
So, for example, you can do something like this:
IMAGES := apple banana ...
all: cleanall releaseall
cleanall: $(IMAGES:%=%.clean)
releaseall: $(IMAGES)
$(IMAGES):
$(MAKE) -f Makefile.$#.mak
$(IMAGES:%=%.clean):
$(MAKE) -f Makefile.$(basename $#).mak clean
.PHONY: all cleanall releaseall $(IMAGES) $(IMAGES:%=%.clean)
This also lets you run make apple.clean to clean just the apple directory, or whatever.
You can add more targets like debug, etc. as well.

How to force a certain groups of targets to be always run sequentially?

Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.

Resources