Makefile -j does useless things - parallel-processing

I have a shell program that generates 3 files from one source file. For instance I can have 'makedocumentation' that take in input foo.tex and generates foo.pdf, foo.dvi and foo.ps.
Because of performances issues I have to call my Makefile with the option -j8.
With this example file:
names = a b
src = $(addsuffix .def,$(names))
a = $(patsubst %.def,%.inc,$(src))
b = $(patsubst %.def,%.asm,$(src))
c = $(patsubst %.def,%.h,$(src))
obj = $(a) $(b) $(c)
command= cp $(1) $(basename $(1)).inc; \
cp $(1) $(basename $(1)).asm; \
cp $(1) $(basename $(1)).h
all: $(obj)
init: clean $(src)
$(src): %:
touch $#
$(a): %.inc: %.def
$(call command, $<)
$(b): %.asm: %.def
$(call command, $<)
$(c): %.h: %.def
$(call command, $<)
clean:
-rm -f *.def
-rm -f *.inc
-rm -f *.asm
-rm -f *.h
If I run it without -j I get:
$ make
cp a.def a.inc; cp a.def a.asm; cp a.def a.h
cp b.def b.inc; cp b.def b.asm; cp b.def b.h
And in the other case:
$ make -j8
cp a.def a.inc; cp a.def a.asm; cp a.def a.h
cp b.def b.inc; cp b.def b.asm; cp b.def b.h
cp a.def a.inc; cp a.def a.asm; cp a.def a.h
cp b.def b.inc; cp b.def b.asm; cp b.def b.h
cp a.def a.inc; cp a.def a.asm; cp a.def a.h
cp b.def b.inc; cp b.def b.asm; cp b.def b.h
We can observe that in parallel mode, make regenerates files that already exists.
I would like to avoid it...
I also tried something like:
a.inc a.asm a.h: a.def
command $<
But I don't get any better result.
Any help ?

As you wrote the original makefile you haven't told make that the output files share any relationship so it can't know that they've already been built and so it has to try to build each one each time.
Your modified idea is the right idea but unforunately make doesn't interpret that sort of definition as indicating that the single command will generate all the files and instead it sees it as a way to build each individual file (written in a compact shorthand).
A rule with multiple targets is equivalent to writing many rules, each
with one target, and all identical aside from that.
The only way to tell make that N output files are generated from 1 rule body is with multiple pattern rules
Pattern rules may have more than one target. Unlike normal rules, this
does not act as many different rules with the same prerequisites and
recipe. If a pattern rule has multiple targets, make knows that the
rule's recipe is responsible for making all of the targets. The recipe
is executed only once to make all the targets. When searching for a
pattern rule to match a target, the target patterns of a rule other
than the one that matches the target in need of a rule are incidental:
make worries only about giving a recipe and prerequisites to the file
presently in question. However, when this file's recipe is run, the
other targets are marked as having been updated themselves.

Related

GNU Make Force Removal of Intermediate Files

I have a compiler that produces .c files from .ec files as an intermediate step. The compiler does not remove the .c file. The compiler cannot be asked to skip invocation of $CC to produce the .o file. I am trying to have GNU make (3.81) treat the .c files produced as intermediate files and clean them up. The following is an isolated example that reproduces the bad behavior with a file extension that has no implied rules.
.INTERMEDIATE: %.delme
%.o: %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $#
all: test.o
To execute the test case:
rm -f test.*
touch test.ec
make
if [[ -e test.delme ]]; then echo "Failure"; else echo "Success"; fi
Try using a pattern rule to tell make their your compiler produces both .o and .c files from the .ec source. And then declare all the c files as INTERMEDIATE. Pattern rules with multiple outputs work differently than non-pattern rules. Make will run pattern rules only once to produce all output files from the rule, while static rules will be run for each output file. Make also understand that a pattern rule will produce all output files, even if it only wanted one of them as a target.
The result is something like this:
SRC := foo.ec bar.ec
OBJS := $(SRC:.ec=.o)
all: program
program: $(OBJS)
cat $^ > $#
%.o %.c: %.ec
cp $< $(<:.ec=.c) ; cp $< $(<:.ec=.o)
.INTERMEDIATE: $(SRC:.ec=.c)
The command to make the .c and .o from the .ec will be run once to produce both those files. Since make knows it made the .c (even though it only wanted the .o), it will know enough to delete it. The .INTERMEDIATE target will only work if the files are listed explicitly, not using a pattern, so we haven't used %.c. Which seems like a bad idea anyway, what if you had C source that wasn't produce from an .ec file and make deleted it for you? Example output:
$ make
cp foo.ec foo.c ; cp foo.ec foo.o
cp bar.ec bar.c ; cp bar.ec bar.o
cat foo.o bar.o > program
rm bar.c foo.c
$ touch foo.ec ; make
cp foo.ec foo.c ; cp foo.ec foo.o
cat foo.o bar.o > program
rm foo.c
Notice how in the second invocation it only deleted foo.c since bar.o/c wasn't rebuilt.
Make can only consider make targets to be intermediate. You can't just declare a random file on the filesystem as intermediate and have make delete it for you.
Here the .delme file is created as a side effect of the recipe that builds the .o file; make doesn't know anything about it, so make will not delete it because there are no targets in the makefile that are intermediate.
In your example you could split the two cp commands into separate rules and that would allow the intermediate setting to work:
%.delme : %.ec
cp $< $#
%.o : %.delme
cp $< $#
I'm assuming that in your real environment you can't do that because it's all one command that generates the intermediate file and the real file. In that case you'll have to deal with the delete yourself inside the recipe:
%.o : %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $# && rm -f $(<:.ec=.delme)
Note this leaves the .delme file existing if the cp command fails; if you want to remove it no matter what you can do that too.
EDIT
To delete the intermediate file even if the command fails you have to preserve the exit code so you can tell make what it was. Something like:
%.o : %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $#; e=$$?; rm -f $(<:.ec=.delme); exit $$e

How to use make to build a static website into a dist directory

I have a website in an src directory that is several levels deep, containing html, markdown, less, coffee, png, jpg and other assets. I'd like to build my website into a dist directory using make. With build, I mean
converting markdown files to html
compiling coffee to js
compiling less to css
minifying html files
minifying js files (that where not compiled from coffee)
minifying css files (that where not compiled from less or sass)
preparing images (logo.png becomes logo#1x.png logo#2x.png logo#3x.png)
I have the following file. The cp statements will be replaced with the respective tools to do the transformation.
sources = $(shell find src -type f)
t1 := $(sources:.md=.html)
t2 := $(t1:.less=.css)
targets := $(t2:.coffee=.js)
targetdirs = $(dir $(targets))
all: $(targets)
%.html: %.md
cp $< $#
%.css: %.less
cp $< $#
%.js: %.coffee
cp $< $#
This creates outputs side by side. So src/index.md becomes src/index.html, src/assets/stylesheets/app.less becomes src/assets/stylesheets/app.css and src/assets/scripts/app.coffee becomes src/assets/scripts/app.js. What I'd like to do is change the make file such that it stores the output in the dist directory, so src/index.md is converted to dist/index.html, src/assets/stylesheets/app.less compiled to dist/assets/stylesheets/app.css and and src/assets/scripts/app.coffee becomes dist/assets/scripts/app.js.
So I changed the makefile as follows:
sources = $(shell find src -type f)
t0 := $(subst src/,dist/,$(sources:.md=.html))
t1 := $(t0:.md=.html)
t2 := $(t1:.less=.css)
targets := $(t2:.coffee=.js)
targetdirs = $(dir $(targets))
all: $(targets)
%.html: %.md
mkdir -p $(targetdirs)
cp $< $#
%.css: %.less
mkdir -p $(targetdirs)
cp $< $#
%.js: %.coffee
mkdir -p $(targetdirs)
cp $< $#
Now make fails with the following:
make: *** No rule to make target `dist/assets/scripts/app.js', needed by `all'. Stop.
Most examples I can find, is limited to a single directory, or compiles multiple source files into a single target.
How would one achieve this without knowing the contents of the source directory?
Environment:
GNU Make 3.81
OS X 10.11.1
In make pattern rules, the stems represented by the patterns in the target and prerequisite must match exactly. Take this rule:
%.html: %.md
mkdir -p $(targetdirs)
cp $< $#
If the target make wants to build is dist/index.html, then the stem is dist/index. So, it will look for the prerequisite dist/index.md. Which doesn't exist. So make ignores that pattern rule as not matching, and continues to look for more implicit rules that might match... and doesn't find any so it fails.
You have to fix your rules so that the change in directory is reflected in the pattern:
dist/%.html: src/%.md
mkdir -p $(#D)
cp $< $#
(I'm not sure why you're creating all directories in every recipe instead of just the current one). Now the stem for dist/index.html is just index, and the prerequisite matches src/index.md and it will work.

Makefile to generate JSON from Python scripts in separate directories

I have a Makefile which generates JSON from several different Python scripts (the scripts print to stdout) in a single directory, e.g.
/src
scriptOne.py
scriptTwo.py
scriptThree.py
Which outputs the JSON to a folder:
/templates
scriptOne.json
scriptTwo.json
scriptThree.json
I'm trying to restructure so that, for example, each script is in its own subdirectory and the Makefile creates the JSON templates in their consequent subdirectories as follows:
/src
/importantTemplates
scriptOne.py
/notSoImportantTemplates
scriptTwo.py
scriptThree.py
And the output:
/templates
/importantTemplates
scriptOne.json
/notSoImportantTemplates
scriptTwo.json
scriptThree.json
The current Makefile is as follows:
SOURCES := $(shell echo src/*.py)
TARGETS := $(patsubst src/%.py,templates/%.json,$(SOURCES))
all: $(TARGETS)
clean:
rm -f $(TARGETS)
templates/%.json: src/%.py
python2 $< > $#
I've tried changing the wildcards to include a subdirectory for each line e.g. /src/*/*.py, although I just end up with the following:
make: Nothing to be done for `all'.
You want a static pattern rule (4.12 Static Pattern Rules) for this.
SOURCES := $(wildcard src/*/*.py)
TARGETS := $(patsubst src/%.py,templates/%.json,$(SOURCES))
all: $(TARGETS)
clean:
rm -rf templates
$(TARGETS) : templates/%.json: src/%.py
mkdir -p $(#D)
python2 $< > $#
You could avoid needing mkdir -p in that rule body if you wanted to (and go with an order-only prerequisite on the directory instead) but I'm not sure the effort is worth the savings in execution cost. You could avoid the extra shell by combining the two lines mkdir -p $(#D) && python2 $< > $# if you wanted to though.
Please note that the second time you run the make, it will give you the message (if there are no new files):
make: Nothing to be done for `all'.
Try to run make clean and see if you get the same message.
Here is the Makefile which will do what you want:
SOURCES := $(wildcard src/*/*.py)
TARGETS := $(patsubst src/%.py,templates/%.json,$(SOURCES))
FOLDERS := $(sort $(dir $(TARGETS)))
all: $(TARGETS)
clean:
rm -rf $(TARGETS) $(FOLDERS)
$(FOLDERS):
mkdir -p $#
$(TARGETS): $(SOURCES) $(FOLDERS)
python2 $< > $#
The FOLDERS variable will contain the folders you need to create in the template directory. (sort will remove duplicates, so each folder will be there only once)
The $(FOLDERS) rule will create the folders.
The clean rule will remove the folders also.
If you need to add more sources, just do it like this:
SOURCES := $(wildcard src/*/*.py)
SOURCES += $(wildcard src/*.py)
...

Makefile: rule that match multiple patterns

I have this rule in my Makefile, that responds to flags I pass:
$(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/disable_$*
rm -f $(BUILD_DIR)/enable_$*
cd $(BUILD_DIR) && rm -f Makefile
$(BUILD_DIR)/enable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/enable_$*
rm -f $(BUILD_DIR)/disable_$*
cd $(BUILD_DIR) && rm -f Makefile
What this means is that when changing the flags by which I invoke the makefile, I can trigger some recompilations that could depend on these flags.
The code presented above is a bit redundant: you see that I remove a file, touch another and remove a Makefile in both cases. The only thing that changes is the name of the files that I touch/remove, and they are related.
For instance,
make clean
make enable_debug=yes enable_video=no # will compile from zero
make enable_debug=no enable_video=no # flag change detected -> recompile some submodules that depend on this flag
Provided that the only thing that changes between the two rules ( [en|dis]able ), what I would like is to only have 1 generic rule, something like that:
# match 2 parts in the rule
$(BUILD_DIR)/%ble_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/(???)ble_$* # should be $#
rm -f $(BUILD_DIR)/(???)able_$* # should be disable if $# is enable and inverse
cd $(BUILD_DIR) && rm -f Makefile
Is this possible ?
PS: Sorry if I didn't get the title correctly, I couldn't figure how to explain it better.
$(BUILD_DIR)/enable_% $(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
rm -f $(BUILD_DIR)/*able_$*
touch $#
cd $(BUILD_DIR) && rm -f Makefile
Not literally what you wanted (multi-wildcards are forbidden in make), but does quite the same.

Expanding dependency's dirname for Makefile wildcard

Following on from Build script to Makefile, which lives in this upstream location. I want to include the Javascript examples that are included into this generated HTML document as dependencies.
INFILES = $(shell find . -name "index.src.html")
OUTFILES = $(INFILES:.src.html=.html)
TEMP:= $(shell mktemp -u /tmp/specs.XXXXXX)
all: $(OUTFILES)
# Problem line:
%.html: %.src.html $(wildcard contacts/*js)
#echo Dependencies: $^
cd $(#D) && m4 -PEIinc index.src.html > $(TEMP)
anolis --max-depth=3 $(TEMP) $#
rm -f $(TEMP)
clean:
rm -f $(OUTFILES)
PHONY: all clean
I want $(wildcard contacts/*js) to be $(wildcard $(#D)/*js) or $(wildcard $(dirname %)/*js), but nothing I've tried works. There must be some sort of keyword to get the parent directory of the target or dependency so I can reference the javascript dependencies.
AFAIK, using $(#D) and other automatic variables inside list of prerequisites can only be achieved using secondary expansion feature of GNU Make.
Thus, your problem probably can be solved as follows:
.SECONDEXPANSION:
%.html: %.src.html $$(wildcard $$(#D)/*js)
However, I'm not sure whether it will work with pattern rules.

Resources