I am trying to write a make rule with pre-dependencies which are inter related.
a: b $(FILES)
b: $(FILE_1)
$(eval FILES := some_function_using_file($(FILE_1)))
$(FILES):
do something ...
Basically, I need $(FILE_1) to get $(FILES). When make first reads in the file, this rule below does not exist because $(FILES) is not defined initially but gets evaluated dynamically.
$(FILES):
do something ...
Is there a way to make this work? I want to dynamically create $(FILES) and then run its rule.
The targets get evaluated before any of them run. Your best bet is to generate the list before that happens.
FILES = $(shell ls) new.txt
all: $(FILES)
$(FILES):
echo $#
If you're really desperate you could generate another makefile from within the makefile and call another copy of make.
Related
I'm extremely new to makefiles and just spent a full day trying to figure out how to automate my testing. For this project I have one program, main.c and it accepts as parameters an input and an output file. The input file is of the form "test-{filler}.txt" so an example file could be "test-invalid-opcode.txt". The output file would be of the form "out-{filler}.txt", so it would be of the form "out-invalid-opcode.txt". I would then want to compare the output with the correct output saved in "correct-{filler}.txt", so this would be "correct-invalid-opcode.txt". Each test would therefore have an input, output, and correct output. I would want to check if the output and correct output have any differences and I would want this to run on every test file with the prefix "test-". I read through a lot of the makefile spec and a lot of different examples, but I'm really confused as to how to handle this. Any help would be really appreciated.
Here is what I have to run one test:
CC=gcc
CFLAGS=-o
main.o: main.c
$(CC) $(CFLAGS) main.o $<
.PHONY: test
test: main.o test-provided.txt test-out.cs
./program test-provided.txt test-out.txt
diff -q test-out.txt out-provided.txt
The layout of what I want the automatic test to be is something like
.PHONY: autotest
autotest: program
$(foreach <file with "test-" prefix> run:
main.o <test-name> <out-name>
diff -q <correct-name> <out-name>
But I really am lost on how to go about implementing that.
Thank you and sorry I couldn't post more code. I tried a lot of different things, but none of them were even close enough to be worth posting.
Edit to show working final version:
program: main.c
$(CC) $(CFLAGS) $a $<
TEST_INPUTS := $(wildcard test-*.txt)
.PHONY: autotest $(TEST_INPUTS)
autotest: $(TEST_INPUTS)
$(TEST_INPUTS): test-%.txt: program
echo '' > out-$*.txt
./$< $# out-$*.txt
diff -q correct-$*.txt out-$*.txt
In general using loops with make is not very "make-ish". A makefile is an entire language fundamentally based on iteration and recursion so trying to do "extra" iteration inside a recipe is often redundant.
If you need to iterate over something, most especially when that something is files, you should attempt to work with make by taking advantage of its target/prerequisite organization. Above you give an algorithm:
$(foreach <file with "test-" prefix> run:
main.o <test-name> <out-name>
diff -q <correct-name> <out-name>
which is perfectly suited to make's default behavior. To translate this into a makefile you write a rule which invokes a single iteration of the loop, then use prerequisites to run it for the files you want. Something like this:
TEST_INPUTS := $(wildcard test-*.txt)
.PHONY: autotest $(TEST_INPUTS)
autotest: $(TEST_INPUTS)
$(TEST_INPUTS): test-%.txt: program
$< $# test-out.txt
diff -q test-out.txt out-$*.txt
Not only is this much more make-like but it has other advantages: for example you can run make test-provided.txt and it will run just that one test instead of all the tests.
You have to use static pattern rules here not normal pattern rules because .PHONY targets can't work with pattern rules.
I am trying to statically build HTML files that requires a markdown file and a meta file called "whatlinkshere" for the HTML file to demonstrate its back links.
I believe it can be effeciently done by a Makefile, by first generating all the "whatlinkshere" files. I don't think this can be done in parallel, because the program that generates these files needs to append to the whatlinkshere files, and there could be race conditions that I am not quite sure how to solve.
Once the "whatlinkshere" files are generated then if a markdown file is edited, say foo.mdwn to point to bar.mdwn, only foo.mdwn needs to be analysed again for "whatlinkshere" changes. And finally only foo.html and bar.html need to be rebuilt.
I am struggling to accomplish this in my backlinks project.
INFILES = $(shell find . -name "*.mdwn")
OUTFILES = $(INFILES:.mdwn=.html)
LINKFILES = $(INFILES:.mdwn=.whatlinkshere)
all: $(OUTFILES)
# These need to be all made before the HTML is processed
$(LINKFILES): $(INFILES)
#echo Creating backlinks $#
#touch $#
#go run backlinks.go $<
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
Current problems here is that *.whatlinkshere** aren't being generated on first run. My workaround is for i in *.mdwn; do go run backlinks.go $i; done. Furthermore there are not rebuilding as I want after editing a file as described earlier. Something is horribly wrong. What am I missing?
I think I finally understood your problem. If I understood well:
You have a bunch of *.mdwn source files.
You generate *.whatlinkshere files from your *.mdwn source files using the backlinks.go utility. But this utility does not produce foo.whatlinkshere from foo.mdwn. It analyzes foo.mdwn, searches for links to other pages in it and, for each link to bar it finds, it appends a [foo](foo.html) reference to bar.whatlinkshere.
From each foo.mdwn source file you want to build a corresponding foo.html file with:
$ cmark foo.mdwn foo.whatlinkshere
Your rule:
$(LINKFILES): $(INFILES)
#echo Creating backlinks $#
#touch $#
#go run backlinks.go $<
contains one error and has several drawbacks. The error is the use of the $< automatic variable in the recipe. It expands as the first prerequisite, that is probably always pageA.mdwn in your case. Not what you want. $^ expands as all prerequisites but it is not the correct solution because:
your go utility takes only one source file name, but even if it was accepting several...
...make will run the recipe several times, one per link file, which is a waste, and...
...as your go utility appends to the link files it will even be worse than a waste: back links will be counted several times each, and...
...if make runs in parallel mode (note that you can prevent this with make -j1 or by adding the .NOTPARALLEL: special rule to your Makefile, but it is a pity) there is a risk of race conditions.
Important: the following works only with a flat organization where all source files and HTML files are in the same directory as the Makefile. Other organizations are possible, of course, but they would require some modifications.
First option using multi-targets pattern rules
One possibility is to use a special property of make pattern rules: when they have several targets make considers that one single execution of the recipe produces all targets. For instance:
pageA.w%e pageB.w%e pageC.w%e: pageA.mdwn pageB.mdwn pageC.mdwn
for m in $^; do go run backlinks.go $$m; done
tells make that pageA.whatlinkshere, pageB.whatlinkshere and pageC.whatlinkshere are all generated by one execution of:
for m in pageA.mdwn pageB.mdwn pageC.mdwn; do go run backlinks.go $m; done
(make expands $^ as all prerequisites and $$m as $m). Of course, we want to automate the computation of the pageA.w%e pageB.w%e pageC.w%e pattern targets list. This should make it:
INFILES := $(shell find . -name "*.mdwn")
OUTFILES := $(INFILES:.mdwn=.html)
LINKFILES := $(INFILES:.mdwn=.whatlinkshere)
LINKPATTERN := $(INFILES:.mdwn=.w%e)
.PHONY: all clean
.PRECIOUS: $(LINKFILES)
all: $(OUTFILES)
# These need to be all made before the HTML is processed
$(LINKPATTERN): $(INFILES)
#echo Creating backlinks
#rm -f $(LINKFILES)
#touch $(LINKFILES)
#for m in $^; do go run backlinks.go $$m; done
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
clean:
rm -f $(LINKFILES) $(OUTFILES)
Notes:
I declared all and clean as phony because... it is what they are.
I declared the whatlinkshere files as precious because (some of them) are considered by make as intermediates and without this declaration make would delete them after building the HTML files.
In the recipe for the whatlinkshere files I added rm -f $(LINKFILES) such that, if the recipe is executed, we restart from a clean state instead of concatenating new stuff to old (possibly outdated) references.
The pattern stem in the $(LINKPATTERN) can be anything but must match at least one character. I used w%e but whatlin%shere would work too. Use whatever is specific enough in your case. If you have a pageB.where file prefer whatlin%shere or what%here.
There is a drawback with this solution but it is due to your particular set-up: each time one single mdwn file changes it must be re-analyzed (which is normal) but any whatlinkshere file can be impacted. This is not predictable, it depends on the links that have been modified in this source file. But more problematic is the fact that the result of this analysis is appended to the impacted whatlinkshere files. They are not "edited" with the old content relative to this source file replaced by the new one. So, if you change just a comment in a source file, all its links will be appended again to the respective whatlinkshere files (while they are already there). This is probably not what you want.
This is why the solution above deletes all whatlinkshere files and re-analyzes all source files each time one single source file changes. And another negative consequence is that all HTML files must also be re-generated because all whatlinkshere files changed (even if their content did not really change, but make does not know this). If the analysis is super fast and you have a small number of mdwn files, it should be OK. Else it is sub-optimal but not easy to solve because of your particular set-up.
Second option using recursive make, separated back link files and marker files
There is a possibility, however, which consists in:
separating all back links references with one whatlinkshere file per from/to pair: foo.backlinks/bar.whatlinkshere contains all references to bar found in foo.mdwn,
using recursive make with one first invocation (when the STEP make variable is unset) to update all whatlinkshere files that need to be and a second invocation (STEP set to 2) to generate the HTML files that need to be,
using empty dummy files to mark that a foo.mdwn file has been analyzed: foo.backlinks/.done,
using the secondary expansion to be able to refer to the stem of a pattern rule in its list of prerequisites (and using $$ to escape the fist expansion).
But it is probably a bit more difficult to understand (and maintain).
INFILES := $(shell find . -name "*.mdwn")
OUTFILES := $(INFILES:.mdwn=.html)
DONEFILES := $(patsubst %.mdwn,%.backlinks/.done,$(INFILES))
.PHONY: all clean
ifeq ($(STEP),)
all $(OUTFILES): $(DONEFILES)
$(MAKE) STEP=2 $#
%.backlinks/.done: %.mdwn
rm -rf $(dir $#)
mkdir -p $(dir $#)
cp $< $(dir $#)
cd $(dir $#); go run ../backlinks.go $<; rm $<
touch $#
else
all: $(OUTFILES)
.SECONDEXPANSION:
%.html: %.mdwn $$(wildcard *.backlinks/$$*.whatlinkshere)
#echo Deps $^
#cmark $^ > $#
endif
clean:
rm -rf *.backlinks $(OUTFILES)
Even if it looks more complicated there are a few advantages with this version:
only outdated targets are rebuilt and only once each,
all whatlinkshere files are updated (if needed) before any HTML file is updated (if needed),
the whatlinkshere files can be built in parallel,
the HTML files can be built in parallel.
Third option using only recursive make and marker files
If you do not care about inaccurate results where back links persist in the results after they disappeared from the source files or where back links are uselessly replicated, we can reuse ideas from the previous solution but drop the separation in individual from/to whatlinkshere files.
INFILES := $(wildcard *.mdwn)
OUTFILES := $(patsubst %.mdwn,%.html,$(INFILES))
LINKFILES := $(patsubst %.mdwn,%.whatlinkshere,$(INFILES))
DONEFILES := $(patsubst %.mdwn,.%.done,$(INFILES))
.PHONY: all clean
.PRECIOUS: $(LINKFILES)
ifeq ($(STEP),)
.NOTPARALLEL:
all $(OUTFILES): $(DONEFILES)
$(MAKE) STEP=2 $#
.%.done: %.mdwn
go run backlinks.go $<
touch $#
else
all: $(OUTFILES)
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
%.whatlinkshere:
touch $#
endif
clean:
rm -f $(OUTFILES) $(LINKFILES) $(DONEFILES)
Notes:
As this works only for a flat organization I replaced the $(shell find...) by the make built-in $(wildcard ...).
I used patsubst instead of the old syntax but it's just a matter of taste.
The %.whatlinkshere: rule is a default rule to create the missing empty whatlinkshere files.
The NOTPARALLEL: special target prevents parallel execution when building the whatlinkshere files.
Let's say that I have a program generator that accepts a list of filenames with suffixes .source and produces one file with suffix .source replaced with .target for each argument.
I have a set of files with suffixes .source and want to write a rule that calls this program once for all files that are newer than targets.
I just want to optimise this working Makefile that calls generator for each changed source.
SOURCES=$(wildcard *.source)
TARGETS=$(SOURCES:%.source=%.target)
all: $(TARGETS)
%.target : %.source
./generator $<
This works as required:
SOURCES=$(wildcard *.source)
all: target.timestamp
target.timestamp : $(SOURCES)
./generator $?
touch target.timestamp
Can I avoid creating the timestamp file?
You could do that by collecting the prerequisites in a list in each recipe
and then doing a roundup operation in a phony target:
SOURCES=$(wildcard *.source)
TARGETS=$(SOURCES:%.source=%.target)
all: collective_build
%.target : %.source
$(eval collective_src += $<)
# we do a cp to update the .target files
collective_build: $(TARGETS)
$(foreach f,$(collective_src),cp $(f) $(subst source,target,$(f)); )
BUT you are breaking Pauls rule #2 "Every non-.PHONY rule must update a file with the exact name of its target." and are unhinging the very basis of make.
make need a timestamp to know if run generator or not; but your generator build a list of files, so there is no a single timestamp.
You wrote two valid makefiles, with the pros and cons you know, and there is no alternative.
I have a large project I'm working on, in which I want to perform include to some .mak file, but only after I make change to this file content via a command inside the original makefile. Since it's a large project it will be hard to write code, so I will give this ridiculous example instead:
I have some small C project that all it's C and header files are in the same directory, and I need to write a makefile. I'm not allowed to use clean rule in the makefile I write, but I have a file named file.mak that I can include in my makefile. Content of file.mak:
.PHONY: clean
cleam:
$(RM) $(objs) test
The problem here is that the rule is cleam and not clean. I'm also not allowed to change manually file.mak , but I'm allowed to do this with a command inside the original makefile. This can be done easily by:
sed -i 's/cleam/clean/g' file.mak
So I thought of writing the makefile like this:
CC = gcc
srcs = $(wildcard ./*.c)
objs = $(srcs:.c=.o)
test: $(objs) change_file include_file
$(CC) $^ -o $#
%.o: %.c
$(CC) -c $< -o $#
change_file:
$(shell sed -i 's/cleam/clean/g' file.mak)
include_file: change_file
include file.mak
But I get the following error:
include: Command not found
So I understand that there is a problem of using include inside a rule, so is there a way to achieve what I want?
(GNU) make has a feature Remaking Makefiles that can be used for scenarios like this, but your approach is wrong. include is a directive and can't be used in a recipe.
Instead, when you include a file, make first checks for rules creating this exact file and executes them. As in your case, the file you want to include already exists, you have to make this rule .PHONY to force its execution. It would look like this:
.PHONY: file.mak
file.mak:
sed -i 's/cleam/clean/g' file.mak
include file.mak
As a more robust alternative (without the need for a phony rule), consider creating a fixed version (copy) and include this:
file_fixed.mak: file.mak
sed -e 's/cleam/clean/g' <file.mak >file_fixed.mak
include file_fixed.mak
I often find myself wanting to refer to the stem not in the recipe but int the prerequisites themselves.
For example here I was playing around with some python code that scans the .cpp and .hpp files of the executable source code, in a recursive fashion, to detect what objects it depends from. The script itself works pretty well but I can't figure out how to connect it with the makefile since the input varies.
$(TESTS): bin/tests/%_a : bin/obj/%.o $(foreach var, $(shell python3 ./autoInc.py ./src/lib/%.cpp), bin/obj/$(var).o)
#echo "#---------------------------"
#echo "# Linking $# "
$(CC) -o $# $^
(Here the makefile executes ./autoInc.py ./src/lib/%.cpp without substitution)
This is the form:
.SECONDEXPANSION:
$(TESTS): %_a : $$(foreach var, $$(shell whatever $$*.cpp), $$(var).o)
...
I advise you to get it working with a very simple toy rule, before trying to incorporate your python.