GNU Make Force Removal of Intermediate Files - makefile

I have a compiler that produces .c files from .ec files as an intermediate step. The compiler does not remove the .c file. The compiler cannot be asked to skip invocation of $CC to produce the .o file. I am trying to have GNU make (3.81) treat the .c files produced as intermediate files and clean them up. The following is an isolated example that reproduces the bad behavior with a file extension that has no implied rules.
.INTERMEDIATE: %.delme
%.o: %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $#
all: test.o
To execute the test case:
rm -f test.*
touch test.ec
make
if [[ -e test.delme ]]; then echo "Failure"; else echo "Success"; fi

Try using a pattern rule to tell make their your compiler produces both .o and .c files from the .ec source. And then declare all the c files as INTERMEDIATE. Pattern rules with multiple outputs work differently than non-pattern rules. Make will run pattern rules only once to produce all output files from the rule, while static rules will be run for each output file. Make also understand that a pattern rule will produce all output files, even if it only wanted one of them as a target.
The result is something like this:
SRC := foo.ec bar.ec
OBJS := $(SRC:.ec=.o)
all: program
program: $(OBJS)
cat $^ > $#
%.o %.c: %.ec
cp $< $(<:.ec=.c) ; cp $< $(<:.ec=.o)
.INTERMEDIATE: $(SRC:.ec=.c)
The command to make the .c and .o from the .ec will be run once to produce both those files. Since make knows it made the .c (even though it only wanted the .o), it will know enough to delete it. The .INTERMEDIATE target will only work if the files are listed explicitly, not using a pattern, so we haven't used %.c. Which seems like a bad idea anyway, what if you had C source that wasn't produce from an .ec file and make deleted it for you? Example output:
$ make
cp foo.ec foo.c ; cp foo.ec foo.o
cp bar.ec bar.c ; cp bar.ec bar.o
cat foo.o bar.o > program
rm bar.c foo.c
$ touch foo.ec ; make
cp foo.ec foo.c ; cp foo.ec foo.o
cat foo.o bar.o > program
rm foo.c
Notice how in the second invocation it only deleted foo.c since bar.o/c wasn't rebuilt.

Make can only consider make targets to be intermediate. You can't just declare a random file on the filesystem as intermediate and have make delete it for you.
Here the .delme file is created as a side effect of the recipe that builds the .o file; make doesn't know anything about it, so make will not delete it because there are no targets in the makefile that are intermediate.
In your example you could split the two cp commands into separate rules and that would allow the intermediate setting to work:
%.delme : %.ec
cp $< $#
%.o : %.delme
cp $< $#
I'm assuming that in your real environment you can't do that because it's all one command that generates the intermediate file and the real file. In that case you'll have to deal with the delete yourself inside the recipe:
%.o : %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $# && rm -f $(<:.ec=.delme)
Note this leaves the .delme file existing if the cp command fails; if you want to remove it no matter what you can do that too.
EDIT
To delete the intermediate file even if the command fails you have to preserve the exit code so you can tell make what it was. Something like:
%.o : %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $#; e=$$?; rm -f $(<:.ec=.delme); exit $$e

Related

How to make a recursive make evaluate sub-directory pre-requisites?

Appreciating that the title is not quite on "target", how can I make it so that when I call make at the top level, it will recursively call the makefiles in the sub-directories?
Having been intrigued by the Kconfig pattern, to learn it, I've applied it to a mark down to pdf generator.
The recursive Makefile resides in ./scripts/Makefile.boilerplate and is defined:
HEADER=$(wildcard section-header.md)
.PHONY:all clean $(md-sub-y)
all clean: $(md-sub-y)
all: $(TARGET)
clean:
# $(RM) $(TARGET)
$(TARGET): $(HEADER) $(md-y) | $(md-sub-y)
# cat /dev/null $^ > $#
$(md-sub-y):
# $(MAKE) -C $(#D) TOPDIR=$(TOPDIR) $(MAKECMDGOALS)
I'm likely using the order-only prerequisite for the $(TARGET) target inappropriately, but it solved a minor problem.
In each directory there is a unique KConfig file (not shown), which lists CONFIG_{OPTION} macros that evaluate to either y or n. Then each directory contains a Makefile that has the form:
include Kconfig
md-$(CONFIG_INTRODUCTION)+= Introduction.md
md-$(CONFIG_FW_UPDATE)+= FW-update.md
md-sub-$(CONFIG_CHAPTERS)+= Chapters/Chapters.md
md-$(CONFIG_CHAPTERS)+= Chapters/Chapters.md
md-$(CONFIG_EXAMPLES)+= Examples.md
md-$(CONFIG_APPENDIX_I)+= Appendix-I.md
md-$(CONFIG_APPENDIX_II)+= Appendix-II.md
md-$(CONFIG_APPENDIX_III)+= Appendix-III.md
include ${TOPDIR}/scripts/Makefile.boilerplate
And finally, the very top level makefile is (abbreviated):
.PHONY: all clean pdf embedded_html
all clean test: JsonAPI/JsonAPI.md
all: pdf embedded_html
pdf: $(MARKDOWN_FILES:.md=.pdf)
embedded_html: $(MARKDOWN_FILES:.md=.html)
MAKEFLAGS += --no-print-directory
clean:
# $(RM) *.pdf *.html
JsonAPI/JsonAPI.md:
# $(MAKE) -C $(#D) TOPDIR=${CURDIR} $(MAKECMDGOALS)
%.html:%.md
# pandoc -s --toc -c /var/www/css/bootstrap.css $< -f markdown -t html -s -o $#
%.pdf:%.md
# pandoc --read=markdown --table-of-contents --toc-depth=3 --preserve-tabs --standalone --template=template.latex $(PANDOC_ENGINE)=pdflatex --listings -V geometry:margin=1in --highlight-style=pygments -H listing-setup.tex -r markdown+simple_tables+table_captions+yaml_metadata_block $< -o $#
If I call make on an unbuilt directory tree, it works fine. But there are a few problems I'm not sure how to address:
How can I ensure that if an updated .md deeply nested in the directory tree will cause the top level PDF file to be updated? Or, How can I force the makefile's in the sub-directories to be called?
The clean target at the top level is problematic, in that it doesn't recurse through the sub-directories. What do I need to do to remedy that?
Is there a better way to include the Makefile.boilerplate makefile, without having to define the TOPDIR on the $(MAKE) command line as I've done?
For 1, and 2, I'm guessing that an empty target dependency (FORCE:) will be required. And for 3, I've tried using $(CURDIR) but it was always evaluating to the directory the Makefile resided in, not the parent directory where the original make command was invoked.
Changing the md-sub-$(CONFIG_EEEE) macro definition to be just the directory was the key, and to make those targets have an empty rule.
Essentially, the per directory Makefile from above becomes:
include Kconfig
md-$(CONFIG_INTRODUCTION)+= Introduction.md
md-$(CONFIG_FW_UPDATE)+= FW-update.md
md-sub-$(CONFIG_CHAPTERS)+= Chapters/Chapters.md
md-$(CONFIG_CHAPTERS)+= Chapters
md-$(CONFIG_EXAMPLES)+= Examples.md
md-$(CONFIG_APPENDIX_I)+= Appendix-I.md
md-$(CONFIG_APPENDIX_II)+= Appendix-II.md
md-$(CONFIG_APPENDIX_III)+= Appendix-III.md
include ${TOPDIR}/scripts/Makefile.boilerplate
and the default Makefile.boilerplate changes the $(md-sub-y) target too:
$(md-sub-y): FORCE
# $(MAKE) -C $# TOPDIR=$(TOPDIR) $(MAKECMDGOALS)
FORCE:
And the top level makefile no longer needs $(#D) on the command line for the JsonAPI recipe, just $#.

GNU make: create targets baed on specific directory contents (1:1 target-directory mapping)

I have a series of directories organized like this:
foo/
foo.file1 foo.file2
bar/
bar.file1 bar.file2
baz/
baz.file1 baz.file2
Right now I'm processing these files using a script that does all the checking for file existence etc but I thought that perhaps I could use a Makefile for it (since said script is very fragile), to avoid reprocessing files that did not change.
The problem is that each directory is independent, and I'd need to do, for example:
foo.file1.processed: foo.file1
run_random_program foo.file1 -o foo.file1.processed
for each of the 71 directories that are in total in that path. This looks like being extremely tedious and I wonder if there's something that would prevent me from writing all of this by hand.
Is such a thing possible?
EDIT: Some examples that show what I have in mind, had I a single Makefile for each directory:
file1.cds.callable: file1.callable
long_script_name -i $< -o $#
file1.rds: file1.cds.callable
another_long_script_name $< additional_file_in_folder $#
file1.csv: file1.rds
yet_another_script $< $#
Seems like pattern rules are exactly what you need:
# These are the original source files (based on the example)
CALLABLE := $(wildcard */*.callable)
# These are the final targets
TARGETS := $(CALLABLE:%.callable=%.csv)
all: $(TARGETS)
%.csv : %.rds
yet_another_script $< $#
%.rds: %.cds.callable
another_long_script_name $< additional_file_in_folder $#
%.cds.callable: %.callable
long_script_name -i $< -o $#

Is it possible to create a target that repeats an action for different files?

I'm trying to build a Makefile that simplifies compilation for a C assignment. The Makefile works fine for now, however, I would like to add a new target that executes a previous target and creates files.
The objective is the following:
Compile a given program (figures.c)
Execute it (this creates a bunch of .gv files)
Transform every .gv file to a .pdf file
I know how to transform a single file (I have the command), but can't seem to figure out how to loop through every file, without typing them all out.
I've already tried doing a different type of target, but does not work (see commented target)
# COMPILATION
CC=gcc
CFLAGS=-Wall -ansi -pedantic
# DOSSIERS
SOURCEDOC=sourcedoc
DOC=doc
SRC=src
INC=inc
OBJ=build
FIGS=images
FILES=$(wildcard $(FIGS)/*.gv)
.PHONY: clean doc archive author all
.SILENT : clean
# Targets
all : clean test images
test : $(OBJ)/Test_arbre.o $(OBJ)/aux.o $(OBJ)/Affichage.o $(OBJ)/ArbreBinaire.o $(OBJ)/arbres.o
$(CC) $^ -o $# $(CFLAGS)
figures : $(OBJ)/figures.o $(OBJ)/Affichage.o $(OBJ)/ArbreBinaire.o $(OBJ)/aux.o $(OBJ)/arbres.o
$(CC) $^ -o $# $(CFLAGS)
%.pdf: $(FIGS)/%.gv
dot -Tpdf -o $(FIGS)/$# $^
#$(FILES): $(FIGS)/%.pdf : $(FIGS)/%.gv
# dot -Tpdf -o $# $^
images : figures $(FILES)
#=========== Objets ===========
$(OBJ)/arbres.o : $(INC)/arbres.h $(INC)/aux.h $(INC)/Affichage.h $(INC)/ArbreBinaire.h
$(OBJ)/Affichage.o : $(INC)/Affichage.h $(INC)/ArbreBinaire.h
$(OBJ)/exemple*_arbre.o : $(INC)/Affichage.h $(INC)/ArbreBinaire.h
$(OBJ)/aux.o : $(INC)/aux.h
$(OBJ)/figures.o : $(INC)/Affichage.h $(INC)/ArbreBinaire.h $(INC)/arbres.h
$(OBJ)/Test_arbre.o : $(INC)/arbres.h $(INC)/ArbreBinaire.h $(INC)/Affichage.h
# Dummy rule
$(OBJ)/%.o : $(SRC)/%.c
#mkdir -p $(#D)
#$(CC) $< $(CFLAGS) -I $(INC)/ -c -o $#
# Miscellaneous
clean:
rm -f *~ */*~
rm -rf __pycache__ src/__pycache__
rm -rf $(DOC)
rm -f $(PROJECT)_$(AUTHOR12)_$(AUTHOR22).zip
rm -f conf.py-e
rm -rf $(OBJ)
rm -f $(FIGS)/*.pdf $(FIGS)/*.gv
rm -f test
The current Makefile works fine on all other commands than images.
If any of you could help, it would mean a lot!
Your definition of FILES should map the *.gv files to the corresponding *.pdf files;
FILES=$(patsubst %.gv,%.pdf,$(wildcard $(FIGS)/*.gv))
The rule which says how to generate a PDF should factor out the directory name;
%.pdf: %.gv
dot -Tpdf -o $# $^
Now, if make tries to create $(FIGS)/ick.pdf, the input will be $(FIGS)/ick.gv - the pattern says to substitute the extension .gv with the extension .pdf, and the rest of the file name stays unmodified, exactly like you'd want. A rule like
%.pdf: $(FIGS)/%.gv # error, don't use
says you need to find the source file in a subdirectory $(FIGS); so if you tried to make $(FIGS)/ick.pdf, that means make would need to find or generate $(FIGS)/$(FIGS)/ick.gv as input according to this rule.
If you absolutely cannot predict what files will be created on step (2) (and so confined to using $(wildcard ...)), you still must execute it after (2) is finished.
It's ugly but I can't think of better alternative than using "recursive make". I mean something like this:
...
.PHONY: images pdf
images: figures
# use figures to generate all .gv files
##figures --create-all-gv-files
# invoke make recursively
#$(MAKE) --no-print-directory pdf
# ensure $(wildcard ...) is invoked only if needed
ifeq ($(MAKECMDGOALS),pdf)
PDF_FILES:=$(patsubst %.gv,%.pdf,$(wildcard $(FIGS)/*.gv))
endif
pdf: $(PDF_FILES)
%.pdf: %.gv
dot -Tpdf -o $# $<

Make rebuilds everytime

I have a Makefile as :
BUILD_DIR= $(BASE_DIR)/build
_OBJ := a.o b.o
CLEAN_OBJECTS := $(_OBJ)
.PHONY: clean
create_code:
python ../script/my_script.py
all: create_code $(_OBJ)
$(_OBJ): %.o: %.c
mkdir -p $(BUILD_DIR)
$(CC) $(CFLAGS) $(INCLUDE_PATH) -c $< -o $#
cp *.o $(BUILD_DIR)
clean:
rm -f $(CLEAN_OBJECTS)
The target create_code executes the python script and generates a set of .c/.h files.
The target _obj compiles them.
Every time I run make all , create_code target is run even though there is no change to .c/.h generated earlier .
Any suggestions on why this is happening and how to make this create_code target run only if make clean was done earlier .
The underlying problem is that you have one or more generated files that depend on something other than the underlying file system -- namely the contents of your database.
One possibility would be to take advantage of the fact that make, having invoked a rule to rebuild a target will, nonetheless, always check the update time of that target when it is specified as a prerequisite in any other rule.
So, given the rule (untested)...
.PHONY: FORCE
%.c: FORCE
command-to-generate-source $#.tmp
diff -q $#.tmp $# || cp $#.tmp $#
Invoking make foo.c from the command line. make will run the commands...
command-to-generate-source foo.c.tmp
diff -q foo.c.tmp foo.c || cp foo.c.tmp foo.c
where command-to-generate-source foo.c.tmp is expected to leave its output in foo.c.tmp. If the newly generated output file is different than the existing file the cp operation will be run and, hence, the target timestamp will be updated and anything dependent on the target will be updated accordingly.
If, however, the newly generated output file is the same as the existing one then no cp will be run, the target file will be left untouched and make will not consider it to be changed when it appears as a prerequisite in other rules.
This is just one possibility but it's the obvious one given that you already have most (if not all) of the required logic in the command python ../script/my_script.py.

How to use make to build a static website into a dist directory

I have a website in an src directory that is several levels deep, containing html, markdown, less, coffee, png, jpg and other assets. I'd like to build my website into a dist directory using make. With build, I mean
converting markdown files to html
compiling coffee to js
compiling less to css
minifying html files
minifying js files (that where not compiled from coffee)
minifying css files (that where not compiled from less or sass)
preparing images (logo.png becomes logo#1x.png logo#2x.png logo#3x.png)
I have the following file. The cp statements will be replaced with the respective tools to do the transformation.
sources = $(shell find src -type f)
t1 := $(sources:.md=.html)
t2 := $(t1:.less=.css)
targets := $(t2:.coffee=.js)
targetdirs = $(dir $(targets))
all: $(targets)
%.html: %.md
cp $< $#
%.css: %.less
cp $< $#
%.js: %.coffee
cp $< $#
This creates outputs side by side. So src/index.md becomes src/index.html, src/assets/stylesheets/app.less becomes src/assets/stylesheets/app.css and src/assets/scripts/app.coffee becomes src/assets/scripts/app.js. What I'd like to do is change the make file such that it stores the output in the dist directory, so src/index.md is converted to dist/index.html, src/assets/stylesheets/app.less compiled to dist/assets/stylesheets/app.css and and src/assets/scripts/app.coffee becomes dist/assets/scripts/app.js.
So I changed the makefile as follows:
sources = $(shell find src -type f)
t0 := $(subst src/,dist/,$(sources:.md=.html))
t1 := $(t0:.md=.html)
t2 := $(t1:.less=.css)
targets := $(t2:.coffee=.js)
targetdirs = $(dir $(targets))
all: $(targets)
%.html: %.md
mkdir -p $(targetdirs)
cp $< $#
%.css: %.less
mkdir -p $(targetdirs)
cp $< $#
%.js: %.coffee
mkdir -p $(targetdirs)
cp $< $#
Now make fails with the following:
make: *** No rule to make target `dist/assets/scripts/app.js', needed by `all'. Stop.
Most examples I can find, is limited to a single directory, or compiles multiple source files into a single target.
How would one achieve this without knowing the contents of the source directory?
Environment:
GNU Make 3.81
OS X 10.11.1
In make pattern rules, the stems represented by the patterns in the target and prerequisite must match exactly. Take this rule:
%.html: %.md
mkdir -p $(targetdirs)
cp $< $#
If the target make wants to build is dist/index.html, then the stem is dist/index. So, it will look for the prerequisite dist/index.md. Which doesn't exist. So make ignores that pattern rule as not matching, and continues to look for more implicit rules that might match... and doesn't find any so it fails.
You have to fix your rules so that the change in directory is reflected in the pattern:
dist/%.html: src/%.md
mkdir -p $(#D)
cp $< $#
(I'm not sure why you're creating all directories in every recipe instead of just the current one). Now the stem for dist/index.html is just index, and the prerequisite matches src/index.md and it will work.

Resources