I have a series of directories organized like this:
foo/
foo.file1 foo.file2
bar/
bar.file1 bar.file2
baz/
baz.file1 baz.file2
Right now I'm processing these files using a script that does all the checking for file existence etc but I thought that perhaps I could use a Makefile for it (since said script is very fragile), to avoid reprocessing files that did not change.
The problem is that each directory is independent, and I'd need to do, for example:
foo.file1.processed: foo.file1
run_random_program foo.file1 -o foo.file1.processed
for each of the 71 directories that are in total in that path. This looks like being extremely tedious and I wonder if there's something that would prevent me from writing all of this by hand.
Is such a thing possible?
EDIT: Some examples that show what I have in mind, had I a single Makefile for each directory:
file1.cds.callable: file1.callable
long_script_name -i $< -o $#
file1.rds: file1.cds.callable
another_long_script_name $< additional_file_in_folder $#
file1.csv: file1.rds
yet_another_script $< $#
Seems like pattern rules are exactly what you need:
# These are the original source files (based on the example)
CALLABLE := $(wildcard */*.callable)
# These are the final targets
TARGETS := $(CALLABLE:%.callable=%.csv)
all: $(TARGETS)
%.csv : %.rds
yet_another_script $< $#
%.rds: %.cds.callable
another_long_script_name $< additional_file_in_folder $#
%.cds.callable: %.callable
long_script_name -i $< -o $#
Related
I've seen a few approaches to making output directories in Make.
These include making all directories ahead of time outside of any rule, and
making an object's destination directory as part of the object's rule.
Both of these approaches involve making directories that likely already exist.
Am I missing any gotchas or drawbacks that explain why I haven't seen the below approach?
.SECONDEXPANSION:
$(OBJDIR)%.o: %.c | $$(#D)/
# Compile command
.PRECIOUS: %/
%/:
# mkdir Command
make is very good at dealing with files. make is not very good at dealing with directories.
So treating directories as implementation detail internal to the target rule makes sense, because then make never has to consider the directory at all:
MKDIR_P = mkdir -p
$(objdir)%.o: %.c
#$(MKDIR_P) $(#D)
$(COMPILE.c) -o $# -c $<
Note that the processing and IO required for the mkdir -p can be neglected next to the processing and IO required for the compilation.
The problem with directories is that (contrary to any other target) you don't care for their timestamp, you only need them to exist. Many Makefiles get directories somehow wrong, and creating them over and over again is what you observe, so make will never detect "Nothing to be done for ...".
In fact, the only thing you need for correct handling of directories with GNU make is an "order only dependency", like shown in your example. The trailing slash normally isn't needed (you seem to use it in order to have a pattern rule, I'm not sure whether this works), and you don't need .PRECIOUS either. Your trick with .SECONDEXPANSION looks quite neat, I guess this will work, given the pattern rule indeed works that way (didn't try).
For an alternative, most Makefiles that handle directories correctly take a simpler approach by concatenating all needed output directories for a rule in a single variable and use this variable as a target for another rule, e.g. like in this simplified example:
MODULES:=src/main
OBJDIR?=obj
OBJS:=$(addprefix $(OBJDIR)/,$(addsuffix .c,$(MODULES)))
DIRS:=$(sort $(addprefix $(OBJDIR)/,$(dir $(OBJS))))
TARGET:= myprogram
all: $(TARGET)
myprogram: $(OBJS)
$(CC) -o$# $^
$(DIRS):
mkdir -p $(DIRS)
$(OBJDIR)/%.o: %.c Makefile | $(DIRS)
$(CC) -c -o$# $<
clean:
rm -fr $(OBJDIR)
.PHONY: all clean
I have a compiler that produces .c files from .ec files as an intermediate step. The compiler does not remove the .c file. The compiler cannot be asked to skip invocation of $CC to produce the .o file. I am trying to have GNU make (3.81) treat the .c files produced as intermediate files and clean them up. The following is an isolated example that reproduces the bad behavior with a file extension that has no implied rules.
.INTERMEDIATE: %.delme
%.o: %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $#
all: test.o
To execute the test case:
rm -f test.*
touch test.ec
make
if [[ -e test.delme ]]; then echo "Failure"; else echo "Success"; fi
Try using a pattern rule to tell make their your compiler produces both .o and .c files from the .ec source. And then declare all the c files as INTERMEDIATE. Pattern rules with multiple outputs work differently than non-pattern rules. Make will run pattern rules only once to produce all output files from the rule, while static rules will be run for each output file. Make also understand that a pattern rule will produce all output files, even if it only wanted one of them as a target.
The result is something like this:
SRC := foo.ec bar.ec
OBJS := $(SRC:.ec=.o)
all: program
program: $(OBJS)
cat $^ > $#
%.o %.c: %.ec
cp $< $(<:.ec=.c) ; cp $< $(<:.ec=.o)
.INTERMEDIATE: $(SRC:.ec=.c)
The command to make the .c and .o from the .ec will be run once to produce both those files. Since make knows it made the .c (even though it only wanted the .o), it will know enough to delete it. The .INTERMEDIATE target will only work if the files are listed explicitly, not using a pattern, so we haven't used %.c. Which seems like a bad idea anyway, what if you had C source that wasn't produce from an .ec file and make deleted it for you? Example output:
$ make
cp foo.ec foo.c ; cp foo.ec foo.o
cp bar.ec bar.c ; cp bar.ec bar.o
cat foo.o bar.o > program
rm bar.c foo.c
$ touch foo.ec ; make
cp foo.ec foo.c ; cp foo.ec foo.o
cat foo.o bar.o > program
rm foo.c
Notice how in the second invocation it only deleted foo.c since bar.o/c wasn't rebuilt.
Make can only consider make targets to be intermediate. You can't just declare a random file on the filesystem as intermediate and have make delete it for you.
Here the .delme file is created as a side effect of the recipe that builds the .o file; make doesn't know anything about it, so make will not delete it because there are no targets in the makefile that are intermediate.
In your example you could split the two cp commands into separate rules and that would allow the intermediate setting to work:
%.delme : %.ec
cp $< $#
%.o : %.delme
cp $< $#
I'm assuming that in your real environment you can't do that because it's all one command that generates the intermediate file and the real file. In that case you'll have to deal with the delete yourself inside the recipe:
%.o : %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $# && rm -f $(<:.ec=.delme)
Note this leaves the .delme file existing if the cp command fails; if you want to remove it no matter what you can do that too.
EDIT
To delete the intermediate file even if the command fails you have to preserve the exit code so you can tell make what it was. Something like:
%.o : %.ec
cp $< $(<:.ec=.delme)
cp $(<:.ec=.delme) $#; e=$$?; rm -f $(<:.ec=.delme); exit $$e
I am looking for Makefile macro to get list of all files in a directory that were generated as rule1 processing and using this list for rule2 processing.
Here's what I am trying to achieve :
Rule 1: Generate source .c files (using xml files) and place them in $(MYDIR) directory.
Rule 2: Get the list of all files in $(MYDIR) and create object files and place them in $(OBJDIR).
Problem is, I want to update list of files in Rule2 after Rule 1 has been processed, else list of files in $(MYDIR) will be empty.
all : rule_1 rule_2
rule1 : $(MYDIR)/generated_source1.c $(MYDIR)/generated_source2.c
$(MYDIR)/generated_source1.c:
xsltproc generator1.xml style_generator.xsl -o $(MYDIR)/generated_source_1.c
$(MYDIR)/generated_source2.c:
xsltproc generator2.xml style_generator.xsl -o $(MYDIR)generated_source_2.c
#Get list of all $(MYDIR).*c , create corresponding $(OBJDIR)/*.o list.
SOURCES := $(wildcard $(MYDIR)/*.c)
OBJECTS := $(notdir ${SOURCES})
GENERATED_OBJS := $(patsubst %.c,$(OBJDIR)/%.o,$(OBJECTS))
#This rule is compiling of all .c generated in rule1.
rule2 : $(GENERATED_OBJS)
ld -r -o $(OBJDIR)/generated_lib.o $(GENERATED_OBJS)
$(OBJDIR)/%.o: $(MYDIR)/%.c
gcc $(CFLAGS) -c -o $# $<
$(SOURCES) is shown empty, but actually it should contain generated_source1.c and generated_source2.c
I am not sure how .SECONDEXPANSION rule will work for my case.
You can't really (and don't really want to) play around with getting make to re-evaluate file existence during the running of the make process.
What you want to do is track the files from start to finish in make and then you have all your lists.
You can start at either direction but starting with the initial source tends to be easier.
So start with
MYDIR:=dir
OBJDIR:=obj
XML_SOURCES := $(wildcard $(MYDIR)/*.xml)
then translate from there to the generated source files
SOURCES := $(subst generator,generated_source,$(XML_SOURCES:.xml=.c))
and from there to the generated object files
GENERATED_OBJS := $(patsubst $(MYDIR)/%.c,$(OBJDIR)/%.o,$(SOURCES))
At which point you can define the default target
all: $(OBJDIR)/generated_lib.o
and then define the rules for each step
$(MYDIR)/%.c:
cat $^ > $#
$(OBJDIR)/%.o: $(MYDIR)/%.c
cat $^ > $#
$(OBJDIR)/generated_lib.o: $(GENERATED_OBJS)
ld -r -o $# $^
The $(MYDIR)/%.c rule needs a bit of extra magic to actually work correctly. You need to define the specific input/output pairs so that they are used correctly by that rule.
$(foreach xml,$(XML_SOURCES),$(eval $(subst generator,generated_source,$(xml:.xml=.c)): $(xml)))
This .xml to .c step would be easier if the input and output files shared a basename as you could then just use this and be done.
%.c: %.xml
cat $^ > $#
first time here. I am relatively new to makefiles. Here is my current makefile:
# Closure compiler php script path
closure = ../../cli/scripts/Compilers/closure.php
# Destination directory
dest = ../../static/js/
# Build directory
build = build/
# Tell "make" to search build and destination dirs
vpath %.o $(build)
vpath %.js $(dest)
all: main.js
#echo "Done.";
main.js: \
date.o \
jquery.autocomplete.o \
jquery.bullseye.o \
jquery.clickopen.o \
jquery.fbmodal.o \
jquery.helpers.o \
jquery.pulljson.o \
jquery.thumbrotate.o \
jquery.timefmt.o \
jquery.tools.o \
layout.main.o
cat $^ > $(dest)$#
%.o: %.js
php $(closure) $*.js $(build)$#
clean:
rm -rf $(build)*.o
rm -rf $(dest)*.js
The problem is with the following line:
cat $^ > $(dest)$#.
It is supposed to cat all the prerequisite objects (minified javascript) into one final js library. According to makefile docs, $^ is an automatic variable which contains a list of prerequisites with directories they are in. From my experience, it behaves differently depending on wether prerequisite needs to be compiled or not.
If prerequisite is up-to-date, this code works perfectly and $^ contains a list like:
build/date.o build/jquery.autocomplete.o build/jquery.bullseye.o....
However if prerequisite needs a fresh compile, then $^ gets directory part stripped and looks like:
date.o jquery.autocomplete.o jquery.bullseye.o
Only the file which needs a fresh compile gets directory part stripped.
I have managed to work around this issue by replacing
cat $^ > $(dest)$#
with
cat $(addprefix $(build), $(^F) ) > $(dest)$#.
I don't like it because:
It's a hack
$(^F) is semi-deprecated
I want to understand why make behaves like this.
thanks
Look here:
# Tell "make" to search build and destination dirs
vpath %.o $(build)
If Make is looking for foo.o, it will look in the local directory first. If it finds no foo.o there, it will look in $(build) (i.e. build/, and you might reconsider your variable names).
And how would Make build foo.o, if it couldn't find it anywhere? With this rule:
%.o: %.js
php $(closure) $*.js $(build)$#
This rule violates an important guideline of makefiles, in that the target (foo.o) is not the name of the thing actually built (build/foo.o).
Now consider what happens when Make tries to execute this rule:
main.js: date.o ...
cat $^ > $(dest)$#
So if date.o is up to date, it's in build/. Make finds it there, and the automatic variable $^ expands to build/date.o ...
But if date.o must be rebuilt, then Make looks to the %.o rule, which promises to build date.o (not build/date.o), so Make takes that rule at its word and $^ expands to date.o ...
There are several ways to solve this problem. I'd do something like this:
OBJS := date.o jquery.autocomplete.o jquery.bullseye.o ...
OBJS := $(addprefix $(build),$(OBJS))
$(dest)main.js: $(OBJS)
cat $^ > $#
# you might have to tinker with this rule a little
$(build)%.o: %.js
php $(closure) $< $#
I've got a make file that generates multiple targets. Something like:
target-a: target-a.src target-include.src
#$(BUILD_TOOL) -f $< -o $#
target-b: target-b.src target-include.src
#$(BUILD_TOOL) -f $< -o $#
target-c: target-c.src target-include.src
#$(BUILD_TOOL) -f $< -o $#
The actual build process (abbreviated as $(BUILD_TOOL) above) is a multiple line thing involving compilers, scripts and various whatnot, but suffice to say, the build process acts on the first target dependency ($<) and produces the output target ($#).
This is quite unwieldly. Would what I've got below be considered a safe way to replace the above (using a pattern rule that doesn't have a suffix)?
all: target-a target-b target-c
% : %.src target-include.src
#$(BUILD_TOOL) -f $< -o $#
The make tool is GNU, and I'm content to use it's powerful extensions.
If target is a literal string, renierpost's solution is very good. If it isn't (or even if it is) this will work:
TARGETS := target-a target-b target-c
all: $(TARGETS)
$(TARGETS): % : %.src target-include.src
#$(BUILD_TOOL) -f $< -o $#
Note that this rule will not build targets you did not intend, not even target-include.
It depends on the rest of your Makefile, but in principle this should work,
if all files are in one directory.
It's better practice to use extensions on your targets.
Is target a literal string? In that case, you can be more specific
(and speed up rule application a tiny little bit, but it's fast already) by using
all: target-a target-b target-c
target-% : target-%.src target-include.src
#$(BUILD_TOOL) -f $< -o $#
GNU make's advanced syntax will come into play if you want to automatically deduce the names of target-a target-b target-c from the target-*.src filenames on the filesystem or something similar.