Hopefully this is a very simple question. I have a makefile pattern rule that looks like this:
%.so : %.f %.pyf
f2py -c -L${LAPACK_DIR} ${GRASPLIBS} -m $* $^ ${SOURCES} --opt='-02' --f77flags='-fcray-pointer' >> silent.txt
I want the makefile to build a number of .so files, so I tried to get it to build two files (radgrd_py.so and lodiso_py.so) by doing this:
radgrd_py.so lodiso_py.so:
%.so : %.f %.pyf
f2py -c -L${LAPACK_DIR} ${GRASPLIBS} -m $* $^ ${SOURCES} --opt='-02' --f77flags='-fcray-pointer' >> silent.txt
and then tried this:
radgrd_py.so:
lodiso_py.so:
%.so : %.f %.pyf
f2py -c -L${LAPACK_DIR} ${GRASPLIBS} -m $* $^ ${SOURCES} --opt='-02' --f77flags='-fcray-pointer' >> silent.txt
But in each case, it only builds the first target that I specify. If I run 'make radgrd_py.so' it works fine, I'm just not sure how to specify a list of files that need to be built so I can just run 'make'.
The usual trick is to add a 'dummy' target as the first that depends on all targets you want to build when running a plain make:
all: radgrd_py.so lodiso_py.so
It is a convention to call this target 'all' or 'default'. For extra correctness, let make know that this is not a real file by adding this line to your Makefile:
.PHONY: all
Best way is to add:
.PHONY: all
.DEFAULT: all
all: radgrd_py.so lodiso_py.so
Explanations:
make uses the first target appearing when no .DEFAULT is specified.
.PHONY informs make that the targets (a coma-separated list, in fact) don't create any file or folder.
all: as proposed by schot
Related
I have a makefile that for various reasons relies on a supporting python script to run every time and grab files from several external locations, copy into working directory, and run through a separate preprocessor before compiling.
This makefile must be able to be run in parallel (-j8) so the order of processing cannot be guaranteed.
In trying to explicitly specify prerequisites, I have created a situation where make skips all object files, goes straight to linking, and fails because the necessary objects do not exist. On a second run, all the objects already exist (the preprocess step skips the files that already exist) and all the files are compiled and linked properly.
When run without -j# everything works fine, but the moment I add -j2, the skipping begins.
Following is an example make file:
GEN_FILES := file1.cpp file2.cpp file3.cpp
CXX_FILES := bin_main.cpp $(GEN_FILES)
OBJ_FILES := $(patsubst %.cpp,%.o,$(CXX_FILES))
.PHONY : all clean prepare
all : bin_file
prepare :
# Copy and preprocess all source files
[ -f file1.cpp ] || cp d1/file1.cpp .
[ -f file2.cpp ] || cp d2/file2.cpp .
[ -f file3.cpp ] || cp d3/file3.cpp .
$(OBJ_FILES) : prepare
bin_file : $(OBJ_FILES)
[ -f file1.o ] && [ -f file2.o ] && [ -f file3.o ] && touch bin_file
%.o : %.cpp
#echo "Compiling $<..."
[ -f $< ] && touch $#
clean :
$(RM) *.o
$(RM) file*
$(RM) bin_file
How can I get this to build in one go, first running prepare to collect all files and then compiling and linking as necessary?
As code_fodder mentions the issue is the creation of the source files.
Basically what happens is, you have not told make how to create these source files, so as far as make knows they don't exist and there's no way to create them. So when make wants to build, for example, file1.o it looks at your pattern rule and finds it could build file1.o from file1.cpp. So then it looks for how to build file1.cpp. No file1.cpp exists, and there is no rule that make knows of that will build it, so make ignores that pattern rule as not matching.
Then make sees the target:
$(OBJ_FILES) : prepare
so it thinks there's no recipe needed to create the object files, and just runs the link line. The next time through, make sees the prepared source files (from the previous build) and then it can use your pattern rule.
If you change your pattern rule to a static pattern rule, where you explicitly tell make exactly what rule to use instead of providing it with a possible rule to use that it can ignore if it doesn't match (which is what a pattern rule is), you'll see the error:
$(OBJ_FILES): %.o : %.cpp
#echo "Compiling $<..."
sleep 1
[ -f $< ] && touch $#
will tell you:
make: *** No rule to make target 'file1.cpp', needed by 'file1.o'. Stop.
Remember, make is looking for a matching pattern rule BEFORE it actually builds anything: it doesn't want to build every possible prerequisite of every possible matching pattern rule, to decide whether or not at the end of it the rule can be used. The rule is matched based on the current state of the filesystem plus rules you have given make about changes it could make. Make has no idea that if it were to invoke the prepare target the source files it was looking for would magically come into existence.
Your basic problem is that this statement is the wrong dependency relationship:
$(OBJ_FILES) : prepare
It's not really true that the object files depend on prepare; what's true is that the PREPARED SOURCE FILES depend on prepare. The object files depend only the "prepared" source files, as your pattern rules shows. This rule should be written, instead:
$(GEN_FILES): prepare
If you do this with -j everything will wait as you want.
Yeah, this gets messy / difficult. The problem you have is that you can specify prerequisite lists - that can work in order, but as soon as you start to use -j then make can start processing prerequisites in any old order. So bin_file requires $(OBJ_FILES) which require prepare. Then %.o requires the same named %.cpp file - which it can do for main.o, but not the filex.o since they don't exist yet - but it tries anyway and fails - in the mean time make (in parallel) is potentially starting to generate the .cpp files, but by this time its too late...etc...
My Prerequisites Build Pattern
I use a very specific prerequisites pattern of my own design - some might frown upon - but I have carefully considered this over the years and found it to be optimal for me.
I create a rule called build or something - which requires build_prerequisites target and then calls make to do the actual build once this is complete:
.PHONY: build
build: build_prerequisites
build:
#echo "start_build"
#$(MAKE) bin_file
This means that build_prerequisites is always run first before the recipe runs. You cant seem to achieve the same forcing of order (at least not easily) using just dependencies. I.e. a list of dependencies can be run in any order with -j, but the rule recipe is always run last.
Now we have this pattern we can fill in the rest. First the build_prerequisites target which does your file generation - I am using echo in my example because I don't have your python script:
.PHONY: build_prerequisites
build_prerequisites:
#echo "build_prerequisites"
echo "create file1" > file1.cpp
echo "create file2" > file2.cpp
echo "create file3" > file3.cpp
Finally add in the c++ compile and link stages - these will be run with the single recursive make call from build - i.e. $(MAKE) bin_file (again I am using echo to create the files in my example):
%.o : %.cpp
#echo "compiling: $<"
##echo "$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $#"
#echo "touch" > $#
bin_file : $(OBJ_FILES)
#echo "linking: $<"
#echo $(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $#
#echo "touch" > $#
Output
Here is the output from my test program (using echo) and main.cpp already exists usingn -j10:
make -j10
build_prerequisites
echo "create file1" > file1.cpp
echo "create file2" > file2.cpp
echo "create file3" > file3.cpp
start_build
make[1]: Entering directory '/mnt/d/software/ubuntu/make'
compile: bin_main.cpp
compile: file1.cpp
compile: file2.cpp
compile: file3.cpp
link: bin_main.o
g++ bin_main.o file1.o file2.o file3.o -o bin_file
make[1]: Leaving directory '/mnt/d/software/ubuntu/make'
Note: if I put a sleep 1 in the "compile" rule - this still takes only 1 second for all 4 files to compile.
Put it all together
GEN_FILES := file1.cpp file2.cpp file3.cpp
CXX_FILES := bin_main.cpp $(GEN_FILES)
OBJ_FILES := $(patsubst %.cpp,%.o,$(CXX_FILES))
###### STAGE 1
.PHONY: build
build: build_prerequisites
build:
#echo "start_build"
#$(MAKE) bin_file
.PHONY: build_prerequisites
build_prerequisites:
#echo "build_prerequisites"
copy_and_pp_files.py $(CXX_FILES) $(SEARCH_DIRS) .
copy_and_pp_files.py $(CFG_FILES) $(SEARCH_DIRS) .
###### STAGE 2
%.o : %.cpp
#echo "compiling: $<"
#$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $#
bin_file : $(OBJ_FILES)
#echo "linking: $<"
#$(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $#
###### OTHER RULES
.PHONY: clean
clean :
#$(RM) *.o
#$(RM) file*
I have attempted to use your actual code, but I have no way to test this so there may be a bug in there. I split it up into 2 "stages" for clarity. Stage 1 is done in your makeor make build call, then state 2 is done in the recursive make call in the build recipe.
I have a large project I'm working on, in which I want to perform include to some .mak file, but only after I make change to this file content via a command inside the original makefile. Since it's a large project it will be hard to write code, so I will give this ridiculous example instead:
I have some small C project that all it's C and header files are in the same directory, and I need to write a makefile. I'm not allowed to use clean rule in the makefile I write, but I have a file named file.mak that I can include in my makefile. Content of file.mak:
.PHONY: clean
cleam:
$(RM) $(objs) test
The problem here is that the rule is cleam and not clean. I'm also not allowed to change manually file.mak , but I'm allowed to do this with a command inside the original makefile. This can be done easily by:
sed -i 's/cleam/clean/g' file.mak
So I thought of writing the makefile like this:
CC = gcc
srcs = $(wildcard ./*.c)
objs = $(srcs:.c=.o)
test: $(objs) change_file include_file
$(CC) $^ -o $#
%.o: %.c
$(CC) -c $< -o $#
change_file:
$(shell sed -i 's/cleam/clean/g' file.mak)
include_file: change_file
include file.mak
But I get the following error:
include: Command not found
So I understand that there is a problem of using include inside a rule, so is there a way to achieve what I want?
(GNU) make has a feature Remaking Makefiles that can be used for scenarios like this, but your approach is wrong. include is a directive and can't be used in a recipe.
Instead, when you include a file, make first checks for rules creating this exact file and executes them. As in your case, the file you want to include already exists, you have to make this rule .PHONY to force its execution. It would look like this:
.PHONY: file.mak
file.mak:
sed -i 's/cleam/clean/g' file.mak
include file.mak
As a more robust alternative (without the need for a phony rule), consider creating a fixed version (copy) and include this:
file_fixed.mak: file.mak
sed -e 's/cleam/clean/g' <file.mak >file_fixed.mak
include file_fixed.mak
first time here. I am relatively new to makefiles. Here is my current makefile:
# Closure compiler php script path
closure = ../../cli/scripts/Compilers/closure.php
# Destination directory
dest = ../../static/js/
# Build directory
build = build/
# Tell "make" to search build and destination dirs
vpath %.o $(build)
vpath %.js $(dest)
all: main.js
#echo "Done.";
main.js: \
date.o \
jquery.autocomplete.o \
jquery.bullseye.o \
jquery.clickopen.o \
jquery.fbmodal.o \
jquery.helpers.o \
jquery.pulljson.o \
jquery.thumbrotate.o \
jquery.timefmt.o \
jquery.tools.o \
layout.main.o
cat $^ > $(dest)$#
%.o: %.js
php $(closure) $*.js $(build)$#
clean:
rm -rf $(build)*.o
rm -rf $(dest)*.js
The problem is with the following line:
cat $^ > $(dest)$#.
It is supposed to cat all the prerequisite objects (minified javascript) into one final js library. According to makefile docs, $^ is an automatic variable which contains a list of prerequisites with directories they are in. From my experience, it behaves differently depending on wether prerequisite needs to be compiled or not.
If prerequisite is up-to-date, this code works perfectly and $^ contains a list like:
build/date.o build/jquery.autocomplete.o build/jquery.bullseye.o....
However if prerequisite needs a fresh compile, then $^ gets directory part stripped and looks like:
date.o jquery.autocomplete.o jquery.bullseye.o
Only the file which needs a fresh compile gets directory part stripped.
I have managed to work around this issue by replacing
cat $^ > $(dest)$#
with
cat $(addprefix $(build), $(^F) ) > $(dest)$#.
I don't like it because:
It's a hack
$(^F) is semi-deprecated
I want to understand why make behaves like this.
thanks
Look here:
# Tell "make" to search build and destination dirs
vpath %.o $(build)
If Make is looking for foo.o, it will look in the local directory first. If it finds no foo.o there, it will look in $(build) (i.e. build/, and you might reconsider your variable names).
And how would Make build foo.o, if it couldn't find it anywhere? With this rule:
%.o: %.js
php $(closure) $*.js $(build)$#
This rule violates an important guideline of makefiles, in that the target (foo.o) is not the name of the thing actually built (build/foo.o).
Now consider what happens when Make tries to execute this rule:
main.js: date.o ...
cat $^ > $(dest)$#
So if date.o is up to date, it's in build/. Make finds it there, and the automatic variable $^ expands to build/date.o ...
But if date.o must be rebuilt, then Make looks to the %.o rule, which promises to build date.o (not build/date.o), so Make takes that rule at its word and $^ expands to date.o ...
There are several ways to solve this problem. I'd do something like this:
OBJS := date.o jquery.autocomplete.o jquery.bullseye.o ...
OBJS := $(addprefix $(build),$(OBJS))
$(dest)main.js: $(OBJS)
cat $^ > $#
# you might have to tinker with this rule a little
$(build)%.o: %.js
php $(closure) $< $#
I have a version.c file in my project that contains current revision of the project and some other stuff that is passed as a definition (-D compiler option) from makefile.
I know that to force make to compile version.c always regardless of modification date I can touch version.c.
Is there a makefile only way to achieve this? If I write .PHONY : version.o the object file doesn't get build at all.
EDIT:
Here is my makefile:
export CC = gcc
export MODULES = $(sort \
sys \
cim \
version \
)
export FILES = $(sort \
main.c \
cim.c \
version.c \
)
VPATH = $(MODULES)
OBJS = $(FILES:.c=.o)
INCLUDES = $(addprefix -I,$(MODULES))
all:$(OBJS)
$(CC) $(INCLUDES) $(OBJS) -o main.exe
clean:
rm -rf *.o *.exe
cim.o: cim.c
main.o: main.c cim.o
version.o: version.c
.PHONY: version.o
.c.o :
$(CC) $(CFLAGS) $(INCLUDES) -c $<
The classic way to do it is:
version.o: .FORCE
.FORCE:
(and you might add .PHONY: .FORCE). The file '.FORCE' is presumed not to exist, so it is always 'created', so version.o is always out of date w.r.t it, so version.o is always compiled.
I'm not sure that making version.o into a phony file is correct; it is actually a real file, not a phony one.
Not a makefile way, but easier than touch:
make -B
‘-B’ ‘--always-make’
Consider all targets out-of-date. GNU make proceeds to consider targets and their prerequisites using the normal algorithms; however,
all targets so considered are always remade regardless of the status
of their prerequisites. To avoid infinite recursion, if MAKE_RESTARTS
(see Other Special Variables) is set to a number greater than 0 this
option is disabled when considering whether to remake makefiles (see
How Makefiles Are Remade).
If you want to do this using the FORCE mechanism the correct solution looks like this:
version.o: FORCE
.PHONY: FORCE
FORCE:
By explicitly declaring FORCE to be phony we make sure things will work right even if .SECONDARY: is used (.SECONDARY: will cause FORCE to be considered an intermediate file, and make doesn't rebuilt intermediate files unless they have prerequisites newer than the ultimate target, and FORCE doesn't have any prerequisites, so .PHONY: FORCE is needed).
The other solution (using $(shell touch version.c)) also has a problem: it may cause your editor to think version.c has been updated, and prompt for a reload of the file, which might end up being destructive if you've been editing the file's buffer but haven't yet saved it. If you don't mind this, it can be made even simpler by observing that the touch command is silent, so the assignment to the hack dummy variable isn't needed:
$(shell touch version.c) # This is enough, but will likely confuse your editor
The .PHONY "trick" referred to in the comments on the question generally DOES NOT work. It may look like it does because it will force a relink iff version.o already exists, but the actual object file won't get rebuilt if the .o file rule is an implicit rule (which it usually is). The problem is that make doesn't do the implicit rule search for explicitly phony targets. This make file shows the failure:
fooprog: test.o
cp $< $#
%.o: %.c
cp $< $#
.PHONY: test.o # WRONG
clean:
rm test.o fooprog
If a static pattern rule is used instead of an implicit rule the .PHONY: version.o trick will work. In general using static pattern rules instead of implicit rules cuts out most of the more confusing Make behaviors. But most make files use implicit rules.
The quick hack version when you just need it to work and you don't want to play Make games:
# Hack to get main.c rebuilt
hack := $(shell touch main.c)
Basically just make Make run touch for you.
I'm sure this is a totally normal thing to do, but I can't figure out how to get make to do this.
I have a compiler that generates make dependencies of the usual form:
M/A.o : M/A.hs
M/B.o : M/A.o
So I write a rule to compile %.hs into %.o, add a rule to link the binary, include the dependencies file, and all is well. But I want to have several binary targets with different flags. E.g. I want build/test built with -DTESTING and build/profile built with -prof. So I need to keep the .o files in a separate tree, where they will be compiled with special flags.
The straightforward way I can think of would be to have dependencies that look something like this:
build/test/M/A.o : M/A.hs
build/test/M/B.o : build/test/M/A.o
build/profile/M/A.o : M/A.hs
... etc.
And then rules so that %.hs to build/test/%.o compiles with -DTESTING, etc. I think this would work, but it's clumsy, means preprocessing the deps file to add all that build/whatever/ prefix stuff, and would multiply its size by however many kinds of builds.
VPATH appears to be designed for this sort of thing and my idea was that I could set the VPATH and compiler flags depending on the target, and it almost works, but:
%.o: %.hs
#mkdir -p build/M
cp $< build/$#
VPATH = build
main: M/A.o M/B.o
cat $^ >$#
M/A.o : M/A.hs
M/B.o : M/B.hs
The first time the main target wants to run 'cat M/A.o M/B.o >main' which seems contrary to the gnu make documentation that says $^ should include the include the VPATH directory in which the dependency was found. Curiously, if I remove 'main' and make again, this time it uses the correct path. This is GNU make, 3.81.
What's going on here? Is there a better way to build with different flags? VPATH seems like a clumsy tool, surely there is a better way?
Make is working correctly. It tries cat M/A.o M/B.o >main the first time because it can't find the prerequisites it needs, but it knows a rule for M/A.o' andM/B.o(<em>not</em>build/M/A.o' and build/M/B.o) and expects that that is what the rule will produce. If you remove main and try again, it will find build/M/A.o' andbuild/M/B.o` via VPATH.
Let's modify this makefile in stages. First we change the VPATH so that it can find the .hs files (Make is good at using things there to build things here, not vise-versa, and that's what VPATH is good for), and change the rules slightly:
build/%.o: %.hs
cp $< $#
VPATH = M
main: build/A.o build/B.o
cat $^ > $#
Now for the different object directories.
build/test/%.o build/project/%.o: %.hs
cp $< $#
VPATH = M
test: build/test/A.o build/test/B.o
cat $^ > $#
project: build/project/A.o build/project/B.o
cat $^ > $#
Then we simplify those last two rules, so that it's easy to add more object files and binary targets:
OBJECTS = A.o B.o
test: $(addprefix build/test/,$(OBJECTS))
project: $(addprefix build/project/,$(OBJECTS))
test project:
cat $^ > $#
Now for the different compiler flags:
build/test/%.o: FLAGS += test_flags
build/project/%.o: FLAGS += proj_flags
build/test/%.o build/project/%.o: %.hs
#echo building $# from $^ using flags $(FLAGS)
cp $< $#
Finally the dependencies. This is a little tricky. Suppose you want the dependency B.o : A.hs to apply to however many object you have. This is one approach:
OBJECT_PATHS = build/test/ build/project/
# The following is from the included file generated by the compiler
$(addsuffix B.o,$(OBJECT_PATHS)) : A.hs
To generate lines like that, I'd pipe the raw lines (e.g. B.o: A.hs) through sed 's/\(.*\):\(.*\)/\1:\2/', and note that if you want to put this in a makefile command, don't forget to double the $ signs to preserve them for the shell.
I know that's a lot to absorb. Take it one step at a time and let us know how it works out.
If you haven't solved your problem by now or are experiencing further problems, best give the autotools (automake and autoconf) a chance. They'll quickly build you a Makefile that supports more configurable and flexible out-of-tree builds.