make recompiles source code files even if they are unchanged. To reproduce this behavior, I need to do:
make clean, so that only *.90 files are left
make
save one of the *.f90 file, e.g. "touch bands.f90"
make: bands.f90 is recompiled, which is correct, since it was changed
make: bands.f90 is recompiled again, which is not correct, since bands.f90 was not touched
In this example, after 4. the file bands.f90 will be recompiled at every make call. If I additionally change any other source code file, the behavior also applies to that file. So after some time I end up recompiling all of my source code. I can partially fix this by calling make clean, which resets the loop. But it is not a permanent fix.
Below you see the output for the five steps above:
1: find . ! -name 'Makefile' -a ! -name '*.f90' -type f -exec rm -f {} +
2: [triggered by changes in const.f90]
mpifort -c const.f90
[triggered by changes in system.f90 const.mod]
mpifort -c system.f90
[triggered by changes in io.f90 system.mod const.mod]
mpifort -c io.f90
[triggered by changes in parallel.f90]
mpifort -c parallel.f90
[triggered by changes in bands.f90 system.mod io.mod const.mod parallel.mod]
mpifort -c bands.f90
[triggered by changes in polmob.f90 io.mod system.mod bands.mod parallel.mod]
mpifort -o polmob polmob.f90 io.f90 system.f90 bands.f90 parallel.f90
3: "no output"
4: [triggered by changes in bands.f90]
mpifort -c bands.f90
5: [triggered by changes in bands.f90]
mpifort -c bands.f90
At step 5 make should actually say that there is nothing to compile. But is says that there were changes in bands.f90 and it therefore needs to be recompiled.
Here is my Makefile:
polmob: polmob.f90 io.mod system.mod bands.mod parallel.mod
#echo [triggered by changes in $?]
mpifort -o polmob polmob.f90 io.f90 system.f90 bands.f90 parallel.f90
io.mod: io.f90 system.mod const.mod
#echo [triggered by changes in $?]
mpifort -c io.f90
system.mod: system.f90 const.mod
#echo [triggered by changes in $?]
mpifort -c system.f90
bands.mod: bands.f90 system.mod io.mod const.mod parallel.mod
#echo [triggered by changes in $?]
mpifort -c bands.f90
const.mod: const.f90
#echo [triggered by changes in $?]
mpifort -c const.f90
parallel.mod: parallel.f90
#echo [triggered by changes in $?]
mpifort -c parallel.f90
clean:
find . ! -name 'Makefile' -a ! -name '*.f90' -type f -exec rm -f {} +
The flag $? tells me, that the recompilation is triggered by changes in the particular .f90 file. So somehow make relates to old changes. Does anybody know that the cause for this behavior might be?
Does anybody know that the cause for this behavior might be?
I found the cause: If I replace "mod" by "o" everywhere in my
Makefile, the problem disappears. However, I don't quite understand
why.
The only plausible explanation is that the timestamps of the existing .mod files are not updated when the original sources are rebuilt. I presume that extends only as long as the module interface is unchanged -- i.e. no functions or subroutines are added or removed, or modified in a way that affects their external interface, and similarly no module variables are added, removed, or changed in type.
I can see that it might be considered desirable to avoid updating .mod files when nothing in them has actually changed, particularly in conjunction with module implementations stored in external libraries. To the best of my knowledge, however, there is no general standardization of even the existence of .mod files, much less the circumstances under which they are created or updated. Your compiler may offer an option that forces them to be updated when their corresponding source is compiled; if so, that adding that option to your compile commands ought to resolve your issue.
Otherwise, the thing that your compiler does surely promise is that if you ask it to compile a source file to an object file, then on success, it will write a new object file. Moreover, although I would not necessarily have guessed that the same would be untrue of the .mod files, it is important to understand that the latter normally describe the interfaces to your modules, not their implementations, so your main rule is semantically incorrect:
polmob: polmob.f90 io.mod system.mod bands.mod parallel.mod
#echo [triggered by changes in $?]
mpifort -o polmob polmob.f90 io.f90 system.f90 bands.f90 parallel.f90
You need to rebuild if any of the implementations change, regardless of whether module interfaces change, and your rule does not adequately capture that requirement.
At this point I observe also that when you rebuild via that rule, you rebuild all the sources, making the per-source build rules pointless. If that's what you want, then a simpler solution would be to rewrite your while makefile to this:
SOURCES = polmob.f90 io.f90 system.f90 bands.f90 parallel.f90
polmob: $(SOURCES)
#echo [triggered by changes in $?]
mpifort -o $# $(SOURCES)
clean:
find . ! -name 'Makefile' -a ! -name '*.f90' -type f -exec rm -f {} +
There, the target is actually built from the specified prerequisites, and the build recipe is sufficient to build the target from those prerequisites. However, if any of the sources change, it will rebuild them all (similar to what your original makefile does).
The alternative is to build individual objects and then link them together in a separate step. Your original makefile seems to lean in that direction, but then it throws it away with the main rule. If you want to take that approach, then it becomes complicated by the facts that
it really is the module interfaces on which the individual objects depend (in addition to their own sources), so expressing prerequisites in terms of other object files is incorrect in these cases;
both the .o and .mod files are built by the same process (i.e. it has multiple outputs);
and apparently, that process uses different logic than make does to determine whether the .mod files are out of date.
The multiple outputs issue is probably the hairiest one; you can find a discussion and alternative solutions of varying rigor in the Automake documentation (but not specific to Automake).
Here's a way you could approach that, inspired by the Automake docs but accounting for the peculiarity you discovered in your Fortran implementation:
# All the object files contributing to the program:
OBJECTS = bands.o const.o io.o parallel.o polmob.o system.o
# Link all the objects together to form the final program
# (.mod files are typically not needed for this step)
polmob: $(OBJECTS)
mpifort -o $# $(OBJECTS)
# Rules for building the objects
bands.o : bands.f90 const.mod io.mod parallel.mod system.mod
mpifort -c bands.f90 -o $#
const.o : const.f90
mpifort -c const.f90 -o $#
io.o : io.f90 const.mod system.mod
mpifort -c io.f90 -o $#
parallel.o : parallel.f90
mpifort -c parallel.f90 -o $#
polmob.o : polmob.f90 bands.mod const.mod io.mod parallel.mod system.mod
mpifort -c polmob.f90 -o $#
system.o : system.f90 const.mod
mpifort -c system.f90 -o $#
# Rules for ensuring that .mod files are (re)created when needed, and
# that their timestamps do not fall behind those of their corresponding
# sources
bands.mod : bands.o
#if test -f $#; then touch $#; else \
rm -f bands.o; \
$(MAKE) bands.o; \
fi
const.mod : const.o
#if test -f $#; then touch $#; else \
rm -f const.o; \
$(MAKE) const.o; \
fi
io.mod : io.o
#if test -f $#; then touch $#; else \
rm -f io.o; \
$(MAKE) io.o; \
fi
parallel.mod : parallel.o
#if test -f $#; then touch $#; else \
rm -f parallel.o; \
$(MAKE) parallel.o; \
fi
system.mod : system.o
#if test -f $#; then touch $#; else \
rm -f system.o; \
$(MAKE) system.o; \
fi
####
clean:
find . ! -name 'Makefile' -a ! -name '*.f90' -type f -exec rm -f {} +
This exhibits the correct dependencies:
The main program depends (only) on all the objects.
Each object depends on its own source and on the .mod files for those modules it uses.
It also accounts for the fact that the same compilation process generates both an object file and (where appropriate) a corresponding .mod file, and it takes care of updating .mod timestamps where that is needed to satisfy make. It may occasionally cause files to be rebuilt when they didn't really need to be, because it will thwart any attempt by the compiler to avoid updating .mod files' timestamps. But that should be on a one-time basis, not every subsequent build.
I found the cause: If I replace "mod" by "o" everywhere in my Makefile, the problem disappears. However, I don't quite understand why.
Related
I'm creating an OS and when I compile the code nothing happened, simply nothing(No errors, warnings or anything) I"m thinking that the make file has some issues.
Makefile:
build_kernel:
echo "Building kernel..."
${ASM} ./src/kernel/kernel_entry.asm -f elf64 -o ${BUILD_DIR}/kernel_entry.o
${C_COMPILER} -c ./src/kernel/kernel.c -o ${BUILD_DIR}/kernel_start.o
${C_COMPILER} -c ./src/kernel/drivers/printutils.c -o ${BUILD_DIR}/kernel_printutils.o
${C_COMPILER} -c ./src/kernel/drivers/port.c -o ${BUILD_DIR}/kernel_ports.o
echo "kernel build complete."
link:
echo "Linking..."
${LINKER} -o ${BUILD_DIR}/kernel.bin ${BUILD_DIR}/kernel_ports.o \
${BUILD_DIR}/kernel_printutils.o ${BUILD_DIR}/kernel_start.o \
${BUILD_DIR}/kernel_entry.o -Ttext 0x1000 --oformat binary
echo "Linking complete"
run:
echo "Running qemu..."
qemu-system-x86_64 -fda ${BUILD_DIR}/os.bin
merge_binary:
echo "Merging binary..."
cat ${BUILD_DIR}/boot.bin ${BUILD_DIR}/kernel.bin > ${BUILD_DIR}/os.bin
echo "Binary merged."
post_build:
rm -f ${BUILD_DIR}/boot.bin
rm -f ${BUILD_DIR}/kernel.bin
rm -f ${BUILD_DIR}/kernel.o
rm -f ${BUILD_DIR}/kernel_entry.o
rm -f ${BUILD_DIR}/kernel_ports.o
rm -f ${BUILD_DIR}/kernel_printutils.o
rm -f ${BUILD_DIR}/kernel_start.o
I'm wondering what is happening with the makefile and is this the correct way to compile all the code into object files and link them.
Any help will be appreciated.
I'm just commenting on your make style -- this does not answer why you are outputting nothing (and it is unclear what you mean by that -- if you run make, it should output at least echo "Building kernel..."...). As far as the makefile style goes, this seems to be built using a scripting mentality rather than a make mentality. Consider your first part:
build_kernel:
echo "Building kernel..."
${ASM} ./src/kernel/kernel_entry.asm -f elf64 -o ${BUILD_DIR}/kernel_entry.o
${C_COMPILER} -c ./src/kernel/kernel.c -o ${BUILD_DIR}/kernel_start.o
${C_COMPILER} -c ./src/kernel/drivers/printutils.c -o ${BUILD_DIR}/kernel_printutils.o
${C_COMPILER} -c ./src/kernel/drivers/port.c -o ${BUILD_DIR}/kernel_ports.o
echo "kernel build complete."
This has several issues. First is the name -- this looks to build a bunch of artifacts rather than building the kernel. Also, the recipe never produces a file named build_kernel, thus this should have been a phony target. Next, this is actually a script, which builds four separate things. These could be separated out into four separate rules, each which builds one thing, and then the main target would be dependent on this. Thus, it might look like:
.PHONY: build_kernel_objs
build_kernel_objs: ${C_OBJS} ${ASM_OBJS}
#echo done building $#
${BUILD_DIR}/kernel_start.o : ./src/kernel/kernel.c
${C_COMPILER} -c $< -o $#
${BUILD_DIR}/kernel_printutils.o : ./src/kernel/kernel_printutils.c
${C_COMPILER} -c $< -o $#
${BUILD_DIR}/kernel_ports.o : ./src/kernel/kernel_ports.c
${C_COMPILER} -c $< -o $#
Note that the above is repetitive, and if you have hundreds of files, will bolat very quickly. This can also be done using static pattern rules:
C_FILES := \
./src/kernel/kernel_start.c
./src/kernel/kernel_printutils.c
./src/kernel/kernel_ports.c
ASM_FILES := \
./src/kernel/kernel_entry.asm
C_OBJS := ${C_FILES :./src/kernel/%.c=${BUILD_DIR}/%.o}
ASM_OBJS := ${ASM_FILES :./src/kernel/%.asm=${BUILD_DIR}/%.o}
${C_OBJS} : ${BUILD_DIR}/%.o : ./src/kernel/%.c
${C_COMPILER} -c $< $#
.PHONY: build_kernel_objs
build_kernel_objs: ${C_OBJS} ${ASM_OBJS}
#echo "done building $#"
These have several advantages over what you've done -- first, make will only ever build the objects that are out of date, so it doesn't do needless work. It can also build the files in parallel if a -j option is specified on the make command line. Next, it's more maintainable -- if you have to add extra files, you can do it in one place, and everything works out. Also, the .PHONY prevents the make from failing if you happen to have a file named build_kernel_objs in your make directory. Lastly, the # in front of the echo lines prevents the actual echo command from being echoed, which will look nicer.
On caveat to this is that it does not handle modification of header files (as written, if a header file is updated, c files that depend on it would not be rebuilt. See here for some notes about getting around that.
The next section, link, the makefile recipe should reflect the target.
.PHONY: link
link : ${BUILD_DIR}/kernel.bin
${BUILD_DIR}/kernel.bin: ${C_OBJS} ${ASM_OBJS}
${LINKER} -o $# $^ -Ttext 0x1000 --oformat binary
This creates a phony target link, so you can type make link. It will only do the link if any of the C objects or ASM objects have been updated. The same concept applies to your merge_binary target
For run, this seems to be somewhat contentious, but a common rule of thumb is that a make should be used to make an executable, not to run it. A separate shell script is better suited if you want to invoke your built target with specific parameters.
Lastly, your post_build rule should likely be renamed to CLEAN, and declared as a phony.
I was given a makefile that looks like this, and told not to change it.
all: clean flex scanner.lex bison -d parser.ypp g++ -std=c++11 -o hw2 *.c *.cpp clean: rm -f lex.yy.c rm -f parser.tab.*pp rm -f hw2
I am trying to run this makefile in a folder with files named: scanner.lex, parser.ypp, output.hpp and output.cpp
I copied it to a file like this:
all:
clean flex scanner.lex bison -d parser.ypp g++ -std=c++11 -o hw2 *.c *.cpp
clean:
rm -f lex.yy.c rm -f parser.tab.*pp rm -f hw2
When I run the make command in my terminal I get an error:
clean flex scanner.lex bison -d parser.ypp g++ -std=c++11 -o hw2 *.c *.cpp
/bin/sh: clean: command not found
make: *** [all] Error 127
Am I doing something wrong? Again, I was given this line and told not to change it.
Thanks a lot.
Line breaks are essential in most computer environments. If you were given a Makefile without the line breaks and you try to cut it randomly you will have difficulties before if finally works. Try this, maybe:
all: clean
flex scanner.lex
bison -d parser.ypp
g++ -std=c++11 -o hw2 *.c *.cpp
clean:
rm -f lex.yy.c
rm -f parser.tab.*pp
rm -f hw2
And use tabs to indent the indented lines, not spaces.
Explanations: all and clean are what is called a target in make parlance. They are the names of the things you want make to do. clean to delete some files, all to do everything else. The
target: prerequisite1 prerequisite2...
recipe1
recipe2
...
template is the basic make template. It means that target depends on prerequisite1, prerequisite2 and that in order to build it make shall pass recipe1 to the shell for execution, then recipe2...
Note that this Makefile is poorly written. As all and clean are not real file names they should be declared as phony, such that, if a file with that name exists make does the job anyway. As is, it wouldn't. Give it a try:
$ make all
$ touch clean
$ make clean
make: 'clean' is up to date.
See? Because a file named clean exists you cannot make clean anymore, make considers that there is nothing to do for clean. Add this at the beginning of your Makefile:
.PHONY: all clean
A second issue is that make works by comparing last modification times of targets and prerequisites to decide if targets must be rebuilt or not. With your Makefile make will always recompile everything, even if the inputs did not change and the outputs are up-to-date. This is a waste. A better (but untested) Makefile would be something like:
.PHONY: all clean
CFILES := $(filter-out lex.yy.c,$(wildcard *.c))
CPPFILES := $(filter-out parser.tab.cpp,$(wildcard *.cpp))
all: hw2
hw2: lex.yy.c parser.tab.cpp $(CFILES) $(CPPFILES)
g++ -std=c++11 -o $# $^
lex.yy.c: scanner.lex
flex $<
parser.tab.cpp: parser.ypp
bison -d $<
clean:
rm -f lex.yy.c
rm -f parser.tab.*pp
rm -f hw2
Understanding it and why it is better is left as an exercise.
I would like to have all folder's content in dependency. My folder is created when the rule is called.
create-content:
mkdir -p sample_dir/subdir1
touch sample_dir/file1
touch sample_dir/subdir1/file2
list-content: create-content $(shell find sample_dir)
echo $^
I would like this output:
make list-content
create-content sample_dir sample_dir/file1 sample_dir/subdir1 sample_dir/subdir1/file2
But I have:
make list-content
create-content
The problem is that the shell function is called on make and not when the rule is running.
So shell is called before create-content, so the directory doesn't exist yet.
Is it possible to solve this?
Thank you.
The answer is no... and yes.
No, you can't because make builds all its dependency graph before it runs the first recipe. So, if a recipe creates new nodes of the dependency graph, it is too late for make to notice.
But there is a simple workaround: tell make to invoke make (recursive make). Despite what many people say, recursive make is not always harmful. It is even sometimes the only solution to a specific problem. This is your case. And with a bit of make magic (conditionals) you can hide this with two different behaviors depending on the existence of sample_dir:
create-content:
#mkdir -p sample_dir/subdir1 && \
touch sample_dir/file1 && \
touch sample_dir/subdir1/file2
ifeq ($(wildcard sample_dir),)
list-content: create-content
#$(MAKE) --no-print-directory $#
else
list-content: create-content $(shell find sample_dir)
#echo $^
endif
Now, the prerequisites of list-content are complete. Demo:
$ rm -rf sample_dir
$ make list-content
create-content sample_dir sample_dir/file1 sample_dir/subdir1 sample_dir/subdir1/file2
$ make list-content
create-content sample_dir sample_dir/file1 sample_dir/subdir1 sample_dir/subdir1/file2
I have this rule in my Makefile, that responds to flags I pass:
$(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/disable_$*
rm -f $(BUILD_DIR)/enable_$*
cd $(BUILD_DIR) && rm -f Makefile
$(BUILD_DIR)/enable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/enable_$*
rm -f $(BUILD_DIR)/disable_$*
cd $(BUILD_DIR) && rm -f Makefile
What this means is that when changing the flags by which I invoke the makefile, I can trigger some recompilations that could depend on these flags.
The code presented above is a bit redundant: you see that I remove a file, touch another and remove a Makefile in both cases. The only thing that changes is the name of the files that I touch/remove, and they are related.
For instance,
make clean
make enable_debug=yes enable_video=no # will compile from zero
make enable_debug=no enable_video=no # flag change detected -> recompile some submodules that depend on this flag
Provided that the only thing that changes between the two rules ( [en|dis]able ), what I would like is to only have 1 generic rule, something like that:
# match 2 parts in the rule
$(BUILD_DIR)/%ble_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/(???)ble_$* # should be $#
rm -f $(BUILD_DIR)/(???)able_$* # should be disable if $# is enable and inverse
cd $(BUILD_DIR) && rm -f Makefile
Is this possible ?
PS: Sorry if I didn't get the title correctly, I couldn't figure how to explain it better.
$(BUILD_DIR)/enable_% $(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
rm -f $(BUILD_DIR)/*able_$*
touch $#
cd $(BUILD_DIR) && rm -f Makefile
Not literally what you wanted (multi-wildcards are forbidden in make), but does quite the same.
I have several widgets denoted by a config.xml in their root in a directory layout.
The GNUmakefile I have here is able to build them. Though if I update the folders, the dependencies aren't tracked. I don't want to depend on a clean target obviously, so how do I track the contents of each folder?
WGTS := $(shell find -name 'config.xml' | while read wgtdir; do echo `dirname $$wgtdir`.wgt; done )
all: $(WGTS)
%.wgt:
#cd $* && zip -q -r ../$(shell basename $*).wgt .
#echo Created $#
clean:
rm -f $(WGTS)
I hoped something like:
%.wgt: $(shell find $* -type f)
Would work, but it doesn't. Help.
Combining Beta's idea with mine:
WGTS := $(shell find -name config.xml)
WGTS := $(WGTS:/config.xml=.wgt)
WGTS_d := $(WGTS:.wgt=.wgt.d)
all: $(WGTS)
clean:
rm -f $(WGTS) $(WGTS_d)
-include $(WGTS_d)
define WGT_RULE
$(1): $(shell find $(1:.wgt=))
$(1:.wgt=)/%:
#
endef
$(foreach targ,$(WGTS),$(eval $(call WGT_RULE,$(targ))))
%.wgt:
#echo Creating $#
#(echo -n "$#: "; find $* -type f | tr '\n' ' ') > $#.d
#cd $* && zip -q -r ../$(shell basename $*).wgt .
Example:
$ mkdir -p foo bar/nested
$ touch {foo,bar/nested}/config.xml
$ make
Creating bar/nested.wgt
Creating foo.wgt
$ make
make: Nothing to be done for `all'.
$ touch foo/a
$ make
Creating foo.wgt
$ rm foo/a
$ make
Creating foo.wgt
$ make
make: Nothing to be done for `all'.
The only potential problem here is the dummy rule that lets make ignore targets it doesn't know how to build which are nested inside the directories. (foo/a in my example.) If those are real targets that make needs to know how to build, the duplicate recipe definition may be a problem.
Probably the best way to do this is to create the prerequisite lists explicitly, beforehand:
define WGT_RULE
$(1).wgt: $(wildcard $(1)/*)
endef
$(foreach targ,$(WGTS),$(eval $(call WGT_RULE,$(targ))))
There is another way that's very clever (a phrase that makes a good programmer wary). Years ago I came up with a left-handed kludge for treating a directory as a prerequisite. I'll see if I can dig up my old notebooks if the above isn't good enough.
EDIT:
Sorry, I didn't consider subdirectories. Here's a complete makefile (I left out the clean rule) that should do the trick.
WGTS := $(shell find -name 'config.xml' | while read wgtdir; do echo `dirname $\
$wgtdir`.wgt; done )
all: $(WGTS)
# This constructs a rule without commands ("foo.wgt: foo/bar.txt foo/baz.dat...").
define WGT_RULE
$(1).wgt: $(shell find $(1))
endef
# This invokes the above to create a rule for each widget.
$(foreach targ,$(WGTS),$(eval $(call WGT_RULE,$(targ))))
%.wgt:
#cd $* && zip -q -r ../$(shell basename $*).wgt .
#echo Created $#