I wanted to keep a log of previously compiled code so I can be consistent when setting the compiling options. I devised the following scheme but it has a weird behavior that I don't quite fully understand. Here is a simplified version that reproduces the error:
Makefile:
.DEFAULT_GOAL = final
MKFILES = Makefile Makefile.checks
include Makefile.checks
.PHONY: final
final: info.txt results.txt
#cat $^
results.txt: $(MKFILES) | .PRELIM
#echo 'First file with opt=$(opt)' > $#
info.txt: info.src $(MKFILES) | .PRELIM
#cat $< > $#
#echo 'Internal record of opt=$(opt)' >> $#
.PHONY: .FORCE
.FORCE:
.PHONY: .PRELIM
.PRELIM: Makefile.log
.PHONY: clean
clean:
rm -rf *.txt *.log
Makefile.checks:
# SETUPS & CHECKS
opt ?= 0
BUILDSTAT := 0
ifneq ($(MAKECMDGOALS),clean)
ifneq ("$(wildcard Makefile.log)","")
include Makefile.log
ifneq ($(opt),$(oldopt))
$(warning Previously compiled opt = $(oldopt))
$(warning Currently requested opt = $(opt))
BUILDSTAT := 1
endif
endif
endif
# RESOLUTION SECTION
ifneq ($(BUILDSTAT),0)
info.txt: .FORCE
Makefile.log: .FORCE
$(warning The build is not consistent, a clean make is recomended. )
ifneq ($(MAKECMDGOALS),anyway)
$(warning To force compilation, run 'make anyway opt=X' )
$(error Aborting compilation)
endif
endif
.PHONY: anyway
anyway: $(.DEFAULT_GOAL)
Makefile.log:
#echo 'Creating new Makefile.log'
#echo '########################################' > Makefile.log
#echo 'oldopt := $(opt)' >> Makefile.log
#echo 'BUILDSTAT := $(BUILDSTAT)' >> Makefile.log
#echo '########################################' >> Makefile.log
The Makefiles create the log and check for the previous options: make followed by make opt=1 behaves as expected by issuing the warning and stopping. However, when I make anyway opt=1 the makefile seems to parse the Makefile.checks many times before actually compiling:
Makefile.checks:12: Previously compiled opt = 0
Makefile.checks:13: Currently requested opt = 1
Makefile.checks:26: The build is not consistent, a clean make is recomended.
Creating new Makefile.log
Makefile.checks:26: The build is not consistent, a clean make is recomended.
Creating new Makefile.log
(...repeated many times...)
Makefile.checks:26: The build is not consistent, a clean make is recomended.
Creating new Makefile.log
This is a personal log
Internal record of opt=1
First file with opt=0
I suspect this may have something to do with how Makefile re-parses itself after modifying the Makefile.log that is itself included in one of the MKFILES that causes the whole list of targets to recompile. I tried using the -d option but it is not helping me much. This design ended up being quite convoluted but I didn't know how to simplify it, and I don't know enough about how makefile works to decipher what is happening on my own either.
I would be really thankful for an explanation of this error. Which dependency is causing the cycle and why it eventually leaves it and moves on to finish compilation? Do you think there is a better way to record compilation options?
EDIT: I will be adding here further requested information/explanations. I modified a bit the code to simplify it.
What I'm trying to do is to store the compilation options used in the last build so as not to be mixing options when recompiling only part of a code (or at least to be warned that this is what I'm doing). The way I implemented this is in the Makefile.checks: options are written into / loaded from a Makefile.log and BUILDSTAT is a kind of boolean to check if there were discrepancies or not and keep track of it in the successive builds.
Makefile.checks first sets defaults, then checks if the log exist and loads the "old" options in separated variables and then compares the two to check for discrepancies. If any is found, it issues a warning here and then later checks again and stops (this separation makes more sense when you have more than one option) unless explicitly asked to ignore them by using the target anyway. In this last case, both the log and the file info.txt (which is another way to store the options) need to be updated with the last build info (and specially the BUILDSTAT=1 to keep record that the build may have a "mixed compilation").
EDIT - "SOLUTION": Apparently if instead of "forcing" the recompilation of the log, I make the log itself a phony target (inside the if), it prevents this behavior. I still have found no drawbacks to this approach, but it does feel a bit "dirty".
Related
I have inherited a large branched project? that requires a volatile set of .a archives $(LIB_FILES) to be included into link target, located in some directories $(LIB_DIRS). I can write an expression like this:
LIBDEP = $(foreach ldir, $(LIB_DIRS), \
$(filter $(addprefix %/, $(LIB_FILES)), $(wildcard $(ldir)/* )))
The problem is that they might not exist at moment of make's invocation and would be built by invoking $(MAKE) inside of another target's rule, which is a prerequisite to the link step.
The problem is actual list of files that should be created varies on external factors determined at their build steps, that I can't hard-code it properly, without turning makefile into a spaghetti mess and said variable is not re-evaluated at the moment of link command invocation.
I have suspicion that $(eval ) function can be used somehow, but manual is not very forthcoming as well as I didn't found examples of its use in this way.
Toolchain: GCC and binutils, make 3.81
Another solution is to create an explicit dependency of your make script on the output of the step which currently creates the variable $(LIB_FILES). This is what the manual is dealing with in the chapter How makefiles are remade and it aims at the technique which make is best at, namely deriving dependencies from the existence and timestamp of files (instead of variables). The following hopefully depicts your situation with the process of deducing a new set of libraries simulated by the two variables $(LIBS_THIS_TIME) and $(LIB_CONFIG_SET).
LIBS_THIS_TIME = foo.a:baz.a:bar.a
LIB_CONFIG_SET = $(subst :,_,$(LIBS_THIS_TIME))
include libdeps.d
linkstep:
#echo I am linking $^ now
touch $#
libdeps.d: $(LIB_CONFIG_SET)
-rm libdeps.d
$(foreach lib,$(subst :, ,$(LIBS_THIS_TIME)),echo linkstep: $(lib) >> libdeps.d;)
$(LIB_CONFIG_SET):
touch $#
If make finds that libdeps.d is not up to date to your current library configuration it is remade before make executes any other rule, although it is not the first target in the makefile. This way, if your build process creates a new or different set of libraries, libdeps.d would be remade first and only then make would carry on with the other targets in your top makefile, now with the correct dependecy information.
It sometimes happens that you need to invoke make several times in succession. One possibility to do this is to use conditionals:
ifeq ($(STEP),)
all:
<do-first-step>
$(MAKE) STEP=2 $#
else ifeq ($(STEP),2)
all:
<do-second-step>
$(MAKE) STEP=3 $#
else ifeq ($(STEP),3)
all:
<do-third-step>
endif
In each step you can generate new files and have them existing for the next step.
Is there a way to let make determine the number of files to be recompiled before actually compiling? The problem is this: Consider having a quite big project with hundreds of source files. It would very convenient to have a rough idea of how long compilation will take, but to know that, one needs to know the number of files to be compiled.
The general answer is no, because your build could generate files which themselves are inputs to other rules which generate more files. And so on. However if a rough answer is good enough you can try the --dry-run flag. From the GNU make documentation...
“No-op”. Causes make to print the recipes that are needed to make the targets up to date, but not actually execute them. Note that some recipes are still executed, even with this flag (see How the MAKE Variable Works). Also any recipes needed to update included makefiles are still executed (see How Makefiles Are Remade).
As you can see, despite its name even the --dry-run flag will change the state of your build.
"make -n" will do the dry run. But you can't get the list of files to be rebuilt. May be you can write shell script to store the last modified time of files and get the list of files.
I think a found a decent solution for unix. Here SRC are your source files, HDR your headers and DEP the dependency files (something like DEP:=$(OBJ:.o=.d) )
isInDepFile+=$(shell grep -q $(modifiedFile) $(depFile) 1>&2 2> /dev/null && echo $(depFile))
COMPFILES=
checkDepFiles=$(foreach depFile,$(DEP), $(eval filesToCompile+=$(isInDepFile))) $(thinOutDepFiles)
thinOutDepFiles=$(foreach fileToCompile,$(filesToCompile),$(eval DEP=$(filter-out $(fileToCompile),$(DEP))))
countFilesToCompile: $(SRC) $(HDR)
$(eval modifiedFiles=$?)
$(foreach modifiedFile,$(modifiedFiles), $(call checkDepFiles))
$(eval numOfFilesToCompile = $(words $(filesToCompile)))
$(eval numDepFiles = $(words $(DEP)))
$(eval NumSRCFiles = $(words $(SRC)))
#echo $(NumSRCFiles) sources
#echo $(numDepFiles) files to leave
#echo $(numOfFilesToCompile) files to compile
#touch $#
This first generates a list of modified files within your source and header files lists. Then for each modified file it checks all dependency files for its filename. If a dependency file contains the current file we are looking at, it is added to the list of filesToCompile. It is also removed from the list of dependency files to avoid duplication.
This can be invoked in the main building rule of your project. The advantage of that over the dry run is that it gives you a simple number to work with.
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.
How can I properly write gmake rule for a header file which is generated by the make itself?
Suppose, that I can pass do make BUILDTYPE=1 and buildtype.h will be created and populated with
#define BUILDTYPE 1
Makefile will simply do something like this:
buildtype.h:
echo #define BUILDTYPE 1 > TMPFILE
//pseudo code:
if(TMPFILE != buildtype.h)
cat TMPFILE > buildtype.h
I need to ensure that this process won't be repeated 1000 times for each cpp file and I want to ensure that this process will be done at least once
What I want to ensure is that this rule runs always and only once. That is, even if buidtype.h exist it still has to be run. I have automatic dependency tracking and it should trigger this rule only once when make runs.
That is, if I run make BUILDTYPE=2 and there is nothing to do, it still has to run that rule for buildtype.h and if buildtype.h will updated by the rule it should recompile all files.
Is something like that possible with gmake?
I need to ensure that this process won't be repeated 1000 times for each cpp file
You shouldn't need to do anything special to ensure that. Make will keep track of the targets it has updated. It will not rerun the rule multiple times just because multiple other targets depend on its output.
and I want to ensure that this process will be done at least once
The canonical way to do that is:
.PHONY: force
buildtype.h: force
You didn't ask for it, but a simple way to implement
//pseudo code:
if(TMPFILE != buildtype.h)
cat TMPFILE > buildtype.h
is
cmp -s TMPFILE buildtype.h || cp TMPFILE buildtype.h
Update: A related "interesting problem" is how to ensure that buildtype.h is up to date before any compilation tries to use it. Automatic dependency tracking systems can fail here for "clean" builds, because their output is only based on what header files they can see on disk; If buildtype.h hasn't yet been created, makedepend or gcc -M cannot know about it, so cannot generate correct dependencies.
One solution for that is to carefully hand-code the right dependencies into the makefile, like
foo.o: buildtype.h # because foo.c includes buildtype.h
A more foolproof but hacky alternative is to write
Makefile: buildtype.h
which ensures that make will update buildtype.h before it does anything else (see the manual). So now buildtype.h will never be missing or out of date.
One disadvantage of that method is that even typing something like make clean will cause buildtype.h to be updated, even though it's not needed at all in that case. That can be mitigated for specific cases by really ugly hackery like
ifneq (clean,$(MAKECMDGOALS))
Makefile: buildtype.h
endif
Here is one way using sed:
deps =
ifdef BUILDTYPE
old = $(shell sed -n 's/\#define *BUILDTYPE *\([0-9]*\)/\1/p' buildtype.h)
ifneq ($(BUILDTYPE),$(old))
deps := buildtype.h
endif
endif
all: $(deps)
#echo $(deps)
Let's say you have a Makefile with two pseudo-targets, 'all' and 'debug'. The 'debug' target is meant to build the same project as 'all', except with some different compile switches (like -ggdb, for example). Since the targets use different compile switches, you obviously need to rebuild the entire project if you switch between the two. But GNUmake doesn't naturally recognize this.
So if you type make all you'll get
Building ...
...
Then if you type make debug, you get
make: Nothing to be done for `debug'.
So my question is: how do you implement a clean solution in the Makefile to notice that the last build used a different pseudo-target, or different compile switches, than the one you want currently? If they are different, the Makefile would rebuild everything.
Put the build products into different directory trees (whilst keeping one copy of the source of course). That way you are always just a short compile from an up-to-date build, be it debug or release (or even others). No possibility of confusion either.
EDIT
Sketch of the above.
src := 1.c 2.c 3.c
bare-objs := ${src:%.c=%.o}
release-objs := ${bare-objs:%=Release/%}
debug-objs := ${bare-objs:%=Debug/%}
Release/prog: ${release-objs}
Debug/prog: ${debug-objs}
${release-objs}: Release/%.o: %.c # You gotta lurve static pattern rules
gcc -c $< -o $#
${debug-objs}: Debug/%.o: %.c
gcc -c $< -o $#
Release/prog Debug/prog:
gcc $^ -o $#
.PHONY: all
all: Release/prog ; echo $# Success
.PHONY: debug
debug: Debug/prog ; echo $# Success
(Disclaimer: not tested, nor even run through make.)
There you go. It's even -j safe so you can do make -j5 all debug. There is a lot of obvious boiler plate just crying out for tidying up.
Keeping variant sets of object files (as in bobbogo's solution) is probably the best way, but if for some reason you don't want to do that, you can use empty files as markers, to indicate which way you last built the executable:
%-marker:
#rm -f $(OBJECTS) *-marker
#touch $#
debug: GCCFLAGS += -ggdb
debug: SOMEOTHERFLAG = WHATEVER
all debug: % : %-marker
#echo making $#
#$(MAKE) -S GCCFLAGS='$(GCCFLAGS)' SOMEOTHERFLAG='$(SOMEOTHERFLAG)' main
There are other variants on this idea; you could have a small file containing the flag settings, which the makefile would build and include. That would be clever, but not really any cleaner than this.
The only clean solution is to incorporate the difference into the target names.
E.g. you can define a variable $(DEBUG) and consistently use it in all targets that depend on the compile step.