Set a variable from within a rule - makefile

I have an external tool that fetches some sources (Rebar). I want to fill in a variable according to the contents of a directory after Rebar runs.
EFLAGS += -I$(PWD)/include
EFLAGS += -pa $(PWD)/ebin
## $(PWD)/deps/* will only have contents after Rebar runs
EFLAGS += $(patsubst %,-pa %,$(wildcard $(PWD)/deps/*/ebin))
build-deps:
./rebar get-deps
./rebar compile
build-main: build-deps
erlc $(EFLAGS) $(INFILE)
The above will work as intended if I run it as two separate invocations:
make build-deps
make build-main
However, if I just make build-main, then EFLAGS gets set while the deps/ directory is empty, then the directory is populated, and then I use EFLAGS.
Is there a good way for me to only set EFLAGS after I've run some rules?
EDIT: Here's a Makefile that may demonstrate the problem more easily:
A=$(wildcard test*)
foo:
touch test1
bar: foo
#echo $A
clean:
-rm test*
Here, the "foo" target is standing in for my call to rebar, so just imagine that you don't know which files I'm going to pass to touch. If you try
make clean
make bar
make bar
you will find that the two invocations of make bar produce different results, because in the second one test1 exists before make begins. I'm looking for a way to get the output of the second make bar invocation immediately after running make clean.

The simplest solution is to use the shell to compute the value, rather than using make rules. Like this:
EFLAGS += -I$(PWD)/include
EFLAGS += -pa $(PWD)/ebin
## $(PWD)/deps/* will only have contents after Rebar runs
EFLAGS += $(patsubst %,-pa %,$(wildcard $(PWD)/deps/*/ebin))
build-deps:
./rebar get-deps
./rebar compile
build-main: build-deps
for f in $(PWD)/deps/*/ebin; do paflags="$$paflags -pa $$f"; done; \
erlc $(EFLAGS) $$paflags $(INFILE)

I believe something like the following, using an included makefile, will work.
Untested but I believe something along these lines will do what you want. Assuming you want to run rebar every time you try to build.
EFLAGS += -I$(PWD)/include
EFLAGS += -pa $(PWD)/ebin
EFLAGS += $(PADIRS)
-include paflags.mk
build-main:
erlc $(EFLAGS) $(INFILE)
paflags.mk: force
./rebar get-deps
./rebar compile
echo 'PADIRS := $$(patsubst %,-pa %,$$(wildcard $$(PWD)/deps/*/ebin))' > '$#'
force: ;
Above edited to remove the .PHONY declaration on paflags.mk as that seemed to cause make to not perform the restart necessary for this trick to work.
Alternatively, since you aren't using any of make's prerequisite testing for this you could just move it all into the build-main rule body and do the globbing/etc. in the shell.
Alternatively alternatively, you could use eval to forcibly evaluate the patsubst in the build-main rule I believe (I'd have to test that to be sure the timing works out correctly with how, GNU make at least, hoists make directives in rule bodies).
This version of the idea above for the simple test case works for me:
-include inc.mk
$(warning A:$A)
bar:
#echo $A
inc.mk:
touch test1
echo 'A=$$(wildcard test*)' > '$#'
force: ;
Edited both sample makefiles to include a force rule on the included makefile to force make to build the included file every time. Without that (and without fancy automatic dependency generation/detection) make will only build the included file the first time and then never touch it again. I believe the force, at a cost of always trying to build it, will avoid that problem.
That being said, for this case, MadScientist's answer is probably the better choice.

Related

How can I add a directory to the search path of GNU Make?

I have a makefile that looks something like this:
include anotherFile.mk
all:
someStuff
The file anotherFile.mk is like this:
include yetAnotherFile.mk
export SOME_VAR = 93
The problem is that anotherFile.mk and yetAnotherFile.mk are in a different directory from my Makefile. So my makefile can't just be changed to this:
include $(OTHER_PROJECT_PATH)/anotherFile.mk
all:
someStuff
The problem with this approach is that the include statement in anotherFile.mk will fail because it will be searching in the current directory.
A partial solution that I found is to pass the --include-dir=$OTHER_PROJECT_PATH flag to the invocation of make, but that's a bit user-unfriendly.
So my question is: Is there something I can put inside my makefile that will add to the directories that make searches for when executing an include? Something like MAKE_INCLUDE_DIRS += $(OTHER_PROJECT_PATH)
Surprisingly there doesn't seem to be a good answer to that question. Forcing .INCLUDE_DIR doesn't help and there doesn't seem to be any way around invoking make with --include-dir=$OTHER_PROJECT_PATH.
It is however possible to put the appropriate recursive make invocation inside the makefile but, in order to get it to work for all reasonable cases it quickly becomes too complicated to be worth it. In summary it requires:
a top level condition to check if the OTHER_PROJECT_PATH is in .INCLUDE_DIR
the appropriate target with the recipe invoking make recursively
possibly additional targets if there are multiple command goals
the real make file enclosed in the else part of the conditional
You Makefile would look like this:
OTHER_PROJECT_PATH := other
ifeq (,$(filter $(OTHER_PROJECT_PATH), $(.INCLUDE_DIRS)))
# this is the mechanism to add the include dir in a recursive make
$(or $(firstword $(MAKECMDGOALS)),all):
$(MAKE) -I$(OTHER_PROJECT_PATH) $(MAKECMDGOALS)
# add empty targets for additional goals if needed
ifneq (,$(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS)))
$(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS)):
endif
else
# this is where the real makefile starts
all more:
echo $#: $< $^
include a.mak
endif
It still does not seem possible from a makefile, but if you have a script that sets up environment variables, you can use MAKEFLAGS (e.g. export MAKEFLAGS=I/your/path ordentlich on Linux, or SET on Windows)

Makefile: Declaring dependencies between included files

Using make's ''Remaking Makefiles'' feature I am generating parts of my makefile with include directives (see Makefile: defining rules and prerequisites in recipes). Now I'm stuck with being unable to see how I can express dependencies between included makefiles. They seem to be all evaluated at once.
Consider the following minimal makefile that illustrates my problem:
all:
-include foo.make
-include bar.make
foo.make: Makefile
echo FOO:=blub bla baz > foo.make
bar.make: Makefile foo.make
echo BAR:=$(FOO) > bar.make
If I now run make I will get:
$ cat foo.make
FOO:=blub bla baz
$ cat bar.make
BAR:=
Why? Since bar.make depends on foo.make, shouldn't the evaluation of bar.make wait until it successfully included foo.make?
And how do I fix this problem and make sure that bar.make is either re-evaluated later or only evaluated once foo.make exists, is included and can define the variable BAR?
The reason I cannot combine foo.make and bar.make into a single makefile and rule is two-fold:
Firstly, in my real setup, bar.make depends on more intermediate targets which in turn transitively depend on foo.make. So at the time foo.make can be created, the content of bar.make cannot yet be made.
Secondly, in my real setup, foo.make and bar.make do not just define variables but also eval() define/endef blocks. So I have to write:
-include makefile_with_prerequisite_variables
define MYDEF
sometarget-$1: $(TARGET_$1_PREREQUISITES)
[...]
endf
-include makefile_with_eval_call_statements
The content of makefile_with_prerequisite_variables and makefile_with_eval_call_statements cannot go into a single makefile snippet:
If I would put makefile_with_eval_call_statements above MYDEF together with makefile_with_prerequisite_variables then the $eval( $call( MYDEF)) statements in it would not work because MYDEF is only declared afterward.
If I would put makefile_with_prerequisite_variables below MYDEF together with makefile_with_eval_call_statements then the recipes defined in MYDEF would not have proper prerequisits because the $(TARGET_$1_PREREQUISITES) variables would then be declared afterward by makefile_with_prerequisite_variables.
In summary, I need to include two different makefiles where one depends upon the other. I do not know how I can express this relationship such that the content of one makefile would only be created after the other makefile is up-to-date and included into the main makefile.
First, your makefile creation in this simple example can easily be fixed by escaping the value of $(FOO) so that it's not expanded when bar.make is created but rather deferred until it's read in. So:
bar.make: Makefile foo.make
echo 'BAR:=$$(FOO)' > $#
However, that might not be sufficient in your more complex real-life makefiles.
GNU make works like this: first parse all the makefiles. Then for every included makefile, treat it as a goal and try to build it (e.g., act as if the user invoked make include1.mk include2.mk include3.mk ...). Then at the end of that, if any of the included makefiles was rebuilt, re-exec ourselves and start the entire process over from scratch.
GNU make does NOT work like this: parse makefiles, try to rebuild the first included makefile and if it's rebuilt, re-exec; if it's not rebuilt go on to the next included makefile, etc.
A simple trick you can use if you have to have this type of order is to put the include of bar.make into foo.make:
all:
-include foo.make
foo.make: Makefile
printf -- '-include bar.make' > $#
echo FOO:=blub bla baz >> $#
bar.make: Makefile foo.make
echo 'BAR:=$$(FOO)' > $#
By doing this you ensure that if foo.make doesn't exist, make can't see the include of bar.make and so it won't try to build it. Only after the first re-exec will make see the include of bar.make and try to build it.
One thing: if you get the latest version of GNU make you no longer need to use the -include trick. You can just use include even with generated makefiles.

How to force a certain groups of targets to be always run sequentially?

Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.

how to use recursive make with an option telling it not to travel down the tree if needed?

I setup make to build my tree using recursive make. So the setup is
A/Makefile a.c
A/B/Makefile a.c
A/B/C/Makefile a.c
where if I issue the command make all from level A/ then make will travel down the tree building everything and then come back up. Each Makefile contains a list of folders below it to build. There is a common.inc file in the root which is read in each Makefile.
This is just a standard layout for recursive make, and nothing new. The details is gives in many places. here and here are examples.
My question is this: many times I'd like to do make all but only build things in the current folder, and not actually travel down the tree, may be because I want to test some changes in the current folder at this time. So I end up editing the current folder's Makefile by commenting out the SUBDIRS=A B C which lists all folders below, or by adding new special targets for this folder only. Both are annoying things to have to keep doing.
Does any one have an idea or a small example of a recursive makefile that uses a switch to tell it if it should travel down the tree or not when called? may be there is a way to call make and pass it some flag at the command line, and this flag is used to remove SUBDIRS=A B C ..... list so it only stops at the current folder level?
Just to be clear. I am using standard SUBDIRS in the Rules.mk, which each Makefile in the tree includes. Here is the part. I copied this from the net long time ago
$(SUBDIRS)::
#if test -d $#; then \
set $(EXIT_ON_ERROR); \
echo "cd $#; make $#"; \
cd $#; make $#; \
set +e; \
else \
echo "Skipping non-directory $#..."; \
fi \
$(CLICK_STOPWATCH);
endif
and in each folder Makefile I write
SUBDIRS = A B C
include Rules.mk
all:: .......
Then I just write make all to build. If there is a way to do make all LOOP=0 where LOOP is some value I pass it or an option or a string or something and then change the above SUBDIRS logic to check for the value of this LOOP and based on the value then do the recursive make or not, then the problem is solved. The default can be to LOOP=1 if it is missing from the command line.
But I do not know enough Make to program this type of logic.
You should use power of rules' depencies. Add your sources files to the dependencies of the rule called from the "root Makefile". If these files are up to date, the recursivity in an folder will stop because the rule is 'up-to-date', and nothing will be done.
Don't add .PHONY for all your rules in the sub-directory Makefile, otherwise recursives rules will be called.
Play with the dependencies of the rules can be the key to not make recursive call, but if you modify sources in each folder and want to build only from the root Makfile, you have to create another rules. With make all, the make binary may not know if you want build all your projet or not (if all your sources has been modified).
EDIT: choice by the command line
You're near the answer, you can set env var while calling your make all and test the value to decide calling recursivly or not.
CC=g++
SUBDIR=a b
all: ${SUBDIR} main.cc
${CC} main.cc
${SUBDIR}:
ifneq ($(MK_LOOP), 0)
#echo "trust the recursivity !"
${MAKE} -C $#
endif
.PHONY: ${SUBDIR}
If you don't set the MK_LOOP var or you set to something else than 0, it will not be equal to 0 so recursive Makefile will be call; if you set to 0, $(SUBDIR) rule do nothing
42SH $ MK_LOOP=0 make # no recur
42SH $ make all
trust the recursivity !
42SH $ make all MK_LOOP=1 # recur by default; same as : make all
trust the recursivity !
42SH $

gmake rule for a header file which is created by the makefile

How can I properly write gmake rule for a header file which is generated by the make itself?
Suppose, that I can pass do make BUILDTYPE=1 and buildtype.h will be created and populated with
#define BUILDTYPE 1
Makefile will simply do something like this:
buildtype.h:
echo #define BUILDTYPE 1 > TMPFILE
//pseudo code:
if(TMPFILE != buildtype.h)
cat TMPFILE > buildtype.h
I need to ensure that this process won't be repeated 1000 times for each cpp file and I want to ensure that this process will be done at least once
What I want to ensure is that this rule runs always and only once. That is, even if buidtype.h exist it still has to be run. I have automatic dependency tracking and it should trigger this rule only once when make runs.
That is, if I run make BUILDTYPE=2 and there is nothing to do, it still has to run that rule for buildtype.h and if buildtype.h will updated by the rule it should recompile all files.
Is something like that possible with gmake?
I need to ensure that this process won't be repeated 1000 times for each cpp file
You shouldn't need to do anything special to ensure that. Make will keep track of the targets it has updated. It will not rerun the rule multiple times just because multiple other targets depend on its output.
and I want to ensure that this process will be done at least once
The canonical way to do that is:
.PHONY: force
buildtype.h: force
You didn't ask for it, but a simple way to implement
//pseudo code:
if(TMPFILE != buildtype.h)
cat TMPFILE > buildtype.h
is
cmp -s TMPFILE buildtype.h || cp TMPFILE buildtype.h
Update: A related "interesting problem" is how to ensure that buildtype.h is up to date before any compilation tries to use it. Automatic dependency tracking systems can fail here for "clean" builds, because their output is only based on what header files they can see on disk; If buildtype.h hasn't yet been created, makedepend or gcc -M cannot know about it, so cannot generate correct dependencies.
One solution for that is to carefully hand-code the right dependencies into the makefile, like
foo.o: buildtype.h # because foo.c includes buildtype.h
A more foolproof but hacky alternative is to write
Makefile: buildtype.h
which ensures that make will update buildtype.h before it does anything else (see the manual). So now buildtype.h will never be missing or out of date.
One disadvantage of that method is that even typing something like make clean will cause buildtype.h to be updated, even though it's not needed at all in that case. That can be mitigated for specific cases by really ugly hackery like
ifneq (clean,$(MAKECMDGOALS))
Makefile: buildtype.h
endif
Here is one way using sed:
deps =
ifdef BUILDTYPE
old = $(shell sed -n 's/\#define *BUILDTYPE *\([0-9]*\)/\1/p' buildtype.h)
ifneq ($(BUILDTYPE),$(old))
deps := buildtype.h
endif
endif
all: $(deps)
#echo $(deps)

Resources