Is it possible to serialize certain targets while parallelizing others in GNU make. I do not have an issue so unfortunately there is no code to share. But I can give an example :
Let's say I have a lot of .c source files that I need to compile and only a few assembly source files that need to be assembled. So, I need to speed up the compiling process by adding the -j8 option however for some reason I need the assembling process to be serial regardless whether I used the -j8 option or not. I tried to find a way to make .NOTPARALLEL work only on specific targets but I was unsuccessful and all the questions, posts, and articles talked about ordering the targets which is not what I am after. So, is it possible in either of the following forms? If it is possible, what is the proper way to do it?
c_sources := a.c b.c c.c
c_objs := a.o b.o c.o
#This process needs to be parallelized.
$(c_sources) : $(c_objs)
#echo 'Command used to compile .c files.'
assembly_sources := d.s e.s f.s
assembly_objs := d.o e.o f.o
#This process needs to be serialized.
$(assembly_sources) : $(assembly_objs)
#echo 'Command used to assemble assembly files (.s files).'
or
#This process needs to be parallelized.
%.c : %.o
#echo 'Command used to compile .c files.'
#This process needs to be serialized.
%.s : %.o
#echo 'Command used to assemble assembly files (.s files).'
There is no way to declare a serialization between two specific targets in GNU make, other than creating a dependency relationship between them. If target A lists target B as a prerequisite then B will be completely built before A.
It may be useful to investigate order-only dependencies.
Or you can use recursion as mentioned in the comment.
Those are your only options. Some other versions of make provide special targets that can be interspersed within a list of prerequisites to force "serialization points" but GNU make does not provide this facility.
Since you haven't provided us with a concrete example of a problem to solve we can't give you a more specific solution.
Related
I have this makefile in a directory with a set of .cpp, each one representing a single program with some header-only dependences. All files are in the same directory.
To compile for example the program a, I do make a. The make's implicit rules will compile successfully a from a.cpp, but make must also remake the target when their header-only dependencies change.
However, once a program compiles, the following makefile doesn't rebuild anything if I do, for example, touch utils.hpp and then retry compilation. What's going on?
src := $(shell find . -maxdepth 1 -name "*.cpp")
exe := $(src:.cpp=)
# In case I want to build every program, w/o a cmd-line target.
all: $(exe)
%: utils.hpp test.hpp
My make version is GNU Make 4.1.
Your last resort rule cannot be used to express the kind of dependencies you want. It is considered only if make needs to build something and knows no other rule to do it. Instead you should express this dependencies on your executable list:
$(exe): utils.hpp test.hpp
Im trying to run this simple makefile commands but get the error - 'Nothing to be done for 'all''
FILES = file1.c file2.c file3.c
all:test
test:
for file in $(FILES);
do
echo $$file;
done
The target test has no dependencies and therefore no reason to be built, which is inherited by the target all. It has instructions, but it should include FILES as its prerequisites. What you're doing appears to be ingredients-first, but test is the target. Working backwards is what make is best at. You may benefit from an article called "Auto-Dependency Generation" which takes the opposite approach (you appear to think like I do.)
test: $(FILES)
Then you could do something like the following:
$(FILES:.o:.c): %.o: %.c
$(CC) -c -o $# $<
The first part is a set of possible targets, the list of objects corresponding to the list of sources, and the second is a specific but nameless object (it will assume the name of the corresponding source.) Later on, the target, e.g. test, can be the name of your executable taking these objects as both dependencies and objects to link statically. For my purposes I typically use shared libraries but this is irrelevant to the question at hand.
Edit: untested, will revise if issues ensue
I have a 500+ line Makefile that is used to handle various automated builds. In its current state, it specifies a few rules, and depends on an argument being passed to make to determine which architecture to build for, ie:
all: release_build clean_all debug_build clean_all
So, on my build server, for example, I can generate tarballs of my project for multiple architectures like so:
make all ARCH=armhf
make all ARCH=x86
make all ARCH=x64
I'm attempting to parallelize builds to speed up how quickly the build server can give me the output of these builds, since CPU and IO-wise, it's underutilized. However, this leads to havok, as all the architectures use the same naming scheme for intermediate object files.
Is there a simple way to have my builds generate an intermediate folder like .obj/${RELEASE_TYPE}/ to store the intermediate files? This would let me run debug and release builds in parallel
Thank you.
This doesn't sound difficult. (I could have sworn there was already a question like this...)
OBJECTS := foo.o bar.o
OBJDIR := ./.obj/$(ARCH)
all: release_build debug_build
RELEASEOBJECTS := $(addprefix $(OBJDIR)/release/, $(OBJECTS))
release_build: $(RELEASEOBJECTS)
do something with $^ to produce $#
DEBUGOBJECTS := $(addprefix $(OBJDIR)/debug/, $(OBJECTS))
debug_build: $(DEBUGOBJECTS)
do something with $^ to produce $#
With some further trickery you could have one rule for both builds, but it would be quite cryptic.
I am using implicit rules only - removing the makefile altogether for a minimal test case. I have an empty (no problem for GNU assembler) program.s file. Executing:
make program
Gives me following output from make:
cc program.s -o program
(and of course the expected errors, which here is of no importance for the question: since my assembler source file is empty, there is no "_start" and all kinds of linking fails.)
I wonder, why does make choose to attempt to build the program in one go? As opposed to first using as program.s ... and then ld program.o ...? Is this because it considers the object file unnecessary in my scenario here?
If I do make program.o, then as program.as ... is invoked, producing my program.o as expected.
Make will always choose one-step implicit rules in preference to multi-step implicit rules, to get the same result. In this case, make contains built-in rules that create an executable both from an object file, but also directly from various source files including assembly:
%: %.s
# recipe to execute (built-in):
$(LINK.s) $^ $(LOADLIBES) $(LDLIBS) -o $#
Since this is shorter than first building the .o then building the executable from the .o, and since your makefile doesn't say you want the .o, make uses the shortest set of steps.
Is there a way how to ask gmake to never run two targets from a set in parallel?
I don't want to use .NOTPARALLEL, because it forces the whole Makefile to be run sequentially, not just the required part.
I could also add dependencies so that one depends on another, but then (apart from being ugly) I'd need to build all of them in order to build the last one, which isn't necessary.
The reason why I need this is that (only a) part of my Makefile invokes ghc --make, which takes care of its dependencies itself. And it's not possible to run it in parallel on two different targets, because if the two targets share some dependency, they can rewrite each other's .o file. (But ghc is fine with being called sequentially.)
Update: To give a specific example. Let's say I need to compile two programs in my Makefile:
prog1 depends on prog1.hs and mylib.hs;
prog2 depends on prog2.hs and mylib.hs.
Now if I invoke ghc --make prog1.hs, it checks its dependencies, compiles both prog1.hs and mylib.hs into their respective object and interface files, and links prog1. The same happens when I call ghc --make prog2.hs. So if they the two commands get to run in parallel, one will overwrite mylib.o of the other one, causing it to fail badly.
However, I need that neither prog1 depends on prog2 nor vice versa, because they should be compilable separately. (In reality they're very large with a lot of modules and requiring to compile them all slows development considerably.)
Hmmm, could do with a bit more information, so this is just a stab in the dark.
Make doesn't really support this, but you can sequential-ise two targets in a couple of ways. First off, a real use for recursive make:
targ1: ; recipe1...
targ2: ; recipe2...
both-targets:
${MAKE} targ1
${MAKE} targ2
So here you can just make -j both-targets and all is fine. Fragile though, because make -j targ1 targ2 still runs in parallel. You can use dependencies instead:
targ1: ; recipe1...
targ2: | targ1 ; recipe2...
Now make -j targ1 targ2 does what you want. Disadvantage? make targ2 will always try to build targ1 first (sequentially). This may (or may not) be a show-stopper for you.
EDIT
Another unsatisfactory strategy is to explicitly look at $MAKECMDGOALS, which lists the targets you specified on the command-line. Still a fragile solution as it is broken when someone uses dependencies inside the Makefile to get things built (a not unreasonable action).
Let's say your makefile contains two independent targets targ1 and targ2. Basically they remain independent until someone specifies on the command-line that they must both be built. In this particular case you break this independence. Consider this snippet:
$(and $(filter targ1,${MAKECMDGOALS)),$(filter targ2,${MAKECMDGOALS}),$(eval targ1: | targ2))
Urk! What's going on here?
Make evaluates the $(and)
It first has to expand $(filter targ1,${MAKECMDGOALS})
Iff targ1 was specified, it goes on to expand $(filter targ2,${MAKECMDGOALS})
Iff targ2 was also specified, it goes on to expand the $(eval), forcing the serialization of targ1 and targ2.
Note that the $(eval) expands to nothing (all its work was done as a side-effect), so that the original $(and) always expands to nothing at all, causing no syntax error.
Ugh!
[Now that I've typed that out, the considerably simpler prog2: | $(filter prog1,${MAKECMDGOALS})
occurs to me. Oh well.]
YMMV and all that.
I'm not familiar with ghc, but the correct solution would be to get the two runs of ghc to use different build folders, then they can happily run in parallel.
Since I got stuck at the same problem, here is another pointer in the direction that make does not provide the functionality you describe:
From the GNU Make Manual:
It is important to be careful when using parallel execution (the -j switch; see Parallel Execution) and archives. If multiple ar commands run at the same time on the same archive file, they will not know about each other and can corrupt the file.
Possibly a future version of make will provide a mechanism to circumvent this problem by serializing all recipes that operate on the same archive file. But for the time being, you must either write your makefiles to avoid this problem in some other way, or not use -j.
What you are attempting, and what I was attempting (using make to insert data in a SQLite3 database) suffers from the exact same problem.
I needed to separate the compilation from other steps (cleaning, building dirs and linking), as I wanted to run the compilation with more core processes and the -j flag.
I managed to solve this, with different makefiles including and calling each other. Only the "compile" make file is running in parallel with all the cores, the rest of the process is syncronous.
I divided my makefile in 3 separate scripts:
settings.mk: contains all the variables and flag definitions
makefile: has all the targets except the compilation one (It has .NOTPARALLEL directive). It calls compile.mk with -j flag
compile.mk: contains only the compile operation (without .NOTPARALLEL)
In settings.mk I have:
CC = g++
DB = gdb
RM = rm
MD = mkdir
CP = cp
MAKE = mingw32-make
BUILD = Debug
DEBUG = true
[... all other variables and flags needed, directories etc ...]
In makefile I have Link and compilation target as these:
include .makefiles/settings.mk
[... OTHER TARGETS (clean, directories etc)]
compilation:
#echo Compilation
#$(MAKE) -f .makefiles/compile.mk --silent -j 8 -Oline
#Link
$(TARGET): compilation
#echo -e Linking $(TARGET)
#$(CC) $(LNKFLAGS) -o $(TARGETDIR)/$(TARGET) $(OBJECTS) $(LIBDIRS) $(LIB)
#Non-File Targets
.PHONY: all prebuild release rebuild clean resources directories run debug
.NOTPARALLEL: all
# include dependency files (*.d) if available
-include $(DEPENDS)
And this is my compile.mk:
include .makefiles/settings.mk
#Defauilt
all: $(OBJECTS)
#Compile
$(BUILDDIR)/%.$(OBJEXT): $(SRCDIR)/%.$(SRCEXT)
#echo -e Compiling: $<
#$(MD) -p $(dir $#)
#$(CC) $(COMFLAGS) $(INCDIRS) -c $< -o $#
#Non-File Targets
.PHONY: all
# include dependency files (*.d) if available
-include $(DEPENDS)
Until now, it's working.
Note that I'm calling compile.mk with -j flag AND -Oline so that parallel processing doesn't mess up with the output.
Any syntax color can be setted in the makefile main script, since the -O flag invalidates escape color codes.
I hope it can help.
I had a similar problem so ended up solving it on the command line, like so:
make target1; make target2
to force it to do the targets sequentially.