I have this command from my bash script
find . -type f -name "*.c" -execdir bash -c "f={}; kos-cc $KOS_CFLAGS -c {} -o $PWD/\${f%.c}.o" \;
Its job it to recursively search the current directory ($PWD) for .c files, compile them with my "kos-cc" compiler then output all the .o files into the current working directory.
I want to move parts of my bash script into a makefile and this line is the last one that stumps me. I know how to make rules that compiles c files from directory A and outputs the .o files into directory B, but here directory A isn't always the same (Since I need to handle sub directories too). How do I do the equivalent task in a Makefile either with a rule or a command?
It is not trivial because you have your sources in different directories and simple make pattern rules cannot easily handle this. But using slightly more advanced make features can make it:
SRC := $(shell find . -type f -name "*.c")
OBJ := $(patsubst %.c,%.o,$(notdir $(SRC)))
.PHONY: all
all: $(OBJ)
define COMPILE_rule
$$(patsubst %.c,%.o,$$(notdir $(1))): $(1)
kos-cc $$(KOS_CFLAGS) -c -o $$# $$<
endef
$(foreach s,$(SRC),$(eval $(call COMPILE_rule,$(s))))
.PHONY: clean
clean:
rm -f $(OBJ)
Beware: you could have several source files with the same base name in different directories. And if this happens, you will end up with a fatal conflict of object file names...
EDIT (add conflicts detection):
The following detects conflicts situations and issues an error (replace error by warning or info depending on the severity level you assign to these conflicts):
SRC := $(shell find . -type f -name "*.c")
OBJ := $(patsubst %.c,%.o,$(notdir $(SRC)))
ifneq ($(words $(OBJ)),$(words $(sort $(OBJ))))
$(error object file name conflicts detected)
endif
(sort sorts and also removes duplicates and words returns the number of space-separated words in its string parameter).
Related
This is my first ever post to stackoverflow. It ought to be straightforward, but I have this problem whenever I try to write a makefile and I've never been able to figure out a satisfactory solution. Apologies if there is already a solution somewhere on the site. I couldn't find one.
What I'm trying to do is as follows:
Search my src directory for matching source files.
Compile the src code into a sandbox.
Here's my directory structure:
makefile
src1
file1.src
file2.src
src2
file3.src
subfolder
src3
file4.src
file5.src
And here's my makefile:
BUILDDIR := ./sandbox
SRC_DIRS := ./
SRCS := $(shell find $(SRC_DIRS) -name *.src)
OBJS := $(addprefix $(BUILDDIR)/o., $(notdir $(SRCS) ) )
# make print-X prints value of variable X
print-%: ; #echo $* = $($*)
.PHONY: help
help:
#echo "make <all|clean>"
.PHONY: all
all: $(OBJS)
#echo "compilation done"
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
$(BUILDDIR)/.create:
#echo "creating sandbox"
mkdir -p $(BUILDDIR) && cd $(BUILDDIR)
#touch $#
.PHONY: clean
clean:
#echo "deleting sandbox"
#rm -rf $(BUILDDIR)
If I type make all, the file works as expected. However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
And the process of creating objects of objects continues recursively as many times as I type make.
Any help would be appreciated.
Incidentally, please don't post solutions that rely on the build in compile functions of make. I'm looking for a general solution that can be used for any task. For example, in this instance, I'm trying to read the source files into a tool using its command line interface.
Well, first, by having your sandbox as a subdirectory of your source directory, then using find on the source directory, every time you run it you're going to find all the files in both the source directory and all its subdirectories, including the sandbox. If the built files in the sandbox have the same names as the files in the source directories, the find will find them all.
Maybe instead of:
SRCS := $(shell find $(SRC_DIRS) -name *.src)
you want something like:
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -o -name *.src -print)
Or, alternatively, don't make your sandbox a subdirectory of your source directory. Or, make sure that whatever name you give to the files in the sandbox directory won't match the *.src pattern you give to find.
But beyond that this is wrong:
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
Suppose SRCS is foo.src bar.src, which means OBJS is sandbox/o.foo.src sandbox/o.bar.src. Then the above expands to this:
sandbox/o.foo.src sandbox/o.bar.src : foo.src bar.src sandbox/.create
This is a common mistake; people seem to think that make will go through the targets and prerequisites and match them up, so the first target depends on the first prerequisite and the second target depends on the second one etc. but of course this cannot work correctly and that's not how make works. Make treats the above as if you'd written one rule for each target, with the same prerequisites; like this:
sandbox/o.foo.src : foo.src bar.src sandbox/.create
sandbox/o.bar.src : foo.src bar.src sandbox/.create
You can see this won't do what you want at all, since the $< will always be foo.src which is clearly wrong.
You need to write a single pattern rule that will build ONE target. Then make sure the pattern applies to all the targets.
You have made things hard for yourself by trying to "flatten" a directory structure of multiple source subdirectories, into a single level of target directory (by using the $(notdir $(SRCS))). Because of this, there's no pattern that will match the same target and directory, unless you write a separate rule for every subdirectory.
Luckily there is a solution for this: VPATH. This should work for you:
VPATH := $(sort $(dir $(SRCS))
$(BUILDDIR)/o.%.src : %.src $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
The VPATH tells make to go look in all the directories that it found any sources in, whenever it can't find one to build.
The basic problem is that your SRCS is all files in all subdirectories that match the pattern *.src (when you run make). That means that all your object files ($(OBJS)) also match, so they copied as well.
The solution is to change your SRCS pattern so it does not match the "object" files in the build directory. Possibilities:
SRCS := $(wildcard *.src)
or
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -false -o -name *.src)
or change the names of your "object" files so they don't end in .src
If I type make all, the file works as expected.
By which I take you to mean that directory ./sandbox exists and contains these files:
o.file1.src
o.file2.src
o.file3.src
o.file4.src
o.file5.src
However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
Of course you do, because everything is not up to date at that point, according to the target list you create. This line ...
SRCS := $(shell find $(SRC_DIRS) -name *.src)
... defines SRCS as a list of all the files under path ./ (which is the value of SRC_DIRS) that ends in .src. That includes any such files in ./sandbox, which include all the files placed there by the first make run. You generate a corresponding target file to build for each source, and those targets corresponding to sources built by the previous make run will not, in general, exist yet. So make builds them, just as you instructed it.
The best solution, short of abandoning that automatic source identification altogether, would probably be to change your naming scheme so that outputs cannot be mistaken so easily for sources. For example, if you want the output names to have the form foo.src, then have the corresponding input named something like foo.src.in. In that particular case, you could convert from source names to target names with
OBJS := $(addprefix $(BUILDDIR)/o., $(basename $(notdir $(SRCS) ) ) )
Alternatively, you could modify the find command to skip the sandbox directory, maybe by modifying SRC_DIRS:
SRC_DIRS = src1 src2 subfolder/src3
(These specific alternatives are not mutually exclusive.)
gcc -o abc/def.o def.c generates def.o file in a directory abc; only when there exists a directory abc.
Is there a way to make gcc to create a directory when the enclosing directory of the generated object file does not exist? If not, what could be the easiest way to make the directory in advance automatically, especially for Makefile?
From this post, it seems like that there is no way to create a directory from gcc.
For makefile, I can use this code snippet.
OBJDIR = obj
MODULES := src src2
...
OBJDIRS := $(patsubst %, $(OBJDIR)/%, $(MODULES))
build: $(OBJDIRS)
echo $^
$(OBJDIRS):
mkdir -p $#
make build will create directories, and echo the results.
We also can make the object files are created automatically without invoking the make build.
PROG := hellomake
LD := gcc
...
$(PROG): $(OBJDIRS) obj/src/hellofunc.o obj/src/hellomake.o
$(LD) $(filter %.o, $^) -o $(PROG)
The way I do it is I find out what the paths will be, and create the dirs for them:
SOURCES:=$(shell find $(SRC)/ -type f -name '*.c') #Get all the c files
#Get the files they're gonna be compiled to
OBJECTS:=$(patsubst $(SRC)/%.c, $(OBJ)/%.o, $(SOURCES))
#Get just the paths so you can create them in advance
OBJDIRS:=$(dir $(OBJECTS))
#call mkdir with the directories (using a dummy var because too much
#trouble to deal with priorities) someone could prob do better
#--parents ignores the repeated and already existing errors
DUMMY:=$(shell mkdir --parents $(OBJDIRS))
Sauce:
https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
In addition to precedent solutions, you can create the directories in your rule for building your .o files
$(OBJ_DIR)/%.o: $(SRCS_DIR)/%.c
#mkdir -p $(shell dirname $#)
$(CC) $(CFLAGS) $(INC) -c $< -o $#
The "#" before mkdir silent the output of mkdir (we don't want a message for each directory created).
The "-p" option tell mkdir to create all intermediate directories in path if they do not exist.
The "dirname" method takes the file path ("$#") and keep only the file directory path.
I have a project with sources in the src/ directory and its subdirectories (e.g. src/foo/ and src/bar/), and the objects in the obj directory and the matching subdirectories (e.g. obj/foo/ and obj/bar/).
I use the following (smimplified) Makefile:
SOURCES=$(shell find src/ -type f -name '*.c')
OBJECTS=$(patsubst src/%.c,obj/%.o,$(SOURCES))
all: $(OBJECTS)
obj/%.o: src/%.c
gcc -c $< -o $#
The problem
The problem is that if obj/ or one of its subdirectories doesn't exist, I get the following error:
Fatal error: can't create obj/foo/f1.o: No such file or directory
How can I tell make that %.o files depend on the creation of their containing directory?
What I tried
One solution when there are no subdirectories is to use "order only prerequisites":
$(OBJECTS): | obj
obj:
mkdir $#
But that fixes the problem only with obj/, but not obj/foo and obj/bar/. I thought about using $(#D), but I don't know how to get all this together.
I have also used hidden marker files in each directory, but that's just a hack, and I have also put a mkdir -p just before the GCC command but that also feels hacky. I'd rather avoid using recursive makefiles, if that were a potential solution.
Minimal example
To create a minimal project similar to mine you can run:
mkdir /tmp/makefile-test
cd /tmp/makefile-test
mkdir src/ src/foo/ src/bar/
echo "int main() { return 0; }" > src/main.c
touch src/foo/f1.c src/bar/b1.c src/bar/b2.c
I don't know why you consider adding mkdir -p before each compiler operation to be "hacky"; that's probably what I'd do. However, you can also do it like this if you don't mind all the directories created all the time:
First, you should use := for assigning shell variables, not =. The former is far more efficient. Second, once you have a list of filenames it's easy to compute the list of directories. Try this:
SOURCES := $(shell find src/ -type f -name '*.c')
OBJECTS := $(patsubst src/%.c,obj/%.o,$(SOURCES))
# Compute the obj directories
OBJDIRS := $(sort $(dir $(OBJECTS))
# Create all the obj directories
__dummy := $(shell mkdir -p $(OBJDIRS))
If you really want to have the directory created only when the object is about to be, then you'll have to use second expansion (not tested):
SOURCES := $(shell find src/ -type f -name '*.c')
OBJECTS := $(patsubst src/%.c,obj/%.o,$(SOURCES))
# Compute the obj directories
OBJDIRS := $(sort $(dir $(OBJECTS))
.SECONDEXPANSION:
obj/%.o : src/%.c | $$(#D)
$(CC) -c $< -o $#
$(OBJDIRS):
mkdir -p $#
I'd do it this way:
SOURCES=$(shell find src -type f -name '*.c') # corrected small error
...
obj/%.o: src/%.c
if [ ! -d $(dir $#) ]; then mkdir -p $(dir $#); fi
gcc -c $< -o $#
I dont know much makefile stuff I've been tending to learn bits as required.
The biggest failing of my makefiles is that I have been listing all the files manually, while this hasn't been a problem my current project is getting unwieldy. I have 4 directories each with sources files.
How can I get all the object file listing without having to list them manually.
This doesn't work, but it shows what I've been trying to do.
VPATH = Lib GameCode Moot/Moot Moot/Impl
OBJS = $(subst .cpp, .o, $(VPATH))
foobar: $(OBJS)
g++ -o $# $^
%.o: %.cpp
g++ -c $< -o $# -I Moot
clean:
rm main.o lib.o foo.o foobar
Personally, I never had any problem in listing all files manually. Listing a file to the makefile takes negligible time compared to adding filling it with useful content.
To get all files from different directories, one might suggest using wildcard function. So my_sources:=$(wildcard *.cpp dir1/*.cpp) will make the variable contain source files that match wildcard expression.
However, I find it less convenient than using usual Linux find command via shell:
# Find all sources
my_sources:=$(shell find -iname '*.cpp')
# Make targets out of them
OBJS=$(my_sources:%.cpp=%.o)
Find is more powerful than Make's builtin wildcard. You might also want to use other shell capabilities, such as pipelines, for example, to filter output of find (if Make's filter-out function is not enough). Or something like this, to avoid excessive variables:
OBJS:=$(shell find -iname '*.cpp' | sed 's/\.cpp$/.o/')
You name it!
Using VPATH or vpath will not work for your problem.. it provides a search path to find files but you still need to list the files. If you just need to compile all and any .c/.cpp files found in those directories then this should work:
foobar: $(shell ls Lib/*.cpp) $(shell ls GameCode/*.cpp) $(shell ls Moot/Moot/*.cpp) $(shell ls Moot/Impl/*cpp)
g++ -o $# $^
clean:
rm foobar $(shell ls Lib/*.o) $(shell ls GameCode/*.o) $(shell ls Moot/Moot/*.o) $(shell ls Moot/Impl/*o)
The VPATH info is not needed, the substitution of .o for .cpp can go as can the override of the implicit rule. Additionally, not the use of ls instead of find to look in, and only in, the specfified directory.
I have a "lib" directory in my applications main directory, which contains an arbitrary number of subdirectories, each having its own Makefile.
I would like to have a single Makefile in the main directory, that calls each subdirectory's Makefile. I know this is possible if I manually list the subdirs, but I would like to have it done automatically.
I was thinking of something like the following, but it obviously does not work. Note that I also have clean, test, etc. targets, so % is probably not a good idea at all.
LIBS=lib/*
all: $(LIBS)
%:
(cd $#; $(MAKE))
Any help is appreciated!
The following will work with GNU make:
LIBS=$(wildcard lib/*)
all: $(LIBS)
.PHONY: force
$(LIBS): force
cd $# && pwd
If there might be something other than directories in lib, you could alternatively use:
LIBS=$(shell find lib -type d)
To address the multiple targets issue, you can build special targets for each directory, then strip off the prefix for the sub-build:
LIBS=$(wildcard lib/*)
clean_LIBS=$(addprefix clean_,$(LIBS))
all: $(LIBS)
clean: $(clean_LIBS)
.PHONY: force
$(LIBS): force
echo make -C $#
$(clean_LIBS): force
echo make -C $(patsubst clean_%,%,$#) clean
There is also a way of listing sub-directories with gmake commands only, without using any shell commands:
test:
#echo $(filter %/, $(wildcard lib/*/))
This will list all sub-directories with trailing '/'. To remove it you can use the substitute pattern:
subdirs = $(filter %/, $(wildcard lib/*/))
test:
#echo $(subdirs:%/=%)
Then to actually create rules executing makefiles in each sub-directory you can use a small trick - a phony target in a non-existent directory. I think in this case an example will tell more than any explanation:
FULL_DIRS =$(filter %/, $(wildcard lib/*/))
LIB_DIRS =$(FULL_DIRS:%/=%)
DIRS_CMD =$(foreach subdir, $(LIB_DIRS), make-rule/$(subdir))
make-rule/%:
cd $* && $(MAKE)
all: DIRS_CMD
Basically, target 'all' lists all sub-directories as prerequisites. For example, if LIB_DIRS contained lib/folder1 lib/folder2 then the expansion would look like this:
all: make-rule/lib/folder1 make-rule/lib/folder2
Then 'make', in order to execute rule 'all', tries to match each prerequisite with an existing target. In this case the target is 'make-rule/%:', which uses '$*' to extract the string after 'make-rule/' and uses it as argument in the recipe. For example, the first prerequisite would be matched and expanded like this:
make-rule/lib/folder1:
cd lib/folder1 && $(MAKE)
What if you want to call different targets than all in an unknown number of subdirectories?
The following Makefile uses macros so create a forwarding dummy-target for a number of subdirectories to apply the given target from the command line to each of them:
# all direct directories of this dir. uses "-printf" to get rid of the "./"
DIRS=$(shell find . -maxdepth 1 -mindepth 1 -type d -not -name ".*" -printf '%P\n')
# "all" target is there by default, same logic as via the macro
all: $(DIRS)
$(DIRS):
$(MAKE) -C $#
.PHONY: $(DIRS)
# if explcit targets where given: use them in the macro down below. each target will be delivered to each subdirectory contained in $(DIRS).
EXTRA_TARGETS=$(MAKECMDGOALS)
define RECURSIVE_MAKE_WITH_TARGET
# create new variable, with the name of the target as prefix. it holds all
# subdirectories with the target as suffix
$(1)_DIRS=$$(addprefix $(1)_,$$(DIRS))
# create new target with the variable holding all the subdirectories+suffix as
# prerequisite
$(1): $$($1_DIRS)
# use list to create target to fullfill prerequisite. the rule is to call
# recursive make into the subdir with the target
$$($(1)_DIRS):
$$(MAKE) -C $$(patsubst $(1)_%,%,$$#) $(1)
# and make all targets .PHONY
.PHONY: $$($(1)_DIRS)
endef
# evaluate the macro for all given list of targets
$(foreach t,$(EXTRA_TARGETS),$(eval $(call RECURSIVE_MAKE_WITH_TARGET,$(t))))
Hope this helps. Really helpfull when dealing with paralelism: make -j12 clean all in a tree with makefiles having these targets... As always: playing with make is dangerous, different meta-levels of programming are too close together ,-)