This question is different from the one at makefiles - compile all c files at once in the sense that I have one extra requirement: I want to redirect all the object files in a separate directory.
Here is the setup:
I have multiple sources in a directory say src/mylib.
I want the objects files to end up in build/mylib.
Please note also that under mylib there are subdirectories.
The first attempt was as follows:
sources = $(shell find src/ -name ".c")
objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This is where things aren't working as expected
$(objects): build $(sources)
$(cc) $(cflags) -o $# $(word 2, $^))
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
For the makefile above, only one object file was being generated. I guessed this might have something to do with GCC only being able to generate one object file at a time. Regardless of that, checking the values of $# and $(word 2, $^) in the $(objects) target shows that only one file is being considered even though I have multiple files.
So I changed my makefile to the following:
sources = $(shell find src/ -name ".c")
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This works as expected but it appears to me like make is generating all the objects files even though source files did not change. This can be seen by checking the timestamps on new object files after running make again.
$(objects): build $(sources)
$(foreach source, $(sources), $(shell $(cc) $(cflags) -o $(subst src/,build/, $(patsubst %.o,%.c,$(source))) $(source)))
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
The second makefile works as expected but objects files are being rebuilt again which defeats another purpose of using make: only recompile those source files that changed from the last compilation.
Hence my question: how does one generate all object files in a separate directory at once (by this I mean perform the compilation of all sources files in one rule) while making sure that if a source file didn't change the associated object file should not be regenerated.
I am not after speeding up compilation. What I seek is one rule that will generate all objects files such that only updated source files should be recompiled.
The last makefile does the job but there is a recompiling of all source files which defeats another purpose of using make: only changed source files should be recompiled.
EDIT
After reading comments, it appears I have not phrased my question properly. As the details of what I have are already present, I leave the question as it is with additional details below.
The second makefile in the source code above does work. But it does only half the job. The build directory effectively mirrors the src directory.
So if I have say a file as src/mylib/point/point.c, I get build/mylib/point/point.o generated. This is the first part.
The second part is that if point.c does not changes, point.o in the build/mylib/point/ directory must not be regenerated. But after checking timestamps on the object file, I can tell that a new object file replaced the old one after running make again. This is not good because for large projects, compilation time remains O(n) with n being the number of source files to compile.
So this question is about how to preserve the second makefile without make regenerating object files.
From what I can gather from comments, I am asking too much from make. But if anyone knows how to make this happen, I leave the question open.
Makefile:
all:
clean:
src_root := src
src_subdirs := foo foo/bar foo/bar/buz
build_root := build
o_suffix := .o
# Build list of sources. Iterate every subfolder from $(src_subdirs) list
# and fetch all existing files with suffixes matching the list.
source_suffixes := .c .cpp .cxx
sources := $(foreach d,$(addprefix $(src_root)/,$(src_subdirs)),$(wildcard $(addprefix $d/*,$(source_suffixes))))
# If src_subdirs make variable is unset, use 'find' command to build list of sources.
# Note that we use the same list of suffixes but tweak them for use with 'find'
ifeq ($(src_subdirs),)
sources := $(shell find $(src_root) -type f $(foreach s,$(source_suffixes),$(if $(findstring $s,$(firstword $(source_suffixes))),,-o) -name '*$s'))
endif
$(info sources=$(sources))
# Build source -> object file mapping.
# We want map $(src_root) -> $(build_root) and copy directory structure
# of source tree but populated with object files.
objects := $(addsuffix $(o_suffix),$(basename $(patsubst $(src_root)%,$(build_root)%,$(sources))))
$(info objects=$(objects))
# Generate rules for every .o file to depend exactly on corresponding source file.
$(foreach s,$(sources),$(foreach o,$(filter %$(basename $(notdir $s)).o,$(objects)),$(info New rule: $o: $s)$(eval $o: $s)))
# This is how we compile sources:
# First check if directory for the target file exists.
# If it doesn't run 'mkdir' command.
$(objects): ; $(if $(wildcard $(#D)),,mkdir -p $(#D) &&) g++ -c $< -o $#
# Compile all sources.
all: $(objects)
clean: ; rm -rf $(build_root)
.PHONY: clean all
Environment:
$ find
.
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp
Run makefile:
$ make -f /cygdrive/c/stackoverflow/Makefile.sample -j
sources=src/foo/bar/bar.cxx src/foo/bar/buz/buz.c src/foo/bar/foo.c src/foo/foo.cpp
objects=build/foo/bar/bar.o build/foo/bar/buz/buz.o build/foo/bar/foo.o build/foo/foo.o
New rule: build/foo/bar/bar.o: src/foo/bar/bar.cxx
New rule: build/foo/bar/buz/buz.o: src/foo/bar/buz/buz.c
New rule: build/foo/bar/foo.o: src/foo/bar/foo.c
New rule: build/foo/foo.o: src/foo/bar/foo.c
New rule: build/foo/bar/foo.o: src/foo/foo.cpp
New rule: build/foo/foo.o: src/foo/foo.cpp
mkdir -p build/foo/bar && g++ -c src/foo/bar/bar.cxx -o build/foo/bar/bar.o
mkdir -p build/foo/bar/buz && g++ -c src/foo/bar/buz/buz.c -o build/foo/bar/buz/buz.o
mkdir -p build/foo/bar && g++ -c src/foo/bar/foo.c -o build/foo/bar/foo.o
mkdir -p build/foo && g++ -c src/foo/bar/foo.c -o build/foo/foo.o
Environment again:
$ find
.
./build
./build/foo
./build/foo/bar
./build/foo/bar/bar.o
./build/foo/bar/buz
./build/foo/bar/buz/buz.o
./build/foo/bar/foo.o
./build/foo/foo.o
./src
./src/foo
./src/foo/bar
./src/foo/bar/bar.cxx
./src/foo/bar/buz
./src/foo/bar/buz/buz.c
./src/foo/bar/foo.c
./src/foo/foo.cpp
Try running this Makefile with 'src_subdirs=' to exercise another approach to locate sources. Output should be the same.
I finally had some time to experiment with this, so here is what I came up with:
BUILD_DIR = build
SRC_DIR = src
SOURCES = $(shell find $(SRC_DIR)/ -name "*.c")
TARGET = program
OBJECTS = $(SOURCES:$(SRC_DIR)/%.c=$(BUILD_DIR)/%.o)
default: $(TARGET)
.SECONDEXPANSION:
$(OBJECTS) : $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$#)
mkdir -p $(#D)
$(CC) -c -o $# $(CFLAGS) $<
$(TARGET): $(OBJECTS)
$(CC) -o $# $(CFLAGS) $^
.PHONY: default
Points of interest:
I had to change the sources find pattern from ".c" to "*.c", I'm not sure if it depends on the exact shell used, but if you want to stay portable, be sure to use a widely accepted pattern.
The .SECONDEXPANSION: is needed to enable the $$ rules for GNU Make. It is needed to allow target based substitution rules in the prerequisites for the $(OBJECTS).
The prerequisite $$(patsubst $(BUILD_DIR)/%.o,$(SRC_DIR)/%.c,$$#) is saying, that the current target depends on a specific source file with the same folder structure and name.
The command mkdir -p $(#D) is ensuring, that the path of the current target is created if it's missing.
If all you want is a single rule to handle all object files, without necessarily needing to "compile all at once" then you could have something like this:
BUILD_DIR = build
SOURCES = ...
TARGET = ...
OBJECTS = $(SOURCES:%.c=$(BUILD_DIR)/%.o)
default: target
target: $(TARGET)
$(TARGET): $(OBJECTS)
$(LD) -o $# $(LDFLAGS) $^ $(LIBS)
$(BUILD_DIR)/%.o: %.c
$(CC) -c -o $# $< $(CFLAGS)
$(BUILD_DIR):
-mkdir $#
[Note: This is written from memory and without testing.]
After reading the GNU make manual again, here is a solution that solves the second problem.
The first attempt was the correct path. And the second attempt has the $(sources) in the prerequisites but does not use it in the commands and this is silly.
So the working makefile follows. It puts object files in a separate directory and it only compiles files that have changed.
sources = $(shell find src/ -name ".c")
$objects_dirs = $(subst src/, build/, $(dir $(sources)) # This variable is used by the build rule to create directories for objects files prior to compilation
objects = $(subst src/, build/, $(patsubst %.c, %.o, $(sources))) # This variable has the paths to the objects files that will be generated in the build directory
# This should now work as expected: object files go into their designated directories under "build/" and only updated files will be recompiled.
$(objects): build $(sources)
# After running say "make clean", make will figure out the need to run the first prerequisite.
# If we are doing a clean build, the number of prerequisites will equal the number of new prerequisites.
ifeq ($(words $?), $(words $^))
# Note the use of "$?" instead of "$^". $? is used since it holds prerequisites that are newer than the target while $^ will holds all prerequisites whether they are new or not.
$(foreach source, $(wordlist 2, $(words $?), $?), $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
else
# If we have a few new targets, no need to exclude "build" from prerequisites because the first prerequisite will be a file that changed.
$(foreach source, $?, $(shell $(cc) $(cflags) -o $(subst src/,build, $(patsubst %.c,%.o, $(source))) $(source)))
endif
.PHONY: build
build:
$(foreach dir, $(objects_dirs), $(shell mkdir -p $(dir)))
.PHONY: clean
clean:
#rm -rf build/
The makefile is heavily commented with changes that made it work. The most important changes were:
Use of $(foreach) to compile each file individually as required by GCC
Use of $? to work only with prerequisites that are newer than the target
Use of conditional to detected whether the first prerequisite has changed depending on circumstances. If we have a clean build (running make for the first time or after running make clean), the number of updated prerequisites will be the same as the number of newer prerequisites compared to the target. In other words $(words $?) == $(words $^) will be true. So we use this fact to exclude the firs prerequisite listed (build in our case) from the list of files to pass to GCC.
Also, when building the executable from the objects files, make sure to use $^ and not $? when selecting prerequisites else you will end up with only newer files in the executable and it will not run.
target = bin/mylib.a
.PHONY: all
all: $(target)
$(target): $(objects)
ar -cvq $# $^ # Notice that we're not using $? else only updated object files will end up in the archive.
I have a website in an src directory that is several levels deep, containing html, markdown, less, coffee, png, jpg and other assets. I'd like to build my website into a dist directory using make. With build, I mean
converting markdown files to html
compiling coffee to js
compiling less to css
minifying html files
minifying js files (that where not compiled from coffee)
minifying css files (that where not compiled from less or sass)
preparing images (logo.png becomes logo#1x.png logo#2x.png logo#3x.png)
I have the following file. The cp statements will be replaced with the respective tools to do the transformation.
sources = $(shell find src -type f)
t1 := $(sources:.md=.html)
t2 := $(t1:.less=.css)
targets := $(t2:.coffee=.js)
targetdirs = $(dir $(targets))
all: $(targets)
%.html: %.md
cp $< $#
%.css: %.less
cp $< $#
%.js: %.coffee
cp $< $#
This creates outputs side by side. So src/index.md becomes src/index.html, src/assets/stylesheets/app.less becomes src/assets/stylesheets/app.css and src/assets/scripts/app.coffee becomes src/assets/scripts/app.js. What I'd like to do is change the make file such that it stores the output in the dist directory, so src/index.md is converted to dist/index.html, src/assets/stylesheets/app.less compiled to dist/assets/stylesheets/app.css and and src/assets/scripts/app.coffee becomes dist/assets/scripts/app.js.
So I changed the makefile as follows:
sources = $(shell find src -type f)
t0 := $(subst src/,dist/,$(sources:.md=.html))
t1 := $(t0:.md=.html)
t2 := $(t1:.less=.css)
targets := $(t2:.coffee=.js)
targetdirs = $(dir $(targets))
all: $(targets)
%.html: %.md
mkdir -p $(targetdirs)
cp $< $#
%.css: %.less
mkdir -p $(targetdirs)
cp $< $#
%.js: %.coffee
mkdir -p $(targetdirs)
cp $< $#
Now make fails with the following:
make: *** No rule to make target `dist/assets/scripts/app.js', needed by `all'. Stop.
Most examples I can find, is limited to a single directory, or compiles multiple source files into a single target.
How would one achieve this without knowing the contents of the source directory?
Environment:
GNU Make 3.81
OS X 10.11.1
In make pattern rules, the stems represented by the patterns in the target and prerequisite must match exactly. Take this rule:
%.html: %.md
mkdir -p $(targetdirs)
cp $< $#
If the target make wants to build is dist/index.html, then the stem is dist/index. So, it will look for the prerequisite dist/index.md. Which doesn't exist. So make ignores that pattern rule as not matching, and continues to look for more implicit rules that might match... and doesn't find any so it fails.
You have to fix your rules so that the change in directory is reflected in the pattern:
dist/%.html: src/%.md
mkdir -p $(#D)
cp $< $#
(I'm not sure why you're creating all directories in every recipe instead of just the current one). Now the stem for dist/index.html is just index, and the prerequisite matches src/index.md and it will work.
gcc -o abc/def.o def.c generates def.o file in a directory abc; only when there exists a directory abc.
Is there a way to make gcc to create a directory when the enclosing directory of the generated object file does not exist? If not, what could be the easiest way to make the directory in advance automatically, especially for Makefile?
From this post, it seems like that there is no way to create a directory from gcc.
For makefile, I can use this code snippet.
OBJDIR = obj
MODULES := src src2
...
OBJDIRS := $(patsubst %, $(OBJDIR)/%, $(MODULES))
build: $(OBJDIRS)
echo $^
$(OBJDIRS):
mkdir -p $#
make build will create directories, and echo the results.
We also can make the object files are created automatically without invoking the make build.
PROG := hellomake
LD := gcc
...
$(PROG): $(OBJDIRS) obj/src/hellofunc.o obj/src/hellomake.o
$(LD) $(filter %.o, $^) -o $(PROG)
The way I do it is I find out what the paths will be, and create the dirs for them:
SOURCES:=$(shell find $(SRC)/ -type f -name '*.c') #Get all the c files
#Get the files they're gonna be compiled to
OBJECTS:=$(patsubst $(SRC)/%.c, $(OBJ)/%.o, $(SOURCES))
#Get just the paths so you can create them in advance
OBJDIRS:=$(dir $(OBJECTS))
#call mkdir with the directories (using a dummy var because too much
#trouble to deal with priorities) someone could prob do better
#--parents ignores the repeated and already existing errors
DUMMY:=$(shell mkdir --parents $(OBJDIRS))
Sauce:
https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html
In addition to precedent solutions, you can create the directories in your rule for building your .o files
$(OBJ_DIR)/%.o: $(SRCS_DIR)/%.c
#mkdir -p $(shell dirname $#)
$(CC) $(CFLAGS) $(INC) -c $< -o $#
The "#" before mkdir silent the output of mkdir (we don't want a message for each directory created).
The "-p" option tell mkdir to create all intermediate directories in path if they do not exist.
The "dirname" method takes the file path ("$#") and keep only the file directory path.
I have a Makefile which generates JSON from several different Python scripts (the scripts print to stdout) in a single directory, e.g.
/src
scriptOne.py
scriptTwo.py
scriptThree.py
Which outputs the JSON to a folder:
/templates
scriptOne.json
scriptTwo.json
scriptThree.json
I'm trying to restructure so that, for example, each script is in its own subdirectory and the Makefile creates the JSON templates in their consequent subdirectories as follows:
/src
/importantTemplates
scriptOne.py
/notSoImportantTemplates
scriptTwo.py
scriptThree.py
And the output:
/templates
/importantTemplates
scriptOne.json
/notSoImportantTemplates
scriptTwo.json
scriptThree.json
The current Makefile is as follows:
SOURCES := $(shell echo src/*.py)
TARGETS := $(patsubst src/%.py,templates/%.json,$(SOURCES))
all: $(TARGETS)
clean:
rm -f $(TARGETS)
templates/%.json: src/%.py
python2 $< > $#
I've tried changing the wildcards to include a subdirectory for each line e.g. /src/*/*.py, although I just end up with the following:
make: Nothing to be done for `all'.
You want a static pattern rule (4.12 Static Pattern Rules) for this.
SOURCES := $(wildcard src/*/*.py)
TARGETS := $(patsubst src/%.py,templates/%.json,$(SOURCES))
all: $(TARGETS)
clean:
rm -rf templates
$(TARGETS) : templates/%.json: src/%.py
mkdir -p $(#D)
python2 $< > $#
You could avoid needing mkdir -p in that rule body if you wanted to (and go with an order-only prerequisite on the directory instead) but I'm not sure the effort is worth the savings in execution cost. You could avoid the extra shell by combining the two lines mkdir -p $(#D) && python2 $< > $# if you wanted to though.
Please note that the second time you run the make, it will give you the message (if there are no new files):
make: Nothing to be done for `all'.
Try to run make clean and see if you get the same message.
Here is the Makefile which will do what you want:
SOURCES := $(wildcard src/*/*.py)
TARGETS := $(patsubst src/%.py,templates/%.json,$(SOURCES))
FOLDERS := $(sort $(dir $(TARGETS)))
all: $(TARGETS)
clean:
rm -rf $(TARGETS) $(FOLDERS)
$(FOLDERS):
mkdir -p $#
$(TARGETS): $(SOURCES) $(FOLDERS)
python2 $< > $#
The FOLDERS variable will contain the folders you need to create in the template directory. (sort will remove duplicates, so each folder will be there only once)
The $(FOLDERS) rule will create the folders.
The clean rule will remove the folders also.
If you need to add more sources, just do it like this:
SOURCES := $(wildcard src/*/*.py)
SOURCES += $(wildcard src/*.py)
...
I'm trying to run a makefile on a directory such that it runs lint on all the cpp files in that directory and save the output in multiple files with the same names as source files.
e.g. in other words I want to save lint out put for abc.cpp to abc.txt and def.cpp to def.txt and so on for all the unknown number of files to a sub-directory lintfiles.
some thing like the following:-
*lint .ALWAYS:*
--lint-nt $(PATHS) $(OPTIONS) *.cpp > ./lintfiles/%f.txt
make util is not understanding Lint's %f option. I also tried the following but it generates no output.
lintfiles/%.txt: %.cpp
-lint-nt $(PATHS) $(OPTIONS) $# $^
please suggest.
I'm not familiar with pc-lint, but if the syntax is something like
lint-nt -o abc.lint abc.cpp
then this makefile should do the job:
SRC_DIR := src # or whatever
SRCS := $(wildcard $(SRC_DIR)/*.cpp)
LINTS := $(patsubst $(SRC_DIR)/%.cpp, $(SRC_DIR)/lint/%.lint, $(SRCS))
all: $(LINTS)
$(LINTS): $(SRC_DIR)/lint/%.lint : $(SRC_DIR)/%.cpp
lint-nt $(PATHS) $(OPTIONS) -o $# $<