I have, for example, the following Makefile to generate PDF files from Markdown files in subdirectories:
FOLDERS = f1 f2 f3
.PHONY: $(FOLDERS)
f1: f1/f1.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f2: f2/f2.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
f3: f3/f3.md
cd $# && pandoc $(notdir $^) -o $(patsubst %.md,%.pdf,$(notdir $^))
The expected result is that make f1 requires the existence of f1/f1.md, and generates the resulting PDF as f1/f1.pdf. The same for f2 and f3. This works, but the declarations seem unnecessarily repetitive.
Is there any way to combine these three rules into one, generic rule? That is, without needing to explicitly write out all of the paths to the PDF files or Markdown files, as I may be dynamically adding subfolders and I'd prefer to just change the definition of FOLDERS in the first line. I've googled around and tried a few things, but I feel like either I can't find the right incantation to use, or I'm missing a piece of knowledge about how Makefiles work. Could someone please point me in the right direction?
First, note that there's no good reason to use PHONY targets here, since these rules appear to be building files whose names are known beforehand. Targets like f1/f1.pdf would be much better.
Unfortunately we can't use a pattern rule when the stem (e.g. f1) is repeated in a prerequisite. But a "canned recipe" can do the trick:
define pdf_template
$(1): $(1)/$(1).md
cd $$# && pandoc $$(notdir $$^) -o $$(patsubst %.md,%.pdf,$$(notdir $$^))
endef
$(eval $(call pdf_template,f1))
$(eval $(call pdf_template,f2))
$(eval $(call pdf_template,f3))
(Note how you must escape the $ signs in the template.)
If those $(eval...) lines look too repetitive, you can replace them with a loop:
$(foreach folder,$(FOLDERS),$(eval $(call pdf_template,$(folder))))
EDIT: Come to think of it, there's another way. You can't construct a pattern rule that uses the stem more than once:
$(FOLDERS): %: %/%.md
cd $# && ... this won't work
And you can't use the automatic variables in the prerequisite list, because they aren't yet defined when they're needed:
$(FOLDERS): $#/$#.md
cd $# && ... this won't work either
But you can use them there if you use Secondary Expansion, which causes Make to expand the prereq list a second time:
.SECONDEXPANSION:
$(FOLDERS): $$#/$$#.md
cd $# && ... this works
Again, note the escaped $ symbols.
I have a top level Makefile that can create multiple targets, for which I have a for loop in place. In this file I am trying to read a text file with a version number that is created only when a sub makefile runs. I have got it to work but it looks very ugly, so I'm thinking there is possibly a better way to do it. Here is the top level Makefile:
T1=_release
T2=_debug
T3=_test
all: bin/mybin_release.bin
debug: bin/mybin_debug.bin
debug: export DBG:=1
test: bin/mybin_test.bin
test: export TST:=1
define build_foo
bin/mybin$$($(1)).bin: bin/abc$$($(1)).bin bin/xyz$$($(1)).bin
cat $$^ > $$#
VER=$$$$(cat bin/rev.txt) && mv bin/mybin$$($(1)).bin bin/mybin$$($(1))_$$$${VER}.bin
bin/abc$$($(1)).bin:
$(MAKE) -C mod1 # <--- rev.txt is produced here
bin/xyz$$($(1)).bin:
$(MAKE) -C mod2
endef
$(foreach suffix, T1 T2 T3, $(eval $(call build_foo,$(suffix))))
clean:
rm bin/*.bin bin/*.txt
Notice the attempt to grab file contents in VER. Is there a better/right way to do this? I cannot use eval since it runs at the beginning and rev.txt only gets created once the sub make runs.
Also, is this a decent way to build multiple targets(using foreach)? I use the exported variables to modify the target built by the sub makefile.
As far as I understand, this bin/rev.txt file, and all the bin/abcXXX.bin are produced when running $(MAKE) -C mod1. So, it is the same for all. What about:
include version.mk
version.mk: bin/rev.txt
{ printf 'VER = '; cat $<; } > $#
bin/rev.txt:
$(MAKE) -C mod1
Demo:
$ ls
Makefile
$ cat Makefile
include version.mk
all:
touch $(VER).txt
version.mk: rev.txt
{ printf 'VER = '; cat $<; } > $#
rev.txt:
echo "1.2.3" > $#
$ make --quiet
Makefile:1: version.mk: No such file or directory
$ ls
1.2.3.txt Makefile rev.txt version.mk
Explanation:
version.mk is a second makefile that make will look for. As per GNU make documentation, section 3.5:
To this end, after reading in all makefiles, make will consider each
as a goal target and attempt to update it. If a makefile has a rule
which says how to update it (found either in that very makefile or in
another one) or if an implicit rule applies to it (see Using Implicit
Rules), it will be updated if necessary. After all makefiles have been
checked, if any have actually been changed, make starts with a clean
slate and reads all the makefiles over again. (It will also attempt to
update each of them over again, but normally this will not change them
again, since they are already up to date.)
Note: as you wrote your makefile, $(MAKE) -C mod1 will be run several times, which is a waste. You could instead exploit a specificity of GNU make pattern rules: when they have multiple targets, make considers that all targets are produced by one single invocation of the recipe. Example:
$ cat Makefile
all: a.bin b.bin c.bin
a.%in b.%in c.%in:
#echo 'building a.bin b.bin c.bin'
$ make all
building a.bin b.bin c.bin
See? The recipe is executed only once to build the 3 targets. The only problem is that the % wildcard must match at least one character. So, in your case you could do something like (the character that % matches is the b of bin/):
T1 := _release
T2 := _debug
T3 := _test
suffixes := T1 T2 T3
pattern := $(foreach suffix,$(suffixes),%in/abc$($(suffix))) %in/rev.txt
$(pattern):
$(MAKE) -C mod1
This will tell make that $(MAKE) -C mod1 builds all the bin/abcXXX.bin and bin/rev.txt at once. Same with $(MAKE) -C mod2.
All in all, you could probably get completely rid of your build_foo generic rule by gluing all these features together. Something like:
T1 := _release
T2 := _debug
T3 := _test
suffixes := T1 T2 T3
patternabc := $(foreach suffix,$(suffixes),%in/abc$($(suffix)).bin) %in/rev.txt
patternxyz := $(foreach suffix,$(suffixes),%in/xyz$($(suffix)).bin)
include version.mk
all: bin/mybin_release_$(VER).bin
debug: bin/mybin_debug_$(VER).bin
debug: export DBG:=1
test: bin/mybin_test_$(VER).bin
test: export TST:=1
version.mk: bin/rev.txt
{ printf 'VER = '; cat $<; } > $#
$(patternabc):
$(MAKE) -C mod1
$(patternxyz):
$(MAKE) -C mod2
bin/mybin%_$(VER).bin: bin/abc%.bin bin/xyz%.bin
cat $^ > $#
I'm trying to do something like this with make:
SRC := $(src/*.md)
DIST := $(subst -,/,$(patsubst src/%.md, dist/%/index.html, $(SRC)))
all: $(DIST)
$(DIST): $(SRC)
mkdir -p $(#D) && pandoc $< -o $#
E.g., the prerequisite src/2014-04-myfile.md is put into target dist/2014/04/myfile/index.html with the transform pandoc
But when I use $< it only refers to the first argument in the $(SRC) variable.
I know normally we would do something like:
dist/%.html: src/%.md
but since I changed the file name in the output to just index.html for all files and used the original file name to create a new path I'm not sure how to go about iterating over the prerequisites.
Here's one way it could be done. The way this works is that it iterates over $(SRC) to create one rule per source file. The $$ in MAKE_DEP are necessary to prevent make from interpreting the functions when it first reads the contents of MAKE_DEP. The documentation on call and eval are also useful.
SRC := $(wildcard src/*.md)
# Set the default goal if no goal has been specified...
.DEFAULT_GOAL:=all
#
# This is a macro that we use to create the rules.
#
define MAKE_DEP
# _target is a temporary "internal" variable used to avoid recomputing
# the current target multiple times.
_target:=$$(subst -,/,$$(patsubst src/%.md, dist/%/index.html, $1))
# Add the current target to the list of targets.
TARGETS:=$$(TARGETS) $$(_target)
# Create the rule proper.
$$(_target):$1
mkdir -p $$(#D) && pandoc $$< -o $$#
endef # MAKE_DEP
# Iterate over $(SRC) to create each rule.
$(foreach x,$(SRC),$(eval $(call MAKE_DEP,$x)))
.PHONY: all
all: $(TARGETS)
If I create:
src/2000-01-bar.md
src/2014-04-foo.md
and run $ make -n, I get:
mkdir -p dist/2000/01/bar && pandoc src/2000-01-bar.md -o dist/2000/01/bar/index.html
mkdir -p dist/2014/04/foo && pandoc src/2014-04-foo.md -o dist/2014/04/foo/index.html
This could also be done using secondary expansion but it did not appear to me to be simpler or nicer.
the makefile below processes files matching the patterncontent/%.md and outputs the targets in the html directory. Source files are named with the convention of putting a leading number in front of them, like content/01.index.md or content/O2.second-page.md and so on. I would like to remove the leading 0x. number sequence in the target file. For instance, content/01.index.html would generate html/index.html.
How can I do this?
Thanks
MD_FILES = $(shell find content/ -type f -name '*.md')
HTML_FILES = $(patsubst content/%.md, html/%.html, $(MD_FILES))
all: $(HTML_FILES) html/static
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
.PHONY: html/static
html/static :
rsync -rupE generator/static html/
.PHONY: clean
clean:
rm -fr html
Replace:
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
with:
html/%.html : content/%.md
mkdir -p $(#D)
file='$(#F)'; python generator/generate.py $< "$(#D)/${file#*.}"
Unfortunately, I can't think of a good way of doing that in make itself. I can think of one way but it isn't as simple as that escaping and it isn't safe for files with spaces (not that that matters much here since make already can't handle those).
IMHO, it is a bad idea to use find or wildcards to list files in makefiles. This is because developers have temporary or debugging files sometimes. It is best to list files explicitly. This way, it forces the developer to think about their intent.
If you agree to list files explicitly, then in this case it is best to list the target files, rather than source files, and here is your answer:
HTML_FILES := html/index.html html/second-page.html
.SECONDEXPANSION:
$(HTML_FILES): html/%.html : $$(wildcard content/*.$$*.md)
(put recipe here, using $# and $<)
I want to create directories using makefile. My project directory is like this
+--Project
+--output
+--source
+Testfile.cpp
+Makefile
I want to put all the objects and output into the respective output folder. I want to create folder structure which would be like this after compiling.
+--Project
+--output
+--debug (or release)
+--objs
+Testfile.o
+Testfile (my executable file)
+--source
+Testfile.cpp
+Makefile
I tried with several options, but could not succeed. Please help me to make directories using make file. I'm posting my Makefile for your consideration.
#---------------------------------------------------------------------
# Input dirs, names, files
#---------------------------------------------------------------------
OUTPUT_ROOT := output/
TITLE_NAME := TestProj
ifdef DEBUG
TITLE_NAME += _DEBUG
else
ifdef RELEASE
TITLE_NAME += _RELEASE
endif
endif
# Include all the source files here with the directory tree
SOURCES := \
source/TestFile.cpp \
#---------------------------------------------------------------------
# configs
#---------------------------------------------------------------------
ifdef DEBUG
OUT_DIR := $(OUTPUT_ROOT)debug
CC_FLAGS := -c -Wall
else
ifdef RELEASE
OUT_DIR := $(OUTPUT_ROOT)release
CC_FLAGS := -c -Wall
else
$(error no build type defined)
endif
endif
# Put objects in the output directory.
OUT_O_DIR := $(OUT_DIR)/objs
#---------------------------------------------------------------------
# settings
#---------------------------------------------------------------------
OBJS = $(SOURCES:.cpp=.o)
DIRS = $(subst /,/,$(sort $(dir $(OBJS))))
DIR_TARGET = $(OUT_DIR)
OUTPUT_TARGET = $(OUT_DIR)/$(TITLE_NAME)
CC_FLAGS +=
LCF_FLAGS :=
LD_FLAGS :=
#---------------------------------------------------------------------
# executables
#---------------------------------------------------------------------
MD := mkdir
RM := rm
CC := g++
#---------------------------------------------------------------------
# rules
#---------------------------------------------------------------------
.PHONY: all clean title
all: title
clean:
$(RM) -rf $(OUT_DIR)
$(DIR_TARGET):
$(MD) -p $(DIRS)
.cpp.o:
#$(CC) -c $< -o $#
$(OBJS): $(OUT_O_DIR)/%.o: %.cpp
#$(CC) -c $< -o $#
title: $(DIR_TARGET) $(OBJS)
In my opinion, directories should not be considered targets of your makefile, either in technical or in design sense. You should create files and if a file creation needs a new directory then quietly create the directory within the rule for the relevant file.
If you're targeting a usual or "patterned" file, just use make's internal variable $(#D), that means "the directory the current target resides in" (cmp. with $# for the target). For example,
$(OUT_O_DIR)/%.o: %.cpp
#mkdir -p $(#D)
#$(CC) -c $< -o $#
title: $(OBJS)
Then, you're effectively doing the same: create directories for all $(OBJS), but you'll do it in a less complicated way.
The same policy (files are targets, directories never are) is used in various applications. For example, git revision control system doesn't store directories.
Note: If you're going to use it, it might be useful to introduce a convenience variable and utilize make's expansion rules.
dir_guard=#mkdir -p $(#D)
$(OUT_O_DIR)/%.o: %.cpp
$(dir_guard)
#$(CC) -c $< -o $#
$(OUT_O_DIR_DEBUG)/%.o: %.cpp
$(dir_guard)
#$(CC) -g -c $< -o $#
title: $(OBJS)
This would do it - assuming a Unix-like environment.
MKDIR_P = mkdir -p
.PHONY: directories
all: directories program
directories: ${OUT_DIR}
${OUT_DIR}:
${MKDIR_P} ${OUT_DIR}
This would have to be run in the top-level directory - or the definition of ${OUT_DIR} would have to be correct relative to where it is run. Of course, if you follow the edicts of Peter Miller's "Recursive Make Considered Harmful" paper, then you'll be running make in the top-level directory anyway.
I'm playing with this (RMCH) at the moment. It needed a bit of adaptation to the suite of software that I am using as a test ground. The suite has a dozen separate programs built with source spread across 15 directories, some of it shared. But with a bit of care, it can be done. OTOH, it might not be appropriate for a newbie.
As noted in the comments, listing the 'mkdir' command as the action for 'directories' is wrong. As also noted in the comments, there are other ways to fix the 'do not know how to make output/debug' error that results. One is to remove the dependency on the the 'directories' line. This works because 'mkdir -p' does not generate errors if all the directories it is asked to create already exist. The other is the mechanism shown, which will only attempt to create the directory if it does not exist. The 'as amended' version is what I had in mind last night - but both techniques work (and both have problems if output/debug exists but is a file rather than a directory).
Or, KISS.
DIRS=build build/bins
...
$(shell mkdir -p $(DIRS))
This will create all the directories after the Makefile is parsed.
make in, and off itself, handles directory targets just the same as file targets. So, it's easy to write rules like this:
outDir/someTarget: Makefile outDir
touch outDir/someTarget
outDir:
mkdir -p outDir
The only problem with that is, that the directories timestamp depends on what is done to the files inside. For the rules above, this leads to the following result:
$ make
mkdir -p outDir
touch outDir/someTarget
$ make
touch outDir/someTarget
$ make
touch outDir/someTarget
$ make
touch outDir/someTarget
This is most definitely not what you want. Whenever you touch the file, you also touch the directory. And since the file depends on the directory, the file consequently appears to be out of date, forcing it to be rebuilt.
However, you can easily break this loop by telling make to ignore the timestamp of the directory. This is done by declaring the directory as an order-only prerequsite:
# The pipe symbol tells make that the following prerequisites are order-only
# |
# v
outDir/someTarget: Makefile | outDir
touch outDir/someTarget
outDir:
mkdir -p outDir
This correctly yields:
$ make
mkdir -p outDir
touch outDir/someTarget
$ make
make: 'outDir/someTarget' is up to date.
TL;DR:
Write a rule to create the directory:
$(OUT_DIR):
mkdir -p $(OUT_DIR)
And have the targets for the stuff inside depend on the directory order-only:
$(OUT_DIR)/someTarget: ... | $(OUT_DIR)
All solutions including the accepted one have some issues as stated in their respective comments. The accepted answer by #jonathan-leffler is already quite good but does not take into effect that prerequisites are not necessarily to be built in order (during make -j for example). However simply moving the directories prerequisite from all to program provokes rebuilds on every run AFAICT.
The following solution does not have that problem and AFAICS works as intended.
MKDIR_P := mkdir -p
OUT_DIR := build
.PHONY: directories all clean
all: $(OUT_DIR)/program
directories: $(OUT_DIR)
$(OUT_DIR):
${MKDIR_P} $(OUT_DIR)
$(OUT_DIR)/program: | directories
touch $(OUT_DIR)/program
clean:
rm -rf $(OUT_DIR)
I've just come up with a fairly reasonable solution that lets you define the files to build and have directories be automatically created. First, define a variable ALL_TARGET_FILES that holds the file name of every file that your makefile will be build. Then use the following code:
define depend_on_dir
$(1): | $(dir $(1))
ifndef $(dir $(1))_DIRECTORY_RULE_IS_DEFINED
$(dir $(1)):
mkdir -p $$#
$(dir $(1))_DIRECTORY_RULE_IS_DEFINED := 1
endif
endef
$(foreach file,$(ALL_TARGET_FILES),$(eval $(call depend_on_dir,$(file))))
Here's how it works. I define a function depend_on_dir which takes a file name and generates a rule that makes the file depend on the directory that contains it and then defines a rule to create that directory if necessary. Then I use foreach to call this function on each file name and eval the result.
Note that you'll need a version of GNU make that supports eval, which I think is versions 3.81 and higher.
given that you're a newbie, I'd say don't try to do this yet. it's definitely possible, but will needlessly complicate your Makefile. stick to the simple ways until you're more comfortable with make.
that said, one way to build in a directory different from the source directory is VPATH; i prefer pattern rules
OS independence is critical for me, so mkdir -p is not an option. I created this series of functions that use eval to create directory targets with the prerequisite on the parent directory. This has the benefit that make -j 2 will work without issue since the dependencies are correctly determined.
# convenience function for getting parent directory, will eventually return ./
# $(call get_parent_dir,somewhere/on/earth/) -> somewhere/on/
get_parent_dir=$(dir $(patsubst %/,%,$1))
# function to create directory targets.
# All directories have order-only-prerequisites on their parent directories
# https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html#Prerequisite-Types
TARGET_DIRS:=
define make_dirs_recursively
TARGET_DIRS+=$1
$1: | $(if $(subst ./,,$(call get_parent_dir,$1)),$(call get_parent_dir,$1))
mkdir $1
endef
# function to recursively get all directories
# $(call get_all_dirs,things/and/places/) -> things/ things/and/ things/and/places/
# $(call get_all_dirs,things/and/places) -> things/ things/and/
get_all_dirs=$(if $(subst ./,,$(dir $1)),$(call get_all_dirs,$(call get_parent_dir,$1)) $1)
# function to turn all targets into directories
# $(call get_all_target_dirs,obj/a.o obj/three/b.o) -> obj/ obj/three/
get_all_target_dirs=$(sort $(foreach target,$1,$(call get_all_dirs,$(dir $(target)))))
# create target dirs
create_dirs=$(foreach dirname,$(call get_all_target_dirs,$1),$(eval $(call make_dirs_recursively,$(dirname))))
TARGETS := w/h/a/t/e/v/e/r/things.dat w/h/a/t/things.dat
all: $(TARGETS)
# this must be placed after your .DEFAULT_GOAL, or you can manually state what it is
# https://www.gnu.org/software/make/manual/html_node/Special-Variables.html
$(call create_dirs,$(TARGETS))
# $(TARGET_DIRS) needs to be an order-only-prerequisite
w/h/a/t/e/v/e/r/things.dat: w/h/a/t/things.dat | $(TARGET_DIRS)
echo whatever happens > $#
w/h/a/t/things.dat: | $(TARGET_DIRS)
echo whatever happens > $#
For example, running the above will create:
$ make
mkdir w/
mkdir w/h/
mkdir w/h/a/
mkdir w/h/a/t/
mkdir w/h/a/t/e/
mkdir w/h/a/t/e/v/
mkdir w/h/a/t/e/v/e/
mkdir w/h/a/t/e/v/e/r/
echo whatever happens > w/h/a/t/things.dat
echo whatever happens > w/h/a/t/e/v/e/r/things.dat
See https://www.oreilly.com/library/view/managing-projects-with/0596006101/ch12.html
REQUIRED_DIRS = ...
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
do \
[[ -d $$d ]] || mkdir -p $$d; \
done)
$(objects) : $(sources)
As I use Ubuntu, I also needed add this at the top of my Makefile:
SHELL := /bin/bash # Use bash syntax
I use the makefiles in windows environment and my simple solution is as follows,
Create a target makedir and add it as a prerequisites to where ever it is required.
# Default goal
all: gccversion makedir build finalize list sizeafter completed
The makedir target is (applicable only in windows environment)
makedir:
#IF NOT EXIST $(subst /,\,$(BUILD_DIR)) mkdir $(subst /,\,$(BUILD_DIR)) 2> NULL
#IF NOT EXIST $(subst /,\,$(OUTPUT_DIR)) mkdir $(subst /,\,$(OUTPUT_DIR)) 2> NULL
#IF NOT EXIST $(subst /,\,$(DEP_DIR)) mkdir $(subst /,\,$(DEP_DIR)) 2> NUL
#IF NOT EXIST $(subst /,\,$(OBJ_DIR)) mkdir $(subst /,\,$(OBJ_DIR)) 2> NUL
$(subst /,\,$(BUILD_DIR)) converts the directory separator / to \ and
mkdir $(subst /,\,$(BUILD_DIR)) 2> NUL redirects the error if any.
src_dir := src
obj_dir := obj
build_dir := build
dirs := $(src_dir) $(obj_dir) $(build_dir) # new variable
all: $(dirs) $(other_dependencies) # added dependency (*before* any others)
$(dirs): # rule which makes missing directories
mkdir $#
Won't clutter your terminal with "cannot create directory" error messages. If the directories exist, they don't need to be built.
Works like any other dependency, only requires one rule and one variable.