I am trying to write a Makefile that builds PDF outputs with LaTeX, using Latexmk. The outputs have basically the same rule, with different prerequisites, so I tried generalising my original Makefile using GNU Make's "secondary expansion". (I also created .PHONY targets, also with secondary expansion, to make it more user-friendly.) However, this is causing the prerequisite rules to always be run, even when they don't need to be. Fortunately, Latexmk is clever enough to avoid doing unnecessary work, but I wonder if I'm doing anything wrong...
To try to abstract what I'm attempting:
,-> foo -> build/foo.pdf
all -{
`-> bar -> build/bar.pdf
That is, the all target builds foo and bar. These targets open the respective PDF file, which have a prerequisite of build/X.pdf (where X is foo or bar). These are genuine targets which build the appropriate PDF file.
This is what I've come up with:
TARGETS = foo bar
BUILD_DIR = build
OUTPUTS = $(TARGETS:%=$(BUILD_DIR)/%.pdf)
commonSRC = src/preamble.tex src/header.tex # etc...
fooSRC = src/foo.tex $(commonSRC) # etc...
barSRC = src/bar.tex $(commonSRC) # etc...
all: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): $(BUILD_DIR)/$$#.pdf
open $<
# FIXME This isn't quite right: This rule is still getting called by the
# above rule, even when it doesn't need to be. Latexmk is clever enough
# not to do any extra work, but it shouldn't run at all.
.SECONDEXPANSION:
$(OUTPUTS): $$($$(subst .pdf,SRC,$$(#F))) $(BUILD_DIR)
latexmk -outdir=$(BUILD_DIR) -auxdir=$(BUILD_DIR) -pdf $<
$(BUILD_DIR):
mkdir $#
clean:
rm -rf $(BUILD_DIR)
.PHONY: all $(TARGETS) clean
Just to be clear: The rule for build/X.pdf should run whenever the files enumerated in XSRC (again, where X is foo or bar) are newer than the PDFs, or the PDFs don't exist; but it should not run otherwise.
I believe that this got somewhat complex, more than it needs to be. Part of these second expansion statements can be just replaced with static pattern rules. The other thing is that .SECONDEXPANSION: makes all further Makefile contents to be subject to second expansion, so you don't need to explicitly state it before every target (it would be much clearer to mark .PHONY targets this way to quickly see if a target is phony or not).
Nevertheless, I believe that most important issue here is that you mention a directory as a prerequisite. Remember that make decides on whether to rebuild the target based on dependencies timestamp, and a directory gets its timestamp always updated whenever a file in this directory is updated. Therefore, whenever you write $(BUILD_DIR)/foo.pdf, $(BUILD_DIR) timestamp gets updated and the next call will build again since the directory is newer. You can avoid it by specifying a directory as an order-only prerequisite (which means: build if it doesn't exist, but do not check timestamp).
Putting it all together I would make it this way:
TARGETS = foo bar
BUILD_DIR = build
commonSRC = src/preamble.tex src/header.tex # etc...
fooSRC = src/foo.tex $(commonSRC) # etc...
barSRC = src/bar.tex $(commonSRC) # etc...
.SECONDEXPANSION:
.PHONY: all
all: $(TARGETS)
.PHONY: $(TARGETS)
$(TARGETS): %: $(BUILD_DIR)/%.pdf
echo open $<
$(BUILD_DIR)/%.pdf: $$($$*SRC) | $(BUILD_DIR)
echo latexmk -outdir=$(BUILD_DIR) -auxdir=$(BUILD_DIR) -pdf $< > $#
$(BUILD_DIR):
mkdir -p $#
.PHONY: clean
clean:
rm -rf $(BUILD_DIR)
Output:
$ make all
mkdir -p build
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo latexmk -outdir=build -auxdir=build -pdf src/bar.tex > build/bar.pdf
echo open build/bar.pdf
open build/bar.pdf
$ make all
echo open build/foo.pdf
open build/foo.pdf
echo open build/bar.pdf
open build/bar.pdf
Note that subsequent call did not attempt to build anything, just open the pdf. It still reacts on the file change however:
$ touch src/foo.tex
$ make all
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo open build/bar.pdf
open build/bar.pdf
$ touch src/header.tex
$ make all
echo latexmk -outdir=build -auxdir=build -pdf src/foo.tex > build/foo.pdf
echo open build/foo.pdf
open build/foo.pdf
echo latexmk -outdir=build -auxdir=build -pdf src/bar.tex > build/bar.pdf
echo open build/bar.pdf
open build/bar.pdf
I am using a Makefile to automate the installation of some tools, some of which I download
all:bin/symfony
bin/symfony: var/bin/symfonyInstaller
#echo making $#
#$< --install-dir bin > /dev/null 2>&1
var/bin/symfonyInstaller: var/bin
#echo making $#
#wget -q https://get.symfony.com/cli/installer -O $# >/dev/null 2>&1
#chmod a+x $# > /dev/null
var/bin:
#echo making $#
#mkdir -p $# > /dev/null
clean:
#echo making $#
#rm -rf bin var
For some reasons, var/bin/symfonyInstaller is always being remade :
$ make
making var/bin
making var/bin/symfonyInstaller
making bin/symfony
$ make
making var/bin/symfonyInstaller
$
Is there a way to tell make "don't rebuild var/bin/symfonyInstaller" ?
Your var/bin/symfonyInstaller target is always rebuilt because it depends on the directory in which it goes. And each time it is built the directory's last modification time is updated and becomes newer than the target. Thus the endless cycle.
What you want is called an order-only prerequisite. Every prerequisite mentioned in a make rule before the | separator is a "normal" one. After the | separator it is an order-only prerequisite.
All prerequisites, normal or order-only, are updated before the target, if they don't exist or if they are outdated. The difference is in how make decides to rebuild or not the corresponding target. Make rebuilds the target if it updated a normal prerequisite, or if the normal prerequisite is newer than the target. But not if it is an order-only prerequisite that has been updated or is newer than the target.
In your case you should use it for var/bin:
var/bin/symfonyInstaller: | var/bin
#echo making $#
#wget -q https://get.symfony.com/cli/installer -O $# >/dev/null 2>&1
#chmod a+x $# > /dev/null
If var/bin does not exist it is created before var/bin/symfonyInstaller, which is good, but var/bin/symfonyInstaller will not be rebuilt just because it is older than var/bin.
My makefile failed. My idea was to extract files from a certain file and put them in a temporary directory. This is then rsync'ed to a source directory. Since rsync only updates when a file has changed, makefiles in the source directory do not remake unless necessary. I have isolated the problem below:
.PHONY: extract clean
FILES = filea1.cc filea2.cc filea3.cc \
filea1.hh filea2.hh filea3.hh \
filea1.py filea2.py filea3.py \
dir-cc/filea1.cc dir-cc/filea2.cc dir-cc/filea3.cc \
dir-cc/filea1.hh dir-cc/filea2.hh dir-cc/filea3.hh \
dir-py/filea1.py dir-py/filea2.py dir-py/filea3.py
PATTERN := $(sort $(addprefix build/%., $(patsubst .%,%,$(suffix $(FILES)))))
extract: $(addprefix build/, $(FILES))
$(PATTERN):
mkdir -p $(dir $#); echo "hello!" > $#
#build/%.cc:;mkdir -p $(dir $#); echo "hello!" > $#
#build/%.hh:;mkdir -p $(dir $#); echo "hello!" > $#
#build/%.py:;mkdir -p $(dir $#); echo "hello!" > $#
temp:
echo $(PATTERN)
clean:
rm -rf build
Doing make temp gets 'build/%.cc build/%.hh build/%.py' but 'make extract' fails to make all the files:
build:
dir-cc dir-py filea1.cc filea2.cc filea3.cc
build/dir-cc:
filea1.cc filea2.cc filea3.cc
build/dir-py:
filea1.py filea2.py filea3.py
The hh files are missing. Unbelievably, in the actual work, the cc files are missing. Anyway, commenting out $(PATTERN), and removing the comments so that the targets in the code above are separate, does get all the files:
build:
dir-cc filea1.cc filea1.py filea2.hh filea3.cc filea3.py
dir-py filea1.hh filea2.cc filea2.py filea3.hh
build/dir-cc:
filea1.cc filea1.hh filea2.cc filea2.hh filea3.cc filea3.hh
build/dir-py:
filea1.py filea2.py filea3.py
Since everything has to be automatic in the actual work, and the rule is the same for all the targets (so I should not have to retype it), I vastly prefer the first version. I read the manual but I am perplexed as to why make does this. I am using GNU Make 4.1.
It's explained in the last paragraph of the Pattern Intro section:
Pattern rules may have more than one target. Unlike normal rules, this does not act as many different rules with the same prerequisites and recipe. If a pattern rule has multiple targets, make knows that the rule’s recipe is responsible for making all of the targets. The recipe is executed only once to make all the targets.
You can't write a pattern rule with many different pattern targets, such that make will run the recipe once per individual target. That's not how it works.
You could do this, for your very simple case:
FILETYPES := $(sort $(suffix $(FILES)))
$(foreach T,$(FILETYPES),$(eval build/%$T: ; mkdir -p $$(#D); echo "hello!" > $$#))
There was a good resolution: use the ordinary (non pattern) multiple target
rule $(FILES):;rule and define pattern specific variable values to adjust
the rule to the file extension.
I have this rule in my Makefile, that responds to flags I pass:
$(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/disable_$*
rm -f $(BUILD_DIR)/enable_$*
cd $(BUILD_DIR) && rm -f Makefile
$(BUILD_DIR)/enable_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/enable_$*
rm -f $(BUILD_DIR)/disable_$*
cd $(BUILD_DIR) && rm -f Makefile
What this means is that when changing the flags by which I invoke the makefile, I can trigger some recompilations that could depend on these flags.
The code presented above is a bit redundant: you see that I remove a file, touch another and remove a Makefile in both cases. The only thing that changes is the name of the files that I touch/remove, and they are related.
For instance,
make clean
make enable_debug=yes enable_video=no # will compile from zero
make enable_debug=no enable_video=no # flag change detected -> recompile some submodules that depend on this flag
Provided that the only thing that changes between the two rules ( [en|dis]able ), what I would like is to only have 1 generic rule, something like that:
# match 2 parts in the rule
$(BUILD_DIR)/%ble_%:
mkdir -p $(BUILD_DIR)
touch $(BUILD_DIR)/(???)ble_$* # should be $#
rm -f $(BUILD_DIR)/(???)able_$* # should be disable if $# is enable and inverse
cd $(BUILD_DIR) && rm -f Makefile
Is this possible ?
PS: Sorry if I didn't get the title correctly, I couldn't figure how to explain it better.
$(BUILD_DIR)/enable_% $(BUILD_DIR)/disable_%:
mkdir -p $(BUILD_DIR)
rm -f $(BUILD_DIR)/*able_$*
touch $#
cd $(BUILD_DIR) && rm -f Makefile
Not literally what you wanted (multi-wildcards are forbidden in make), but does quite the same.
I want to create directories using makefile. My project directory is like this
+--Project
+--output
+--source
+Testfile.cpp
+Makefile
I want to put all the objects and output into the respective output folder. I want to create folder structure which would be like this after compiling.
+--Project
+--output
+--debug (or release)
+--objs
+Testfile.o
+Testfile (my executable file)
+--source
+Testfile.cpp
+Makefile
I tried with several options, but could not succeed. Please help me to make directories using make file. I'm posting my Makefile for your consideration.
#---------------------------------------------------------------------
# Input dirs, names, files
#---------------------------------------------------------------------
OUTPUT_ROOT := output/
TITLE_NAME := TestProj
ifdef DEBUG
TITLE_NAME += _DEBUG
else
ifdef RELEASE
TITLE_NAME += _RELEASE
endif
endif
# Include all the source files here with the directory tree
SOURCES := \
source/TestFile.cpp \
#---------------------------------------------------------------------
# configs
#---------------------------------------------------------------------
ifdef DEBUG
OUT_DIR := $(OUTPUT_ROOT)debug
CC_FLAGS := -c -Wall
else
ifdef RELEASE
OUT_DIR := $(OUTPUT_ROOT)release
CC_FLAGS := -c -Wall
else
$(error no build type defined)
endif
endif
# Put objects in the output directory.
OUT_O_DIR := $(OUT_DIR)/objs
#---------------------------------------------------------------------
# settings
#---------------------------------------------------------------------
OBJS = $(SOURCES:.cpp=.o)
DIRS = $(subst /,/,$(sort $(dir $(OBJS))))
DIR_TARGET = $(OUT_DIR)
OUTPUT_TARGET = $(OUT_DIR)/$(TITLE_NAME)
CC_FLAGS +=
LCF_FLAGS :=
LD_FLAGS :=
#---------------------------------------------------------------------
# executables
#---------------------------------------------------------------------
MD := mkdir
RM := rm
CC := g++
#---------------------------------------------------------------------
# rules
#---------------------------------------------------------------------
.PHONY: all clean title
all: title
clean:
$(RM) -rf $(OUT_DIR)
$(DIR_TARGET):
$(MD) -p $(DIRS)
.cpp.o:
#$(CC) -c $< -o $#
$(OBJS): $(OUT_O_DIR)/%.o: %.cpp
#$(CC) -c $< -o $#
title: $(DIR_TARGET) $(OBJS)
In my opinion, directories should not be considered targets of your makefile, either in technical or in design sense. You should create files and if a file creation needs a new directory then quietly create the directory within the rule for the relevant file.
If you're targeting a usual or "patterned" file, just use make's internal variable $(#D), that means "the directory the current target resides in" (cmp. with $# for the target). For example,
$(OUT_O_DIR)/%.o: %.cpp
#mkdir -p $(#D)
#$(CC) -c $< -o $#
title: $(OBJS)
Then, you're effectively doing the same: create directories for all $(OBJS), but you'll do it in a less complicated way.
The same policy (files are targets, directories never are) is used in various applications. For example, git revision control system doesn't store directories.
Note: If you're going to use it, it might be useful to introduce a convenience variable and utilize make's expansion rules.
dir_guard=#mkdir -p $(#D)
$(OUT_O_DIR)/%.o: %.cpp
$(dir_guard)
#$(CC) -c $< -o $#
$(OUT_O_DIR_DEBUG)/%.o: %.cpp
$(dir_guard)
#$(CC) -g -c $< -o $#
title: $(OBJS)
This would do it - assuming a Unix-like environment.
MKDIR_P = mkdir -p
.PHONY: directories
all: directories program
directories: ${OUT_DIR}
${OUT_DIR}:
${MKDIR_P} ${OUT_DIR}
This would have to be run in the top-level directory - or the definition of ${OUT_DIR} would have to be correct relative to where it is run. Of course, if you follow the edicts of Peter Miller's "Recursive Make Considered Harmful" paper, then you'll be running make in the top-level directory anyway.
I'm playing with this (RMCH) at the moment. It needed a bit of adaptation to the suite of software that I am using as a test ground. The suite has a dozen separate programs built with source spread across 15 directories, some of it shared. But with a bit of care, it can be done. OTOH, it might not be appropriate for a newbie.
As noted in the comments, listing the 'mkdir' command as the action for 'directories' is wrong. As also noted in the comments, there are other ways to fix the 'do not know how to make output/debug' error that results. One is to remove the dependency on the the 'directories' line. This works because 'mkdir -p' does not generate errors if all the directories it is asked to create already exist. The other is the mechanism shown, which will only attempt to create the directory if it does not exist. The 'as amended' version is what I had in mind last night - but both techniques work (and both have problems if output/debug exists but is a file rather than a directory).
Or, KISS.
DIRS=build build/bins
...
$(shell mkdir -p $(DIRS))
This will create all the directories after the Makefile is parsed.
make in, and off itself, handles directory targets just the same as file targets. So, it's easy to write rules like this:
outDir/someTarget: Makefile outDir
touch outDir/someTarget
outDir:
mkdir -p outDir
The only problem with that is, that the directories timestamp depends on what is done to the files inside. For the rules above, this leads to the following result:
$ make
mkdir -p outDir
touch outDir/someTarget
$ make
touch outDir/someTarget
$ make
touch outDir/someTarget
$ make
touch outDir/someTarget
This is most definitely not what you want. Whenever you touch the file, you also touch the directory. And since the file depends on the directory, the file consequently appears to be out of date, forcing it to be rebuilt.
However, you can easily break this loop by telling make to ignore the timestamp of the directory. This is done by declaring the directory as an order-only prerequsite:
# The pipe symbol tells make that the following prerequisites are order-only
# |
# v
outDir/someTarget: Makefile | outDir
touch outDir/someTarget
outDir:
mkdir -p outDir
This correctly yields:
$ make
mkdir -p outDir
touch outDir/someTarget
$ make
make: 'outDir/someTarget' is up to date.
TL;DR:
Write a rule to create the directory:
$(OUT_DIR):
mkdir -p $(OUT_DIR)
And have the targets for the stuff inside depend on the directory order-only:
$(OUT_DIR)/someTarget: ... | $(OUT_DIR)
All solutions including the accepted one have some issues as stated in their respective comments. The accepted answer by #jonathan-leffler is already quite good but does not take into effect that prerequisites are not necessarily to be built in order (during make -j for example). However simply moving the directories prerequisite from all to program provokes rebuilds on every run AFAICT.
The following solution does not have that problem and AFAICS works as intended.
MKDIR_P := mkdir -p
OUT_DIR := build
.PHONY: directories all clean
all: $(OUT_DIR)/program
directories: $(OUT_DIR)
$(OUT_DIR):
${MKDIR_P} $(OUT_DIR)
$(OUT_DIR)/program: | directories
touch $(OUT_DIR)/program
clean:
rm -rf $(OUT_DIR)
I've just come up with a fairly reasonable solution that lets you define the files to build and have directories be automatically created. First, define a variable ALL_TARGET_FILES that holds the file name of every file that your makefile will be build. Then use the following code:
define depend_on_dir
$(1): | $(dir $(1))
ifndef $(dir $(1))_DIRECTORY_RULE_IS_DEFINED
$(dir $(1)):
mkdir -p $$#
$(dir $(1))_DIRECTORY_RULE_IS_DEFINED := 1
endif
endef
$(foreach file,$(ALL_TARGET_FILES),$(eval $(call depend_on_dir,$(file))))
Here's how it works. I define a function depend_on_dir which takes a file name and generates a rule that makes the file depend on the directory that contains it and then defines a rule to create that directory if necessary. Then I use foreach to call this function on each file name and eval the result.
Note that you'll need a version of GNU make that supports eval, which I think is versions 3.81 and higher.
given that you're a newbie, I'd say don't try to do this yet. it's definitely possible, but will needlessly complicate your Makefile. stick to the simple ways until you're more comfortable with make.
that said, one way to build in a directory different from the source directory is VPATH; i prefer pattern rules
OS independence is critical for me, so mkdir -p is not an option. I created this series of functions that use eval to create directory targets with the prerequisite on the parent directory. This has the benefit that make -j 2 will work without issue since the dependencies are correctly determined.
# convenience function for getting parent directory, will eventually return ./
# $(call get_parent_dir,somewhere/on/earth/) -> somewhere/on/
get_parent_dir=$(dir $(patsubst %/,%,$1))
# function to create directory targets.
# All directories have order-only-prerequisites on their parent directories
# https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html#Prerequisite-Types
TARGET_DIRS:=
define make_dirs_recursively
TARGET_DIRS+=$1
$1: | $(if $(subst ./,,$(call get_parent_dir,$1)),$(call get_parent_dir,$1))
mkdir $1
endef
# function to recursively get all directories
# $(call get_all_dirs,things/and/places/) -> things/ things/and/ things/and/places/
# $(call get_all_dirs,things/and/places) -> things/ things/and/
get_all_dirs=$(if $(subst ./,,$(dir $1)),$(call get_all_dirs,$(call get_parent_dir,$1)) $1)
# function to turn all targets into directories
# $(call get_all_target_dirs,obj/a.o obj/three/b.o) -> obj/ obj/three/
get_all_target_dirs=$(sort $(foreach target,$1,$(call get_all_dirs,$(dir $(target)))))
# create target dirs
create_dirs=$(foreach dirname,$(call get_all_target_dirs,$1),$(eval $(call make_dirs_recursively,$(dirname))))
TARGETS := w/h/a/t/e/v/e/r/things.dat w/h/a/t/things.dat
all: $(TARGETS)
# this must be placed after your .DEFAULT_GOAL, or you can manually state what it is
# https://www.gnu.org/software/make/manual/html_node/Special-Variables.html
$(call create_dirs,$(TARGETS))
# $(TARGET_DIRS) needs to be an order-only-prerequisite
w/h/a/t/e/v/e/r/things.dat: w/h/a/t/things.dat | $(TARGET_DIRS)
echo whatever happens > $#
w/h/a/t/things.dat: | $(TARGET_DIRS)
echo whatever happens > $#
For example, running the above will create:
$ make
mkdir w/
mkdir w/h/
mkdir w/h/a/
mkdir w/h/a/t/
mkdir w/h/a/t/e/
mkdir w/h/a/t/e/v/
mkdir w/h/a/t/e/v/e/
mkdir w/h/a/t/e/v/e/r/
echo whatever happens > w/h/a/t/things.dat
echo whatever happens > w/h/a/t/e/v/e/r/things.dat
See https://www.oreilly.com/library/view/managing-projects-with/0596006101/ch12.html
REQUIRED_DIRS = ...
_MKDIRS := $(shell for d in $(REQUIRED_DIRS); \
do \
[[ -d $$d ]] || mkdir -p $$d; \
done)
$(objects) : $(sources)
As I use Ubuntu, I also needed add this at the top of my Makefile:
SHELL := /bin/bash # Use bash syntax
I use the makefiles in windows environment and my simple solution is as follows,
Create a target makedir and add it as a prerequisites to where ever it is required.
# Default goal
all: gccversion makedir build finalize list sizeafter completed
The makedir target is (applicable only in windows environment)
makedir:
#IF NOT EXIST $(subst /,\,$(BUILD_DIR)) mkdir $(subst /,\,$(BUILD_DIR)) 2> NULL
#IF NOT EXIST $(subst /,\,$(OUTPUT_DIR)) mkdir $(subst /,\,$(OUTPUT_DIR)) 2> NULL
#IF NOT EXIST $(subst /,\,$(DEP_DIR)) mkdir $(subst /,\,$(DEP_DIR)) 2> NUL
#IF NOT EXIST $(subst /,\,$(OBJ_DIR)) mkdir $(subst /,\,$(OBJ_DIR)) 2> NUL
$(subst /,\,$(BUILD_DIR)) converts the directory separator / to \ and
mkdir $(subst /,\,$(BUILD_DIR)) 2> NUL redirects the error if any.
src_dir := src
obj_dir := obj
build_dir := build
dirs := $(src_dir) $(obj_dir) $(build_dir) # new variable
all: $(dirs) $(other_dependencies) # added dependency (*before* any others)
$(dirs): # rule which makes missing directories
mkdir $#
Won't clutter your terminal with "cannot create directory" error messages. If the directories exist, they don't need to be built.
Works like any other dependency, only requires one rule and one variable.