Consider the following setup:
$ touch 1.src 2.src 3.src
$ cat Makefile
%.dst: %.src
#convert -o "$#" "$<"
We can compile our .src files into .dst files by running make 1.dst 2.dst 3.dst which calls the convert (just a placeholder) tool three times.
This setup is fine if there is little overhead in calling convert. However, in my case, it has a startup penalty of a few seconds for every single call. Luckily, the tool can convert multiple files in a single call while paying the startup penalty only once, i.e. convert -o '{}.dst' 1.src 2.src 3.src.
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
Edit: To be more precise, what feature I am looking for: Say that 1.dst is already newer than 1.src so it doesn't need to be recompiled. If I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
A quick and dirty way would be creating a .PHONY rule that simply converts all src files to dst files but that way I would convert every src file each and every time. Further more, specifying dst files as prerequisites in other rules would also no longer be possible.
Thanks in advance!
If you have GNU make 4.3 or above, you can use grouped targets like this:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(_DST_FILES:.dst=.src)
all: $(DST_FILES)
$(DST_FILES) &: $(SRC_FILES)
convert -o '{}.dst' $?
#touch $(DST_FILES)
If your convert is only updating some of the targets then you need the explicit touch to update the rest.
Here's a way to do it with passing a goal on the command line that might work; change DST_FILES to:
DST_FILES := $(or $(filter %.dst,$(MAKECMDGOALS)),1.dst 2.dst 3.dst)
Is there a way in GNU make to specify that multiple src files should be batched into a single call to convert?
It is possible, but messy, to write make rules for build steps that produce multiple targets with a single run of the recipe, such that the recipe is executed just once if any of the targets needs to be updated. However, you clarify that
[if] 1.dst is already newer than 1.src [, and] I run make 1.dst 2.dst 3.dst, I would like GNU make to execute convert -o '{}.dst' 2.src 3.src.
. That's a slightly different problem. You can use the $? automatic variable in a recipe to get the prerequisites that are newer than the rule's target, but for that to serve the purpose, you need a rule with a single target.
Here's one slightly convoluted way to make it work:
DST_FILES = 1.dst 2.dst 3.dst
SRC_FILES = $(DST_FILES:.dst=.src)
$(DST_FILES): dst.a
ar x $< $#
dst.a: $(SRC_FILES)
convert -o '{}.dst' $?
x='$?'; ar cr $# $${x//src/dst}
The dst.a archive serves as the one target with all the .src files as prerequisites, so as to provide a basis for use of $?. Additionally, it provides a workaround for the problem that whenever that target is updated, it becomes newer than all the then-existing .dst files: .dst files that are out of date with respect to the archive but not with respect to the corresponding .src file are extracted from the archive instead of being rebuilt from scratch.
I am working on a make-script for a hobby OS project. When playing with the script I noticed that two different variable expansions of (almost) the same variable yields different results (even though they are placed 'directly' after each other). I'll provide the important part of the makefile and the result when ran.
Makefile:
####################
# KERNEL #
####################
.PHONY: kernel
KERNEL_OBJS = $(patsubst %.c,%.o,$(wildcard kernel/*.c))
KERNEL_OBJS += $(patsubst %.asm,%.o,$(wildcard kernel/*.asm))
KERNEL_OBJS += $(DRIVER_OBJS)
KERNEL_NAME = kernel32.elf
kernel: $(KERNEL_OBJS)
#echo $^
#echo $(KERNEL_OBJS)
####################
# DRIVERS #
####################
.PHONY: drivers
DRIVER_OBJS := $(patsubst %.c,%.o,$(wildcard drivers/*/*.c))
Makefile executed via terminal in the following manner (GNU Make 4.2.1):
make kernel
And this yields the following result:
kernel/kmain.o kernel/boot.o
kernel/kmain.o kernel/boot.o drivers/vga/vga.o
The output lines are of-course from the two 'echo lines' in the kernel recipe. It is worth mentioning that all variables used in this code-snippet are used here and only here in the otherwise larger make-script. Two regular suffix rules are used to build the KERNEL_OBJS but they should not alter the output here in anyway. Apart from the suffix rules, this snippet and it's variables is completely seperated from the rest of the script.
Any ideas why the two variable expansions differ? Yours, Mikael.
The big difference between the two contexts is that the prerequisite list is expanded immediately when the makefile is parsed, but the recipe is only expanded later, when make is about to build that target.
When make first parses your makefile it finds this line:
kernel: $(KERNEL_OBJS)
It expands this variable immediately. When the variable is expanded the DRIVER_OBJS variable has not been set yet, so it is the empty string and you get this:
kernel: kernel/kmain.o kernel/boot.o
Then make finishes parsing all the makefiles and as part of that, the DRIVER_OBJS variable is set... but that doesn't matter to the above line because it's already been expanded.
Now make decides it wants to build the kernel target and in order to do that it has to expand the recipe:
#echo $^
#echo $(KERNEL_OBJS)
Here $^ is the prerequisite list: kernel/kmain.o kernel/boot.o. Now KERNEL_OBJS is expanded and now DRIVER_OBJS is set, so you get the full list.
See How make Reads a Makefile for full details on when expansion happens.
You can activate second expansion and then use it in the dependency:
.SECONDEXPANSION:
kernel: $$(KERNEL_OBJS)
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
So I have a variable with file names, however I am not sure whether these file names exist. I want to put the ones that exist into some other variable.
All this is happening in a Makefile.
Here is one of my many tries to accomplish it:
FILES is the preset variable containing the set of the files.
OUTPUT += $(foreach file, $(FILES), \
ifneq (,$(wildcard $(file))
$(file)
endif
Obviously it doesn't work for many reasons, just trying to make clear what I want to achieve.
Also I would like to avoid using $(shell *) if possible.
Thanks!
The wildcard function takes multiple targets, and unlike shell globs non-matching patterns are omitted from the output, so I would expect this to work:
OUTPUT = $(wildcard $(FILES))
I have a makefile that depending on some properties sets vpath and generates a list of source files into one variable. I need to run the makefile without compiling anything (the compilation is actually handled by a different makefile) and just see to which real files the filenames get matched depending on the vpath settings.
Option 1: Let make do its path search:
.PHONY: whichfiles
whichfiles: $(LIST_OF_SOURCE_FILES)
#echo $+
Option 2: Simulate the path search using $(wildcard):
.PHONY: whichfiles
whichfiles:
#echo $(foreach f,$(LIST_OF_SOURCE_FILES),$(firstword $(wildcard $(VPATH:%=%/$f)) not-found:$f))
Either way, "make whichfiles" will print the list of matched files.
If some of the files can't be found, option 1 will fail with "no rule to make" reporting the first file that could not be found. Option 2 will print "not-found:" for each missing file.