Here is a simplified use case of my Makefile. What I want to do is use the names of these files and create the respective .d and .hpp files. However, for some reason, I am unable to do it.
XML_DATA_FILES := a.xml b.xml c.xml d.xml
build:
#for var in $(XML_DATA_FILES); do \
echo "$(basename $$var)";\
$(eval var1 = $(basename $$var)) \
echo "$(join var1,.hpp)"; \
echo "$(join var1,.d)"; \
done
The output that I get when I run make is as follows
a.xml
var1.hpp
var1.d
b.xml
var1.hpp
var1.d
c.xml
var1.hpp
var1.d
d.xml
var1.hpp
var1.d
But what I want is a.d, a.hpp and so on for all the four xml files input.
I have already referred to this question and GNU Manual but it hasnt helped so far.How can I achieve this?
There're a number of problems here :). But, fundamentally you cannot combine make functions like basename and eval inside a shell loop like for and expect it to do anything useful. Make always expands the entire recipe for all make variables and function FIRST, then it passes the entire expanded string to the shell to run, then it waits for the shell to finish.
Consider: how would the shell, running its for loop, communicate the current value of var back up to make each time through the shell's loop so that make could run the proper functions etc.? It's just not possible.
You need to write your entire loop using only shell constructs, plus simple make variables that have the same value throughout the recipe.
However, this is useless as a makefile since you just have one target that does everything. Why not just write a shell script? There's no point to using make for this. If you want to write it the make way, you'll need to declare the targets and prerequisites and create pattern rules, like this:
XML_DATA_FILES := a.xml b.xml c.xml d.xml
OUTPUTS := $(foreach X,$(XML_DATA_FILES:.xml=),$X.d $X.hpp)
build: $(OUTPUTS)
%.d %.hpp: %.xml
echo "$*.d $*.hpp"
Of course since you don't say exactly what the real commands do I can't be sure this is correct; if you actually have two different commands, one that builds the .d file and one that builds the .hpp file, you should create two different pattern rules.
Related
I have a shell command where it outputs multiple lines. I want to store it in a variable in makefile for later processing in the target.
A simplified example:
I have this file called zfile1
#zfile1
some text
$echo 123
more text
$$$#&^$
more text
The makefile:
a:
#$(eval v1 = $(shell cat zfile1))
# need to process the variable here, example:
#echo "$(v1)"
# I want to prevent expansion of values in the file and print in multi-line
If you have GNU make 4.2 or above you can use the $(file <zfile1) function. See https://www.gnu.org/software/make/manual/html_node/File-Function.html
If you don't have a new-enough version of GNU make, you can't do it. Of course in your example there's no real need to put the contents of the file into a make variable at all: you can just have your recipe use the file itself. But maybe your real use-case isn't so simple.
ETA
You should never use either the make function eval or the make function shell in a recipe [1].
You can just write:
v1 := $(file <zfile1)
.ONESHELL:
a:
#echo "$(v1)"
You must have the .ONESHELL because otherwise each line of the recipe (after it expands into multiple lines) is considered a separate recipe line. Note that .ONESHELL is in effect for the entire make process so could cause other recipes to break if they rely on each line being invoked in a different shell.
Another option is to export the result into the environment, and use a shell variable like this:
export v1 := $(file <zfile1)
a:
#echo "$$v1"
There are probably better ways to do it but since you've only given us this example and not what you really want to do, that's about all we can say.
[1] There are times where it can be useful but if you have a sufficiently sophisticated requirement to need this you'll probably know how to use it.
I think you're making things too complicated.
Start by writing your recipes as proper self-contained shell scripts.
You can then either store the whole script in a file and run it from make, or you can include it directly in your makefile as a single logical line, as in the following:
a:
#v1=$$(< zfile1); \
echo $$v1
Note the need to "escape" the dollar sign by repeating it.
You could also use global make variables, depending on the actual logic of your real-world use.
Is there a way to force a target-rule to run as part of setting a something in a variable?
For example let's say we have a target and rule:
all_mp3s:
find / -name "*.mp3" > all_mp3s
And then a variable:
MP3S := $(file < all_mp3s)
Is there a way to make sure all_mp3s file is getting created before evaluating the MP3S variable?
There is no simple straightforward way to force a rule to be evaluated before a variable gets assigned. There are more complex ways. The following is for GNU make.
Let's first assume that you want to run the (slow) find command only if the file all_mp3s does not exist, else use its content. You can use GNU make conditionals:
ifeq ($(wildcard all_mp3s),all_mp3s)
MP3S := $(shell cat all_mp3s)
else
MP3S := $(shell $(MAKE) all_mp3s ; cat all_mp3s)
endif
all_mp3s:
find / -name "*.mp3" > $#
But I if your Makefile is more complex than this, uses MP3S several times, and what you really want is:
avoid running your super-slow find several times,
run it only if needed (and only once),
get the result in a file (all_mp3s) plus a make variable (MP3S),
MadScientist has a wonderful GNU make trick that can be used here:
MP3S = $(eval MP3S := $$(shell find / -name "*.mp3"))$(MP3S)
all_mp3s:
printf '%s\n' '$(MP3S)' > all_mp3s
.PHONY: help clean
help:
printf 'MP3 finder\n'
clean:
rm -f all_mp3s
If the MP3S recursively expanded make variable is expanded because some part of your Makefile is evaluated and needs its value (e.g. if you run make all_mp3s while all_mp3s does not exist), the find command will be run, its result stored in the variable... and the variable will be turned into a simply expanded make variable, which further expansions, if any, will reuse the same, already computed, value.
Else, if your invocation of make (e.g. make cleanor make help) does not need MP3S value, the find command will not even be run.
The all_mp3s file is generated from the value of MP3S (instead of the opposite in the other solution).
However, there is another important thing to decide: do you want to declare all_mp3s as a phony target:
.PHONY: all_mp3s
or not?
If you declare it as phony, the find command will be run once and only once each time you invoke make all_mp3s (or another goal that depends on all_mp3s). But targets depending on all_mp3s will always be rebuilt too, which is not necessarily what you want.
If you don't declare it as phony, and the file exists already, the find command will not be run at all (unless the value of MP3S is needed elsewhere in your Makefile), and the content of all_mp3s will not be updated, which is not necessarily what you want.
As you do not give enough information in your question to decide, it is up to you.
I'm trying to write a Makefile where multiple source files (in my case they are markdown) create multiple target files (pdfs). However, the target files generated have extra characters in the file name that can't be predicted (it happens to be a version number encoded in the source), but ideally the Makefile would not have to read the source itself.
So, for example:
file1.md => file1-v1.pdf
file2.md => file2-v2.pdf
...
I can calculate source name given a target name (by excluding anything after the hyphen and adding .md), but cannot calculate target name given the source.
Is it possible to write a Makefile that builds only the targets where the source have been updated?
This will be ugly, but it will work.
As it often is with Make, our problem divides into these two problems:
1. construct a list of targets
2. build them
Suppose we have five md files which map to pdf files (whose names we don't know beforehand):
file1.md => file1-v1.pdf
file2.md => file2-v1.pdf
file3.md => file3-v1.pdf
file4.md => file4-v1.pdf
file5.md => file5-v1.pdf
We can't use the real output file names as targets, because we don't know them beforehand, but we see five input files and know that we must build one output file for each. For now, a fake target name will do:
file1-dummy.pdf: file1.md
zap file1.md
When Make executes this rule, it produces the file file1-v1.pdf. The fact that it doesn't produce a file named file1-dummy.pdf is disquieting, but not a serious problem. We can turn this into a pattern rule:
%-dummy.pdf: %.md
zap $<
Then all we have to do is turn the list of existing input files (file1.md, file2.md, ...) into a list of dummy targets (file1-dummy.pdf, file2-dummy.pdf, ...), and build them. So far, so good.
But suppose some of the output files already exist. If file2-v2.pdf already exists -- and is newer than file2.md -- then we would prefer that Make not rebuild it (by attempting to build file2-dummy.pdf). In that case we would prefer that file2-v2.pdf be in the target list, with a rule that worked like this:
file2-v2.pdf: file2.md
zap $<
This is not easy to turn into a pattern rule, because Make does not handle wildcards very well, and cannot cope with multiple wildcards in a single phrase, not without a lot of clumsiness. But there is a way to write one rule that will cover both cases. First note that we can obtain the part of a variable before the hyphen with this kludge:
$(basename $(subst -,.,$(VAR)))
Armed with this, and with secondary expansion, we can write a pattern rule that will work with both cases, and construct a target list that will exploit it:
# There are other ways to construct these two lists, but this will do.
MD := $(wildcard *.md)
PDF := $(wildcard *.pdf)
PDFROOTS := $(basename $(subst -,.,$(basename $(PDF))))
MDROOTS := $(filter-out $(PDFROOTS), $(basename $(MD)))
TARGETS:= $(addsuffix -foo.pdf, $(MDROOTS)) $(PDF)
.SECONDEXPANSION:
%.pdf: $$(basename $$(subst -,., $$*)).md
# perform actions on $<
Make's algorithm always starts with the final output product and works its way backwards to the source files, to see what needs to be updated.
Therefore, you HAVE to be able to enumerate the final output product as a target name and correlate that back to the inputs that generate that output, for make to work.
This is also why make is not a great tool for building Java, for example, since the output filenames don't map easily to the input file names.
So, you must have at least one target/prerequisite pair which is derivable (for implicit rules), or state-able (for explicit rules)--that is, known at the time you write the makefile. If you don't then a marker file is your only alternative. Note you CAN add extra generated, non-derivative prerequisites (for example, in compilers you can add header files as prerequisites that are not related to the source file name), in addition to the known prerequisite.
#Beta's answer is informative and helpful, but I needed a solution (using GNU Make 4.1) that worked when the destination filename bears no resemblance to the input filename, for example, if it is generated from its content. I came up with the following, which takes every file matching *.in, and creates a file by reading the contents of the source file, appending a .txt, and using it as a filename to create. (For example, if test.in exists and contains foo, the makefile will create a foo.txt file.)
SRCS := $(wildcard *.in)
.PHONY: all
all: all_s
define TXT_template =
$(2).txt: $(1)
touch $$#
ALL += $(2).txt
endef
$(foreach src,$(SRCS),$(eval $(call TXT_template, $(src), $(shell cat $(src)))))
.SECONDARY_EXPANSION:
all_s: $(ALL)
The explanation:
The define block defines the recipe needed to make the text file from the .in file. It's a function that takes two parameters; $(1) is the .in. file and $(2) is the contents of it, or the base of the output filename. Replace touch with whatever makes the output. We have to use $$# because eval will expand everything once, but we want $# to left after this expansion. Since we have to collect all the generated targets so we known what all the make, the ALL line accumulates the targets into one variable. The foreach line goes through each source file, calls the function with the source filename and the contents of the file (i.e. what we want to be the name of the target, here you'd normally use whatever script generates the desired filename), and then evaluates the resulting block, dynamically adding the recipe to make. Thanks to Beta for explaining .SECONDARY_EXPANSION; I needed it for reasons not entirely clear to me, but it works (putting all: $(ALL) at the top doesn't work). The all: at the top depends on the secondary expansion of all_s: at the bottom and somehow this magic makes it work. Comments welcome.
maybe try this ? or something along those lines
# makefile
SRCS=$(wildcard *.md)
PDFS=$(shell printf *.pdf)
$(PDFS): $(SRCS)
command ...
the printf *.pdf is meant to either expand to the first of the pdf files if they exist, else fail if they don't and that will signal to make that it should build. if this doesn't work i suggest maybe experimenting with find, ls or other listing tools (e.g. compgen, complete), maybe even in combination with xargs to get everything on one line.
So I have a variable with file names, however I am not sure whether these file names exist. I want to put the ones that exist into some other variable.
All this is happening in a Makefile.
Here is one of my many tries to accomplish it:
FILES is the preset variable containing the set of the files.
OUTPUT += $(foreach file, $(FILES), \
ifneq (,$(wildcard $(file))
$(file)
endif
Obviously it doesn't work for many reasons, just trying to make clear what I want to achieve.
Also I would like to avoid using $(shell *) if possible.
Thanks!
The wildcard function takes multiple targets, and unlike shell globs non-matching patterns are omitted from the output, so I would expect this to work:
OUTPUT = $(wildcard $(FILES))
How can I write the contents of a makefile variable to file, without invoking a shell command?
The problem is that the contents of the variable is possible longer than the shell allows for a command (i.e. longer than MAX_ARG_STRLEN (131072) characters).
In particular, in a makefile I have a variable containing a long list of filenames to process (including their absolute pathes for out-of-source builds). Now I need to write those filenames to a (temporary) file, which I can then pass to another command.
So far, we had a rule like ($COLLATED_FILES is the variable containing the paths):
$(outdir)/collated-files.tely: $(COLLATED_FILES)
$(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" \
--author="$(AUTHOR)" $^
This breaks if COLLATED_FILES is longer than about 130000 characters, we get the error message:
make[2]: execvp: /bin/sh: Argument list too long
As a solution, we are now trying to write the contents of the variable to a file and use that file in the $(LYS_TO_TELY) command. Unfortunately, I have not yet found a way to do this without invoking the shell.
My attempts include:
$(outdir)/collated-files.list: $(COLLATED_FILES)
echo "" > $#
$(foreach f,$^,echo $f >> $#;)
But this also invokes all echo commands at once in a shell, so the shell command is just as long.
Is there any way to write the contents of $(COLLATED_FILES) to a file on disk without passing them on the command line to a shell command?
I also searched whether I could pipe the contents of the variable to the shell, but I couldn't find anything in that direction, either...
Assuming you are using GNU Make, there is the file function!
https://www.gnu.org/software/make/manual/html_node/File-Function.html
$(file op filename,text)
where op is either > or >>.
This requires GNU Make 4.0+
You could move whatever makefile code you use to build up the value of COLLATED_FILES to a trivial helper makefile, then invoke make recursively from your original makefile and use trivial shell redirection to capture the stdout of the recursive make invocation -- basically using make as a rudimentary text-processing tool in that context. For example, create a makefile called get_collated_files.mk with these contents:
COLLATED_FILES=abc
COLLATED_FILES+=def
COLLATED_FILES+=ghi
# ... etc ...
# Use $(info) to print the list to stdout. If you want each filename on a
# separate line, use this instead:
#
# $(foreach name,$(COLLATED_FILES),$(info $(name)))
$(info $(COLLATED_FILES))
all: ;##shell no-op to quell make complaints
Then, in your original makefile:
collated-files.list:
$(MAKE) -f get_collated_files.mk > $#
$(outdir)/collated-files.tely: collated-files.list
$(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" \
--author="$(AUTHOR)" --filelist=collated-files.list
This will be quite a lot more efficient than using hundreds or thousands of individual echo invocations to append to the file one path at a time.
EDIT: One final option, if you really want to have each filename on a separate line, and you have a lot of control over how COLLATED_FILES is defined:
define NL
endef
COLLATED_FILES=abc
COLLATED_FILES+=$(NL)def
COLLATED_FILES+=$(NL)ghi
$(info $(COLLATED_FILES))
all: ;##no-op
This approach allows you to again use just one call to $(info), if that's important to you for some reason.
Here's a patch to gnu make that lets you directly write a variable into a file. It creates a new 'writefile' function, similar to the existing 'info' function, except it takes a filename argument and writes to the file:
https://savannah.gnu.org/bugs/?35384
It looks to me as if you should rethink your build design-- surely there's a better way than letting a variable get this big. But here's a way to do it:
# Make sure this doesn't collide with any of your other targets.
NAMES_TO_WRITE = $(addprefix write_,$(COLLATED_FILES))
collated-files.list: $(NAMES_TO_WRITE)
write_blank:
echo "" > collated-files.list
.PHONY: $(NAMES_TO_WRITE)
$(NAMES_TO_WRITE) : write_% : write_blank
echo $* >> collated-files.list