Insert a new-line in a Makefile $(foreach ) loop - makefile

I'm compiling a large list of files in a Makefile.
my.list : ${deps}
rm -f $#
$(foreach F,$^,echo "${F}" >> $#;)
but ${deps} can be large and the generated command-line could be too large for one SHELL call. Is it possible to replace ';' by a newline '\n' ?

As already mentioned in Jonathan Wakely's answer, the straightforward answer is
define newline
endef
Interestingly enough, for all chars except newline there is an easier way to get non-printable characters into variables with some help from the shell, e.g.:
tab := $(shell printf '\011')
It won't work here, though, because the builtin shell function replaces all newlines by spaces.
The version above is still a bit fragile though, in particular when combining with automake which will silently remove the second consecutive newline from the input file, turning newline into an empty variable. To force it to keep the newline, we can extend the snippet:
blank :=
define newline
$(blank)
endef
Finally, to actually get a separate shell command for each dependency, you need to run the generated string through eval:
define generate-rule =
my.list : $(1)
rm -f $$#
$(foreach F,$$^,$(newline)$(tab)echo "${F}" >> $#)
endef
$(eval $(call generate-rule,$(deps)))

You can define a variable that expands to a newline like so:
define NEWLINE
endef
Now you can use $(NEWLINE) to expand to a newline.
This won't change anything though, the foreach will still generate a single command with embedded newlines between the statements, rather than a single command with ; between the statements.

Possibly the most straight-forward answer is to use:
#printf "\n"
You can also use this, as I commonly do, to show a description when each new make target executes. An example target:
.PHONY: clean-version
clean-version: ## Reset version back to committed version number
#printf "\n## Reset version to last committed version number ##\n\n"; \
$(GIT) checkout $(VERSION_FILE);

Replacing ';' by a carriage-return will produce a string of the same size, subject to the same problem.
"foreach" is simply a string expansion. If you want to execute a separate command for each item, you can use a for loop.
my.list : ${deps}
rm -f $#
(for F in $^; do echo $$F >> $# ; done)

Edit -- after some revisions, it looks like the only problem with my original was not due to the whitespaces, but with MAKE_O. I've fixed it in my version, but I'll mostly be removing them below.
By the way the original post was written, I'm not sure if my solution will be relevant. However, I found myself in the middle of a define already inside a foreach, and couldn't figure out how to insert a newline using any of the other answers as given.
Solution:
define BR
$(1)
endef
define MAKE_O
$(1): $(wildcard $(1:obj/%.o=src/%.cpp)); \
$(CXX) ... -c $$< \$(call BR)-o $1 ... \
\$(call BR)&& touch $$#
endef
$(foreach N,main.o,$(eval $(call MAKE_O,$(N))))
Desired output (compilation is truncated from auto-dependency generation, hence the touch):
> make obj/main.o
g++ ... -c src/main.cpp \
-o obj/main.o ... \
&& touch obj/main.o
I changed BR to perform the indentation but leave the end of the line up to you:
define BR
$(1)
$(1:%= ) #<remove this comment
endef
define MAKE_O
$(1): $(wildcard $(1:obj/%.o=src/%.cpp)); \
$(CXX) ... -c $$< $(call BR,\)-o $1 ... \
$(call BR,\)&& touch $$#
endef
$(foreach N,main.o,$(eval $(call MAKE_O,$(N))))
The markup won't help to show this, but line 2 of BR is $(1:%=_space_)_tab_ (the comment itself is not allowed.) Result:
> make -n obj/main.o
g++ obj/main.o -c \
-o obj/main.o && \
echo statement on new line
I used $(call BR,\) so that the newline was not parsed as an escape of the new line, and $(1:%=space)tab so that the tab is forced (many similar rules have been defined, like SPACE:=$(SPACE) $(SPACE) without a prior value.) The variable left of the whitespace must evaluate to something. To be clear, removing the whitespace before and after the call: ...lastword$(call BR,\)firstword... yields ...lastword\n\tfirstword..., or written out,
[^]...lastword \[$]
[^]$(call BR,\)firstword...[$]
...yields...
[^]...lastword \[$]
[^] firstword...[$]
to achieve the same (using ^,$ to denote the beginning and end of the line. Someone else will probably know how to format/annotate this better.)
My syntax highlighter is decidedly unimpressed with the 'escaped' parentheses and trailing whitespace, but the result is decent.

Related

Makefile evaluation question - foreach list doesnt evaluate before execution

Here's a snippet from a Makefile which in my environment is recursive and it appears this piece of code has a n issue and I don't understand why foreach doesn't evaluate. The variables TOOLVERSION and TOOLSDIR are assigned values based on running a Ruby script. These provide the include paths for the C_FLAGS. The target test2 produces the right result where every element of C_FLAGS seperated by space appears on a new line which is the desired result. However test1 does not evaluate TOOLVERSION and TOOLSDIR so produces garbage. If I use $(shell getenvval.rb version) for assigning TOOLVERSION and TOOLSDIR the foreach work but the value of running the script is incorrect. I thought this was because something defined in the Makefile environment doesn't get to the shell so I used export but it didn't make a difference.
So the question comes down to why does the following foreach loop not work:
#$(foreach flag, $(C_FLAGS), `echo $(flag) >> $(FILE_C1_LIST)`)
while this works:
#echo $(C_FLAGS) >> $(FILE_C2_LIST)
Appreciate any help in understanding the evaluation.
Snippet from Makefile:
export
TOOLVERSION:= `getenvval.rb version`
TOOLSDIR:= `getenvval.rb directory`
FILE_C1_LIST := test_c1.f
FILE_C2_LIST := test_c2.f
C_FILES =\
./a.c \
./b.c
C_FLAGS := \
-I$(TOOLSDIR)/$(TOOLVERSION)/tools/include \
-I./aa/include \
-I./bb/editline \
-g -DDEBUG -DPLISIM -DINCLUDE_EDITLINE -DSYS_UNIX
$(FILE_C1_LIST): $(C_FILES)
rm -f $(FILE_C1_LIST)
touch $(FILE_C1_LIST)
#(echo $(C_FLAGS) )
#$(foreach flag, $(C_FLAGS), `echo $(flag) >> $(FILE_C1_LIST)`)
#$(foreach file, $(C_FILES), `echo $(file) >> $(FILE_C1_LIST)` )
$(FILE_C2_LIST): $(C_FILES)
#rm -f $(FILE_C2_LIST)
#touch $(FILE_C2_LIST)
#echo $(C_FLAGS) >> $(FILE_C2_LIST)
#$(foreach file, $(C_FILES), `echo $(file) >> $(FILE_C2_LIST)` )
test1: $(FILE_C1_LIST)
test2: $(FILE_C2_LIST)
Your expectations are odd. You appear to try to use shell syntax in constructs which are not going to be evaluated by a shell; and even if they were, the expressions would not be syntactically valid.
Without delving too much into the details, try this instead.
printf '%s\n' $(C_FLAGS) >> $(FILE_C1_LIST)
In some more detail, your loop would create the text
`echo one >>file` `echo two >>file` `echo three >>file`
where the backticks imply that the output from echo should be the text of a command which you want the shell to execute; but of course, because of the redirection, there is no output to standard output from any of these commands. (The superficial problem of having all the commands on a single line could be worked around with a semicolon after each.)
As a further aside, there is no need to rm or touch a file you are going to overwrite. Just write it. Your recipes can be substantially simplified.
$(FILE_C1_LIST): $(C_FILES)
printf '%s\n' $(C_FLAGS) $(C_FILES) >$#
$(FILE_C2_LIST): $(C_FILES)
echo $(C_FLAGS) >$#
printf '%s\n' $(C_FILES) >>$#
The parentheses you had around the first echo would needlessly run that in a separate subshell. I'm guessing you had that purely to see what you were doing; removing the # on your recipes does that much better. You can run make silently witth make -s once you have everything debugged and sorted.

Makefile extract part of filepath as compiler flag

as i am currently working on my makefile i encountered another problem. I am using this rule as part of my building process which transforms .mid files into .s files.
$(MIDAS): $(BLDPATH)/%.s: %.mid
$(shell mkdir -p $(dir $#))
#test $($< | sed "*")
$(MID2AGB) $(MIDFLAGS) -G$($< | sed ".*mus/vcg([0-9]{3})/.*\.mid") $< $#
All .mid input files follow the same format: .mus/vcg[0-9]{3}/..mid, meaning they are stored in different directories following the naming convention vcgXXX where X can be any digit from 0-9. (Maybe my regex is even wrong for this).
When i am calling $(MID2AGB) i want to use a compiler flag -GXXX. However the XXX of this flag has to match the XXX from the input file path.
My makefile code does not work. Any idea how to fix this problem?
There is a crude but effective way to do this using Make's string manipulation tools:
# suppose the source is .mus/vcg456/Z.mid
$(MIDAS): $(BLDPATH)/%.s: %.mid
#echo $* # this gives .mus/vcg456/Z
#echo $(subst /, ,$*) # this gives .mus vcg456 Z
#echo $(word 2,$(subst /, ,$*) # this gives vcg456
#echo $(subst vcg,,$(word 2,$(subst /, ,$*)) # this gives 456

Second expansion and substitution on makefile targets

I am trying to do second expansion in a makefile with substitution.
A sample makefile:
# We have src{0..3}.md documents. Generate them with
#
# for i in src{0..3}.md; do echo "Hello in $i" > $i; done
#
# doc0.txt and doc2.txt are capitalized and doc1.txt and doc3b.txt are
# lowercased.
CAP_DOCS := doc0.txt doc2.txt
LOW_DOCS := doc1.txt doc3.txt
DOCS := $(CAP_DOCS) $(LOW_DOCS)
all: $(DOCS)
.SECONDEXPANSION:
$(CAP_DOCS): $$(#:doc%.txt=src%.md)
tr '[a-z]' '[A-Z]' < $< > $#
.SECONDEXPANSION:
$(LOW_DOCS): $$(#:doc%.txt=src%.md)
tr '[A-Z]' '[a-z]' < $< > $#
And the error I get is
$ make
Makefile:15: *** target pattern contains no `%'. Stop.
I also tried setting perc=% and replacing the % characters with $(perc) and even $$(perc) as I figured it was trying to expand them before the second expansion and failed. It didn't help.
I also couldn't get hiding the % from make with variables to work though that was also my first thought.
I was able to get $(CAP_DOCS): $$(patsubst doc%.txt,src%.md,$$#) and $(LOW_DOCS): $$(patsubst doc%.txt,src%.md,$$# working though.
I haven't tried it, but I wonder if you could just hide the substitution ref in a define or something and achieve the same effect.

multiple stems in makefile rule

I am trying to write a makefile that does something like the following:
%-foo-(k).out : %-foo-(k-1).out
# do something, e.g.
cat $< $#
i.e. there are files with arbitrary stems, then -foo-, then an integer, followed by .out. Each file depends on the one with the same name, with integer one smaller.
For instance, if the file blah/bleh-foo-1.out exists, then
make blah/bleh-foo-2.out
would work.
I could do this with multiple stems if there were such a thing... what's another way to do this sort of thing in (gnu) make?
There is no easy way to do something like this. You basically have two options: you can use auto-generated makefiles, or you can use $(eval ...). To me auto-generated makefiles are easier, so here's a solution:
SOURCELIST = blah/bleh-foo-1.out
all:
-include generated.mk
generated.mk: Makefile
for f in $(SOURCELIST); do \
n=`echo "$$f" | sed -n 's/.*-\([0-9]*\)\.out$/\1/p'`; \
echo "$${f%-foo-[0-9]*.out}-foo-`expr $$n + 1`.out: $$f ; cat $$< > $$#"; \
done > $#

multi-wildcard pattern rules of GNU Make

I want to write something like regex:
SRC:="a.dat.1 a.dat.2"
$(SRC): %.dat.%: (\\1).rlt.(\\2)
dat2rlt $^ $#
so that a.dat.1 and a.dat.2 will give a.rlt.1 and a.rlt.2.
In GNU Make info page, it says "the % can be used only once".
Is there some trick to achieve this in GNU Make?
I'm afraid what you are trying to do is not possible the way you suggest to do it, since - as you already mention - (GNU) make only allows a single stem '%', see http://www.gnu.org/software/make/manual/make.html#Pattern-Rules:
A pattern rule looks like an ordinary rule, except that its target
contains the character ‘%’ (exactly one of them).
Without it, creating such 'multi-dimensional' targets is cumbersome.
One way around this is by rebuilding the name of the dependency in the command (rather than in the dependency list):
SRC := a.dat.1 a.dat.2
all : $(SRC:%=%.dat2rlt)
%.dat2rlt :
dat2rtl $(word 1,$(subst ., ,$*)).rlt.$(word 2,$(subst ., ,$*)) $*
Of course, however, this way you would lose the dependency, it will not rebuild once the rlt has been updated.
The only way I can see to address that is by generating the rules explicitly:
SRC := a.dat.1 a.dat.2
all : $(SRC)
define GEN_RULE
$1.dat.$2 : $1.rlt.$2
dat2rtl $$< $$#
endef
$(foreach src,$(SRC),$(eval $(call GEN_RULE,$(word 1,$(subst ., ,$(src))),$(word 3,$(subst ., ,$(src))))))
Using named variables, we can write more readable code (based on answer of Paljas):
letters:=a b c
numbers:=1 2 3 4
define GEN_RULE
$(letter).dat.$(number) : $(letter).rlt.$(number)
./rlt2dat $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(GEN_RULE)) \
) \
)
We can generate SRC in a similar way. Note that using that method SRC will contain all the combinations. That may or may not be beneficial.
Building on the answer of Erzsébet Frigó, you might additionally choose to:
in the inner loop, eval not the macro itself but the result of calling it
name the macro after program you're calling, dat2rtl
in combination, allowing you to
refer to the program name using make's ${0}
define a target, ${0}s (expanding to dat2rts - note the pluralization) with preconditions of all combinations of letters and numbers on which dat2r2 was called
Like this:
letters:=a b c
numbers:=1 2 3 4
define rlt2dat
${0}s::$(letter).dat.$(number)
$(letter).dat.$(number): $(letter).rlt.$(number)
./${0} $$< $$#
endef
$(foreach number,$(numbers), \
$(foreach letter,$(letters), \
$(eval $(call rlt2dat))))
allowing you to build all rlt2dat targets as:
make rlt2dats
For the limited example you gave, you can use a pattern with one %.
SRC := a.dat.1 a.dat.2
${SRC}: a.dat.%: a.rlt.%
dat2rlt $^ $#
$* in the recipe will expand to whatever the % matched.
Note that the "s around your original macro are definitely wrong.
Have a look at .SECONDEXPANSION in the manual for more complicated stuff (or over here).

Resources