I am trying to write a makefile that does something like the following:
%-foo-(k).out : %-foo-(k-1).out
# do something, e.g.
cat $< $#
i.e. there are files with arbitrary stems, then -foo-, then an integer, followed by .out. Each file depends on the one with the same name, with integer one smaller.
For instance, if the file blah/bleh-foo-1.out exists, then
make blah/bleh-foo-2.out
would work.
I could do this with multiple stems if there were such a thing... what's another way to do this sort of thing in (gnu) make?
There is no easy way to do something like this. You basically have two options: you can use auto-generated makefiles, or you can use $(eval ...). To me auto-generated makefiles are easier, so here's a solution:
SOURCELIST = blah/bleh-foo-1.out
all:
-include generated.mk
generated.mk: Makefile
for f in $(SOURCELIST); do \
n=`echo "$$f" | sed -n 's/.*-\([0-9]*\)\.out$/\1/p'`; \
echo "$${f%-foo-[0-9]*.out}-foo-`expr $$n + 1`.out: $$f ; cat $$< > $$#"; \
done > $#
Related
I am currently using a makefile for cocotb, similar to this.
That consists of a bunch of variable definitions followed by and
include $(shell cocotb-config --makefiles)/Makefile.sim
Where the final commands will be executed.
I run it in Visual studio code, the simulator output message format is (%file,%line|%column). I want them to be shown as (%file:%line:%col). I can easily do it piping the output to sed.
make MODULE=x TESTCASE=y | sed -e 's/,\([0-9]\+\)|\([0-9]\+\)):/:\1:\2):/g'
I was expecting to be able to change the make file so that the output will be edited before shown
Something like
...
.DEFAULT:
make -C $PWD -f $(shell cocotb-config --makefiles)/Makefile.sim \
(all variables/environment) (rule) \
| sed -e 's/,\([0-9]\+\)|\([0-9]\+\)):/:\1:\2):/g
Where "magic rule" would apply the specified rule and pipe its output to sed.
How can I achieve this?
Thank you.
Given your example, it seems like the .DEFAULT target would do what you want:
.DEFAULT:
$(MAKE) -f $$(cocotb-config --makefiles)/Makefile.sim $# \
| sed -e 's/,\([0-9]\+\)|\([0-9]\+\)):/:\1:\2):/g
Edit (from the OP)
The default works when the rule is given, e.g. if I invoke make sim instead of make. For this I can create a default-delegate rule.
More importantly, I have to pass all the variables to the sub make, that can be achieved by using export by itself.
Then, replacing the include command but the following snippets, works for the more common cases.
export
default-delegate: sim
.DEFAULT:
$(MAKE) -f $$(cocotb-config --makefiles)/Makefile.sim $# \
| sed -e 's/,\([0-9]\+\)|\([0-9]\+\)):/:\1:\2):/g'
I have two lists of files as prerequisites
input_i.xx
config_j.yy
and I need to run all of their combinations. A single one looks like this:
input1_config3.output: input1.xx config3.yy
run_script $^
Also in reality, their names are not numbered, but I already have their stems defined in INPUTS and CONFIGS. With that, I can generate all the targets together
TARGETS:=$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(input)_$(config).output))
But I have difficulty with the prerequisites. It seems I need to
get basename
split on _
add the extensions .xx and .yy
.SECONDEXPANSION
$(TARGETS): $(basename $#)
run_script $^
Can someone show me how to do that? Not sure if this the proper way, maybe a bottom-up way is easier?
make is not really suitable for keeping track of an M x N matrix of results. The fundamental problem is that you can't have two stems in a rule, so you can't say something like
# BROKEN
input%{X}_config%{Y}.output: input%{X}.xx config%{Y}.yy
As a rough approximation, you could use a recursive make rule to set a couple of parameters, and take it from there, but this is rather clumsy.
.PHONY: all
all:
$(MAKE) -$(MAKEFLAGS) X=1 Y=6 input1_config6.output
$(MAKE) -$(MAKEFLAGS) X=1 Y=7 input1_config7.output
$(MAKE) -$(MAKEFLAGS) X=2 Y=6 input2_config6.output
:
input$X_config$Y.output: input$X.xx config$Y.yy
run_script $^
It would be a lot easier if you provided a complete sample example with a complete set of targets and prerequisites and exactly what you wanted to happen.
Using .SECONDEXPANSION might work, but you're not using it correctly; please re-read the documentation. The critical aspect of .SECONDEXPANSION is that you have to escape the variables that you want to avoid expanding until the second pass. In your example you've not escaped anything, so .SECONDEXPANSION isn't actually doing anything at all here. However, as #tripleee points out it's not easy to use multiple variable values in a single target.
To do this more easily you'll probably want to use eval. Something like this:
define DECLARE
$1_$2.output: $1.xx $2.yy
TARGETS += $1_$2.output
endef
TARGETS :=
$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(eval $(call DECLARE,$(input),$(config)))))
$(TARGETS):
run_script $^
I have another solution using include and bash for loop.
include trees.mk
trees.mk:
#for input in $(INPUTS); do \
for config in $(CONFIGS); do \
echo $${input}_$$config.output : $${input}.xx $$config.yy; \
echo -e '\t run_scipt $$^ ';\
done \
done > $#
At the beginning, trees.mk doesn't exist. The double for loops write out the rule to the target using file redirection >$#.
I got this idea from Managing Projects with GNU Make, Third Edition By Robert Mecklenburg, on
page 56
I'm compiling a large list of files in a Makefile.
my.list : ${deps}
rm -f $#
$(foreach F,$^,echo "${F}" >> $#;)
but ${deps} can be large and the generated command-line could be too large for one SHELL call. Is it possible to replace ';' by a newline '\n' ?
As already mentioned in Jonathan Wakely's answer, the straightforward answer is
define newline
endef
Interestingly enough, for all chars except newline there is an easier way to get non-printable characters into variables with some help from the shell, e.g.:
tab := $(shell printf '\011')
It won't work here, though, because the builtin shell function replaces all newlines by spaces.
The version above is still a bit fragile though, in particular when combining with automake which will silently remove the second consecutive newline from the input file, turning newline into an empty variable. To force it to keep the newline, we can extend the snippet:
blank :=
define newline
$(blank)
endef
Finally, to actually get a separate shell command for each dependency, you need to run the generated string through eval:
define generate-rule =
my.list : $(1)
rm -f $$#
$(foreach F,$$^,$(newline)$(tab)echo "${F}" >> $#)
endef
$(eval $(call generate-rule,$(deps)))
You can define a variable that expands to a newline like so:
define NEWLINE
endef
Now you can use $(NEWLINE) to expand to a newline.
This won't change anything though, the foreach will still generate a single command with embedded newlines between the statements, rather than a single command with ; between the statements.
Possibly the most straight-forward answer is to use:
#printf "\n"
You can also use this, as I commonly do, to show a description when each new make target executes. An example target:
.PHONY: clean-version
clean-version: ## Reset version back to committed version number
#printf "\n## Reset version to last committed version number ##\n\n"; \
$(GIT) checkout $(VERSION_FILE);
Replacing ';' by a carriage-return will produce a string of the same size, subject to the same problem.
"foreach" is simply a string expansion. If you want to execute a separate command for each item, you can use a for loop.
my.list : ${deps}
rm -f $#
(for F in $^; do echo $$F >> $# ; done)
Edit -- after some revisions, it looks like the only problem with my original was not due to the whitespaces, but with MAKE_O. I've fixed it in my version, but I'll mostly be removing them below.
By the way the original post was written, I'm not sure if my solution will be relevant. However, I found myself in the middle of a define already inside a foreach, and couldn't figure out how to insert a newline using any of the other answers as given.
Solution:
define BR
$(1)
endef
define MAKE_O
$(1): $(wildcard $(1:obj/%.o=src/%.cpp)); \
$(CXX) ... -c $$< \$(call BR)-o $1 ... \
\$(call BR)&& touch $$#
endef
$(foreach N,main.o,$(eval $(call MAKE_O,$(N))))
Desired output (compilation is truncated from auto-dependency generation, hence the touch):
> make obj/main.o
g++ ... -c src/main.cpp \
-o obj/main.o ... \
&& touch obj/main.o
I changed BR to perform the indentation but leave the end of the line up to you:
define BR
$(1)
$(1:%= ) #<remove this comment
endef
define MAKE_O
$(1): $(wildcard $(1:obj/%.o=src/%.cpp)); \
$(CXX) ... -c $$< $(call BR,\)-o $1 ... \
$(call BR,\)&& touch $$#
endef
$(foreach N,main.o,$(eval $(call MAKE_O,$(N))))
The markup won't help to show this, but line 2 of BR is $(1:%=_space_)_tab_ (the comment itself is not allowed.) Result:
> make -n obj/main.o
g++ obj/main.o -c \
-o obj/main.o && \
echo statement on new line
I used $(call BR,\) so that the newline was not parsed as an escape of the new line, and $(1:%=space)tab so that the tab is forced (many similar rules have been defined, like SPACE:=$(SPACE) $(SPACE) without a prior value.) The variable left of the whitespace must evaluate to something. To be clear, removing the whitespace before and after the call: ...lastword$(call BR,\)firstword... yields ...lastword\n\tfirstword..., or written out,
[^]...lastword \[$]
[^]$(call BR,\)firstword...[$]
...yields...
[^]...lastword \[$]
[^] firstword...[$]
to achieve the same (using ^,$ to denote the beginning and end of the line. Someone else will probably know how to format/annotate this better.)
My syntax highlighter is decidedly unimpressed with the 'escaped' parentheses and trailing whitespace, but the result is decent.
I am attempting to do a data pipeline with a Makefile. I have a big file that I want to split in smaller pieces to process in parallel. The number of subsets and the size of each subset is not known beforehand. For example, this is my file
$ for i in {1..100}; do echo $i >> a.txt; done
The first step in Makefile should compute the ranges,... lets make them fixed for now
ranges.txt: a.txt
or i in 0 25 50 75; do echo $$(($$i+1))'\t'$$(($$i+25)) >> $#; done
Next step should read from ranges.txt, and create a target file for each range in ranges.txt, a_1.txt, a_2.txt, a_3.txt, a_4.txt. Where a_1.txt contains lines 1 through 25, a_2.txt lines 26-50, and so on... Can this be done?
You don't say what version of make you're using, but I'll assume GNU make. There are a few ways of doing things like this; I wrote a set of blog posts about metaprogramming in GNU make (by which I mean having make generate its own rules automatically).
If it were me I'd probably use the constructed include files method for this. So, I would have your rule above for ranges.txt instead create a makefile, perhaps ranges.mk. The makefile would contain a set of targets such as a_1.txt, a_2.txt, etc. and would define target-specific variables defining the start and stop values. Then you can -include the generated ranges.mk and make will rebuild it. One thing you haven't described is when you want to recompute the ranges: does this really depend on the contents of a.txt?
Anyway, something like:
.PHONY: all
all:
ranges.mk: a.txt # really? why?
for i in 0 25 50 75; do \
echo 'a_$$i.txt : RANGE_START := $$(($$i+1))'; \
echo 'a_$$i.txt : RANGE_END := $$(($$i+25))'; \
echo 'TARGETS += a_$$i.txt'; \
done > $#
-include ranges.mk
all: $(TARGETS)
$(TARGETS) : a.txt # seems more likely
process --out $# --in $< --start $(RANGE_START) --end $(RANGE_END)
(or whatever command; you don't give any example).
Suppose I have a following rule in the Makefile:
%.foo: %.bar %.spam %.bot
<tab> echo "hello1" > $<
how can I also echo "hello2" into the second dependency (but not the .bot file), i.e. the .spam file? Thanks
%.foo: %.bar %.spam %.bot
echo "hello1" > $<
echo hello2 > $(word 2,$^)
(Note that > overwrites, at least in the shells I know, which makes the whole exercise pretty pointless. To append, use >>.)
What you're asking is antithetical to the normal operation of make: a rule should modify the files named on the left of the colon, not the files on the right of the colon. You haven't given much context here, so it's hard to give you more specific advise than that.
As far as the particular question you've asked, you could use something like this:
%.foo: %.bar %.spam %.bot
echo "hello1" > $<
echo "hello2" > $*.spam
This uses the $* automatic variable, which is defined as the part of the filename that matched the % character in a pattern rule.