A part of my makefile is as follow:
list1: all
for f in \
`less fetch/list1.txt`; \
do \
...
./$(BIN) $$f & \
...
done
list2: all
for f in \
`less fetch/list2.txt`; \
do \
...
./$(BIN) $$f & \
...
done
fetch/list1.txt and fetch/list2.txt contains two lists of files (path+filename), and make list1 and make list2 will respectively go through the 2 lists and run $(BIN) again the files. This works fine.
The problem is that, I have a couple of file lists as list1 and list2, and the process of the make is always the same. Does anyone know how to simplify makefile such that make listA, make list4, etc. does what they are supposed to do?
You can use a Pattern Rule:
all:
#echo "The all recipe"
list%: all
#echo "This recipe does something with $#.txt"
Output is:
$ make list1
The all recipe
This recipe does something with list1.txt
$ make list256
The all recipe
This recipe does something with list256.txt
$
I do not recommend performing scripting within makefiles. It will very often lead to arbitrary, inconsistent bugs, and other forms of frustration.
Use Make for execution control with dependencies, (as in, determining what gets executed when) but write your actual primitives (scripts or other programs) seperately, and call them from within Make.
bar: foo1 foo2 foo3
# bar is the make target. Foo1, 2, and 3 are sub-targets needed to make bar.
foo1:
fooscript1 # Written as an individual script, outside Make.
fooscript2
How about:
all:
#echo "All"
action_%:
#./$(BIN) $*
ACTION=$(patsubst %,action_%,$(shell cat $(ACT_FILE)))
actionList:
#make $(ACTION)
list%: all
#make ACT_FILE=fetch/list$*.txt actionList
Supports all list :-)
Rather than allow infinite parallelism (you were using ./$(BIN) fileName &). You can control actual parallelism using Make's built in features.
make -j8 list1
# ^ Parallelism set to 8
Related
I have two lists of files as prerequisites
input_i.xx
config_j.yy
and I need to run all of their combinations. A single one looks like this:
input1_config3.output: input1.xx config3.yy
run_script $^
Also in reality, their names are not numbered, but I already have their stems defined in INPUTS and CONFIGS. With that, I can generate all the targets together
TARGETS:=$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(input)_$(config).output))
But I have difficulty with the prerequisites. It seems I need to
get basename
split on _
add the extensions .xx and .yy
.SECONDEXPANSION
$(TARGETS): $(basename $#)
run_script $^
Can someone show me how to do that? Not sure if this the proper way, maybe a bottom-up way is easier?
make is not really suitable for keeping track of an M x N matrix of results. The fundamental problem is that you can't have two stems in a rule, so you can't say something like
# BROKEN
input%{X}_config%{Y}.output: input%{X}.xx config%{Y}.yy
As a rough approximation, you could use a recursive make rule to set a couple of parameters, and take it from there, but this is rather clumsy.
.PHONY: all
all:
$(MAKE) -$(MAKEFLAGS) X=1 Y=6 input1_config6.output
$(MAKE) -$(MAKEFLAGS) X=1 Y=7 input1_config7.output
$(MAKE) -$(MAKEFLAGS) X=2 Y=6 input2_config6.output
:
input$X_config$Y.output: input$X.xx config$Y.yy
run_script $^
It would be a lot easier if you provided a complete sample example with a complete set of targets and prerequisites and exactly what you wanted to happen.
Using .SECONDEXPANSION might work, but you're not using it correctly; please re-read the documentation. The critical aspect of .SECONDEXPANSION is that you have to escape the variables that you want to avoid expanding until the second pass. In your example you've not escaped anything, so .SECONDEXPANSION isn't actually doing anything at all here. However, as #tripleee points out it's not easy to use multiple variable values in a single target.
To do this more easily you'll probably want to use eval. Something like this:
define DECLARE
$1_$2.output: $1.xx $2.yy
TARGETS += $1_$2.output
endef
TARGETS :=
$(foreach input,$(INPUTS),$(foreach config,$(CONFIGS),$(eval $(call DECLARE,$(input),$(config)))))
$(TARGETS):
run_script $^
I have another solution using include and bash for loop.
include trees.mk
trees.mk:
#for input in $(INPUTS); do \
for config in $(CONFIGS); do \
echo $${input}_$$config.output : $${input}.xx $$config.yy; \
echo -e '\t run_scipt $$^ ';\
done \
done > $#
At the beginning, trees.mk doesn't exist. The double for loops write out the rule to the target using file redirection >$#.
I got this idea from Managing Projects with GNU Make, Third Edition By Robert Mecklenburg, on
page 56
I want to check whether a string variable contains a specified substring in the Makefile. The purpose is to clean the sub folders.
I used the below code, but it did not work.
SERVICES_LIST = A_Service B_Service C_Service #example
SPECIFIC_SERVICE_LIST = A_Service B_Service
clean:
#list='$(SERVICES_LIST)';for subdir in $$list;do \
echo "clean in $$subdir";\
if [[ "*$$subdir*" == "$(SPECIFIC_SERVICE_LIST)" ]];then\
make $$subdir clean;\
fi;\
done;\
This hasn't much to do with make, because substantially all the logic involved is expressed in the shell language. In particular, you seem to be assuming bash.
The problem is here:
if [[ "*$$subdir*" == "$(SPECIFIC_SERVICE_LIST)" ]];then\
You seem to by trying to match (make's expansion of) "$(SPECIFIC_SERVICE_LIST)" against a glob pattern formed as (make's expansion of) "*$$subdir*". But the left-hand side is quoted, so it is not interpreted as a pattern, and the == operator performs (exact) string matching, not pattern matching.
One of the main ways to apply such pattern-matching tests in the shell language is with a case construct, because the selection expressions used with it are always interpreted as globs. That might look like so in your makefile:
case "$(SPECIFIC_SERVICE_LIST)" in *$$subdir*) make $$subdir clean ;; esac
But the whole thing seems pretty non-idiomatic. Generally speaking, a makefile is tuned to the project. Even if it is dynamically generated in part or in whole, it is reasonable and appropriate to design your build system so that the clean target can do something more like this:
clean:
for subdir in $(SPECIFIC_SERVICE_LIST); do make -C $$subdir clean; done
... or maybe like this:
clean: clean_services
...
clean_services:
for subdir in $(SPECIFIC_SERVICE_LIST); do make -C $$subdir clean; done
I would make it more make way by defining a target for cleaning up any supported service and then call all required clean targets as a prerequisite to clean. This has additional advantage to make clean in parallel when running with -j option as opposed to strictly sequential shell loop.
$ cat Makefile
SERVICES_LIST = A_Service B_Service C_Service #example
SPECIFIC_SERVICE_LIST = A_Service B_Service
.PHONY: $(addsuffix -clean, $(SERVICES_LIST))
$(addsuffix -clean, $(SERVICES_LIST)): %-clean:
$(MAKE) -C $* clean
.PHONY: clean
clean: $(addsuffix -clean, $(SPECIFIC_SERVICE_LIST))
I am looking for a clean way to change a few target declarations I created in a makefile into a more functional type declaration in which I am able to pass variables and the output will remain consistent.
For example:
default: clean run1 run2
run1:
for entity in $(FLIST_01); do \
echo $entity ; \
done
run2:
for entity in $(FLIST_02); do \
echo $entity ; \
done
Ideally, I would like to remove the repetitive run target declarations and only have 1.
FLIST_01 = my_flist.txt
FLIST_02 = other.txt
default: clean run
run:
$(run_func $(FLIST_01))
$(run_func $(FLIST_02))
How do I create a custom function in make to do what run_func is supposed to be doing (the for loop reading of the file list variable passed to it?
UPDATE:
My attempt so far as been this:
run:
runfunc() { \
echo "test" \
for entity in $1; do \
echo $(entity); \
done \
}
runfunc $(FLIST_01)
runfunc $(FLIST_02)
But I get a syntax error on the do line: syntax error near unexpected token `do'
First off, unless you specify .ONESHELL, the commands in a target will be run in separate subshells. So your runfunc() declaration will run in a shell which declares the function, then exits. The next statement will run in a new shell which will know nothing about the function which was declared and then basically forgotten.
Secondly,
echo "test" \
for entity in $1;
will be expanded by make into
echo "test" for entity in ;
which obviously contains multiple errors. You mean
echo "test"; \
for entity in $$1;
to properly pass the dollar sign through to the shell. But on the whole, I would say your approach is flawed. You can refactor this to a make function like you originally hoped;
define run
for entity in $(1); do \
echo $$entity; \
done
endef
Now you can call this like
run:
$(call run,$(FLIST_01))
$(call run,$(FLIST_02))
But this particular loop can quite easily be replaced with a single shell statement.
run:
printf '%s\n' $(FLIST_01)
printf '%s\n' $(FLIST_02)
Another twopenn'orth.
First off, we want to expand $FLIST_01 for target run1, and FLIST_02 for target run2.
A clean way might be:
FLIST_run1 := a b c
FLIST_run2 := 1 2 3
.PHONY: run1 run2
run1 run2:
for entity in ${FLIST_$#}; do echo $entity; done
For a slightly more obscure solution we notice that what you are trying to do can be described with make's noddy pattern matching:
.PHONY: run1 run2
run%: run1 run2:
for entity in ${FLIST_$*}; do echo $entity; done
Here we use a static pattern rule.
In the recipe, $* expands to whatever matched the % in the pattern.
This solution looks quite clean to my jaundiced eye.
I urge you to seek out idiomatic make rather than idiomatic shell.
Your makefiles will nearly always thank you for it.
In the first instance, keep your two run targets rather than coalesce them into the same recipe.
.PHONY: run1 run2
run%: run1 run2:
for entity in ${FLIST_$*}; do echo $entity; done
.PHONY: run
run: run1
run: run2
run: ; echo $# Success
Here, when you make run for instance, make first carries out the recipe for run1, then the recipe for run2, and finally the recipe for run.
Is there any advantage to this? Sure. If you run make -j2 run then the commands for run1 and run2 will be run in parallel. The whole point of make really.
The final issue is to get rid of the shell loop, and to replace it with separate commands. That way make will check the exit code of the echo for you, and even run them in parallel if that is suitable. Sure, not helpful in this noddy example, but they could be compile commands for instance.
Usually, it is not easier to maintain shell functions inside Makefile. Main reasons:
Each shell '$' need to be escaped to '$$'
The $(...) construct has different meaning in shell vs Make.
Each line in the Makefile is executing by new shell instance, meaning that variables assignment, functions definitions, etc, are not shared between lines.
Consider few options: write shell helper script (recommended), use make user-defined functions or inline the function into the Make. For the specific case of the "echo", it might be possible to use one of the other approaches, see "Alternative Solution"
From personal experience, for any nontrivial functions, better to use shell helper script. They are much easier to develop and test.
Using Shell Helper Script
Write a small helper script "print-list.sh"
#! /bin/bash
echo "test" \
for entity ; do
echo $entity
done
And then invoke it from the target
FLIST_01 = a.txt b.txt c.txt
t1:
print-list.sh ${FLIST_01)
Inlining the shell function
As indicated above, embedding the shell function into the Makefile requires matching make rules about quoting, escapes, etc. It also require lot of effort debugging the script, as error messages are very cryptic in this setup.
Note that RUNFUNC must be included before every action lines that uses the command.
RUNFUNC = runfunc() { \
echo "test" ; \
for entity in $$*; do \
echo $$entity; \
done \
} ;
FLIST_01 = a.txt b.txt c.txt
t2:
$(RUNFUNC) runfunc ${FLIST_01)
Note that the function can be written as one liner, or using the 'define'/'endef' to simplify end-of-line escapes.
Using make user-defined functions
It is possible to create simple functions using make. This option require experience and time.
UFUNC = \
echo "test" ; \
for entity in $1; do \
echo $$entity ; \
done
FLIST_01 = a.txt b.txt c.txt
t3:
$(call UFUNC, ${FLIST_01))
Alternative Solution
Last option is to use existing construct (assuming that the only goal of the function is to convert the space-seperate list of files to new-line separated
t4:
echo ${FLIST_01} | tr " " "\n"
I want to create a makefile to be included in others, kind of as a "library", let's call it library.mak:
TARGET_FILES = a.txt b.txt c.txt d.txt
SRC_FOLDER = somewhere/else
.PHONY: all
all:
for target in ${TARGET_FILES} ; do \
echo -e "\nMaking $${target}:" ;\
${MAKE} ${MFLAGS} --no-print-directory $${target} ;\
done
.PHONY: ${TARGET_FILES}
${TARGET_FILES}:
cp ${SRC_FOLDER}/$# $#
The two rules are there to make all, as well as one specific target, respectively; using a default rule (which is the purpose of library.mak).
In my "user makefile" called Makefile, I want to then do this:
include library.mak
# special handling of c.txt
c.txt:
grep -v 'all except this' ${SRC_FOLDER}/$# > $#
As you can see, the user wants to be able to override the behaviour for some special cases. While this works, it always greets the user with the dreaded warning: overriding recipe for target and warning: ignoring old commands for target messages, even though the behaviour is as intended.
So here's the question: Can this be done in a different way that avoids these warnings, or is there a means to suppress them?
The warning says you overwrite a recipe for an explicit rule. And this is really wrong. It makes sense to use a pattern rule instead, like: %.txt: ${SRC_FOLDER}/%.txt. Then it's ok to have some explicit rule (c.txt: ${SRC_FOLDER}/c.txt) overwriting a recipe. Of course, it's not 100% the same, but, I think that should not be a problem. Anyway, putting an explicit rule into a reusable file is a crime.
Next, you constantly use "phonies" and rules w/o prerequisites even when you definitely should have them. This is bad. You're trying to make "make" work as a non-branching shell script. That's not only inefficient but also is a misuse. Basically, make can be viewed as "shell extension" capable of "branching on file timestamps in a sophisticated way". If you don't need it, don't use make at all.
Next, I see absolutely no reason to go into recursive make. Whatever, you're going to do, I believe, you can do without it. And even if you really need recursion, then write simply $(MAKE). $(MFLAGS) shoud not be used anymore (read this).
P.S. There's yet another (more flexible) option to define parameterized variables (macros). For example:
define nl :=
endef
define myrule.text
$1: $2/$1
#cp $$< $$#
endef
myrule = $(eval $(foreach foo$0,$1,$(call myrule.text,$(foo$0),$2)$(nl)))
...
# user makefile
# for everything except c.txt call predefined macro
$(call myrule,$(filter-out c.txt,$(TARGET_FILES)),$(SRC_FOLDER))
# for c.txt only
c.txt: $(SRC_FOLDER)/c.txt
grep -v 'all except this' $< >$#
But as you see, it's a bit of a "cryptic" stuff, and I don't feel like recommending it for a beginner.
How can I use $(MAKEFLAGS) (or another way of passing variables defined on the command line to sub-make) in a way that supports invocation from shell with both make VAR=val and make -args?
I need my subprojects configurable, but I hate autotools, so I'm using make variables for this, e.g. invoking from shell:
$ make USE_SSE3=1
and USE_SSE3 needs to apply to all builds in all sub-makefiles.
The manual states that:
if you do ‘make -ks’ then MAKEFLAGS gets the value ‘ks’.
Therefore I'm using -$(MAKEFLAGS) (with a dash prefix) in my Makefile.
However, that expands into invalid arguments when variables with no flags are used. If I run:
$ make FOO=bar
then sub-make gets invalid -FOO=bar. OTOH without the dash prefix variable definitions work, then but make -s, etc. don't.
Is there a syntax/variable/hack that makes passing of arguments and lone variable definitions work with sub-makefiles?
The legacy $(MKFLAGS) doesn't have the weird dash prefix problem, but it doesn't include variable definitions either. I've tried fixing the variable with $(patsubst), but that only made things worse by trimming whitespace.
I need the solution to be compatible with the outdated GNU Make 3.81 shipped with Mac OS X Mavericks.
foo:
$(MAKE) -C subproject -$(MAKEFLAGS)
$ make foo -s # MAKEFLAGS = 's'
$ make foo BAR=baz # MAKEFLAGS = 'BAR=baz'
$ make foo -j8 # MAKEFLAGS = ' --job-server=…'
You shouldn't set MAKEFLAGS at all. Why do you want to? You didn't give any reason to do so.
MAKEFLAGS is intended, really, to be an internal implementation passing arguments from a parent make to a child make. It's not intended, generally, to be modified by a makefile. About the only thing you can usefully do to it is add new flags.
If you just run the recursive make using the $(MAKE) variable rather than hardcoding make, it will Just Work:
recurse:
#$(MAKE) all FOO=bar
or whatever.
Years too late I got your answer if I got it right.
You can construct $(MAKEARGS) manually yourself like:
MAKEARGS := $(strip \
$(foreach v,$(.VARIABLES),\
$(if $(filter command\ line,$(origin $(v))),\
$(v)=$(value $(v)) ,)))
MAKEARGS := assign static
strip cleans leading and trailing whitespaces.
foreach v iterate over all variable names.
origin $(v) check if variable origin is "command line".
$(v)=$(value $(v)) output env assignment string.
Alternatively you can unpick the $(MAKEFLAGS) like:
MAKEARGS := $(wordlist 2,$(words $(MAKEFLAGS)),$(MAKEFLAGS))
MAKEFLAGS := $(firstword $(MAKEFLAGS))
Which can leave you with cleaner code for further recursions IMHO. I say this because I sometimes need to keep apart arguments and flags in certain cases. Especially as you get caught in debugging a recursion djungle.
But for any specific case one should consult the manual about recursive options processing.
Changing the $(MAKEFLAGS) can lead to unwanted malfunction.
Another useful information for the willing user could be that the $(MAKEFLAGS) variable is basically the whole argument list passed to make, not only the flag characters. So $(info MAKEFLAGS = $(MAKEFLAGS)) can give you something like:
MAKEFLAGS = rRw -- VAR=val
Cheers
To check if -B is present in make flags i do :
BB_CLOBBER := $(shell echo $(MAKEFLAGS) | grep wB)
ifeq (,$(BB_CLOBBER))
# also force clobber make if these files are missing
BB_CLOBBER := $(shell (test -e $(bb_gen)/minimal/.config && test -e $(bb_gen)/full/.config) || echo "B")
endif
bb_prepare:
ifneq (,$(BB_CLOBBER))
#rm -rf $(bb_gen)/full
...