I have some rules in a makefile like below
...
$(MODEL_DIR)/2001.model : $(DATA_DIR)/2000.dat
python fit.py --year 2001 --data-dir $(DATA_DIR) --output-dir $(MODEL_DIR)
$(MODEL_DIR)/2002.model : $(DATA_DIR)/2000.dat $(DATA_DIR)/2001.dat
python fit.py --year 2002 --data-dir $(DATA_DIR) --output-dir $(MODEL_DIR)
$(MODEL_DIR)/2003.model : $(DATA_DIR)/2000.dat $(DATA_DIR)/2001.dat $(DATA_DIR)/2002.dat
python fit.py --year 2003 --data-dir $(DATA_DIR) --output-dir $(MODEL_DIR)
$(MODEL_DIR)/2004.model : $(DATA_DIR)/2000.dat $(DATA_DIR)/2001.dat $(DATA_DIR)/2002.dat $(DATA_DIR)/2003.dat
python fit.py --year 2004 --data-dir $(DATA_DIR) --output-dir $(MODEL_DIR)
...
This basically specifies that the model for a year depends on data from previous years having been created.
I would like a simpler pattern based rule to make this less prone to copy paste errors. I tried
$(MODEL_DIR)/%.model : $(foreach num,$(shell seq 2000 %),$(DATA_DIR)/$(num).dat)
python fit.py --year $* --data-dir $(DATA_DIR) --output-dir $(MODEL_DIR)
But I get the error
seq: invalid floating point argument: %
indicating that the wildcard is not being expanded in the shell command.
You're very close, but such function calls are expanded before matching the pattern rule to a target, so % has no value to be fed to seq. The way to do this kind of calculation in the prerequisite list is by using secondary expansion, and escaping each variable with an extra '$' to prevent it being expanded in the first round:
.SECONDEXPANSION:
$(MODEL_DIR)/%.model : $$(foreach num,$$(shell seq 2000 %),$(DATA_DIR)/$$(num).dat)
...
Finally, notice that seq will give all years up to and including the year of the model, so we must remove the last prerequisite:
.SECONDEXPANSION:
$(MODEL_DIR)/%.model : $$(filter-out $$*.dat,$$(foreach num,$$(shell seq 2000 %),$(DATA_DIR)/$$(num).dat))
...
Related
I have a project which use makefile to control vagrant, I want to put the vagrant parameter into the makefile, such as cpu, memory, ip, hostname, forwarded_port and the like. I find a way that vagrantfile read yaml file to parameterize vagrantfile. So makefile needs a target to read all the user option variables and write them to config.yaml as key-value pairs.
The sample is as follows
# === BEGIN USER OPTIONS ===
BOX_OS ?= fedora
# Box setup
#BOX_IMAGE
# Disk setup
DISK_COUNT ?= 1
DISK_SIZE_GB ?= 25
# VM Resources
MASTER_CPUS ?= 2
MASTER_MEMORY_SIZE_GB ?= 2
NODE_CPUS ?= 2
NODE_MEMORY_SIZE_GB ?= 2
NODE_COUNT ?= 2
# Network
MASTER_IP ?= 192.168.26.10
NODE_IP_NW ?= 192.168.26.
POD_NW_CIDR ?= 10.244.0.0/16
...
...
# === END USER OPTIONS ===
The echo command does achieve it
# Makefile
envInit:
#echo "POD_NW_CIDR : \"$(POD_NW_CIDR)\"" > ${FILECWD}/configs.yaml
But too many variables can be too complex.
Is there a way to bulk read variables and their values and write them to a yml file
I would very appreciate it if you guys can tell me how to achieve it that bulk read variables and their values and write them to a yml file.
Define all user options (along with the default values) as a list, so that they are iterable:
# list of user options with default values
userOptions = \
BOX_OS=2 \
DISK_COUNT=1 \
MASTER_IP=192.168.26.10
# replace each default value with the env value, if any
userOptionValues = $(foreach i, $(userOptions), \
$(word 1, $(subst =, ,$i))=$(or \
$($(word 1, $(subst =, ,$i))), $($(word 1, $(subst =, ,$i))), $(word 2, $(subst =, ,$i))))
# write the yaml file
envInit:
# empty the file
#printf "" > configs.yaml
# write a line for each option
#for i in $(userOptionValues); do \
printf "%s : %s\n" "$$(printf $$i | cut -d= -f1)" "$$(printf $$i | cut -d= -f2)" >> configs.yaml; \
done
#flyx Thank you for you answer, your code does work great. But I seem to have found a more convenient way, and I've partially modified it.
printvars:
#echo$(foreach V,$(sort $(.VARIABLES)), \
$(if $(filter-out environment% default automatic,$(origin $V)),$(info $V: $($V))))
But there is still a gap between achieving the goal.
# the Makefile test file
FILECWD = $(shell pwd)
# === BEGIN USER OPTIONS ===
CLOUD_IP ?= 192.168.79.222
CLOUD_NAME ?= cloud
CLOUD_CPU ?= 6
CLOUD_MEMORY ?= 8
# === END USER OPTIONS ===
printvars:
#echo$(foreach V,$(sort $(.VARIABLES)), \
$(if $(filter-out environment% default automatic,$(origin $V)),$(info $V: $($V))))
make printvars's output contains a number of other variables
$ make printvars
.DEFAULT_GOAL: printvars
CLOUD_IP: 192.168.79.222
CLOUD_MEMORY: 8
CLOUD_NAME: cloud
CURDIR: /testmakecreateyml0930
FILECWD: /testmakecreateyml0930
GNUMAKEFLAGS:
MAKEFILE_LIST: Makefile
MAKEFLAGS:
SHELL: /bin/sh
And it can only be printed and not exported to the yaml file.This is only one step away from success.
I would appreciate it if you could help me modify it to achieve my goal
You can write directly to a file with GNUmakes $(file) function:
define newline :=
$(strip)
$(strip)
endef
space := $(strip) $(strip)#
-never-matching := ¥# character 165, this is used as a list element that should never appear as a real element
option-names = $(subst $(-never-matching),,$(filter $(-never-matching)%,$(subst $(-never-matching)$(space),$(-never-matching),$(-never-matching)$(strip $(subst $(newline), $(-never-matching),$1)))))
# define your user options in as many separate parts as you like, spaces and empty lines included:
define USER_OPTIONS +=
a = spaces are no problem
b = "neither nearly all 'other' characters: 8&)("
endef
define USER_OPTIONS +=
c = baz baf
d = foobar
endef
# make all definition make variables verbatim
$(eval $(USER_OPTIONS))
YAML_FORMAT := $(foreach name,$(call option-names,$(USER_OPTIONS)),$(newline)$(name) : $($(name)))
# write the file. Warning: this happens before any rule is run!
$(file >test.yaml,$(YAML_FORMAT))
$(info $(foreach name,$(call option-names,$(USER_OPTIONS)),<$(name) : $($(name))> ))
The trick lies in the clustering of all relevant user option variables in one multi-line make variable. The function option-names pulls all identifiers from that variable into a separate list.
I took the newline etc. character definitions from the GNUmake table toolkit which has many functions for "programmatic" make.
I have the example makefile below. I am trying to create a directory structure with multiple substitutions, and it seems to be doing a cartesian product when I want by-index substitution:
genomes = C57B6NJ AKJ
GENOME_DIRS = ${genomes:%=${BASE_DATA_DIR}/genomes/%}
TWO_BITS = ${genomes:%=${GENOME_DIRS}/%.2bit}
all:
#echo ${TWO_BITS}
# OUTPUT
# Thu Apr 30 16:35 ~ $make all
# /genomes/C57B6NJ /genomes/AKJ/C57B6NJ.2bit /genomes/C57B6NJ /genomes/AKJ/AKJ.2bit
But the output I desire is
/genomes/C57B6NJ/C57B6NJ.2bit /genomes/AKJ/AKJ.2bit
How can I get this kind of positional substitution?
Your second substitution is asking make to replace each word in genomes with the entire contents of GENOME_DIRS followed by the original word followed by the string .2bit which seems to be exactly what it is doing.
You appear to want to be adding .2bit to the end of the values in GENOME_DIRS in which case you should just do that.
Either with:
TWO_BITS = $(GENOME_DIRS:%=%.2bit)
or with:
TWO_BITS = $(addsuffix .2bit,$(GENOME_DIRS))
I got confused with Makefile. I am trying to run a simple command in the Makefile but it gives me the error "/bin/bash: line 3: :=: command not found". I am using shell to run this makefile
This is my part of my Makefile:
all:
vlog Benchmarks/$(NAME)/Syn/*.v
$(eval tux_number := 1)
$(eval range := 1)
$(eval ssh_log := 255)
echo "Start Range: ${range}"
echo "tux-number: ${tux_number}"
while [[ $$range -le 50 ]] ; do \
ssh -l yazdanbakhsh tux-$(tux_number).cae.wisc.edu exit ; \
echo "range: ${range}" ; \
eval $$range := $$((${range}+1)) ; \
done
Thanks
all:
#range=1; \
while [ $$range -le 10 ] ; \
do echo Range: $$range; \
let range=range+1 ; \
done;
Note that the whitespace in front of #range... is the only TAB.
Just to fix your obvious problems with Makefile syntax, here is an attempt at refactoring your attempt into valid code.
tux_number := 1
ssh_log := 255 # not used anywhere
all:
vlog Benchmarks/$(NAME)/Syn/*.v
echo "Start Range: 1" # This is probably no longer very useful output
echo "tux-number: ${tux_number}"
range=1; while [ $$range -le 50 ] ; do \
ssh -l yazdanbakhsh tux-$(tux_number).cae.wisc.edu exit ; \
echo "range: $$range" ; \
range=$$(expr "$$range + 1); \
done
Notice how tux_number and ssh_log are Makefile variables, while range only exists in the shell which executes the while loop. I have avoided the Bashisms in order to make this portable. (If portability is not important, you might want to refactor it back to Bash syntax and use for ((range=1; range<=50; range++)); do... instead.)
Your use of eval is misguided. As you can see, I simply lifted out the Makefile variables outside the recipe where they don't belong. What you were doing was (1) have Make evaluate the expression range := 1 (which evaluates to itself) and (2) use the output as a shell command in a recipe. Since it's not a valid shell command, you got the syntax error from Bash. Without further ado, I'll just take the easy way out here and say that eval is a complex subject, and until you get more experience with Make, it's probably just best to forget that it exists.
In order to properly make use of Make's facilities, I would make this parallelizable, i.e. split it up into 50 individual targets. This is a bit clumsy (there's probably a better way to define range here), but at least it should illustrate a number of differences to your approach. (If you don't insist on having range count up from 1, making it zero-based would make this a little less clumsy. This exploits the fact that the empty string is harmless in a shell snippet, so we can use it instead of a zero prefix. Again, this could be simplifed if you don't care about the human readability of the range index.)
digits := 0 1 2 3 4 5 6 7 8 9
deca := "" 1 2 3 4
range := $(filter-out ""0,$(foreach d,$(deca),$(foreach i,$(digits),$d$i))) 50
# Or, at the expense of an external process,
# range := $(shell perl -le 'print $$_ for 1..50')
.PHONY: all
all: $(patsubst %,ssh-%,$(range))
.PHONY: ssh-%
ssh-%:
ssh -l yazdanbakhsh tux-$(tux_number).cae.wisc.edu exit
echo "range: $*"
This can be run with something like make -j 5 to execute these in parallel batches of five, for example.
Incidentally, the commented-out $(shell ...) call might be the actual answer to your question, if what you really wanted to do was to use Make to drive an external program to calculate something for you.
I would like to generate a number of files using GNU Make using the following recipe.
ina_as%.dat: ina_driver.m ina_as$(word 1,$(subst _epsi, , %)).m
echo "modelType = '$(word 1,$(subst _epsi, , $*))'; ofile = '$#'; epsi = '$(word 2,$(subst _epsi, , $*))';" | cat - $< | nohup matlab -nodesktop -nosplash
The targets are in a format -- ina_as%d_epsi%.2f.dat (e.g. ina_as1_epsi0.50.dat) and the second prerequisite is ina_as%d.m (e.g. ina_as1.m) (notice, the second part _epsi%.2f missing in the prerequisite file name).
I have tried several combination for the implicit rule ($, $$, $(eval $*) etc.), but it still does not work. I think it could be because Make could not understand the functions ( '$(word 1,$(subst _epsi, , %))' ) in the dependency definition.
There is any way to overcome this problem?
Thanks.
Questions like this come up from time to time. The short answer is that Make simply can't do this in a clean way; the text manipulation statements expand before executing any rule (i.e. before % has any value), and Make doesn't handle wildcards (or regular expressions) very well.
The longer answer is that it can be done, but only by resorting to one kludge or another. If your version of Make supports SECONDEXPANSION, you can do it this way:
.SECONDEXPANSION:
ina_as%.dat: ina_as$$(word 1,$$(subst _, ,%)).m
#echo "modelType = '$(word 1,$(subst _epsi, , $*))'; ofile = '$#'; epsi\
= '$(word 2,$(subst _epsi, , $*))';" | cat - $< | nohup matlab -nodesktop\
-nosplash
If not, you can resort to recursive Make (useful sometimes, no matter what they say):
ina_as%.dat :
#$(MAKE) dummy MODELTYPE=`echo $* | sed "s/_.*//"` EPSI=`echo $* | sed \
"s/.*_epsi//"`
dummy: ina_as$(MODELTYPE).m
#echo "modelType = $(MODELTYPE); ofile = ina_as$(MODELTYPE)_epsi$(EPSI)\
; epsi = $(EPSI);" | cat - ina_as$(MODELTYPE).m | nohup matlab -nodesktop\
-nosplash
For variable assignment in Make, I see := and = operator. What's the difference between them?
Simple assignment :=
A simple assignment expression is evaluated only once, at the very first occurrence.
For example, if CC :=${GCC} ${FLAGS} during the first encounter is evaluated to gcc -W then
each time ${CC} occurs it will be replaced with gcc -W.
Recursive assignment =
A Recursive assignment expression is evaluated everytime the variable is encountered
in the code. For example, a statement like CC = ${GCC} {FLAGS} will be evaluated only when
an action like ${CC} file.c is executed. However, if the variable GCC is reassigned i.e
GCC=c++ then the ${CC} will be converted to c++ -W after the reassignment.
Conditional assignment ?=
Conditional assignment assigns a value to a variable only if it does not have a value
Appending +=
Assume that CC = gcc then the appending operator is used like CC += -w
then CC now has the value gcc -W
For more check out these tutorials
This is described in the GNU Make documentation, in the section titled 6.2 The Two Flavors of Variables
.
In short, variables defined with := are expanded once, but variables defined with = are expanded whenever they are used.
For me, the best way to see it in practice is during this Makefile snippet:
Simple assignment
XX := $(shell date) // date will be executed once
tt:
#echo $(XX)
$(shell sleep 2)
#echo $(XX)
Running
make tt
Will produce:
sex 22 jan 2021 14:56:08 -03
sex 22 jan 2021 14:56:08 -03
( Same value )
Expanded assignment
XX = $(shell date) // date will be executed every time you use XX
tt:
#echo $(XX)
$(shell sleep 2)
#echo $(XX)
Running
make tt
Will produce:
sex 22 jan 2021 14:56:08 -03
sex 22 jan 2021 14:56:10 -03
Different values
From http://www.gnu.org/software/make/manual/make.html#Flavors:
= defines a recursively-expanded variable. := defines a simply-expanded variable.
This is an old question but this example helps me understand the difference whenever I forget.
Running make with the following Makefile will instantly exit:
a = $(shell sleep 3)
Running make with the following Makefile will sleep for 3 seconds, and then exit:
a := $(shell sleep 3)
In the former Makefile, a is not evaluated until it's used elsewhere in the Makefile, while in the latter a is evaluated immediately even though it's not used.
Recursive assignment = is evaluated everytime it is used, but not in the order of when it is encountered among the recipe commands, but rather before running any recipe command.
Based on the following example:
default: target1 target2
target1 target2:
#echo "Running at: `gdate +%s.%N`"
#echo "Simple assignment: $(SIMPLE_ASSIGNMENT)"
#echo "Recursive assignment: $(RECURSIVE_ASSIGNMENT)"
sleep 1
#echo "Running at: `gdate +%s.%N`"
#echo "Simple assignment: $(SIMPLE_ASSIGNMENT)"
#echo "Recursive assignment: $(RECURSIVE_ASSIGNMENT)"
#echo
SIMPLE_ASSIGNMENT := $(shell gdate +%s.%N)
RECURSIVE_ASSIGNMENT = $(shell gdate +%s.%N)
Outputs:
❯ make
Running at: 1645056840.980488000
Simple assignment: 1645056840.949181000
Recursive assignment: 1645056840.958590000
sleep 1
Running at: 1645056842.008998000
Simple assignment: 1645056840.949181000
Recursive assignment: 1645056840.969616000
Running at: 1645056842.047367000
Simple assignment: 1645056840.949181000
Recursive assignment: 1645056842.027600000
sleep 1
Running at: 1645056843.076696000
Simple assignment: 1645056840.949181000
Recursive assignment: 1645056842.035901000