Makefile: do operation on files with specific extension from a variable - makefile

I'm working on a Makefile which needs to be able to do the following:
From a user given variable, SRVS, containing file names, e.g.,
SRVS=Test1.java,test2.c,test3.c,test4.java,test5.c
produce a list of all files with a certain extension to be used by a foreach loop.
This list should not contain the extension anymore.
Currently, I can parse and get all the files into a usable list, but am unable to do so for an extension.
What I have:
$(foreach SRV, $(shell echo $(SRVS) | tr ',' ' '), \
CLASSPATH=$(SELF)/default_runtime javac $(UJ_DIR)/services/java/$(SRV).java && \
find $(UJ_DIR)/services/java/ -iname "$(SRV).class" -type f >> $(SELF)/files && \
) true
Which will take a list of SRVS and produce list usable by foreach and the code therein. For instance, the above example would result in "test1 test2 test3 test4 test5" to be used by the foreach loop
I'd like now to specify the extension, for instance .c, so that the above example would result in "test2 test3 test5".
Can you help me out?

First, you do not need to call shell (this is uselessly slow) because there are make built-in functions that do what you want. If, in the definition of variable SRVS you really cannot separate your file names with spaces (the standard make word separator) instead of commas, subst can do it for you. But there is a little trick because the comma is itself the arguments separator of make functions:
comma := ,
SRVS1 := $(subst $(comma), ,$(SRVS))
Next, filter is the make function that allows to select files by extension:
SRVS2 := $(filter %.c,$(SRVS1))
And finally, basename removes the suffix:
SRVS3 := $(basename $(SRVS2))
Demo:
$ cat Makefile
SRVS=Test1.java,test2.c,test3.c,test4.java,test5.c
comma := ,
SRVS1 := $(subst $(comma), ,$(SRVS))
SRVS2 := $(filter %.c,$(SRVS1))
SRVS3 := $(basename $(SRVS2))
all:
$(info SRVS1 = $(SRVS1))
$(info SRVS2 = $(SRVS2))
$(info SRVS3 = $(SRVS3))
$ make -q
SRVS1 = Test1.java test2.c test3.c test4.java test5.c
SRVS2 = test2.c test3.c test5.c
SRVS3 = test2 test3 test5
Or, all at once:
$ cat Makefile
TAGS := $(basename $(filter %.c,$(subst $(comma), ,$(SRVS))))
all:
$(info TAGS = $(TAGS))
$ make -q
TAGS = test2 test3 test5

Related

How does the makefile write user-defined variables to the yaml file to support vagrantfile fetch parameters

I have a project which use makefile to control vagrant, I want to put the vagrant parameter into the makefile, such as cpu, memory, ip, hostname, forwarded_port and the like. I find a way that vagrantfile read yaml file to parameterize vagrantfile. So makefile needs a target to read all the user option variables and write them to config.yaml as key-value pairs.
The sample is as follows
# === BEGIN USER OPTIONS ===
BOX_OS ?= fedora
# Box setup
#BOX_IMAGE
# Disk setup
DISK_COUNT ?= 1
DISK_SIZE_GB ?= 25
# VM Resources
MASTER_CPUS ?= 2
MASTER_MEMORY_SIZE_GB ?= 2
NODE_CPUS ?= 2
NODE_MEMORY_SIZE_GB ?= 2
NODE_COUNT ?= 2
# Network
MASTER_IP ?= 192.168.26.10
NODE_IP_NW ?= 192.168.26.
POD_NW_CIDR ?= 10.244.0.0/16
...
...
# === END USER OPTIONS ===
The echo command does achieve it
# Makefile
envInit:
#echo "POD_NW_CIDR : \"$(POD_NW_CIDR)\"" > ${FILECWD}/configs.yaml
But too many variables can be too complex.
Is there a way to bulk read variables and their values and write them to a yml file
I would very appreciate it if you guys can tell me how to achieve it that bulk read variables and their values and write them to a yml file.
Define all user options (along with the default values) as a list, so that they are iterable:
# list of user options with default values
userOptions = \
BOX_OS=2 \
DISK_COUNT=1 \
MASTER_IP=192.168.26.10
# replace each default value with the env value, if any
userOptionValues = $(foreach i, $(userOptions), \
$(word 1, $(subst =, ,$i))=$(or \
$($(word 1, $(subst =, ,$i))), $($(word 1, $(subst =, ,$i))), $(word 2, $(subst =, ,$i))))
# write the yaml file
envInit:
# empty the file
#printf "" > configs.yaml
# write a line for each option
#for i in $(userOptionValues); do \
printf "%s : %s\n" "$$(printf $$i | cut -d= -f1)" "$$(printf $$i | cut -d= -f2)" >> configs.yaml; \
done
#flyx Thank you for you answer, your code does work great. But I seem to have found a more convenient way, and I've partially modified it.
printvars:
#echo$(foreach V,$(sort $(.VARIABLES)), \
$(if $(filter-out environment% default automatic,$(origin $V)),$(info $V: $($V))))
But there is still a gap between achieving the goal.
# the Makefile test file
FILECWD = $(shell pwd)
# === BEGIN USER OPTIONS ===
CLOUD_IP ?= 192.168.79.222
CLOUD_NAME ?= cloud
CLOUD_CPU ?= 6
CLOUD_MEMORY ?= 8
# === END USER OPTIONS ===
printvars:
#echo$(foreach V,$(sort $(.VARIABLES)), \
$(if $(filter-out environment% default automatic,$(origin $V)),$(info $V: $($V))))
make printvars's output contains a number of other variables
$ make printvars
.DEFAULT_GOAL: printvars
CLOUD_IP: 192.168.79.222
CLOUD_MEMORY: 8
CLOUD_NAME: cloud
CURDIR: /testmakecreateyml0930
FILECWD: /testmakecreateyml0930
GNUMAKEFLAGS:
MAKEFILE_LIST: Makefile
MAKEFLAGS:
SHELL: /bin/sh
And it can only be printed and not exported to the yaml file.This is only one step away from success.
I would appreciate it if you could help me modify it to achieve my goal
You can write directly to a file with GNUmakes $(file) function:
define newline :=
$(strip)
$(strip)
endef
space := $(strip) $(strip)#
-never-matching := ¥# character 165, this is used as a list element that should never appear as a real element
option-names = $(subst $(-never-matching),,$(filter $(-never-matching)%,$(subst $(-never-matching)$(space),$(-never-matching),$(-never-matching)$(strip $(subst $(newline), $(-never-matching),$1)))))
# define your user options in as many separate parts as you like, spaces and empty lines included:
define USER_OPTIONS +=
a = spaces are no problem
b = "neither nearly all 'other' characters: 8&)("
endef
define USER_OPTIONS +=
c = baz baf
d = foobar
endef
# make all definition make variables verbatim
$(eval $(USER_OPTIONS))
YAML_FORMAT := $(foreach name,$(call option-names,$(USER_OPTIONS)),$(newline)$(name) : $($(name)))
# write the file. Warning: this happens before any rule is run!
$(file >test.yaml,$(YAML_FORMAT))
$(info $(foreach name,$(call option-names,$(USER_OPTIONS)),<$(name) : $($(name))> ))
The trick lies in the clustering of all relevant user option variables in one multi-line make variable. The function option-names pulls all identifiers from that variable into a separate list.
I took the newline etc. character definitions from the GNUmake table toolkit which has many functions for "programmatic" make.

Writing version info to a file using Makefile

I am writing FW version to a file and then reading it while build I my project via Makefile. The SET rule writes the FW version info to the file and GET reads it from it.
When I do
make SET FW_VERSION_MAJOR=1 FW_VERSION_MINOR=2 FW_VERSION_PATCH=3 FW_VERSION_REVISION=4
make GET returns 1.2.3.4
But when I just do
make SET FW_VERSION_MAJOR=1
make GET returns 1...
What I want to achieve is if some one wants to do make SET and want to change only 1 or 2 or 3 parameters out of 4 in the FW version info then my version.h file should be able to retrieve the info left and shouldn't updated with an empty field.
FILE_CONFIG := path/to/version.h
.PHONY: SET GET
SET: VER_MAJOR ?= $(FW_VERSION_MAJOR)
SET: VER_MINOR ?= $(FW_VERSION_MINOR)
SET: VER_PATCH ?= $(FW_VERSION_PATCH)
SET: VER_REVISION ?= $(FW_VERSION_REVISION)
SET:
#echo '#define FW_VERSION_MAJOR $(FW_VERSION_MAJOR)\r\n#define
FW_VERSION_MINOR $(FW_VERSION_MINOR)\r\n#define FW_VERSION_PATCH
$(FW_VERSION_PATCH)\r\n#define FW_VERSION_REVISION $(FW_VERSION_REVISION)'
> $(FILE_CONFIG)
GET:
getnum = $(shell sed -n 's/.*$1 *\([0-9*]\)/\1/p' $(FILE_CONFIG))
FW_VERSION := $(call getnum,MAJOR).$(call getnum,MINOR).$(call
getnum,PATCH).$(call getnum,REVISION)
Not sure I fully understand what you try to achieve but there are several things to fix in your Makefile and examples:
your SET-specific variable definitions (VER_MAJOR...) are useless
your GET rule is empty
your example uses letters-only version numbers while your getnum macro retrieves digits-only version numbers
you must define the FW_VERSION_XXX if you want to use them in a rule when they are not passed on the command line
Try this, maybe:
FILE_CONFIG := version.h
.PHONY: SET GET
SET:
#echo '#define FW_VERSION_MAJOR $(FW_VERSION_MAJOR)\r\n#define FW_VERSION_MINOR $(FW_VERSION_MINOR)\r\n#define FW_VERSION_PATCH $(FW_VERSION_PATCH)\r\n#define FW_VERSION_REVISION $(FW_VERSION_REVISION)' > $(FILE_CONFIG)
GET:
#printf '%s\r\n' '$(FW_VERSION)'
getnum = $(shell sed -n 's/.*$1 *\([0-9*]\)/\1/p' $(FILE_CONFIG))
FW_VERSION_MAJOR := $(call getnum,MAJOR)
FW_VERSION_MINOR := $(call getnum,MINOR)
FW_VERSION_PATCH := $(call getnum,PATCH)
FW_VERSION_REVISION := $(call getnum,REVISION)
FW_VERSION := $(FW_VERSION_MAJOR).$(FW_VERSION_MINOR).$(FW_VERSION_PATCH).$(FW_VERSION_REVISION)
Demo:
$ make SET FW_VERSION_MAJOR=1 FW_VERSION_MINOR=2 FW_VERSION_PATCH=3 FW_VERSION_REVISION=4
$ make GET
1.2.3.4
$ make SET FW_VERSION_REVISION=5
$ make GET
1.2.3.5
Note: we can factorize a bit the FW_VERSION_XXX definitions and also the SET recipe:
FILE_CONFIG := version.h
VERSIONS := MAJOR MINOR PATCH REVISION
.PHONY: SET GET
SET:
#printf '' > $(FILE_CONFIG)
#$(foreach v,$(VERSIONS),$(call setnum,$(v)))
GET:
#printf '%s\r\n' '$(FW_VERSION)'
setnum = printf '\#define FW_VERSION_$1 $(FW_VERSION_$1)\r\n' >> $(FILE_CONFIG);
getnum = $(shell sed -n 's/.*$1 *\([0-9*]\)/\1/p' $(FILE_CONFIG))
$(foreach v,$(VERSIONS),$(eval FW_VERSION_$(v) := $(call getnum,$(v))))
FW_VERSION := $(FW_VERSION_MAJOR).$(FW_VERSION_MINOR).$(FW_VERSION_PATCH).$(FW_VERSION_REVISION)

Makefile does not find target

I have the following Makefile, but it does not work. When I call
make html
I get a
make: *** No rule to make target `docs/index.html', needed by `html'. Stop.
error, even though I think I have defined it.
SRCDIR = source
OUTDIR = docs
RMD = $(wildcard $(SRCDIR)/*.Rmd)
TMP = $(RMD:.Rmd=.html)
HTML = ${subst $(SRCDIR),$(OUTDIR),$(TMP)}
test:
echo $(RMD)
echo $(TMP)
echo $(HTML)
all: clean update html
html: $(HTML)
%.html: %.Rmd
echo $(HTML)
#Rscript -e "rmarkdown::render('$<', output_format = 'prettydoc::html_pretty', output_dir = './$(OUTDIR)/')"
update:
#Rscript -e "devtools::load_all(here::here()); microcosmScheme:::updateFromGoogleSheet(token = './source/googlesheets_token.rds')"
## from https://stackoverflow.com/a/26339924/632423
list:
#$(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^$#$$' | xargs
.PHONY: update clean cleanhtml all list
The variables seem to be correct:
15:21 $ make test
echo source/index.Rmd
source/index.Rmd
echo source/index.html
source/index.html
echo docs/index.html
docs/index.html
If I change it as follow it works, but the target points to the SRCDIR, but I want it to point to the OUTDIR:
RMD = $(wildcard $(SRCDIR)/*.Rmd)
HTML = $(RMD:.Rmd=.html)
# HTML = ${subst $(SRCDIR),$(OUTDIR),$(TMP)}
I am sure it is one small thing...
This rule:
%.html : %.Rmd
....
tells make how to build a file foo.html from a file foo.Rmd, or a file source/foo.html from a file source/foo.Rmd, or a file docs/foo.html from a file docs/foo.Rmd.
It doesn't tell make how to build a file docs/foo.html from a file source/foo.Rmd, because the stem that matches the pattern % is not the same.
If you want to write a pattern for docs/foo.html to be built from source/foo.Rmd, you have to write it like this:
$(OUTDIR)/%.html : $(SRCDIR)/%.Rmd
....
so that the part that matches the pattern % is identical.
ETA Some other notes: you should be using := with the wildcard function as it's much better performing. Also you shouldn't use subst here because it replaces all occurrences of the string which could break things if any of your .Rmd files contain the string source for example (e.g., source/my_source_file.Rmd. This is much better written with patsubst, as in:
RMD := $(wildcard $(SRCDIR)/*.Rmd)
HTML := $(patsubst $(SRCDIR)/%.Rmd,$(OBJDIR)/%.html,$(RMD))
Finally, you don't show what the clean target does but it's unusual to have the clean target depended on by all. Usually it's a separate target that is invoked only when you want it, like make clean.

How Validate Id is correct before creating file using makefile

I need to Validate the ID with pattern (Abbbbb-yyy)
Example :
ID := A12345-789 B98765-123 C58730-417
VARIANT := test1 test2 test3
Build and post processing will generate files depends up on VARIANTS :
`sw_main_test1.hex ,sw_main_test1.hex and sw_main_test1.hex `
.PHONY : SW_TEST
SW_TEST :
if <ID is correct>
cp sw_main_test1.hex --> A12345-789.hex
cp sw_main_test2.hex --> B98765-123.hex
cp sw_main_test3.hex --> C58730-417.hex
I am facing issue in validating the ID with pattern
`Abbbbb-yyy.txt`
Where : A=[A-Z]; b=[0-9]; y=[0-9]
Please let me know how to verify ID is correct using regular expressions inside the Makefile using any tool or utility
In this script, I assume, you get your ID from a file (I called it here someidcontent.txt). Then you could write a script like this (assuming, you only working on Linux).
getID = $(shell cat someidcontent.txt)
all:
if [ "$(getID)" == "1234567890" ]; then \
cp -v output.txt ./delivery/$(getID).txt; \
fi
.PHONY: all
Edit
I made a mistake in my previous script. I did not check, if the ID is correct. Now my newer script does this: I read from a file the ID and check it for correctness. If ID is correct, then some file will be copied into target dir with ID number.
# get ID from a file
getID := $(shell cat someidcontent.txt)
# need a hack for successful checking
idToCheck := $(getID)
# check procedure
checkID := $(shell echo $(idToCheck) | grep "[A-Z][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9]$$")
all:
ifeq "$(checkID)" "$(idToCheck)"
echo found
cp -v output.txt ./delivery/$(idToCheck).txt;
endif
.PHONY: all
Edit 2
Ok, this was a little bit challenging, but I solved it somehow. Maybe there are also other ways to solve this better. In my solution, I assume that the file with IDs and source filenames look like this (in other words, this is the content of my someidcontent.txt):
A2345-678:output1.txt
B3456-123:output.txt
C0987-987:thirdfile.txt
And this is my makefile with comments for additional explanation. I hope, they are sufficient
# retrieve id and filename data from other file
listContent := $(shell cat someidcontent.txt)
# extract only IDs from other files
checkIDs = $(shell echo $(listContent) | grep -o "[A-Z][0-9][0-9][0-9][0-9]-[0-9][0-9][0-9]")
all:
# iterate only over IDs
# first, give me the ID
# second retrieve the filename part for successful copy procedure
# and copy the file to the target dir with ID as filename
#$(foreach x,$(checkIDs), \
echo $(x); \
cp -v $(shell echo $(listContent) | grep -o "$(x):[A-Z0-9a-z\.]*" | sed "s/[-A-Z0-9]*://g") ./delivery/$(x).t$
)
.PHONY: all
You can check simple string patterns quite ok (don't want to say "nicely") from within make:
[A-F] := A B C D E F#
[a-f] := a b c d e f#
[A-Z] := $([A-F]) G H I J K L M N O P Q R S T U V W X Y Z#
[a-z] := $([a-f]) g h i j k l m n o p q r s t u v w x y z#
[0-9] := 0 1 2 3 4 5 6 7 8 9#
######################################################################
##### $(call explode,_stringlist_,_string_)
## Insert a blank after every occurrence of the strings from _stringlist_ in _string_.
## This function serves mainly to convert a string into a list.
## Example: `$(call explode,0 1 2 3 4 5 6 7 8 9,0xl337c0de)` --> `0 xl3 3 7 c0 de`
explode = $(if $1,$(subst $(firstword $1),$(firstword $1) ,$(call explode,$(wordlist 2,255,$1),$2)),$2)
ID := A12345-789 B98765-123 C58730-417 123456+328
############################################################
# $(call check-id,_id-string_)
# Return 'malformed' or the given id
check-id = $(if $(call check-id-1,$(call explode,- $([A-Z]) $([0-9]),$1)),malformed,$1)
check-id-1 = $(strip $(filter-out $([A-Z]),$(wordlist 1,1,$1)) $(filter-out $([0-9]),$(wordlist 2,6,$1)) $(filter-out -,$(word 7,$1)) $(filter-out $([0-9]),$(wordlist 8,10,$1)) )
$(info $(foreach w,$(ID),$(call check-id,$(w))))

gnu make: list the values of all variables (or "macros") in a particular run

How can I list the current value of all variables (also called macros) in a Makefile when running make?
E.g. if this is in the Makefile:
CUR-DIR := $(shell /bin/pwd)
LOG-DIR := $(CUR-DIR)/make-logs
Then I would like it to tell me:
CUR-DIR = /home/johv/src/test
LOG-DIR = /home/johv/src/test/make-logs
GNU make provides .VARIABLES
which holds all global variables' names.
However, this includes built-in variables(like MAKEFLAGS).
If you have to exclude built-in variables, some filtering like the following
might be needed.
The following makefile prints user-defined variables(CUR-DIR, LOG-DIR)
using info:
# Place this line at the top of your Makefile
VARS_OLD := $(.VARIABLES)
# Define your variables
CUR-DIR := $(shell pwd)
LOG-DIR := $(CUR-DIR)/make-logs
# Put this at the point where you want to see the variable values
$(foreach v, \
$(filter-out $(VARS_OLD) VARS_OLD,$(.VARIABLES)), \
$(info $(v) = $($(v))))
Thanks to #Ise Wisteria, condensed down, this shows all variables, useful for large projects with multiple makefiles (Buildroot).
$(foreach v, $(.VARIABLES), $(info $(v) = $($(v))))
output: BR2_GCC_TARGET_TUNE = "cortex-a8" ...
If you get an error like: insufficient number of arguments (1) to function 'addprefix' this project had some broken variables... I trimmed the list of variables to show, only with a prefix BR2_
$(foreach v, $(filter BR2_%,$(.VARIABLES)), $(info $(v) = $($(v))))
I ended up doing it like this:
gmake -pn | grep -A1 "^# makefile"| grep -v "^#\|^--" | sort | uniq > makevars.txt
which gives:
CUR-DIR := /home/johv/src/test
LOG-DIR := /home/johv/src/test/make-logs
MAKEFILE_LIST := Makefile
MAKEFLAGS = pn
SHELL = /bin/sh
VARS_OLD := [...]
gmake -pn is really verbose and looks kinda like this:
# environment
GNOME2_PATH = /usr/local:/opt/gnome:/usr:/usr/local:/opt/gnome:/usr
# automatic
#F = $(notdir $#)
# makefile
SHELL = /bin/sh
# default
RM = rm -f
It's also doable without saving all the .VARIABLES and filtering them out.
Moreover, if one of the original .VARIABLES was modified in your makefile, the two most voted answers won't catch it.
Check out $(origin) function. This target filters out and prints all the variables that were defined in a makefile:
print_file_vars:
$(foreach v, $(.VARIABLES), $(if $(filter file,$(origin $(v))), $(info $(v)=$($(v)))))
I get only a few excess variables this way: CURDIR SHELL MAKEFILE_LIST .DEFAULT_GOAL MAKEFLAGS.
One can replace file with environment or command line to print the respective kinds of variables.
There are a lot of good answers here, but you're going to have problems using $($(v)) if some of your variables are of the recursive flavor. This is why you should use $(value $(v)).
This variation cleans this up a little bit, sorts variables by name and makes the output a bit more readable.
dump:
$(foreach v, \
$(shell echo "$(filter-out .VARIABLES,$(.VARIABLES))" | tr ' ' '\n' | sort), \
$(info $(shell printf "%-20s" "$(v)")= $(value $(v))) \
)
Thanks to #kevinf for the great idea. I would suggest a minor change to prevent .VARIABLE itself from printing out in the variable list:
$(foreach v, $(filter-out .VARIABLES,$(.VARIABLES)), $(info $(v) = $($(v))))
Thanks to #kevinf for the foreach solution -- if one wants to export this list as a somewhat machine-readable file, one will have a hard time with uneven quotes or newlines when using echo or printf, since Make isn't able to quote the data correctly -- one needs to use the $(file ...) function to write the data to avoid sh/bash complaining about invalid syntax. For example, use this in your rule -- it prints variable name, definition and expanded value:
$(file > $(MAKEFILE_ENV_FILE),)
$(foreach v, $(.VARIABLES), \
$(file >> $(MAKEFILE_ENV_FILE),$(v)) \
$(file >> $(MAKEFILE_ENV_FILE), := $(value $(v))) \
$(file >> $(MAKEFILE_ENV_FILE), == $($(v))) \
$(file >> $(MAKEFILE_ENV_FILE),) \
)
(This will still not allow to always distinguish malicious variables with double newlines from two variables, for this one now add a sufficiently unique separator infront of each Makefile-generated newline just after each comma inside $(file >> NAME,TEXT))
Set MAKEFILE_ENV_FILE to some filename, e.g.:
MAKEFILE_ENV_FILE := $(abspath $(lastword $(MAKEFILE_LIST))).env

Resources