How can I import variables from a nested YAML to use in a Makefile. A subset of YAML could be ok, but nested is important.
I understand I could use a python or ruby or x script, but so far I dont have to use either and would like to avoid and keep universal if possible.
thanks.
I don't think there's any good way to do this. If you wanted, you could probably write a Make plugin in C that calls a YAML parser and provides an interface to the data, but unless someone has already written one it's likely not worth the time.
There is a way to do it with the help of yq command-line YAML processor. It can print all or subset of paths with values in a YAML file:
yq r file.yaml -p pv "**"
We can use foreach function in make to define variables for each path:
yaml_file = file.yaml
yaml := $(shell yq r $(yaml_file) -p pv "**" | sed 's/ /|/g' | sed 's/:|/:/g')
yaml_path_value = $(subst :, ,$(yaml_element))
yaml_path = $(word 1,$(yaml_path_value))
yaml_value = $(subst |, ,$(word 2,$(yaml_path_value)))
yaml_variable = $(eval $(yaml_path) := $(yaml_value))
$(foreach yaml_element,$(yaml),$(yaml_variable))
Make uses spaces as a separator in lists, so spaces in YAML values have to be replaced with another character (| in this example) before using foreach and then put back before defining variables.
After some research I came up with this solution, which is ok for me. It combines the https://stackoverflow.com/a/21189044/6869629 and http://make.mad-scientist.net/constructed-include-files/ approaches and adds content of the YAML files as variables to the Makefile:
This is the YAML config file:
project.yaml
PROJECT_NAME: test-project
ENVIRONMENT: test
and this is the Makefile:
PROJECT_CONFIG_YAML_PATH := "../../project.yaml"
PROJECT_CONFIG_VAR_PREFIX := "CONF_"
READ_PROJECT_CONFIG := $(shell prefix=$(PROJECT_CONFIG_VAR_PREFIX) \
&& lf= $$(echo #|tr # '\012')\
&& s='[[:space:]]*' \
&& w='[a-zA-Z0-9_]*' \
&& fs=$$(echo #|tr # '\034') \
&& sed -ne "s|^\($$s\):|\1|" \
-e "s|^\($$s\)\($$w\)$$s:$$s[\"']\(.*\)[\"']$$s\$$|\1$$fs\2$$fs\3|p" \
-e "s|^\($$s\)\($$w\)$$s:$$s\(.*\)$$s\$$|\1$$fs\2$$fs\3|p" $(PROJECT_CONFIG_YAML_PATH) | \
awk -F$$fs '{indent = length($$1)/2; \
vname[indent] = $$2; \
for (i in vname) {if (i > indent) {delete vname[i]}} \
if (length($$3) > 0) { \
vn=""; for (i=0; i<indent; i++) {vn=(vn)(vname[i])("_")} \
printf("%s%s%s=\"%s\"\n", "'$$prefix'",vn, $$2, $$3); \
} \
}'| tr '\n' '\1')
add-project-config: Makefile
#echo "$(READ_PROJECT_CONFIG)" | tr '\1' '\n' > $#
print-%:
#echo "$(READ_PROJECT_CONFIG)" | tr '\1' '\n'
#echo $*=$($*)
-include add-project-config
The print-%: target echoes the content of the YAML file, as it will be added to the Makefile and prints the variable, which has been passed as parameter.
$ make print-CONF_PROJECT_NAME results in:
CONF_PROJECT_NAME=test-project
CONF_ENVIRONMENT=test
CONF_PROJECT_NAME=test-project
Related
I would like to edit a .mk file using Bash.
Inside the file, it looks like this:
SRC_PATHS = src/lib \
src/Application \
src/win \
src/prj
I would like to add a new source, which should look like this:
SRC_PATHS = src/lib \
src/Application \
src/win \
src/prj \
src/New
I am trying a sed command, but cannot add a new line.
Note: the last src path (src/prj) is not always the same.
If ed is available/acceptable.
#!/usr/bin/env bash
ed -s file.mk <<-'EOF'
$t.
-1s/$/ \\/
+s|\(^[[:blank:]]\{1,\}\) \(.\{1,\}\)$|\1 scr/new|
,p
Q
EOF
In-one-line
printf '%s\n' '$t.' '-1s/$/ \\/' '+s|\(^[[:blank:]]*\) \(.*\)$|\1 scr/new|' ,p Q | ed -s file.mk
with a shell variable to store the replacement.
#!/usr/bin/env bash
var='scr/new'
ed -s file.mk <<-EOF
\$t.
-1s/\$/ \\\/
+s|\(^[[:blank:]]\{1,\}\) \(.\{1,\}\)\$|\1 $var|
,p
Q
EOF
Remove the ,p to silence the output to stdout , it is there just to see what is the new outcome of the edited buffer.
Change Q to w if in-place editing is needed
JFYI, both the script and the one-liner are not limited to just bash it should work on any POSIX compliant shell.
With sed how about:
sed -i '$s#.\+#& \\'\\$'\n'' src/New#' file.mk
Result:
SRC_PATHS = src/lib \
src/Application \
src/win \
src/prj \
src/New
Considering the indentation of the input and of the desired result, which is not uniform between the first line and the others, I suspect that it is not important at all. If this is the case, then this sed command might work:
sed -z 's#\n$# \\\nsrc/New\n#' file.mk
where
-z is to treat the file as a single line/stream with embedded \ns
\n$ targets the EOF together with the last \n
the replacement string is \\\nsrc/New\n.
Thanks to all who answered, I tried all your suggestions, and here are the code snippets working and applicable to my needs:
sed -i '/^SRC_PATHS[\t ]*=/{:a;/\\$/{N;ba;};s,$, \\\n\tsrc/New,}' file.mk
there are some instances where there is already a "\" in the file, so I added new code to clean up those lines
sed -i '/^$*.\\/d' file.mk
then to add another path in the EOF:
sed -i '$s#.\+#& \\'\\$'\n'' src/New#' file.mk
I am trying to capture the output in one of the file using
cat <<EOF> /var/log/awsmetadata.log
timestamp= $TIME, \
region= $REGION, \
instanceIp= $INSTANCE_IP, \
availabilityZone= $INSTANCE_AZ, \
instanceType= $INSTANCE_TYPE, \
EOF
Where the output created in the format of
cat /var/log/awsmeta.log
timestamp= 2020-11-04 18:51:17, region= us-west-2, instanceIp= 1.2.3.4, availabilityZone= us-west-2a,
How can i eliminate the wide spaces between each output line?
If you don't want redundant whitespaces simply do not add them:
$ cat <<EOF> /var/log/awsmetadata.log
> timestamp= $TIME, \
> region= $REGION, \
> instanceIp= $INSTANCE_IP, \
> availabilityZone= $INSTANCE_AZ, \
> instanceType= $INSTANCE_TYPE
> EOF
I often use sed or tr instead of cat for this sort of thing:
tr -s ' ' <<EOF > /var/log/awsmetadata.log
timestamp= $TIME, \
region= $REGION, \
instanceIp= $INSTANCE_IP, \
availabilityZone= $INSTANCE_AZ, \
instanceType= $INSTANCE_TYPE,
EOF
But it seems cleaner to not escape the newlines at all and do something like:
{ tr -d \\n <<-EOF; echo; } > /var/log/awsmetadata.log
timestamp= $TIME,
region= $REGION,
instanceIp= $INSTANCE_IP,
availabilityZone= $INSTANCE_AZ,
instanceType= $INSTANCE_TYPE,
EOF
(That solution uses the <<- form of the heredoc which redacts hardtabbed indenation. It will not remove leading spaces.)
OTOH, it seems weird to be using a here doc when you're just wanting to generate one line of output. Why not just use echo?
I have a reoccurring piece of code in my makefile, which I want to put in a function. A simplified version of my code looks as follows:
IDS=4 5
MY_FUNC = $(shell echo "max=2; \
counter=1;\
while [ \$$counter -le \$$max ]; do\
id=$$(echo $(IDS) | cut -d" " -f \$${counter}); \
counter=\$$((counter+1)); \
done");
.PHONY: all
all:
#$(call MY_FUNC)
Unfortunately, cut interprets $counter literally and consequently throws the error message cut: invalid field value ‘$counter’.
I do not understand why this is the case, since the command \$$command accesses the value stored inside this variable. Do you may know how to properly call pass counter to cut?
Here is a fixed form of your Makefile.
IDS=4 5
MY_FUNC = $(shell echo 'max=2; \
counter=1; \
while [ $$counter -le $$max ]; do\
id=$$(echo $(IDS) | cut -d" " -f $${counter}); \
counter=$$((counter+1)); \
echo "debug: id: $$id"; \
done');
.PHONY: all
all:
#$(call MY_FUNC)
I have added a echo "debug: id: $$id"; command to help with debugging and prove that the script is behaving as intended. Here is the output:
$ make
debug: id: 4
debug: id: 5
Here are the important points worth noting in the fixed script:
The outermost delimiter for the argument to the outermost echo statement is single-quote (not double-quotes) in order to prevent the $counter, $max, etc. from expanding to empty strings when the echo statement is being executed by the $(shell echo ...) call from Makefile. This also allows proper nesting of the double-quotes used in cut -d" " within the outer single-quotes.
Now that we are using single-quotes as the outer delimiters, the $ symbols within (escaped as $$ in Makefile) need not be escaped with \ anymore.
If however you want to stick with double-quotes as the outermost delimiter, then the alternative solution with minimal changes to your code looks like this:
IDS=4 5
MY_FUNC = $(shell echo "max=2; \
counter=1;\
while [ \$$counter -le \$$max ]; do\
id=\$$(echo $(IDS) | cut -d\" \" -f \$${counter}); \
counter=\$$((counter+1)); \
echo \"debug: id: \$$id\"; \
done");
.PHONY: all
all:
#$(call MY_FUNC)
Once again the output is:
$ make
debug: id: 4
debug: id: 5
Here are the important points to note in this alternative solution:
All $ symbols for the shell (escaped as $$ in Makefile) need to be escaped carefully. They should all occur as \$$ in Makefile. In your code, this was missing for $$(echo. This has been fixed to \$$(echo.
Further all double-quotes within the outer double-quotes need to be carefully escaped as \", so cut -d" " should be written as cut -d\" \".
Susam Pal's answer explains why your use of double quotes was wrong. Use one or the other of the two proposed solutions.
And then, there are a few other aspects you could consider:
You are using the $(shell...) make function in a recipe which does not realy make sense: recipes are already shell scripts. And you do not need the call function neither. A simpler Makefile could be:
IDS = 4 5
define MY_FUNC
max=2; \
counter=1; \
while [ $$counter -le $$max ]; do \
id=$$(echo $(IDS) | cut -d" " -f $${counter}); \
counter=$$((counter+1)); \
done
endef
.PHONY: all
all:
#$(MY_FUNC)
As your recipe has no side effect it is not very useful. But I guess you know it already and this was just an example.
Hard-wiring the number of items in IDS (max=2) in your recipe is not optimal. If you are using GNU make you could use its words function:
max=$(words $(IDS)); \
There are much simpler ways to achieve what you want with the shell. Assuming you just want to print the id values:
for id in $(IDS); do; \
echo $$id; \
done
is easier. But I guess you know it already and this was just an example.
If you want to use the call function you could pass it a parameter (the current id) and iterate with the foreach make function instead of using a shell loop:
IDS = 4 5
define MY_FUNC
echo $(1)
endef
.PHONY: all
all:
$(foreach id,$(IDS),#$(call MY_FUNC,$(id)))
Note the empty last line of MY_FUNC. It is needed to obtain a true multi-line recipe. Alternate solution with a single-line recipe:
IDS = 4 5
MY_FUNC = echo $(1)
.PHONY: all
all:
#$(foreach id,$(IDS),$(call MY_FUNC,$(id));)
GNU make offers many handy functions and has many very useful features. In your case (and assuming you just want to print each id on standard output) you could use patsubst to create a list of phony targets, one per word in IDS and write a static pattern rule for all of them:
IDS = 4 5
ALLS = $(patsubst %,all-%,$(IDS))
.PHONY: all $(ALLS)
all: $(ALLS)
$(ALLS): all-%:
#echo $*
An advantage of this last solution is that your ids are distributed to as many independent rules (the all-X) and their recipes can be run in parallel by make if you allow it to do so (make -j) while with single rule solutions they necessarily run sequentially.
I have an file named "Version.h" with the content of:
#define APP_VERSION_MAJOR 5
#define APP_VERSION_MINOR 6
#define APP_VERSION_PATCH 0
I have a Makefile, and I want to assign a variable in the Makefile, according to the "Version.h" file, in this case:
MY_APP_VERSION = 5.6.0
I managed to find the line with the following command:
#echo "The result is: $$(grep "#define APP_VERSION_MINOR " Version.h)"
Output:
The result is: #define APP_VERSION_MINOR 6
So, how can I put the version in a variable in the Makefile?
Thanks!
To set a make variable you can do something like this:
getnum = $(shell sed -n 's/.*$1 *\([0-9*]\)/\1/p' Version.h)
MY_APP_VERSION := $(call getnum,MAJOR).$(call getnum,MINOR).$(call getnum,PATCH)
This does invoke sed 3 times though.
Try putting this is your makefile:
all:
#echo The result is: \
$$(grep '#define APP_VERSION_MAJOR' Version.h | cut -d' ' -f5).\
$$(grep '#define APP_VERSION_MINOR' Version.h | cut -d' ' -f5).\
$$(grep '#define APP_VERSION_PATCH' Version.h | cut -d' ' -f5)
Then a simple make gives you
$ make
The result is: 5.6.0
Try something like:
APP_VERSION_MAJOR := $(shell awk '// { if ($$2 = 'APP_VERSION_MAJOR) { print $$3 } }' < Version.h)
The := inhibits repeated expansion of the shell command, and the awk command extracts the the value of the assignment.
With more recent GNU make versions, this should work as well (assuming Version.h doesn't use tabs):
$(foreach i, \
$(shell sed 's/#define \([^ ]*\) *\([^ ]*\)/\1:=\2/' < Version.h), \
$(eval $i))
It has the advantage that it will translate many preprocessor defines in one go, no matter what the specifics are, but it can easily go very wrong if Version.h contains unexpected text.
If you manage to read in the file into a variable (my version of make fails to read with $(file < version.h) but yours may work) then you can use the GNU make table toolkit. It was designed exactly for this purpose:
include gmtt.mk
tbl := 3 $(strip $(file < version.h)) # make a table with 3 columns from the file
versions := $(strip $(call select,3,$(tbl),t)) # select 3rd column from table, t(rue) as where-clause
MY_APP_VERSION := $(subst $(space),.,$(versions))
The file must obey the format that you displayed above so that it can be interpreted as a table with three columns. It has the additional benefit that you don't have to care for which shell (e.g. on Windows) is available.
I have a Makefile with a set of booleans which must be used to control the flags for an external application. The problem is that the flag must be passed as a comma-separated string.
Something like this (non-working pseudo code):
WITH_LIST = ""
WITHOUT_LIST = ""
ifeq ($(BOOL_A),y)
# Append A to list "WITH_LIST"
else
# Append A to list "WITHOUT_LIST"
endif
ifeq ($(BOOL_B),y)
# Append B to list "WITH_LIST"
else
# Append B to list "WITHOUT_LIST"
endif
ifeq ($(BOOL_C),y)
# Append C to list "WITH_LIST"
else
# Append C to list "WITHOUT_LIST"
endif
Now assuming BOOL_A == y, BOOL_B == n and BOOL_C == y, I need to run the following command:
./app --with=A,C --with-out=B
How can I generate these string using Gnu Make?
First you create the two white-space separated lists, either using your method, or thiton's.
Then you use the little trick from the end of section 6.2 of the GNU make manual to create a variable holding a single space, and one holding a comma. You can then use these in $(subst ...) to change the two lists to comma-separated.
PARTS := A B C
BOOL_A := y
BOOL_B := n
BOOL_C := y
WITH_LIST := $(foreach part, $(PARTS), $(if $(filter y, $(BOOL_$(part))), $(part)))
WITHOUT_LIST := $(filter-out $(WITH_LIST), $(PARTS))
null :=
space := $(null) #
comma := ,
WITH_LIST := $(subst $(space),$(comma),$(strip $(WITH_LIST)))
WITHOUT_LIST := $(subst $(space),$(comma),$(strip $(WITHOUT_LIST)))
all:
./app --with=$(WITH_LIST) --with-out=$(WITHOUT_LIST)
A construct like
OPTIONS+=$(if $(filter y,$(BOOL_A)),--with=A,--with-out=A)
should work.
Edit: Sorry, overlooked the necessary collation.
PARTS=A B C
YESSES=$(foreach i,$(PARTS),$(if $(filter y,$(BOOL_$(i))),$(i)))
all:
echo with=$(shell echo $(YESSES) | tr ' ' ',')
The idea is to check for each possible part X whether it's set to yes and insert it into a list if it is yes. This list is whitespace-separated and hard to comma-separate with make, but easy to do this in shell.
Or just use sed: ugly (and untested) but straightforward
WITH_LIST = $(shell echo A$(BOOL_A) B$(BOOL_B) C$(BOOL_C) | sed -e 's/[ABC][^yABC]*//g' -e 's/y//g' -e 's/ /,/g')
WITHOUT_LIST = $(shell echo A$(BOOL_A) B$(BOOL_B) C$(BOOL_C) | sed -e 's/[ABC]y[^ABC]*//g' -e 's/[^ABC ]//g' -e 's/ /,/g')