Makefile (running outside scripts that have makefile macros) - bash

I have some .PHONY targets such as 'clean', 'backup', and 'help'
the rule for some of these targets is very large.
For example:
.PHONY: backup
backup:
#$(GREEN)
#mkdir -p backup/include #make an backup include folder if it doesn't already exist
#mkdir -p backup/src #make a backup src folder if it doesn't already exist
#mkdir -p backup/docs #make a backup docs folder if it doesn't already exist
#total=0; headerCount=0; sourceCount=0; documentCount=0; \
for file in $(HEADER_PATH)*; do \
if ls $$file[~] >/dev/null 2>&1; then \
mv -fu $$file[~] backup/$$file; \
let "headerCount+=1"; \
echo $(DATE)[Backed Up] $$file~ >> $(LOG); \
fi; \
done; \
for file in $(SOURCE_PATH)*; do \
if ls $$file[~] >/dev/null 2>&1; then \
mv -fu $$file[~] backup/$$file; \
let "sourceCount+=1"; \
echo $(DATE)[Backed Up] $$file~ >> $(LOG); \
fi; \
done; \
for file in $(DOC_PATH)*; do \
if ls $$file[~] >/dev/null 2>&1; then \
mv -fu $$file[~] backup/$$file; \
let "documentCount+=1"; \
echo $(DATE)[Backed Up] $$file~ >> $(LOG); \
fi; \
done; \
let "total= headerCount + sourceCount + documentCount"; \
echo -n $(OUTPUT_PROMPT)" "; \
if test $$total -eq 0; then \
echo Nothing To Back up; \
else \
if test $$headerCount -eq $$total; then \
echo -n $$total" "; \
echo -n "Header"; \
if test $$total -ge 2; then \
echo -n "s"; \
fi; \
echo " Backed Up"; \
elif test $$sourceCount -eq $$total; then \
echo -n $$total" "; \
echo -n "Source"; \
if test $$total -ge 2; then \
echo -n "s"; \
fi; \
echo " Backed Up"; \
elif test $$documentCount -eq $$total; then \
echo -n $$total" "; \
echo -n "Document"; \
if test $$total -ge 2; then \
echo -n "s"; \
fi; \
echo " Backed Up"; \
else \
$(UNDERLINE); echo $$total " Files Backed Up"; $(UNUNDERLINE); \
if test $$headerCount -eq 1; then \
echo $(OUTPUT_PROMPT)" "$$headerCount header; \
elif test $$headerCount -ge 2; then \
echo $(OUTPUT_PROMPT)" "$$headerCount headers; \
fi; \
if test $$sourceCount -eq 1; then \
echo $(OUTPUT_PROMPT)" "$$sourceCount source; \
elif test $$sourceCount -ge 2; then \
echo $(OUTPUT_PROMPT)" "$$sourceCount sources; \
fi; \
if test $$documentCount -eq 1; then \
echo $(OUTPUT_PROMPT)" "$$documentCount document; \
elif test $$documentCount -ge 2; then \
echo $(OUTPUT_PROMPT)" "$$documentCount documents; \
fi; \
fi; \
fi;
#$(DEFAULT_TEXT)
what the code does is not important, but I wanted to illustrate that it has macros in which 'make' must expand, and that it also performs shell code (bash), and that some indication on what the script did, is displayed in the terminal.
I want to put this script outside of 'make' in another directory, and turn that code into something like this:
.PHONY: backup
backup:
#run scripts/backup.scr
#or something similar to that
How can I put the rule of my target (which is makefile/bash code) into a separate file, and have make practically paste it in so that it runs how I had it originally?
I thought I might be able to use the "include" command inside 'make'.
It looks like it is used to include other makefiles though..
maybe I should just paste the entire target/rule into another makefile, and then include that makefile into my main one?
Would that be the best way?

In your case, you have quite few output variables. It might be worth the hassle to separate the generation and execution, like:
clean : clean-script
sh clean-script
rm -f clean-script
clean-script : clean-script.in
sed -e 's:[#]HEADER_PATH[#]:$(HEADER_PATH):g' $<.in > $#
And write clean-script.in as a clean sh script with a few substitutions.
If you use GNU make, you can of course build a list of output varables like:
clean-script : clean-script.in
sed $(foreach var,$(SUBSTVARS),-e 's:[#]$(var)[#]:$($(var)):g') $<.in > $#

I don't know if it can help you, but you can run make inside some Makefile usually with a command (inside a rule) e.g.
$(MAKE) subtarget
See the section Recursive use of Make in the GNU make documentation.
I tend to dislike using make for complex projects (but unfortunately, I have to). If you are free to chose some other tool, you might consider omake and many others (cmake, scons, bake, ...)

Related

Makefile cache creating false positive outcome

I have a make target, which i usually need to run twice to get accurate outcome. I.e the 1st run if accurate thenn on the 2nd run, if the variable is changed, it still displays the previous output, which is wrong, is there a way to get rid of cache or clear it in between.
.PHONY:check-tf-lint
check-tf-lint: configure ## TF Linting
$(eval list_of_dir := $(shell cd ${deployment} && ls -ld */ | awk '{print $$NF}'| grep -v 'test_cases'| sed 's|/||g'))
$(shell touch ${quality-metrics}/formatting.txt)
#for i in aws_bot; do \
make set-tf-version -e infra_module_path=$$i; \
terraform fmt -check -list=false ${deployment}/$$i ; \
if [ "$$?" != "0" ]; then \
echo "Need Formatting in $$i" >> ${quality-metrics}/formatting.txt; \
terraform fmt -check ${deployment}/$$i >> ${quality-metrics}/formatting.txt; \
echo "" >> ${quality-metrics}/formatting.txt; \
fi \
done
$(eval TMP := $(shell (cat ${quality-metrics}/formatting.txt | wc -l)))
echo "${TMP}"
#if [ "$(TMP)" = "0" ]; then \
echo "All Good! No Formatting Needed."; \
else \
echo "Kindly Format Below Mentioned code and check in Again"; \
cat ${quality-metrics}/formatting.txt; \
fi
$(shell rm -rf ${quality-metrics}/formatting.txt)
#if [ "$(TMP)" != "0" ]; then \
exit 1; \
fi
Rule of thumb: you should never use eval or shell functions in a make recipe. If you are doing that it's a pretty sure sign that something has gone wrong somewhere.
In your case the reason you see this behavior is that make will expand ALL variables and functions for all lines in a recipe before the first line in the recipe is invoked. So as far as make is concerned your recipe is handled like this:
.PHONY:check-tf-lint
check-tf-lint: configure ## TF Linting
$(eval list_of_dir := $(shell cd ${deployment} && ls -ld */ | awk '{print $$NF}'| grep -v 'test_cases'| sed 's|/||g'))
$(shell touch ${quality-metrics}/formatting.txt)
$(eval TMP := $(shell (cat ${quality-metrics}/formatting.txt | wc -l)))
$(shell rm -rf ${quality-metrics}/formatting.txt)
#for i in aws_bot; do \
make set-tf-version -e infra_module_path=$$i; \
terraform fmt -check -list=false ${deployment}/$$i ; \
if [ "$$?" != "0" ]; then \
echo "Need Formatting in $$i" >> ${quality-metrics}/formatting.txt; \
terraform fmt -check ${deployment}/$$i >> ${quality-metrics}/formatting.txt; \
echo "" >> ${quality-metrics}/formatting.txt; \
fi \
done
echo "${TMP}"
#if [ "$(TMP)" = "0" ]; then \
echo "All Good! No Formatting Needed."; \
else \
echo "Kindly Format Below Mentioned code and check in Again"; \
cat ${quality-metrics}/formatting.txt; \
fi
#if [ "$(TMP)" != "0" ]; then \
exit 1; \
fi
You should always write your recipes using shell facilities and not make facilities. Set shell variables, don't use eval to set make variables, and run shell commands directly (you're in a recipe after all!) rather than using make's shell function.
You may need to put all the lines in a single script (with semicolon / backslash) to allow this to work. Or consider .ONESHELL but that's a much bigger set of changes.
This worked for me !
.PHONY:check-tf-lint
check-tf-lint: ## TF Linting
$(eval list_of_dir := $(shell cd ${deployment} && ls -ld */ | awk '{print $$NF}'| grep -v 'test_cases'| sed 's|/||g'))
#for i in $(list_of_dir); do \
make set-tf-version -e infra_module_path=$$i; \
terraform fmt -check -list=false ${deployment}/$$i ; \
if [ "$$?" != "0" ]; then \
echo "Need Formatting in $$i" >> ${quality-metrics}/formatting.txt; \
terraform fmt -check ${deployment}/$$i >> ${quality-metrics}/formatting.txt; \
echo "" >> ${quality-metrics}/formatting.txt; \
fi \
done
#if [ -e "${quality-metrics}/formatting.txt" ]; then \
export MNC=`cat ${quality-metrics}/formatting.txt | wc -l`; \
if [ "$${MNC}" = "0" ]; then \
echo "All Good! No Formatting Needed."; \
else \
echo ""; \
echo "Kindly Format Below Mentioned code and check in Again"; \
cat ${quality-metrics}/formatting.txt; \
fi; \
rm -rf ${quality-metrics}/formatting.txt; \
if [ "$${MNC}" != "0" ]; then \
exit 1; \
fi \
fi

How to expand wildcard inside shell code block in a Makefile?

I got this script which reads a delimited part of my .gitignore file and remove all files after the given mark # #:
# https://stackoverflow.com/questions/55527923/how-to-stop-makefile-from-expanding-my-shell-output
RAW_GITIGNORE_CONTENTS := $(shell while read -r line; do printf "$$line "; done < ".gitignore")
GITIGNORE_CONTENTS := $(shell echo "$(RAW_GITIGNORE_CONTENTS)" | sed -E $$'s/[^\#]+\# //g')
# https://stackoverflow.com/questions/4210042/exclude-directory-from-find-command
DIRECTORIES_TO_CLEAN := $(shell /bin/find -not -path "./**.git**" -not -path "./pictures**" -type d)
clean:
# https://stackoverflow.com/questions/10586153/split-string-into-an-array-in-bash
# https://stackoverflow.com/questions/11289551/argument-list-too-long-error-for-rm-cp-mv-commands
readarray -td' ' GARBAGE_DIRECTORIES <<<"$(DIRECTORIES_TO_CLEAN) "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<"$(GITIGNORE_CONTENTS) "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "$${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="$${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "$${arraylength}" "$$filename"; \
for extension in "$${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$$filename" ]] || continue; \
[[ ! -z "$$extension" ]] || continue; \
full_expression="$${filename}/$${extension}" ;\
printf '%s\n' "$$full_expression"; \
rm -v "$$full_expression"; \
done; \
done;
Running it with the following .gitignore file:
*.txt
*.var
# Comment #
*.aux
The rm command is not expanding the wildcards and keeps telling me rm: cannot remove './*.aux': No such file or directory and do not remove the *.aux files from the ./ directory.
Update
After asked on a comment by #Beta, I simplified the Makefile to this:
GITIGNORE_CONTENTS := "*.aux" "*.lof"
DIRECTORIES_TO_CLEAN := "./setup/cache" "./setup/cache/chapters"
clean:
readarray -td' ' GARBAGE_DIRECTORIES <<<"$(DIRECTORIES_TO_CLEAN) "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<"$(GITIGNORE_CONTENTS) "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "$${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="$${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "$${arraylength}" "$$filename"; \
for extension in "$${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$$filename" ]] || continue; \
[[ ! -z "$$extension" ]] || continue; \
full_expression="$${filename}/$${extension}" ;\
printf '%s\n' "$$full_expression"; \
rm -vf "$$full_expression"; \
done; \
done;
Which results on this output after running it:
$ make
readarray -td' ' GARBAGE_DIRECTORIES <<<""./setup/cache" "./setup/cache/chapters" "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<""*.aux" "*.lof" "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "${arraylength}" "$filename"; \
for extension in "${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$filename" ]] || continue; \
[[ ! -z "$extension" ]] || continue; \
full_expression="${filename}/${extension}" ;\
printf '%s\n' "$full_expression"; \
rm -vf "$full_expression"; \
done; \
done;
declare -a GARBAGE_DIRECTORIES=([0]="./setup/cache" [1]="./setup/cache/chapters")
declare -a GARBAGE_EXTENSIONS=([0]="*.aux" [1]="*.lof")
Cleaning 2 extensions on ./setup/cache
./setup/cache/*.aux
./setup/cache/*.lof
Cleaning 2 extensions on ./setup/cache/chapters
./setup/cache/chapters/*.aux
./setup/cache/chapters/*.lof
More simplification
I reduced to the more simple version it could be:
clean:
rm -v "./setup/cache/*.aux";
Running this, also do not remove the files:
$ make
rm -v "./setup/cache/*.aux";
rm: cannot remove './setup/cache/*.aux': No such file or directory
make: *** [Makefile:3: clean] Error 1
$ ls ./setup/cache/*.aux
./setup/cache/main.aux
On above, after running ls, you can see the file still exists and it is there.
I managed to fix it by changing:
rm -vf "$$full_expression"; \
To:
rm -vf $${full_expression}; \

Makefile: Splitting a string and looping through results

I am trying to write a target in a makefile which will read a variable(having IPs) from one of the .mk file and if a space separated list found split it and take some action.
Issue i am facing that the string do not split and in for loop do not get the value either.
Have tried following
GWTS_FE_IPS=2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5
test:
$(eval IPS=$(shell echo "$(GW_IPS)" |awk -F " " '{print NF}'))
if [ ${IPS} -gt 1 ]; then \
echo "Multiple Ips [$(GW_IPS)]"; \
for ip in $(shell echo "${GW_IPS}" | sed -e 's/ /\n/g'); \
do \
echo ".... $(ip) ...."; \
done \
else \
echo "Single IP [$(GW_IPS)]"; \
fi
Result i get is
2600:40f0:3e::2n2600:40f0:3e::3n2600:40f0:3e::4n2600:40f0:3e::5
if [ 4 -gt 1 ]; then \
echo "Multiple Ips [2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5]"; \
for ip in 2600:40f0:3e::2n2600:40f0:3e::3n2600:40f0:3e::4n2600:40f0:3e::5; \
do \
echo ".... ...."; \
done \
else \
echo "Single IP [2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5]"; \
fi
Multiple Ips [2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5]
.... ....
Can any one give some pointers.
You are trying to do too many things at once without testing any of them. When you try new tools, try them one at a time.
GWTS_FE_IPS=2600:40f0:3e::2 2600:40f0:3e::3 2600:40f0:3e::4 2600:40f0:3e::5
IPS := $(words $(GWTS_FE_IPS))
test:
#if [ ${IPS} -gt 1 ]; then \
echo "Multiple Ips [$(GW_IPS)]"; \
for ip in $(GWTS_FE_IPS) ; \
do \
echo ".... $$ip ...."; \
done \
else \
echo "Single IP [$(GW_IPS)]"; \
fi

where the target all-recursive is given in makefile

in a makefile,i got the target all and its dependency all-recursive.
i search the whole file,but i can not get the all-recursive defined. i think all-recursive must be also a target, or how can make do next? so someone can tell me how to deal with this, i will really appreciate your help.
all: $(BUILT_SOURCES)
$(MAKE) $(AM_MAKEFLAGS) all-recursive
i can not get the defination of "all-recursive".
if i delete this, the make program will continuously deal with the all target. is that "all-recursive" builtin ?
I spent hours finding them too. No, it's not a built-in feature of make; it turned out that this is a characteristic of Autotools-generated Makefile.
Those THING-recursive targets are actually defined in that very Makefile, but in a complicated way that you cannot use simple grep to find them.
It starts from RECURSIVE_TARGETS variable definition in the Makefile, which goes like this:
RECURSIVE_TARGETS = all-recursive check-recursive dvi-recursive \
html-recursive info-recursive install-data-recursive \
install-dvi-recursive install-exec-recursive \
install-html-recursive install-info-recursive \
install-pdf-recursive install-ps-recursive install-recursive \
installcheck-recursive installdirs-recursive pdf-recursive \
ps-recursive uninstall-recursive
Followed by a real definition of those targets somewhere below:
$(RECURSIVE_TARGETS):
#fail= failcom='exit 1'; \
for f in x $$MAKEFLAGS; do \
case $$f in \
*=* | --[!k]*);; \
*k*) failcom='fail=yes';; \
esac; \
done; \
dot_seen=no; \
target=`echo $# | sed s/-recursive//`; \
list='$(SUBDIRS)'; for subdir in $$list; do \
echo "Making $$target in $$subdir"; \
if test "$$subdir" = "."; then \
dot_seen=yes; \
local_target="$$target-am"; \
else \
local_target="$$target"; \
fi; \
($(am__cd) $$subdir && $(MAKE) $(AM_MAKEFLAGS) $$local_target) \
|| eval $$failcom; \
done; \
if test "$$dot_seen" = "no"; then \
$(MAKE) $(AM_MAKEFLAGS) "$$target-am" || exit 1; \
fi; test -z "$$fail"
Which resolves into this:
all-recursive check-recursive dvi-recursive \
html-recursive info-recursive install-data-recursive \
install-dvi-recursive install-exec-recursive \
install-html-recursive install-info-recursive \
install-pdf-recursive install-ps-recursive install-recursive \
installcheck-recursive installdirs-recursive pdf-recursive \
ps-recursive uninstall-recursive:
#fail= failcom='exit 1'; \
for f in x $$MAKEFLAGS; do \
case $$f in \
*=* | --[!k]*);; \
*k*) failcom='fail=yes';; \
esac; \
done; \
dot_seen=no; \
target=`echo $# | sed s/-recursive//`; \
list='$(SUBDIRS)'; for subdir in $$list; do \
echo "Making $$target in $$subdir"; \
if test "$$subdir" = "."; then \
dot_seen=yes; \
local_target="$$target-am"; \
else \
local_target="$$target"; \
fi; \
($(am__cd) $$subdir && $(MAKE) $(AM_MAKEFLAGS) $$local_target) \
|| eval $$failcom; \
done; \
if test "$$dot_seen" = "no"; then \
$(MAKE) $(AM_MAKEFLAGS) "$$target-am" || exit 1; \
fi; test -z "$$fail"
This target's recipe's is essentially a boilerplate code to loop through each subdirectory found in current folder and start make inside it, using the same target name but with "-recursive" stripped out.
Note that these THING-recursive targets are not meant to be called directly by user; it will be run automatically as a part of normal THING target (without "-recursive"), as a mechanism to trigger same-target building in subprojects' tree.
Addendum: The example code is taken from configured root Makefile of GNU Flash Player (version e9eb84e).
see if it is inclusively defined in the target $(RECURSIVE_TARGETS)

Unix shell bash script unable to put an echo debug statement

I have the following make file, which i think is a shell script.
I am trying to loop through FILE_DIR to perform some operations. However, i feel that the implementation isn't working as expected. So i am trying to insert some echo breakpoints.
Source:
# Target to recurse through the DIR_LIST and make each makefile found in that DIRS
ALLDIRS:
for se in $(FILE_DIR); do \
if [ -d $se ]; then \
cd $se; \
$(MAKE) -f Makefile.mk all; \
cd ..; \
fi \
done
Running:
$ make -f Makefile.batch
h: syntax error at line 3: `then' unexpected
*** Error code 2
The following command caused the error:
for se in `ls -p /app/internal|grep "/"`; do \
echo "Test" \
if [ -d e ]; then \
cd e; \
/usr/ccs/bin/make -f Makefile.mk all; \
cd ..; \
fi \
done
make: Fatal error: Command failed for target `ALLDIRS'
Can i please get help on this. Would like to insert an echo breakpoint.
One common error in Makefiles is using spaces instead of tabs in a command line. Check the whole for loop and make sure there are only tabs at the beginning of each line
ALLDIRS:
<tab>for se in $(FILE_DIR); do \
<tab><tab>if [ -d $se ]; then \
<tab><tab>cd $se; \
<tab><tab>$(MAKE) -f Makefile.mk all; \
<tab><tab>cd ..; \
<tab><tab>fi \
<tab>done
Another error is the dollar sign $. If you want a dollar sign in the shell command, you must double it in your commands, because otherwise dollar introduces a make variable and will be expanded before the shell sees it.
for se in $(FILE_DIR); do \
if [ -d $$se ]; then \
cd $$se; \
$(MAKE) -f Makefile.mk all; \
cd ..; \
fi \
done
And the final one, echo Test needs a semicolon as well
for se in $(FILE_DIR); do \
if [ -d $$se ]; then \
echo "Test"; \
cd $$se; \
...

Resources