Makefile dir function doesn't expand as expected in for loop - makefile

I want to iterate through a list of files, execute cd in their respective directories before executing a command.
So I would like to use $(dir $(FILES)) function to get the directory, but it seems not to work properly in the for loop.
FILES=../dir1/file1 ../dir2/file2
.PHONY: all
all:
#for f in $(FILES); do \
echo $$f in $(dir $${f}) ; \
done
outputs
../dir1/file1 in ./
../dir2/file2 in ./
The $(dir $${f}) gets expanded to ./ instead of ../dirN.
Note :
Writing only echo $(dir $(FILES)) outside of the for loop outputs ../dir1/ ../dir2/ as expected.
using $(abspath ...) doesn't work either.
What am I missing ?

You are mixing make constructs (dir) and shell constructs. Unfortunately they are not evaluated at the same time. Make evaluates its constructs before passing the recipe to the shell, once for all, not during the execution of the recipe by the shell. When expanding your recipe (before passing it to the shell) it transforms it into:
for f in ../dir1/file1 ../dir2/file2; do echo $f in ./; done
because when make expands $(dir $${f}) it first expands the parameter:
$(dir ${f})
and then, as there is no / in the string ${f}, the expansion of the dir function replaces it with ./. From the GNU make manual:
$(dir names…)
Extracts the directory-part of each file name in names. The directory-part of the file name is everything up through (and
including) the last slash in it. If the file name contains no slash,
the directory part is the string ./. For example,
$(dir src/foo.c hacks)
produces the result src/ ./.
Anyway, mixing make and shell constructs is usually not what you want to do (there are exceptions). You could use only shell constructs:
all:
#for f in $(FILES); do \
echo $$f in $$(dirname $$f) ; \
done
Or you could use a static pattern rule to get completely rid of the shell loop and let make iterate over a list of similar targets:
FILES = ../dir1/file1 ../dir2/file2
FOOS = $(addsuffix .foo,$(FILES))
.PHONY: all $(FOOS)
all: $(FOOS)
$(FOOS): %.foo:
#echo $* in $(dir $*)

Related

GNU make: how to ensure pattern to match only filename without directory

I just want to copy files from other directories (before doing something). Because it's tedious to write the copy command for each file, I tried
%: ../src1/%
#echo cp $^ .
%: ../src2/%
#echo cp $^ .
all: file1 file2 file3 file4
# do something
but this doesn't work because make tries to look into ../src2/../src2/../src2/../ . . . . (I included echo for testing to prevent actual copying from happening. I keep forgetting what the "dry run" command line options is . . .)
I naïvely thought that there must be a way to force matches only to filenames that don't include directories.
Is there a way?
You can mark the "make anything" rules terminal with a double colon:
%:: ../src1/%
#echo cp $^ .
%:: ../src2/%
#echo cp $^ .
This does not answer your specific question of how to get Make to match only filenames without directories, but it does get Make to do what you want.
There is another approach that works and is closer to what you asked for: add pattern rules to satisfy Make:
../src1/%:
#: # do nothing
%: ../src1/%
#echo cp $^ .
EDIT: Or better still, us one dummy pattern rule to cover all source directories:
../%:
#: # do nothing

Make does not follow symbolic links to find files

When I extract files from a tar ball to create a list of files make does not follow symbolic links and keeps extracting the same tar ball for each file specified in the list. Forinstance I have created 2 tar balls with the following commands in linux:
design1 is a directory that holds a link to another directory with c design files
% mkdir design1
% mkdir c_files
% cd c_files/
% touch a.c b.c c.c d.c e.c f.c
% cd ../design1/
% ln -s ../c_files/ .
% cd ..
% gtar zcvf design1.tgz design1/
design2 is the same as design1 except that c_files is a directory and not a link
% mkdir design2
%
% cp -rp c_files design2
% gtar zcvf design2.tgz design2/
Now there are 2 zipped tarfiles design1.tgz and design2.tgz and design1 and design2 directories are removed
% \rm -rf design1 design2
Here's my Makefile:
----------------Begin Makefile -------------------
DESIGN_EXTRACT_DIR := .
DESIGN_TOP1 := c_proj1
FILE_LIST1 := $(strip $(DESIGN_TOP1)).f
DESIGN_TAR_FILE1 := design1.tgz
DESIGN_RELEASE_DIR1 := $(DESIGN_EXTRACT_DIR)/design1
DESIGN_FILES1 =\
$(DESIGN_RELEASE_DIR1)/c_files/a.c \
$(DESIGN_RELEASE_DIR1)/c_files/b.c \
$(DESIGN_RELEASE_DIR1)/c_files/c.c \
$(DESIGN_RELEASE_DIR1)/c_files/d.c \
$(DESIGN_RELEASE_DIR1)/c_files/e.c \
$(DESIGN_RELEASE_DIR1)/c_files/f.c \
IP_EXTRACT1: $(DESIGN_FILES1)
$(DESIGN_FILES1): $(DESIGN_TAR_FILE1)
gtar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST1): $(DESIGN_FILES1)
#( rm -f $(FILE_LIST1) )
#( touch $(FILE_LIST1) )
#$(foreach file, $(DESIGN_FILES1), `echo $(file) >> $(FILE_LIST1)`)
FL1: $(FILE_LIST1)
DESIGN_TOP2 := c_proj2
FILE_LIST2 := $(strip $(DESIGN_TOP2)).f
DESIGN_TAR_FILE2 := design2.tgz
DESIGN_RELEASE_DIR2 := $(DESIGN_EXTRACT_DIR)/design2
DESIGN_FILES2 =\
$(DESIGN_RELEASE_DIR2)/c_files/a.c \
$(DESIGN_RELEASE_DIR2)/c_files/b.c \
$(DESIGN_RELEASE_DIR2)/c_files/c.c \
$(DESIGN_RELEASE_DIR2)/c_files/d.c \
$(DESIGN_RELEASE_DIR2)/c_files/e.c \
$(DESIGN_RELEASE_DIR2)/c_files/f.c \
IP_EXTRACT2: $(DESIGN_FILES2)
$(DESIGN_FILES2): $(DESIGN_TAR_FILE2)
gtar zxvfmhC $(DESIGN_TAR_FILE2) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST2): $(DESIGN_FILES2)
#( rm -f $(FILE_LIST2) )
#( touch $(FILE_LIST2) )
#$(foreach file, $(DESIGN_FILES2), `echo $(file) >> $(FILE_LIST2)`)
FL2: $(FILE_LIST2)
clean:
#( rm -rf $(FILE_LIST1)* $(FILE_LIST2)* )
----------------End Makefile----------------------
Now when I run make to create a file list I use the commands make FL1 and make FL2. In the case of FL1 make iterates of gtar as many times as I have files but doesn't do so with FL2. The only difference here is FL1 operates on a link called c_files in design1 while FL2 iterates over c_files as a directory.
Here's what I see:
% make FL1
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
%make FL2
%
Issue1:
FL1 creates c_proj1.f and FL2 creates c_proj2.f but FL2 doesnt iterate like FL1 and I am not sure how to prevent this iteration especially since I could have large tar ball with hundreds of files.
Issue2:
When DESIGN_FILES1 is a long list because of the number of files then I get the following error because the variable is too long:
make: execvp: /bin/sh: Argument list too long
Is there a way to check the size of the variable and maybe write to a file and process the variable a bit at a time so DESIGN_FILES will not be too long for each iteration. Or is there a better way to do this.
Thanks
Niel
For me it works the same (iterates once) whether there is a symlink or not. But from the contents it seems that you are only interested in generating file list, so file existence is not really required. If you omit dependency, you may generate file list without even extracting the files.
For the actual file list write the oversized command line error comes from the fact, that the generated command is a single line with multiple subshell calls, which might be too long for the command line indeed:
$ make FL1 --trace
...
Makefile:24: update target 'c_proj1.f' due to: design1/c_files/a.c design1/c_files/b.c design1/c_files/c.c design1/c_files/d.c design1/c_files/e.c design1/c_files/f.c
( rm -f c_proj1.f )
( touch c_proj1.f )
`echo ./design1/c_files/a.c >> c_proj1.f` `echo ./design1/c_files/b.c >> c_proj1.f` `echo ./design1/c_files/c.c >> c_proj1.f` `echo ./design1/c_files/d.c >> c_proj1.f` `echo ./design1/c_files/e.c >> c_proj1.f` `echo ./design1/c_files/f.c >> c_proj1.f`
This part can be resolved with a $(file) function (if you happen to use GNU make 4.1+):
.PHONY: $(FILE_LIST1)
$(FILE_LIST1):
$(file >$#)
$(foreach file,$(DESIGN_FILES1),$(file >>$#,$(file)))
If an older make is used, this could be done with regular commands, you just need to generate a newline character after each echo to make it work as a separate command. The only tricky part is that it has to be done as a specially crafted variable:
# Note double empty lines below
define newline
endef
.PHONY: $(FILE_LIST2)
$(FILE_LIST2):
rm -f $#
$(foreach file,$(DESIGN_FILES2),echo $(file) >> $#$(newline))
Result is now generated in separate shell calls, which should avoid command length limit:
$ make FL2
rm -f c_proj2.f
echo ./design2/c_files/a.c >> c_proj2.f
echo ./design2/c_files/b.c >> c_proj2.f
echo ./design2/c_files/c.c >> c_proj2.f
echo ./design2/c_files/d.c >> c_proj2.f
echo ./design2/c_files/e.c >> c_proj2.f
echo ./design2/c_files/f.c >> c_proj2.f
EDIT:
I was able to reproduce your multiple iteration of extraction. Your tgz file does not actually contain the files you claim to extract, so the files are not there when the first one is trying to be extracted and the extraction continues for the next file (which is also absent). This is also suggested by tar output, which says:
design1/
design1/c_files
instead of:
design1/
design1/c_files
design1/c_files/a.c
design1/c_files/b.c
design1/c_files/c.c
design1/c_files/d.c
design1/c_files/e.c
design1/c_files/f.c
Your tgz only contains a directory symlink and it only extracts a symlink. If the target files do not exist in the target location, it will attempt to re-extract for every file.
EDIT2:
When actual files are not included in the tgz file, but are available through a symlink, there is a chance that they are older than the tgz file itself. In such situation make checks the file timestamp through a symlink and discovers that it needs to be remade - but the file is not actually remade (since it's not included in the tgz). This check is done for every single file, so for every file the archive is extracted:
$ make FL1 -d
...
Finished prerequisites of target file 'design1/c_files/a.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/a.c'.
Must remake target 'design1/c_files/a.c'.
tar zxvfmhC design1.tgz .
Putting child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286 on the chain.
Live child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286
design1/
design1/c_files
Reaping winning child 0x7fffe11ae730 PID 6286
Removing child 0x7fffe11ae730 PID 6286 from chain.
Successfully remade target file 'design1/c_files/a.c'.
Considering target file 'design1/c_files/b.c'.
Pruning file 'design1.tgz'.
Finished prerequisites of target file 'design1/c_files/b.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/b.c'.
Must remake target 'design1/c_files/b.c'.
tar zxvfmhC design1.tgz .
...
In general extracting archive in this way is not reliable anyway (image running in parallel with -j), so it would be better to use a group target rule to explicitly point out that this rule generates all the files at the same time:
$(DESIGN_FILES1) &: $(DESIGN_TAR_FILE1)
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
This however is a new feature introduced in GNU make 4.3, so to make it work with previous versions, we could use a phony target that is only used for synchronization, e.g.:
$(DESIGN_FILES1): extract-$(DESIGN_TAR_FILE1)
.PHONY: extract-$(DESIGN_TAR_FILE1)
extract-$(DESIGN_TAR_FILE1):
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
Output:
$ make FL1
tar zxvfmhC design1.tgz .
design1/
design1/c_files
Note that the archive is now extracted only once.
I followed your advice and changed the dependency and it resolves the tar issue. The file function in make resolves the length of the variable but I have 2 undesired effect due to it. I have added one more thing to the mix here and I cannot explain the behavior:
On linux:
% mkdir -p design3/c_files
% cd design3/c_files
% touch x.c y.c z.c
% cd ../..
% ls ./design3/c_files/*.c > design3/c_proj3.f
Here's what I have done to the Makefile:
DIR_LIST := design3
IP_EXTRACT1: $(DESIGN_TAR_FILE1)
gtar zxvfmC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(DESIGN_FILES1):
$(MAKE) IP_EXTRACT1
$(FILE_LIST1): $(DESIGN_FILES1)
$(file > $(FILE_LIST1))
#$(if $(strip $(DIR_LIST)), cat $(foreach dir, $(DIR_LIST), $(wildcard $(dir)/*.f)) >> $(FILE_LIST1))
#$(foreach file,$(DESIGN_FILES1),$(file >>$(FILE_LIST1),$(file)))
This resolves the tar issue but now I have added DIR_LIST and I want to add the list of files in each directory to the file list. It does the right thing but it has a couple of undesired effect:
1) I expect the content of design3/c_proj3.f at the top but it appears in the end.
%make FL1
%more c_proj1.f
./design1/c_files/a.c
./design1/c_files/b.c
./design1/c_files/c.c
./design1/c_files/d.c
./design1/c_files/e.c
./design1/c_files/f.c
./design3/c_files/x.c
./design3/c_files/y.c
./design3/c_files/z.c
I expected x, y, z at the top as there is order dependence but its at the bottom. Also, if I do make FL1 --dry-run after removing c_proj1.f the files a through f gets into it but not x,y,z because that is from a cat shell command. I would prefer no files get created. Is there a way to resolve this?. This is a partial solution so I am not sure if this needs to go back into the question.

simple loop over files in some directory makefile

I can easily print all the files inside some directory from bash:
$ cat go.sh
BASEDIR=~/Downloads
MYDIR=${BASEDIR}/ddd
for f in $(ls ${MYDIR}); do echo $f; done
$ ./go.sh
m.txt
d.txt
When I try to do a similar thing from makefile it doesn't work well:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
for f in $(ls ${MYDIR}); do echo ${f}; done
$ make
for f in ; do echo ; done
And here is another trial that doesn't work:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
for f in $(shell ls ${MYDIR}); do echo ${f}; done
$ make
for f in d.txt m.txt; do echo ; done
Maybe you can do it purely Makefile way?
MYDIR = .
list: $(MYDIR)/*
#echo $^
You can still run command from Makefile like this
MYDIR = .
list: $(MYDIR)/*
for file in $^ ; do \
echo "Hello" $${file} ; \
done
If I were you, I'd rather not mix Makefile and bash loops based on $(shell ...). I'd rather pass dir name to some script and run loop there - inside script.
Also almost "true way" from documentation
TEMPLATES_DIR = ./somedir
list:
$(foreach file, $(wildcard $(TEMPLATES_DIR)/*), echo $(file);)
Here is the edited answer based on #Oo.oO:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
#for f in $(shell ls ${MYDIR}); do echo $${f}; done
$ make
d.txt
m.txt
There is a little problem with #Oo.oO's answer.
If there is any file/folder has the same name with a target in makefile, and that target has some prerequisites, and you want to loop through that folder, you will get that target recipe being executed.
For example: if you have a folder named build, and you have a rule like:
build: clean server client
clean:
#echo project cleaned!
server:
#echo server built!
client:
#echo client built!
To loop through the folder contains that special build folder, let's says you have the following rules:
MYDIR = .
ls: $(MYDIR)/*
#echo $^
The result will be:
$ make ls
project cleaned!
server built!
client built!
build Makefile
I would suggest to use #Mike Pylypyshyn's solution. According to the make documentation, the foreach function is more suitable in this case.

Makefile, substitute paths in prerequisites automatic variable

I have following make rule:
$ ls files
a b c
$ cat Makefile
files.7z: ./files/*
7z a $# $?
The rule will be executed as follows:
7z a files.7z files/a files/b files/c
7z treats paths beginning with ./ especially in that, it will include the file in the archive without the path. What would be the shortest way to replace the paths, to begin with, ./ in the $?? (Files names can have spaces.)
Makefile doesn't handle spaces very well at all (see here). The simplest thing is to make make fail if there are spaces and then it's easy:
check:
# ls ./files/* | grep -q " "; \
if [ $$? -ne 1 ]; then \
echo "WE DO NOT SUPPORT FILENAMES WITH SPACES"; \
exit 1; \
fi
files.7z: ./files/* | check
7z a $# $(addprefix ./,$^)
Note: there are workarounds on the web for filenames with spaces, but they're overly complex, not very maintainable and will cost you a lot more headaches than telling your customers/coworkers "We don't support file names with spaces"...

Nested for loop not working in makefile

I am trying to use a nested for loop for searching and copying some files inside the recipe of one of the targets inside a makefile:
DIR = $(DIR_A) $(DIR_B)
install:
for dirs in $(DIR); do \
for file in $(shell find $(dirs) -type f -and -not -path "*/.svn*" | sed -e "s|$(dirs)||"); do \
folder=$${file%/*}; \
$(INSTALL) -d $(DEST_DIR)$$folder/log; \
$(INSTALL) $(dirs)/$$file $(DEST_DIR)$$folder/log; \
done \
done
However $(dirs) variable always evaluates to empty inside the second for loop and the current working directory gets passed to "find" instead of first directory path from $(DIR).
Can someone please suggest if I am missing something ?
Thanks.
The $(dirs) in the find command is being expanded by make to the make variable dirs which is unset and thus the empty string. To reference the shell variable, you need to escape the $:
for file in $$(find $${dirs} -type f -and \
-not -path "*/.svn*" | sed -e "s|$${dirs}||"); do
But don't do this. It is much cleaner to explicitly list the files you intend to install. If there are many, it is fine to write a script to generate the Makefile. What you are doing is a fragile mess.
You have made several errors, and you will find it almost impossible to solve them as l;ong as you insist on trying to solve them all at once.
Let's try this in stages. Suppose your DIR_A and DIR_B are north and south. On the command line, if you try this:
for dirs in north south; do echo $dirs; done
you will get the correct output:
north
south
If you try it as a makefile rule:
install:
for dirs in $(DIR); do echo $$dirs; done
again, it works correctly.
If you try your makefile recipe:
install:
for dirs in $(DIR); do \ for file in $(shell ls $$dirs); do \ echo "file is $$file"; \ done \ done
it fails, because Make expands the $(shell ...) command before passing the entire for command to the shell, when dirs has not yet been assigned a value. One way to construct a sensible shell command is to use backticks:
for dirs in north south; do for file in `ls $dirs`; do echo "file is $file"; done done
This works on the command line. A makefile rule built around it:
install:
for dirs in $(DIR); do for file in `ls $$dirs`; do echo "file is $$file"; done done
also works.
That should be enough to allow you to rewrite your makefile.

Resources