Makefile: depend on every file of a directory - makefile

I'd like to do a Makefile that runs either with gnumake or makepp that packs all the files under given directiories:
DIRS:=$(shell find . -mindepth 2 -maxdepth 2 -not -name mp3 -not -name ".*" -type d)
PACKAGES = $(DIRS:%=%.npk)
all: packages
packages: $(PACKAGES)
%.npk: %/*
npack c $# #^
.PHONY: all packages
the problem is that there's no such thing as %/* in the dependencies.
I need the targets (X.npk) to depend on every file in directory X, but I don't know what the files are when I write the Makefile, 'cause they're generated later.
An example:
./dirA/x
./dirA/y
./dirB/e
./dirB/f
I'd like to create ./dirA.npk (depending on x,y), ./dirB.npk (e,f)
There's nothing I know about the dirs or the files in advance except that the find used in the 1st line finds all the dirs.

Try using the wildcard directive:
DEPS := $(foreach dir, $(DIRS), $(wildcard $(dir)/*))
%.npk: $(DEPS)
npack c $# $^
EDIT:
The above is just an example of using wildcard and makes each .npk file dependent on the files in all of the other folders. Your usage would be slightly different.
I think there may be an easier way to go about this. Why are you wanting to have a dependency on all of the files in the folder? Is it just to use the $^ operator? Or do you need to rebuild the .npk if any of the files changed?
One alternate (and possibly cleaner) solution would be to use the find utility in your recipe instead of $^ and use the .FORCE directive to always force the .npk file to be rebuilt. The downside is that .npk files may be rebuilt unnecessarily.
EDIT 2:
If there's not a way to do this cleanly with make commands, you can work around it by using .FORCE to ensure that the recipe is always run and move the "should I rebuild this file" check into the body of the recipe:
%.npk: .FORCE
check_for_rebuild.sh $# && npack c $# $^
where check_for_rebuild.sh is a shell script that does something like this:
#!/bin/bash
# Returns non-zero if the archive needs to be rebuilt
if [ -e $1 ]; then
folder_name=$(basename $1 .npk)
[ -z "$(find $folder_name -newer $1 -not -type d)" ] && return 0
fi
return 1
I don't really like that solution because it works around the problem instead of solving it directly, but it may be able to get you going in the meantime. If you are going to go that route, it's probably cleaner and easier to do everything in the shell script and either have the makefile simply invoke the script or get rid of the makefile entirely.

This is the solution I found:
it is based on the makedepend idea, with some "meta" scripting. Not very nice, but works.
PACKAGES :=
all: packages
-include Makefile.depend
packages: Makefile.depend $(PACKAGES)
depend: clean Makefile.depend
Makefile.depend:
#(PACKAGES= ; \
for DIR in `find . -mindepth 2 -maxdepth 2 -not -name mp3 -not -name ".*" -type d` ; \
do \
PACKAGE=`basename $${DIR}.npk` ; \
PACKAGES="$${PACKAGES} $${PACKAGE}" ; \
DEPS=`find $${DIR} -not -type d | sed -e 's#\([: ]\)#\\\\\1#' -e 's#^\./\(.*\)# \1#' | tr -d "\n"` ; \
SUBDIR=`echo $${DIR} | sed -e 's#^\./\([^/]\+\)/.*#\1#'` ; \
FILES=`echo \ $${DEPS} | sed -e "s# $${SUBDIR}/# #g"` ; \
echo "$${PACKAGE}:$${DEPS}" ; \
echo " #cd $${SUBDIR} ; \\" ; \
echo " npack c ../\$$# $${FILES} ; \\" ; \
echo ; \
done ; \
echo "PACKAGES = $${PACKAGES}" \
)>> Makefile.depend ; \
cleanall: clean
rm -f *.npk
clean:
#rm -f Makefile.depend
.PHONY: all packages depend clean

With makepp you can do this in 2 steps, via the :foreach rule modifier:
$(foreach).txt: $(foreach)/*: foreach */
&echo $(inputs) -o $(output)
This provides a rule for every subdirectory, which reexecutes whenever there is a change in the list of files therein.

Related

Make does not follow symbolic links to find files

When I extract files from a tar ball to create a list of files make does not follow symbolic links and keeps extracting the same tar ball for each file specified in the list. Forinstance I have created 2 tar balls with the following commands in linux:
design1 is a directory that holds a link to another directory with c design files
% mkdir design1
% mkdir c_files
% cd c_files/
% touch a.c b.c c.c d.c e.c f.c
% cd ../design1/
% ln -s ../c_files/ .
% cd ..
% gtar zcvf design1.tgz design1/
design2 is the same as design1 except that c_files is a directory and not a link
% mkdir design2
%
% cp -rp c_files design2
% gtar zcvf design2.tgz design2/
Now there are 2 zipped tarfiles design1.tgz and design2.tgz and design1 and design2 directories are removed
% \rm -rf design1 design2
Here's my Makefile:
----------------Begin Makefile -------------------
DESIGN_EXTRACT_DIR := .
DESIGN_TOP1 := c_proj1
FILE_LIST1 := $(strip $(DESIGN_TOP1)).f
DESIGN_TAR_FILE1 := design1.tgz
DESIGN_RELEASE_DIR1 := $(DESIGN_EXTRACT_DIR)/design1
DESIGN_FILES1 =\
$(DESIGN_RELEASE_DIR1)/c_files/a.c \
$(DESIGN_RELEASE_DIR1)/c_files/b.c \
$(DESIGN_RELEASE_DIR1)/c_files/c.c \
$(DESIGN_RELEASE_DIR1)/c_files/d.c \
$(DESIGN_RELEASE_DIR1)/c_files/e.c \
$(DESIGN_RELEASE_DIR1)/c_files/f.c \
IP_EXTRACT1: $(DESIGN_FILES1)
$(DESIGN_FILES1): $(DESIGN_TAR_FILE1)
gtar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST1): $(DESIGN_FILES1)
#( rm -f $(FILE_LIST1) )
#( touch $(FILE_LIST1) )
#$(foreach file, $(DESIGN_FILES1), `echo $(file) >> $(FILE_LIST1)`)
FL1: $(FILE_LIST1)
DESIGN_TOP2 := c_proj2
FILE_LIST2 := $(strip $(DESIGN_TOP2)).f
DESIGN_TAR_FILE2 := design2.tgz
DESIGN_RELEASE_DIR2 := $(DESIGN_EXTRACT_DIR)/design2
DESIGN_FILES2 =\
$(DESIGN_RELEASE_DIR2)/c_files/a.c \
$(DESIGN_RELEASE_DIR2)/c_files/b.c \
$(DESIGN_RELEASE_DIR2)/c_files/c.c \
$(DESIGN_RELEASE_DIR2)/c_files/d.c \
$(DESIGN_RELEASE_DIR2)/c_files/e.c \
$(DESIGN_RELEASE_DIR2)/c_files/f.c \
IP_EXTRACT2: $(DESIGN_FILES2)
$(DESIGN_FILES2): $(DESIGN_TAR_FILE2)
gtar zxvfmhC $(DESIGN_TAR_FILE2) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST2): $(DESIGN_FILES2)
#( rm -f $(FILE_LIST2) )
#( touch $(FILE_LIST2) )
#$(foreach file, $(DESIGN_FILES2), `echo $(file) >> $(FILE_LIST2)`)
FL2: $(FILE_LIST2)
clean:
#( rm -rf $(FILE_LIST1)* $(FILE_LIST2)* )
----------------End Makefile----------------------
Now when I run make to create a file list I use the commands make FL1 and make FL2. In the case of FL1 make iterates of gtar as many times as I have files but doesn't do so with FL2. The only difference here is FL1 operates on a link called c_files in design1 while FL2 iterates over c_files as a directory.
Here's what I see:
% make FL1
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
%make FL2
%
Issue1:
FL1 creates c_proj1.f and FL2 creates c_proj2.f but FL2 doesnt iterate like FL1 and I am not sure how to prevent this iteration especially since I could have large tar ball with hundreds of files.
Issue2:
When DESIGN_FILES1 is a long list because of the number of files then I get the following error because the variable is too long:
make: execvp: /bin/sh: Argument list too long
Is there a way to check the size of the variable and maybe write to a file and process the variable a bit at a time so DESIGN_FILES will not be too long for each iteration. Or is there a better way to do this.
Thanks
Niel
For me it works the same (iterates once) whether there is a symlink or not. But from the contents it seems that you are only interested in generating file list, so file existence is not really required. If you omit dependency, you may generate file list without even extracting the files.
For the actual file list write the oversized command line error comes from the fact, that the generated command is a single line with multiple subshell calls, which might be too long for the command line indeed:
$ make FL1 --trace
...
Makefile:24: update target 'c_proj1.f' due to: design1/c_files/a.c design1/c_files/b.c design1/c_files/c.c design1/c_files/d.c design1/c_files/e.c design1/c_files/f.c
( rm -f c_proj1.f )
( touch c_proj1.f )
`echo ./design1/c_files/a.c >> c_proj1.f` `echo ./design1/c_files/b.c >> c_proj1.f` `echo ./design1/c_files/c.c >> c_proj1.f` `echo ./design1/c_files/d.c >> c_proj1.f` `echo ./design1/c_files/e.c >> c_proj1.f` `echo ./design1/c_files/f.c >> c_proj1.f`
This part can be resolved with a $(file) function (if you happen to use GNU make 4.1+):
.PHONY: $(FILE_LIST1)
$(FILE_LIST1):
$(file >$#)
$(foreach file,$(DESIGN_FILES1),$(file >>$#,$(file)))
If an older make is used, this could be done with regular commands, you just need to generate a newline character after each echo to make it work as a separate command. The only tricky part is that it has to be done as a specially crafted variable:
# Note double empty lines below
define newline
endef
.PHONY: $(FILE_LIST2)
$(FILE_LIST2):
rm -f $#
$(foreach file,$(DESIGN_FILES2),echo $(file) >> $#$(newline))
Result is now generated in separate shell calls, which should avoid command length limit:
$ make FL2
rm -f c_proj2.f
echo ./design2/c_files/a.c >> c_proj2.f
echo ./design2/c_files/b.c >> c_proj2.f
echo ./design2/c_files/c.c >> c_proj2.f
echo ./design2/c_files/d.c >> c_proj2.f
echo ./design2/c_files/e.c >> c_proj2.f
echo ./design2/c_files/f.c >> c_proj2.f
EDIT:
I was able to reproduce your multiple iteration of extraction. Your tgz file does not actually contain the files you claim to extract, so the files are not there when the first one is trying to be extracted and the extraction continues for the next file (which is also absent). This is also suggested by tar output, which says:
design1/
design1/c_files
instead of:
design1/
design1/c_files
design1/c_files/a.c
design1/c_files/b.c
design1/c_files/c.c
design1/c_files/d.c
design1/c_files/e.c
design1/c_files/f.c
Your tgz only contains a directory symlink and it only extracts a symlink. If the target files do not exist in the target location, it will attempt to re-extract for every file.
EDIT2:
When actual files are not included in the tgz file, but are available through a symlink, there is a chance that they are older than the tgz file itself. In such situation make checks the file timestamp through a symlink and discovers that it needs to be remade - but the file is not actually remade (since it's not included in the tgz). This check is done for every single file, so for every file the archive is extracted:
$ make FL1 -d
...
Finished prerequisites of target file 'design1/c_files/a.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/a.c'.
Must remake target 'design1/c_files/a.c'.
tar zxvfmhC design1.tgz .
Putting child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286 on the chain.
Live child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286
design1/
design1/c_files
Reaping winning child 0x7fffe11ae730 PID 6286
Removing child 0x7fffe11ae730 PID 6286 from chain.
Successfully remade target file 'design1/c_files/a.c'.
Considering target file 'design1/c_files/b.c'.
Pruning file 'design1.tgz'.
Finished prerequisites of target file 'design1/c_files/b.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/b.c'.
Must remake target 'design1/c_files/b.c'.
tar zxvfmhC design1.tgz .
...
In general extracting archive in this way is not reliable anyway (image running in parallel with -j), so it would be better to use a group target rule to explicitly point out that this rule generates all the files at the same time:
$(DESIGN_FILES1) &: $(DESIGN_TAR_FILE1)
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
This however is a new feature introduced in GNU make 4.3, so to make it work with previous versions, we could use a phony target that is only used for synchronization, e.g.:
$(DESIGN_FILES1): extract-$(DESIGN_TAR_FILE1)
.PHONY: extract-$(DESIGN_TAR_FILE1)
extract-$(DESIGN_TAR_FILE1):
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
Output:
$ make FL1
tar zxvfmhC design1.tgz .
design1/
design1/c_files
Note that the archive is now extracted only once.
I followed your advice and changed the dependency and it resolves the tar issue. The file function in make resolves the length of the variable but I have 2 undesired effect due to it. I have added one more thing to the mix here and I cannot explain the behavior:
On linux:
% mkdir -p design3/c_files
% cd design3/c_files
% touch x.c y.c z.c
% cd ../..
% ls ./design3/c_files/*.c > design3/c_proj3.f
Here's what I have done to the Makefile:
DIR_LIST := design3
IP_EXTRACT1: $(DESIGN_TAR_FILE1)
gtar zxvfmC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(DESIGN_FILES1):
$(MAKE) IP_EXTRACT1
$(FILE_LIST1): $(DESIGN_FILES1)
$(file > $(FILE_LIST1))
#$(if $(strip $(DIR_LIST)), cat $(foreach dir, $(DIR_LIST), $(wildcard $(dir)/*.f)) >> $(FILE_LIST1))
#$(foreach file,$(DESIGN_FILES1),$(file >>$(FILE_LIST1),$(file)))
This resolves the tar issue but now I have added DIR_LIST and I want to add the list of files in each directory to the file list. It does the right thing but it has a couple of undesired effect:
1) I expect the content of design3/c_proj3.f at the top but it appears in the end.
%make FL1
%more c_proj1.f
./design1/c_files/a.c
./design1/c_files/b.c
./design1/c_files/c.c
./design1/c_files/d.c
./design1/c_files/e.c
./design1/c_files/f.c
./design3/c_files/x.c
./design3/c_files/y.c
./design3/c_files/z.c
I expected x, y, z at the top as there is order dependence but its at the bottom. Also, if I do make FL1 --dry-run after removing c_proj1.f the files a through f gets into it but not x,y,z because that is from a cat shell command. I would prefer no files get created. Is there a way to resolve this?. This is a partial solution so I am not sure if this needs to go back into the question.

exclude file from rm in makefile

I have a makefile that has a clean PHONY like this:
.PHONY: clean
clean: clean_target
# Remove files created by the build process.
clean_target:
rm -rf *.mcs *.bit *.bin *.twr *.pwr *.tsi *.twx *.ncd *.pcf *.ngd *.ngc
I want to exclude a file named "FIFO.ngc" from being removed. How can I do that?
thanks,
If you're using GNU make you could do it like this:
clean_target:
rm -rf *.mcs *.bit *.bin *.twr *.pwr *.tsi *.twx *.ncd *.pcf *.ngd \
$(filter-out FIFO.ngc,$(wildcard *.ngc))
A bit lengthy, but you could remove the '*.ngc' from the list and add another line:
find . -type f -name '*.ngc' -not -name 'FIFO.ngc' -delete

Nested for loop not working in makefile

I am trying to use a nested for loop for searching and copying some files inside the recipe of one of the targets inside a makefile:
DIR = $(DIR_A) $(DIR_B)
install:
for dirs in $(DIR); do \
for file in $(shell find $(dirs) -type f -and -not -path "*/.svn*" | sed -e "s|$(dirs)||"); do \
folder=$${file%/*}; \
$(INSTALL) -d $(DEST_DIR)$$folder/log; \
$(INSTALL) $(dirs)/$$file $(DEST_DIR)$$folder/log; \
done \
done
However $(dirs) variable always evaluates to empty inside the second for loop and the current working directory gets passed to "find" instead of first directory path from $(DIR).
Can someone please suggest if I am missing something ?
Thanks.
The $(dirs) in the find command is being expanded by make to the make variable dirs which is unset and thus the empty string. To reference the shell variable, you need to escape the $:
for file in $$(find $${dirs} -type f -and \
-not -path "*/.svn*" | sed -e "s|$${dirs}||"); do
But don't do this. It is much cleaner to explicitly list the files you intend to install. If there are many, it is fine to write a script to generate the Makefile. What you are doing is a fragile mess.
You have made several errors, and you will find it almost impossible to solve them as l;ong as you insist on trying to solve them all at once.
Let's try this in stages. Suppose your DIR_A and DIR_B are north and south. On the command line, if you try this:
for dirs in north south; do echo $dirs; done
you will get the correct output:
north
south
If you try it as a makefile rule:
install:
for dirs in $(DIR); do echo $$dirs; done
again, it works correctly.
If you try your makefile recipe:
install:
for dirs in $(DIR); do \ for file in $(shell ls $$dirs); do \ echo "file is $$file"; \ done \ done
it fails, because Make expands the $(shell ...) command before passing the entire for command to the shell, when dirs has not yet been assigned a value. One way to construct a sensible shell command is to use backticks:
for dirs in north south; do for file in `ls $dirs`; do echo "file is $file"; done done
This works on the command line. A makefile rule built around it:
install:
for dirs in $(DIR); do for file in `ls $$dirs`; do echo "file is $$file"; done done
also works.
That should be enough to allow you to rewrite your makefile.

Makefile rule to conditionally run script and then copy files

I have a Makefile, I'm trying to create a rule that will build RPMs if they don't already exist (based on looking at particular directory) otherwise just copy those RPMs to some other directory.
Essentially what I'd like to do is:
if somedir/build does NOT has RPMs:
somedir/build.sh
cp -R somedir/build/* destination
I've been toying with using $(shell if [[ -d somedir/build ]] ....) but not having much luck.
This is working:
someproject_rpms:
#if [[ -d somedir/build ]] ; then \
count=$$(find somedir/build -name "*.rpm" -type f | wc -l); \
if [[ $$count -eq 0 ]]; then \
$$(cd somedir && ./build.sh); \
fi \
else \
$$(cd somedir && ./build.sh); \
fi; \
find somedir/build -name "*.rpm" -type f -exec cp {} somedestination \;
Not the prettiest, but it works.
You should use so-called "sentinel" files. These are empty files, that are used only for their timestamp, and the timestamp signifies the completion of some action, typically creating a bunch of other files, whose names we may or may not know. The two files named rpms are the sentinel files, and here is your solution Makefile:
destination/rpms: somedir/build/rpms Makefile
find $(<D) -name *.rpm -exec cp {} $(#D) \;
touch $#
somedir/build/rpms: $(PACKEES) Makefile
somedir/build.sh
touch $#
PACKEES right now is empty, but you should set it to be all the files you want to have packed into any of your rpms, in other words, all the files that build.sh reads. If you insist on using one build.sh script to build all your rpms at once (which is not a good idea), the above is the best you are going to get :)

Linux bash-script to run make in all subdirectories

I'm trying to write a bash-script in Linux which traverses the current directory and, in every subdirectory, it launches the existing makefile. It should work for each subdirectory, regardless of depth.
Some restrictions:
I cannot use Python;
I don't know in advance how many subdirectories and their names;
I don't know in advance the name of current directory;
the make command for each directory should only be launched if there is makefile in such folder.
Any ideas on how to do it?
Using -exec and GNU make
find -type f \( -name 'GNUmakefile' -o -name 'makefile' -o -name 'Makefile' \) \
-exec bash -c 'cd "$(dirname "{}")" && make' \;
Given that this is make-related. I'd try to use a makefile at the top-level instead of a script. Something like this:
MAKEFILES:=$(shell find . -mindepth 2 -name Makefile -type f)
DIRS:=$(foreach m,$(MAKEFILES),$(realpath $(dir $(m))))
.PHONY: all
all: $(DIRS)
.PHONY: $(DIRS)
$(DIRS):
$(MAKE) -C $#
I'd accept what #MLSC says about using for with find, and that kind of applies here too .. the problem with that is when you have a space in the directory name. However, in many cases that's not going to happen, and IMHO there are benefits in using a makefile instead of a script. (There might be a solution using make that can cope with spaces in the directory name, but I can't think of it off the top of my head.)
You can use this script https://gist.github.com/legeyda/8b2cf2c213476c6fe6e25619fe22efd0.
Example usage is:
foreach */ 'test -f Makefile && make'
This should work if dont care about the execution order or if parent directory also has a Makefile.
#!/bin/bash
for f in $(find . -name Makefile); do
pushd $(dirname $f)
make
popd
done

Resources