Nested for loop not working in makefile - shell

I am trying to use a nested for loop for searching and copying some files inside the recipe of one of the targets inside a makefile:
DIR = $(DIR_A) $(DIR_B)
install:
for dirs in $(DIR); do \
for file in $(shell find $(dirs) -type f -and -not -path "*/.svn*" | sed -e "s|$(dirs)||"); do \
folder=$${file%/*}; \
$(INSTALL) -d $(DEST_DIR)$$folder/log; \
$(INSTALL) $(dirs)/$$file $(DEST_DIR)$$folder/log; \
done \
done
However $(dirs) variable always evaluates to empty inside the second for loop and the current working directory gets passed to "find" instead of first directory path from $(DIR).
Can someone please suggest if I am missing something ?
Thanks.

The $(dirs) in the find command is being expanded by make to the make variable dirs which is unset and thus the empty string. To reference the shell variable, you need to escape the $:
for file in $$(find $${dirs} -type f -and \
-not -path "*/.svn*" | sed -e "s|$${dirs}||"); do
But don't do this. It is much cleaner to explicitly list the files you intend to install. If there are many, it is fine to write a script to generate the Makefile. What you are doing is a fragile mess.

You have made several errors, and you will find it almost impossible to solve them as l;ong as you insist on trying to solve them all at once.
Let's try this in stages. Suppose your DIR_A and DIR_B are north and south. On the command line, if you try this:
for dirs in north south; do echo $dirs; done
you will get the correct output:
north
south
If you try it as a makefile rule:
install:
for dirs in $(DIR); do echo $$dirs; done
again, it works correctly.
If you try your makefile recipe:
install:
for dirs in $(DIR); do \ for file in $(shell ls $$dirs); do \ echo "file is $$file"; \ done \ done
it fails, because Make expands the $(shell ...) command before passing the entire for command to the shell, when dirs has not yet been assigned a value. One way to construct a sensible shell command is to use backticks:
for dirs in north south; do for file in `ls $dirs`; do echo "file is $file"; done done
This works on the command line. A makefile rule built around it:
install:
for dirs in $(DIR); do for file in `ls $$dirs`; do echo "file is $$file"; done done
also works.
That should be enough to allow you to rewrite your makefile.

Related

Bash: Go up and down directories

Dear stackoverflow community,
I am new to bash and I've got a problem regarding loops and directories (code below). So I am working in the opensmile directory and want to look for .wav files in the subdirectory opensmile/test-audio/. But if I change my directory in the "for" section to test-audio/*.wav, it probably could find the .wav-files but then the main-action does have access to the necessary config file "IS10_paraling.conf". Within the main-action the directories have to be written like after "-C", so without a "/" before the directory.
My loop works, if the wav files are inside the opensmile directory, but not inside a sub-directory. I would like to look for files in the subdirectory test-audio while the main-action still has access to all of the opensmile-directory.
So basically: How do I go up and down directories within a for loop?
Thank you very much in advance!
This works
#! /bin/bash
cd /usr/local/opensmile/
for f in *.wav;
do
/usr/local/opensmile/build/progsrc/smilextract/SMILExtract -C config/is09-13/IS10_paraling.conf -I $f -D output/$f.csv ;
done
This does not work
#! /bin/bash
cd /usr/local/opensmile/
for f in test-audio/*.wav;
do
/usr/local/opensmile/build/progsrc/smilextract/SMILExtract -C config/is09-13/IS10_paraling.conf -I $f -D output/$f.csv ;
done
Saying "this does not work", doesn't tell us anything. What happens? Is there an error message?
Nevertheless, your question was "So basically: How do I go up and down directories within a for loop?"
If I'm tempted to go up and down directories within a loop, I'll do it in a subshell, so that I can be sure that the next time I enter the loop I'll be where I was originally. So I'll put all my commands in ( ).
#! /bin/bash
cd /usr/local/opensmile/
CONFIG=$PWD/config
OUTPUT=$PWD/output
for f in test-audio/*.wav;
do
(
cd test-audio
/usr/local/opensmile/build/progsrc/smilextract/SMILExtract -C $CONFIG/is09-13/IS10_paraling.conf -I `basename $f` -D $OUTPUT/$f.csv
)
done
though why one would need to to it for this case, I can't fathom
Instead of using a for loop, could you use find for this:
find /usr/local/opensmile/ -type f -name "*.wav" -exec /usr/local/opensmile/build/progsrc/smilextract/SMILExtract -C config/is09-13/IS10_paraling.conf -I $1 -D output/$1.csv "{}" \;

Makefile trailing error "Command not found" when calling custom shell function

I have a simple Makefile:
define git_all
ls --recursive --directory --color=never */.git \
| sed 's/\/.git//' \
| xargs --no-run-if-empty --max-procs=10 --replace={} git -C '{}' $1 || true
endef
gitpull:
#$(call git_all,pull -v)
The 'gitpull' rule is supposed to detect any and all repositories in the subfolders that are under the folder that the Makefile is in. For example:
Makefile
\- a/.git
\- b/.git
...
When I run 'make gitpull' the scripts works fine with one minor glitch. If a repository is already up to date I get this weird trailing error:
> make gitpull
make: Already: Command not found
make: *** [Makefile:xxx: gitpull] Error 127
I guess the shell is trying to interpret the text 'Already up to date' as a command and its failing. But why does it try to do that in the first place? I'm obviously missing something but what?
You seem to be confused as to what $(shell ...) does. If you want this to happen when the target is run, not when the Makefile is parsed, you want to take out the $(shell ...).
Also, don't use ls in scripts.
If I'm able to guess what you are trying to do, probably something like
define git-all
find . -type d -name '.git' -execdir sh -c 'cd ..; git -C $1' _ {} \;
endef
here:
$(call git-all,pull -v)

Makefile, substitute paths in prerequisites automatic variable

I have following make rule:
$ ls files
a b c
$ cat Makefile
files.7z: ./files/*
7z a $# $?
The rule will be executed as follows:
7z a files.7z files/a files/b files/c
7z treats paths beginning with ./ especially in that, it will include the file in the archive without the path. What would be the shortest way to replace the paths, to begin with, ./ in the $?? (Files names can have spaces.)
Makefile doesn't handle spaces very well at all (see here). The simplest thing is to make make fail if there are spaces and then it's easy:
check:
# ls ./files/* | grep -q " "; \
if [ $$? -ne 1 ]; then \
echo "WE DO NOT SUPPORT FILENAMES WITH SPACES"; \
exit 1; \
fi
files.7z: ./files/* | check
7z a $# $(addprefix ./,$^)
Note: there are workarounds on the web for filenames with spaces, but they're overly complex, not very maintainable and will cost you a lot more headaches than telling your customers/coworkers "We don't support file names with spaces"...

Makefile rule to conditionally run script and then copy files

I have a Makefile, I'm trying to create a rule that will build RPMs if they don't already exist (based on looking at particular directory) otherwise just copy those RPMs to some other directory.
Essentially what I'd like to do is:
if somedir/build does NOT has RPMs:
somedir/build.sh
cp -R somedir/build/* destination
I've been toying with using $(shell if [[ -d somedir/build ]] ....) but not having much luck.
This is working:
someproject_rpms:
#if [[ -d somedir/build ]] ; then \
count=$$(find somedir/build -name "*.rpm" -type f | wc -l); \
if [[ $$count -eq 0 ]]; then \
$$(cd somedir && ./build.sh); \
fi \
else \
$$(cd somedir && ./build.sh); \
fi; \
find somedir/build -name "*.rpm" -type f -exec cp {} somedestination \;
Not the prettiest, but it works.
You should use so-called "sentinel" files. These are empty files, that are used only for their timestamp, and the timestamp signifies the completion of some action, typically creating a bunch of other files, whose names we may or may not know. The two files named rpms are the sentinel files, and here is your solution Makefile:
destination/rpms: somedir/build/rpms Makefile
find $(<D) -name *.rpm -exec cp {} $(#D) \;
touch $#
somedir/build/rpms: $(PACKEES) Makefile
somedir/build.sh
touch $#
PACKEES right now is empty, but you should set it to be all the files you want to have packed into any of your rpms, in other words, all the files that build.sh reads. If you insist on using one build.sh script to build all your rpms at once (which is not a good idea), the above is the best you are going to get :)

Makefile: depend on every file of a directory

I'd like to do a Makefile that runs either with gnumake or makepp that packs all the files under given directiories:
DIRS:=$(shell find . -mindepth 2 -maxdepth 2 -not -name mp3 -not -name ".*" -type d)
PACKAGES = $(DIRS:%=%.npk)
all: packages
packages: $(PACKAGES)
%.npk: %/*
npack c $# #^
.PHONY: all packages
the problem is that there's no such thing as %/* in the dependencies.
I need the targets (X.npk) to depend on every file in directory X, but I don't know what the files are when I write the Makefile, 'cause they're generated later.
An example:
./dirA/x
./dirA/y
./dirB/e
./dirB/f
I'd like to create ./dirA.npk (depending on x,y), ./dirB.npk (e,f)
There's nothing I know about the dirs or the files in advance except that the find used in the 1st line finds all the dirs.
Try using the wildcard directive:
DEPS := $(foreach dir, $(DIRS), $(wildcard $(dir)/*))
%.npk: $(DEPS)
npack c $# $^
EDIT:
The above is just an example of using wildcard and makes each .npk file dependent on the files in all of the other folders. Your usage would be slightly different.
I think there may be an easier way to go about this. Why are you wanting to have a dependency on all of the files in the folder? Is it just to use the $^ operator? Or do you need to rebuild the .npk if any of the files changed?
One alternate (and possibly cleaner) solution would be to use the find utility in your recipe instead of $^ and use the .FORCE directive to always force the .npk file to be rebuilt. The downside is that .npk files may be rebuilt unnecessarily.
EDIT 2:
If there's not a way to do this cleanly with make commands, you can work around it by using .FORCE to ensure that the recipe is always run and move the "should I rebuild this file" check into the body of the recipe:
%.npk: .FORCE
check_for_rebuild.sh $# && npack c $# $^
where check_for_rebuild.sh is a shell script that does something like this:
#!/bin/bash
# Returns non-zero if the archive needs to be rebuilt
if [ -e $1 ]; then
folder_name=$(basename $1 .npk)
[ -z "$(find $folder_name -newer $1 -not -type d)" ] && return 0
fi
return 1
I don't really like that solution because it works around the problem instead of solving it directly, but it may be able to get you going in the meantime. If you are going to go that route, it's probably cleaner and easier to do everything in the shell script and either have the makefile simply invoke the script or get rid of the makefile entirely.
This is the solution I found:
it is based on the makedepend idea, with some "meta" scripting. Not very nice, but works.
PACKAGES :=
all: packages
-include Makefile.depend
packages: Makefile.depend $(PACKAGES)
depend: clean Makefile.depend
Makefile.depend:
#(PACKAGES= ; \
for DIR in `find . -mindepth 2 -maxdepth 2 -not -name mp3 -not -name ".*" -type d` ; \
do \
PACKAGE=`basename $${DIR}.npk` ; \
PACKAGES="$${PACKAGES} $${PACKAGE}" ; \
DEPS=`find $${DIR} -not -type d | sed -e 's#\([: ]\)#\\\\\1#' -e 's#^\./\(.*\)# \1#' | tr -d "\n"` ; \
SUBDIR=`echo $${DIR} | sed -e 's#^\./\([^/]\+\)/.*#\1#'` ; \
FILES=`echo \ $${DEPS} | sed -e "s# $${SUBDIR}/# #g"` ; \
echo "$${PACKAGE}:$${DEPS}" ; \
echo " #cd $${SUBDIR} ; \\" ; \
echo " npack c ../\$$# $${FILES} ; \\" ; \
echo ; \
done ; \
echo "PACKAGES = $${PACKAGES}" \
)>> Makefile.depend ; \
cleanall: clean
rm -f *.npk
clean:
#rm -f Makefile.depend
.PHONY: all packages depend clean
With makepp you can do this in 2 steps, via the :foreach rule modifier:
$(foreach).txt: $(foreach)/*: foreach */
&echo $(inputs) -o $(output)
This provides a rule for every subdirectory, which reexecutes whenever there is a change in the list of files therein.

Resources