Make "make" change-track two directories in one rule - makefile

I'm using GNU Make, but I'm willing to change to something else if necessary.
I've got two directories ALPHABETIC and NUMERIC with the following contents.
./ALPHABETIC:
A.txt B.txt C.txt
./NUMERIC:
1.txt 2.txt 3.txt
and a program foo which takes as input a file in ALPHABETIC and a file in NUMERIC and outputs something.
I want to create a rule such that: If a file c in ALPHABETIC changes, re-run foo on c and every file in NUMERIC. If a file n in NUMERIC changes, re-run foo on n and every file in ALPHABETIC.
Example: A.txt changes. Then what should happen is:
foo A.txt 1.txt
foo A.txt 2.txt
foo A.txt 3.txt
Let's say now 2.txt changes. Then what should happen is
foo A.txt 2.txt
foo B.txt 2.txt
foo C.txt 2.txt
I tried doing this with pattern rules but it didn't work.

As make is designed to manage build systems that produce files from files, let's first assume that foo A.txt 1.txt produces a file named A.1.txt in the main directory. You could try something like:
.PHONY: all
.DEFAULT_GOAL := all
ALPHA := $(patsubst ALPHABETIC/%.txt,%,$(wildcard ALPHABETIC/*.txt))
NUM := $(patsubst NUMERIC/%.txt,%,$(wildcard NUMERIC/*.txt))
# $(1): ALPHA
# $(2): NUM
define ALPHANUM_rule
$(1).$(2).txt: ALPHABETIC/$(1).txt NUMERIC/$(2).txt
#echo foo $(1).txt $(2).txt && \
touch $$#
all: $(1).$(2).txt
endef
$(foreach a,$(ALPHA),$(foreach n,$(NUM),$(eval $(call ALPHANUM_rule,$(a),$(n)))))
Note the use of echo and touch to emulate the effect of your true foo command. The most important thing to understand is the foreach-eval-call construct, including the reason why some $ signs need to be doubled ($$). See The eval Function from the GNU make manual for a detailed explanation.
Demo (host> is the shell prompt):
host> make
foo B.txt 2.txt
foo B.txt 3.txt
foo B.txt 1.txt
foo A.txt 2.txt
foo A.txt 3.txt
foo A.txt 1.txt
foo C.txt 2.txt
foo C.txt 3.txt
foo C.txt 1.txt
host> make
make: Nothing to be done for 'all'.
host> touch NUMERIC/1.txt
host> make
foo B.txt 1.txt
foo A.txt 1.txt
foo C.txt 1.txt
host> make
make: Nothing to be done for 'all'.
host> touch ALPHABETIC/C.txt
host> make
foo C.txt 2.txt
foo C.txt 3.txt
foo C.txt 1.txt
host> make
make: Nothing to be done for 'all'.
But what if foo A.txt 1.txt does not produce a file? In this case, the simplest thing to do is to produce empty files anyway. We will create them in a separate directory for easier cleaning:
.PHONY: all
.DEFAULT_GOAL := all
ALPHA := $(patsubst ALPHABETIC/%.txt,%,$(wildcard ALPHABETIC/*.txt))
NUM := $(patsubst NUMERIC/%.txt,%,$(wildcard NUMERIC/*.txt))
TAGSDIR := tags
$(TAGSDIR):
#mkdir -p $#
# $(1): ALPHA
# $(2): NUM
define ALPHANUM_rule
$(TAGSDIR)/$(1).$(2).txt: ALPHABETIC/$(1).txt NUMERIC/$(2).txt | $(TAGSDIR)
#echo foo $(1).txt $(2).txt && \
touch $$#
all: $(TAGSDIR)/$(1).$(2).txt
endef
$(foreach a,$(ALPHA),$(foreach n,$(NUM),$(eval $(call ALPHANUM_rule,$(a),$(n)))))
There are two more things to understand:
The use of empty files ($(TAGSDIR)/A.1.txt...) to keep track of what has been done already and when.
The Order-only prerequisite $(TAGSDIR) used to guarantee that the directory containing the empty files is created before being used, without forcing re-builds every time its content is modified.

Related

filter-out on shell output

There are four folders in my src directory: dir1, dir2, dir3, dir4.
PARSE_MODULES = $$(MODULES=$$(ls -l $1 | grep '^d' | awk '{print $$9}'); echo $$MODULES);
MODULES:=$(call PARSE_MODULES,src)
all:
#echo $(MODULES)
Hence, this little piece of code outputs
dir1 dir2 dir3 dir4
Now, I add the following two lines before the all rule:
X:=dir1 dir3
MODULES:=$(filter-out $(X),$(MODULES))
The output remains unchanged, i.e., filter-out is not working as expected. However, if I manually define MODULES, I get the expected output:
MODULES:=dir1 dir2 dir3 dir4
X:=dir1 dir3
MODULES:=$(filter-out $(X),$(MODULES))
Output: dir2 dir4
Why is it impossible to process the output of awk in filter-out? Is there any way to do so?
Your problem is that you have deferred expansion of the contents of PARSE_MODULES by using $$, so that it's being run by the shell in the recipe, not by make.
This:
PARSE_MODULES = $$(MODULES=$$(ls -l $1 | grep '^d' | awk '{print $$9}'); echo $$MODULES);
MODULES:=$(call PARSE_MODULES,src)
leaves the MODULES variable containing this text:
$(MODULES=$(ls -l src | grep '^d' | awk '{print $9}'); echo $MODULES);
Then when you use $(MODULES) in the recipe, it expands like this:
all:
#echo $(MODULES=$(ls -l src | grep '^d' | awk '{print $9}'); echo $MODULES);
then the shell will run the command inside $(...) and echo the results.
When you try to filter out names like dir1 etc. from the value of the make variable MODULES, nothing happens because the make variable MODULES contains the shell script itself, as text, it doesn't contain the results of executing the shell script.
I'm not sure why you are trying to do things in such an incredibly convoluted way, with so much escaping and using difficult to understand shell scripting. Maybe there's a reason that your full environment needs something so complicated but you can FAR more easily implement the example above just using some basic make functions:
PARSE_MODULES = $(sort $(notdir $(patsubst %/.,%,$(wildcard $1/*/.))))
MODULES := $(call PARSE_MODULES,src)
Now the PARSE_MODULES is expanded by make, not by the recipe, and filter-out etc. will work fine on it. Plus it's a lot simpler then doing a shell with grep and awk.
As far as I understand the shell list $(MODULES=$(ls -l src | grep '^d' | awk '{print $9}'); echo $MODULES) is evaluated by the shell when the recipe of all is executed. This is too late. Try this, maybe:
PARSE_MODULES = $(shell ls -l $1 | grep '^d' | awk '{print $$9}')
MODULES := $(call PARSE_MODULES,src)
X := dir1 dir3
MODULES := $(filter-out $(X),$(MODULES))
all:
#echo $(MODULES)
Minor simplification from OP version:
# possibly: find -maxdepth 1
PARSE_MODULES = $(shell find $1 -type d -printf "%P\n")
SKIP := dir1 dir3
MODS := $(call PARSE_MODULES,src)
FLTR := $(filter-out $(SKIP), $(MODS))
all:
#echo MODS: $(MODS)
#echo SKIP: $(SKIP)
#echo FLTR: $(FLTR)
Output:
MODS: dir1 dir2 dir3 dir4
SKIP: dir1 dir3
FLTR: dir2 dir4

simple loop over files in some directory makefile

I can easily print all the files inside some directory from bash:
$ cat go.sh
BASEDIR=~/Downloads
MYDIR=${BASEDIR}/ddd
for f in $(ls ${MYDIR}); do echo $f; done
$ ./go.sh
m.txt
d.txt
When I try to do a similar thing from makefile it doesn't work well:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
for f in $(ls ${MYDIR}); do echo ${f}; done
$ make
for f in ; do echo ; done
And here is another trial that doesn't work:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
for f in $(shell ls ${MYDIR}); do echo ${f}; done
$ make
for f in d.txt m.txt; do echo ; done
Maybe you can do it purely Makefile way?
MYDIR = .
list: $(MYDIR)/*
#echo $^
You can still run command from Makefile like this
MYDIR = .
list: $(MYDIR)/*
for file in $^ ; do \
echo "Hello" $${file} ; \
done
If I were you, I'd rather not mix Makefile and bash loops based on $(shell ...). I'd rather pass dir name to some script and run loop there - inside script.
Also almost "true way" from documentation
TEMPLATES_DIR = ./somedir
list:
$(foreach file, $(wildcard $(TEMPLATES_DIR)/*), echo $(file);)
Here is the edited answer based on #Oo.oO:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
#for f in $(shell ls ${MYDIR}); do echo $${f}; done
$ make
d.txt
m.txt
There is a little problem with #Oo.oO's answer.
If there is any file/folder has the same name with a target in makefile, and that target has some prerequisites, and you want to loop through that folder, you will get that target recipe being executed.
For example: if you have a folder named build, and you have a rule like:
build: clean server client
clean:
#echo project cleaned!
server:
#echo server built!
client:
#echo client built!
To loop through the folder contains that special build folder, let's says you have the following rules:
MYDIR = .
ls: $(MYDIR)/*
#echo $^
The result will be:
$ make ls
project cleaned!
server built!
client built!
build Makefile
I would suggest to use #Mike Pylypyshyn's solution. According to the make documentation, the foreach function is more suitable in this case.

Ignoring file with similar names from compiling in Makefile

Suppose we have a directories two foo and bar. Where foo contains files example.S, run.c, exec.S and bar contains example.S
when I pass foo and bar as dependencies.
I want to compile only example.S in the directory bar by ignoring example.S in directory foo
DEP := foo bar
DIR := $(foreach $dirs, $(DEP), $(shell $(HOME) -find -type d -name $(dirs))),
would provide me the absolute path of the foo and bar directories.
FILES := $(foreach file, $(DIR), $(wildcard $(file)/*)), would provide me the list of files foo and bar as
foo/example.S foo/run.c foo/exec.S bar/example.S
I would like to check for multiple occurance of file with same name. and only compile the latest, i.e, instead of foo/example.S I like to compile bar/example.S...
I have no idea how to do it, As I am very new to creation of Makefile.
This is actually more of a shell question than a makefile one. What you want to do is create a shell command that generates two column list for each file -- the first column being the full filename, and the second being just the filename. Then sort it, and remove the ones with duplicate filenames, and then output only the first column.
Basically you would use:
DIRS = ./A ./B
files=`find $(DIRS) -type f -exec sh -c 'echo {} $$(basename {})' \; | sort -u --stable -k2,2 | awk '{print $$1}'`
You would have to replace the ./B ./A with a sorted list of directories you wanted to search in. Notice the $$'s -- Make resolves these to $ before running the shell command. The $(DIRS) only has a single $, so it is expanded before the command is executed.
Explanation:
find ./B ./A -type f -exec sh -c 'echo {} $(basename {})' \; : This searches the directories B and then A (in that order)
type -f specifies files only -- it won't return directories.
-exec sh -c 'echo {} $(basename {})' \; : for each file found, it will run the echo command where {} resolves to the file it found. It will therefore print the full file path, followed by just the filename.
sort -u --stable -k2,2: Sort the list of files based on column 2 (the filename). The -u means unique, so if there are matching files, it only prints the first. The --stable means that if there are two matching lines, it will always output the first one.
awk '{print $1}': print the first column of the output (the full path name).
You then have a list of unique filenames, including their directories in $(files)
There's a (relatively) simple way to get what you want, starting with reversing a list (with thanks to #simona). After you have calculated DIR your way,
reverse = $(if $(1),$(call reverse,$(wordlist 2,$(words $(1)),$(1)))) $(firstword $(1))
DIR := $(call reverse,$(DIR))
Then use VPATH:
VPATH = $(DIR)
Now you can tell Make to search for a file (such as example.S), and it will find the last instance (in this case .../bar/example.S):
example: example.S
#echo building $# from $^

why are the prerequisites not set as expected?

I'm trying to write this makefile where the prerequisites for a target should be the output of a shell command. here's what I have
foo: ($shell find mydir -name "*.ext")
# command goes here
where mydir is a directory under the same directory where the makefile itself. however, executing make foo executes the recipes even if none of the *.ext files were modified (i.e., it always executes the recipes acting as if there are prerequisites set).
I'm sure that the find command outputs the expected results.
any idea what might be going wrong here and how to get it to behave expectedly?
A little test to see if it is working. Here the Makefile:
foo: $(shell find mydir -name "*.ext")
cat $^ > $#
And the file tree:
.
├── Makefile
└── mydir
├── bar.ext
└── foo.ext
Then run the Makefile:
$ make
cat mydir/foo.ext mydir/bar.ext > foo
$ make
make: `foo' is up to date.
$ touch mydir/foo.ext
$ make
cat mydir/foo.ext mydir/bar.ext > foo
So it's working well. Just be sure that your foo file is created by the build rule.

Makefile loop results from ls command

I have a list of text files, which when executing the following command:
ls *.txt
Will result in something like:
foo.txt bar.txt baz.txt
Now if I want to have the output be something like:
File: foo.txt
File: bar.txt
File: baz.txt
How would I achieve that in a Makefile?
I've been trying:
txtfiles = $$(ls *.txt)
list: $(txtfiles)
$(txtfiles):
echo "File:" $#
But when running this with make list, it results with:
File: $
File: $
I was thinking of trying to achieve this using placeholders % but wasn't sure how to do that.
Note: instead of $$(ls *.txt) I'm guessing I could use $(wildcard *.txt) ?
Note: instead of $$(ls *.txt) I'm guessing I could use $(wildcard *.txt)?
You have to.
Also add .PHONY: $(txtfiles) before $(txtfiles): rule to make an explicit request.
OK so I figured this out:
my_list = $(addsuffix .dep, $(wildcard *.txt))
print_list: $(my_list)
%.dep: %
#echo "Text File:" $<
Run it with make print_list to get back the expected result.

Resources