This question already has answers here:
Makefile removes object files for no reason
(2 answers)
How to undo intermediate file deletion
(3 answers)
Closed 8 years ago.
Here is a Makefile which creates several pdfs from DOT sources. It creates PostScript files as an intermediate step.
all: maps.pdf
D=source_directory
S := $(shell find $D -name '*.dot')
P = $(patsubst $D/%.dot, $D/%.ps, $S)
F = $(patsubst $D/%.dot, $D/%.pdf, $S)
$D/%.ps: $D/%.dot Makefile
dot -Tps:cairo $< > $#
$D/%.pdf: $D/%.ps
ps2pdf $< $#
maps.pdf: $F
pdfjam -q $D/*.pdf -o maps.pdf
It runs fine except that it deletes all the PostScript files after with it runs pdfjam. In fact, it even prints rm source_dir/foo.ps source_dir/bar.ps to standard output!
Where is the rm command coming from?
Related
I just want to copy files from other directories (before doing something). Because it's tedious to write the copy command for each file, I tried
%: ../src1/%
#echo cp $^ .
%: ../src2/%
#echo cp $^ .
all: file1 file2 file3 file4
# do something
but this doesn't work because make tries to look into ../src2/../src2/../src2/../ . . . . (I included echo for testing to prevent actual copying from happening. I keep forgetting what the "dry run" command line options is . . .)
I naïvely thought that there must be a way to force matches only to filenames that don't include directories.
Is there a way?
You can mark the "make anything" rules terminal with a double colon:
%:: ../src1/%
#echo cp $^ .
%:: ../src2/%
#echo cp $^ .
This does not answer your specific question of how to get Make to match only filenames without directories, but it does get Make to do what you want.
There is another approach that works and is closer to what you asked for: add pattern rules to satisfy Make:
../src1/%:
#: # do nothing
%: ../src1/%
#echo cp $^ .
EDIT: Or better still, us one dummy pattern rule to cover all source directories:
../%:
#: # do nothing
When I extract files from a tar ball to create a list of files make does not follow symbolic links and keeps extracting the same tar ball for each file specified in the list. Forinstance I have created 2 tar balls with the following commands in linux:
design1 is a directory that holds a link to another directory with c design files
% mkdir design1
% mkdir c_files
% cd c_files/
% touch a.c b.c c.c d.c e.c f.c
% cd ../design1/
% ln -s ../c_files/ .
% cd ..
% gtar zcvf design1.tgz design1/
design2 is the same as design1 except that c_files is a directory and not a link
% mkdir design2
%
% cp -rp c_files design2
% gtar zcvf design2.tgz design2/
Now there are 2 zipped tarfiles design1.tgz and design2.tgz and design1 and design2 directories are removed
% \rm -rf design1 design2
Here's my Makefile:
----------------Begin Makefile -------------------
DESIGN_EXTRACT_DIR := .
DESIGN_TOP1 := c_proj1
FILE_LIST1 := $(strip $(DESIGN_TOP1)).f
DESIGN_TAR_FILE1 := design1.tgz
DESIGN_RELEASE_DIR1 := $(DESIGN_EXTRACT_DIR)/design1
DESIGN_FILES1 =\
$(DESIGN_RELEASE_DIR1)/c_files/a.c \
$(DESIGN_RELEASE_DIR1)/c_files/b.c \
$(DESIGN_RELEASE_DIR1)/c_files/c.c \
$(DESIGN_RELEASE_DIR1)/c_files/d.c \
$(DESIGN_RELEASE_DIR1)/c_files/e.c \
$(DESIGN_RELEASE_DIR1)/c_files/f.c \
IP_EXTRACT1: $(DESIGN_FILES1)
$(DESIGN_FILES1): $(DESIGN_TAR_FILE1)
gtar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST1): $(DESIGN_FILES1)
#( rm -f $(FILE_LIST1) )
#( touch $(FILE_LIST1) )
#$(foreach file, $(DESIGN_FILES1), `echo $(file) >> $(FILE_LIST1)`)
FL1: $(FILE_LIST1)
DESIGN_TOP2 := c_proj2
FILE_LIST2 := $(strip $(DESIGN_TOP2)).f
DESIGN_TAR_FILE2 := design2.tgz
DESIGN_RELEASE_DIR2 := $(DESIGN_EXTRACT_DIR)/design2
DESIGN_FILES2 =\
$(DESIGN_RELEASE_DIR2)/c_files/a.c \
$(DESIGN_RELEASE_DIR2)/c_files/b.c \
$(DESIGN_RELEASE_DIR2)/c_files/c.c \
$(DESIGN_RELEASE_DIR2)/c_files/d.c \
$(DESIGN_RELEASE_DIR2)/c_files/e.c \
$(DESIGN_RELEASE_DIR2)/c_files/f.c \
IP_EXTRACT2: $(DESIGN_FILES2)
$(DESIGN_FILES2): $(DESIGN_TAR_FILE2)
gtar zxvfmhC $(DESIGN_TAR_FILE2) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST2): $(DESIGN_FILES2)
#( rm -f $(FILE_LIST2) )
#( touch $(FILE_LIST2) )
#$(foreach file, $(DESIGN_FILES2), `echo $(file) >> $(FILE_LIST2)`)
FL2: $(FILE_LIST2)
clean:
#( rm -rf $(FILE_LIST1)* $(FILE_LIST2)* )
----------------End Makefile----------------------
Now when I run make to create a file list I use the commands make FL1 and make FL2. In the case of FL1 make iterates of gtar as many times as I have files but doesn't do so with FL2. The only difference here is FL1 operates on a link called c_files in design1 while FL2 iterates over c_files as a directory.
Here's what I see:
% make FL1
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
%make FL2
%
Issue1:
FL1 creates c_proj1.f and FL2 creates c_proj2.f but FL2 doesnt iterate like FL1 and I am not sure how to prevent this iteration especially since I could have large tar ball with hundreds of files.
Issue2:
When DESIGN_FILES1 is a long list because of the number of files then I get the following error because the variable is too long:
make: execvp: /bin/sh: Argument list too long
Is there a way to check the size of the variable and maybe write to a file and process the variable a bit at a time so DESIGN_FILES will not be too long for each iteration. Or is there a better way to do this.
Thanks
Niel
For me it works the same (iterates once) whether there is a symlink or not. But from the contents it seems that you are only interested in generating file list, so file existence is not really required. If you omit dependency, you may generate file list without even extracting the files.
For the actual file list write the oversized command line error comes from the fact, that the generated command is a single line with multiple subshell calls, which might be too long for the command line indeed:
$ make FL1 --trace
...
Makefile:24: update target 'c_proj1.f' due to: design1/c_files/a.c design1/c_files/b.c design1/c_files/c.c design1/c_files/d.c design1/c_files/e.c design1/c_files/f.c
( rm -f c_proj1.f )
( touch c_proj1.f )
`echo ./design1/c_files/a.c >> c_proj1.f` `echo ./design1/c_files/b.c >> c_proj1.f` `echo ./design1/c_files/c.c >> c_proj1.f` `echo ./design1/c_files/d.c >> c_proj1.f` `echo ./design1/c_files/e.c >> c_proj1.f` `echo ./design1/c_files/f.c >> c_proj1.f`
This part can be resolved with a $(file) function (if you happen to use GNU make 4.1+):
.PHONY: $(FILE_LIST1)
$(FILE_LIST1):
$(file >$#)
$(foreach file,$(DESIGN_FILES1),$(file >>$#,$(file)))
If an older make is used, this could be done with regular commands, you just need to generate a newline character after each echo to make it work as a separate command. The only tricky part is that it has to be done as a specially crafted variable:
# Note double empty lines below
define newline
endef
.PHONY: $(FILE_LIST2)
$(FILE_LIST2):
rm -f $#
$(foreach file,$(DESIGN_FILES2),echo $(file) >> $#$(newline))
Result is now generated in separate shell calls, which should avoid command length limit:
$ make FL2
rm -f c_proj2.f
echo ./design2/c_files/a.c >> c_proj2.f
echo ./design2/c_files/b.c >> c_proj2.f
echo ./design2/c_files/c.c >> c_proj2.f
echo ./design2/c_files/d.c >> c_proj2.f
echo ./design2/c_files/e.c >> c_proj2.f
echo ./design2/c_files/f.c >> c_proj2.f
EDIT:
I was able to reproduce your multiple iteration of extraction. Your tgz file does not actually contain the files you claim to extract, so the files are not there when the first one is trying to be extracted and the extraction continues for the next file (which is also absent). This is also suggested by tar output, which says:
design1/
design1/c_files
instead of:
design1/
design1/c_files
design1/c_files/a.c
design1/c_files/b.c
design1/c_files/c.c
design1/c_files/d.c
design1/c_files/e.c
design1/c_files/f.c
Your tgz only contains a directory symlink and it only extracts a symlink. If the target files do not exist in the target location, it will attempt to re-extract for every file.
EDIT2:
When actual files are not included in the tgz file, but are available through a symlink, there is a chance that they are older than the tgz file itself. In such situation make checks the file timestamp through a symlink and discovers that it needs to be remade - but the file is not actually remade (since it's not included in the tgz). This check is done for every single file, so for every file the archive is extracted:
$ make FL1 -d
...
Finished prerequisites of target file 'design1/c_files/a.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/a.c'.
Must remake target 'design1/c_files/a.c'.
tar zxvfmhC design1.tgz .
Putting child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286 on the chain.
Live child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286
design1/
design1/c_files
Reaping winning child 0x7fffe11ae730 PID 6286
Removing child 0x7fffe11ae730 PID 6286 from chain.
Successfully remade target file 'design1/c_files/a.c'.
Considering target file 'design1/c_files/b.c'.
Pruning file 'design1.tgz'.
Finished prerequisites of target file 'design1/c_files/b.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/b.c'.
Must remake target 'design1/c_files/b.c'.
tar zxvfmhC design1.tgz .
...
In general extracting archive in this way is not reliable anyway (image running in parallel with -j), so it would be better to use a group target rule to explicitly point out that this rule generates all the files at the same time:
$(DESIGN_FILES1) &: $(DESIGN_TAR_FILE1)
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
This however is a new feature introduced in GNU make 4.3, so to make it work with previous versions, we could use a phony target that is only used for synchronization, e.g.:
$(DESIGN_FILES1): extract-$(DESIGN_TAR_FILE1)
.PHONY: extract-$(DESIGN_TAR_FILE1)
extract-$(DESIGN_TAR_FILE1):
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
Output:
$ make FL1
tar zxvfmhC design1.tgz .
design1/
design1/c_files
Note that the archive is now extracted only once.
I followed your advice and changed the dependency and it resolves the tar issue. The file function in make resolves the length of the variable but I have 2 undesired effect due to it. I have added one more thing to the mix here and I cannot explain the behavior:
On linux:
% mkdir -p design3/c_files
% cd design3/c_files
% touch x.c y.c z.c
% cd ../..
% ls ./design3/c_files/*.c > design3/c_proj3.f
Here's what I have done to the Makefile:
DIR_LIST := design3
IP_EXTRACT1: $(DESIGN_TAR_FILE1)
gtar zxvfmC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(DESIGN_FILES1):
$(MAKE) IP_EXTRACT1
$(FILE_LIST1): $(DESIGN_FILES1)
$(file > $(FILE_LIST1))
#$(if $(strip $(DIR_LIST)), cat $(foreach dir, $(DIR_LIST), $(wildcard $(dir)/*.f)) >> $(FILE_LIST1))
#$(foreach file,$(DESIGN_FILES1),$(file >>$(FILE_LIST1),$(file)))
This resolves the tar issue but now I have added DIR_LIST and I want to add the list of files in each directory to the file list. It does the right thing but it has a couple of undesired effect:
1) I expect the content of design3/c_proj3.f at the top but it appears in the end.
%make FL1
%more c_proj1.f
./design1/c_files/a.c
./design1/c_files/b.c
./design1/c_files/c.c
./design1/c_files/d.c
./design1/c_files/e.c
./design1/c_files/f.c
./design3/c_files/x.c
./design3/c_files/y.c
./design3/c_files/z.c
I expected x, y, z at the top as there is order dependence but its at the bottom. Also, if I do make FL1 --dry-run after removing c_proj1.f the files a through f gets into it but not x,y,z because that is from a cat shell command. I would prefer no files get created. Is there a way to resolve this?. This is a partial solution so I am not sure if this needs to go back into the question.
I want to iterate through a list of files, execute cd in their respective directories before executing a command.
So I would like to use $(dir $(FILES)) function to get the directory, but it seems not to work properly in the for loop.
FILES=../dir1/file1 ../dir2/file2
.PHONY: all
all:
#for f in $(FILES); do \
echo $$f in $(dir $${f}) ; \
done
outputs
../dir1/file1 in ./
../dir2/file2 in ./
The $(dir $${f}) gets expanded to ./ instead of ../dirN.
Note :
Writing only echo $(dir $(FILES)) outside of the for loop outputs ../dir1/ ../dir2/ as expected.
using $(abspath ...) doesn't work either.
What am I missing ?
You are mixing make constructs (dir) and shell constructs. Unfortunately they are not evaluated at the same time. Make evaluates its constructs before passing the recipe to the shell, once for all, not during the execution of the recipe by the shell. When expanding your recipe (before passing it to the shell) it transforms it into:
for f in ../dir1/file1 ../dir2/file2; do echo $f in ./; done
because when make expands $(dir $${f}) it first expands the parameter:
$(dir ${f})
and then, as there is no / in the string ${f}, the expansion of the dir function replaces it with ./. From the GNU make manual:
$(dir names…)
Extracts the directory-part of each file name in names. The directory-part of the file name is everything up through (and
including) the last slash in it. If the file name contains no slash,
the directory part is the string ./. For example,
$(dir src/foo.c hacks)
produces the result src/ ./.
Anyway, mixing make and shell constructs is usually not what you want to do (there are exceptions). You could use only shell constructs:
all:
#for f in $(FILES); do \
echo $$f in $$(dirname $$f) ; \
done
Or you could use a static pattern rule to get completely rid of the shell loop and let make iterate over a list of similar targets:
FILES = ../dir1/file1 ../dir2/file2
FOOS = $(addsuffix .foo,$(FILES))
.PHONY: all $(FOOS)
all: $(FOOS)
$(FOOS): %.foo:
#echo $* in $(dir $*)
I have a set of files of type *.x, processing which yields a corresponding set of files *.x.y (where each *.x file produces a corresponding *.x.y file). Simple enough so far.
The two issues I have are:
The set of *.x files keeps changing, so I don't want to use their names as a static list in the dependency portion of a rule.
I only want to process those *.x files that have a size greater than 0.
Currently I use:
for a in $(find . -iname '*.x' -size +0); do make $a.y; done
and it works, but I'd prefer a cleaner, more make-y solution.
Using GNU make 3.81 on OS X 10.7.5.
You can compute the list of source files with a shell function:
SRCFILES := $(shell find . -iname '*.x' -size +0)
OUTFILES := $(SRCFILES:%=%.y)
all: $(OUTFILES)
%.x.y : %.x
$(CONVERT) -to $# $<
One of the many makefiles in my project shows errors in the error console when there are no files to delete even though the -f flag is being used. Here's the offending line in the makefile
-rm -f *.o
If I remove the dash at the beginning of the line, it stops dead in its tracks - so I know that this is the line that's generating the error.
This is a big project with a dozen or so makefiles. What I don't understand is that some of the others don't have this problem.
This is the embedded programming world - I'm using WindRiver 2.6 (so the rm utility is "rm.EXE", though it seems to have the usual options).
Does anyone know why an "rm -f" call would still generate an error if there's nothing to delete?
I'm not familiar with the WindRiver tools, but here's an educated guess based on the behavior of typical Unix shells and make tools.
When you run rm -f *.o, the following happens:
the shell tries to expand *.o, leaving it as is if there are no files matching it (try issuing echo *.o in a dir containing no such files, and one with such files)
the result of the expansion is passed to rm, so if there are no matching files, rm is called with the literal string *.o as its argument
rm interprets as arguments as path names, without expanding patterns (because that's the shell's job); the -f shuts it up if a file cannot be found
From the error message, it looks like your rm is broken in that it still complains when no file called *.o can be found. You can work around this by first checking whether any file matches the pattern. I must admit that I don't know how to do that in your environment.
In Windows environment it can't expand "*.o" and sends it as is to rm command, in fact rm.exe from cygwin for example, which then fails with error exit code. It is difficult to see while running rm.exe from Windows's cmd command prompt as it's appearing to complete OK with the same arguments.
The solution is to use $(wildcard *.o) in makefile. It's important to have "*", as otherwise nothing will be expanded even if file exists.
As a result, the line in make file will look similar to this:
rm -f $(wildcard *.o) $(wildcard *.elf) $(wildcard *.lst) $(wildcard *.map) $(wildcard *.sym) $(wildcard *.lss) $(wildcard *.eep) $(wildcard *.srec) $(wildcard *.bin) $(wildcard *.hex) $(wildcard *.tmp.sh)
In Windows use del command to remove files in Makefile
clean:
del *.o *.exe