I have following make rule:
$ ls files
a b c
$ cat Makefile
files.7z: ./files/*
7z a $# $?
The rule will be executed as follows:
7z a files.7z files/a files/b files/c
7z treats paths beginning with ./ especially in that, it will include the file in the archive without the path. What would be the shortest way to replace the paths, to begin with, ./ in the $?? (Files names can have spaces.)
Makefile doesn't handle spaces very well at all (see here). The simplest thing is to make make fail if there are spaces and then it's easy:
check:
# ls ./files/* | grep -q " "; \
if [ $$? -ne 1 ]; then \
echo "WE DO NOT SUPPORT FILENAMES WITH SPACES"; \
exit 1; \
fi
files.7z: ./files/* | check
7z a $# $(addprefix ./,$^)
Note: there are workarounds on the web for filenames with spaces, but they're overly complex, not very maintainable and will cost you a lot more headaches than telling your customers/coworkers "We don't support file names with spaces"...
Related
When I extract files from a tar ball to create a list of files make does not follow symbolic links and keeps extracting the same tar ball for each file specified in the list. Forinstance I have created 2 tar balls with the following commands in linux:
design1 is a directory that holds a link to another directory with c design files
% mkdir design1
% mkdir c_files
% cd c_files/
% touch a.c b.c c.c d.c e.c f.c
% cd ../design1/
% ln -s ../c_files/ .
% cd ..
% gtar zcvf design1.tgz design1/
design2 is the same as design1 except that c_files is a directory and not a link
% mkdir design2
%
% cp -rp c_files design2
% gtar zcvf design2.tgz design2/
Now there are 2 zipped tarfiles design1.tgz and design2.tgz and design1 and design2 directories are removed
% \rm -rf design1 design2
Here's my Makefile:
----------------Begin Makefile -------------------
DESIGN_EXTRACT_DIR := .
DESIGN_TOP1 := c_proj1
FILE_LIST1 := $(strip $(DESIGN_TOP1)).f
DESIGN_TAR_FILE1 := design1.tgz
DESIGN_RELEASE_DIR1 := $(DESIGN_EXTRACT_DIR)/design1
DESIGN_FILES1 =\
$(DESIGN_RELEASE_DIR1)/c_files/a.c \
$(DESIGN_RELEASE_DIR1)/c_files/b.c \
$(DESIGN_RELEASE_DIR1)/c_files/c.c \
$(DESIGN_RELEASE_DIR1)/c_files/d.c \
$(DESIGN_RELEASE_DIR1)/c_files/e.c \
$(DESIGN_RELEASE_DIR1)/c_files/f.c \
IP_EXTRACT1: $(DESIGN_FILES1)
$(DESIGN_FILES1): $(DESIGN_TAR_FILE1)
gtar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST1): $(DESIGN_FILES1)
#( rm -f $(FILE_LIST1) )
#( touch $(FILE_LIST1) )
#$(foreach file, $(DESIGN_FILES1), `echo $(file) >> $(FILE_LIST1)`)
FL1: $(FILE_LIST1)
DESIGN_TOP2 := c_proj2
FILE_LIST2 := $(strip $(DESIGN_TOP2)).f
DESIGN_TAR_FILE2 := design2.tgz
DESIGN_RELEASE_DIR2 := $(DESIGN_EXTRACT_DIR)/design2
DESIGN_FILES2 =\
$(DESIGN_RELEASE_DIR2)/c_files/a.c \
$(DESIGN_RELEASE_DIR2)/c_files/b.c \
$(DESIGN_RELEASE_DIR2)/c_files/c.c \
$(DESIGN_RELEASE_DIR2)/c_files/d.c \
$(DESIGN_RELEASE_DIR2)/c_files/e.c \
$(DESIGN_RELEASE_DIR2)/c_files/f.c \
IP_EXTRACT2: $(DESIGN_FILES2)
$(DESIGN_FILES2): $(DESIGN_TAR_FILE2)
gtar zxvfmhC $(DESIGN_TAR_FILE2) $(DESIGN_EXTRACT_DIR)
$(FILE_LIST2): $(DESIGN_FILES2)
#( rm -f $(FILE_LIST2) )
#( touch $(FILE_LIST2) )
#$(foreach file, $(DESIGN_FILES2), `echo $(file) >> $(FILE_LIST2)`)
FL2: $(FILE_LIST2)
clean:
#( rm -rf $(FILE_LIST1)* $(FILE_LIST2)* )
----------------End Makefile----------------------
Now when I run make to create a file list I use the commands make FL1 and make FL2. In the case of FL1 make iterates of gtar as many times as I have files but doesn't do so with FL2. The only difference here is FL1 operates on a link called c_files in design1 while FL2 iterates over c_files as a directory.
Here's what I see:
% make FL1
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
gtar zxvfmhC design1.tgz .
design1/
design1/c_files
%make FL2
%
Issue1:
FL1 creates c_proj1.f and FL2 creates c_proj2.f but FL2 doesnt iterate like FL1 and I am not sure how to prevent this iteration especially since I could have large tar ball with hundreds of files.
Issue2:
When DESIGN_FILES1 is a long list because of the number of files then I get the following error because the variable is too long:
make: execvp: /bin/sh: Argument list too long
Is there a way to check the size of the variable and maybe write to a file and process the variable a bit at a time so DESIGN_FILES will not be too long for each iteration. Or is there a better way to do this.
Thanks
Niel
For me it works the same (iterates once) whether there is a symlink or not. But from the contents it seems that you are only interested in generating file list, so file existence is not really required. If you omit dependency, you may generate file list without even extracting the files.
For the actual file list write the oversized command line error comes from the fact, that the generated command is a single line with multiple subshell calls, which might be too long for the command line indeed:
$ make FL1 --trace
...
Makefile:24: update target 'c_proj1.f' due to: design1/c_files/a.c design1/c_files/b.c design1/c_files/c.c design1/c_files/d.c design1/c_files/e.c design1/c_files/f.c
( rm -f c_proj1.f )
( touch c_proj1.f )
`echo ./design1/c_files/a.c >> c_proj1.f` `echo ./design1/c_files/b.c >> c_proj1.f` `echo ./design1/c_files/c.c >> c_proj1.f` `echo ./design1/c_files/d.c >> c_proj1.f` `echo ./design1/c_files/e.c >> c_proj1.f` `echo ./design1/c_files/f.c >> c_proj1.f`
This part can be resolved with a $(file) function (if you happen to use GNU make 4.1+):
.PHONY: $(FILE_LIST1)
$(FILE_LIST1):
$(file >$#)
$(foreach file,$(DESIGN_FILES1),$(file >>$#,$(file)))
If an older make is used, this could be done with regular commands, you just need to generate a newline character after each echo to make it work as a separate command. The only tricky part is that it has to be done as a specially crafted variable:
# Note double empty lines below
define newline
endef
.PHONY: $(FILE_LIST2)
$(FILE_LIST2):
rm -f $#
$(foreach file,$(DESIGN_FILES2),echo $(file) >> $#$(newline))
Result is now generated in separate shell calls, which should avoid command length limit:
$ make FL2
rm -f c_proj2.f
echo ./design2/c_files/a.c >> c_proj2.f
echo ./design2/c_files/b.c >> c_proj2.f
echo ./design2/c_files/c.c >> c_proj2.f
echo ./design2/c_files/d.c >> c_proj2.f
echo ./design2/c_files/e.c >> c_proj2.f
echo ./design2/c_files/f.c >> c_proj2.f
EDIT:
I was able to reproduce your multiple iteration of extraction. Your tgz file does not actually contain the files you claim to extract, so the files are not there when the first one is trying to be extracted and the extraction continues for the next file (which is also absent). This is also suggested by tar output, which says:
design1/
design1/c_files
instead of:
design1/
design1/c_files
design1/c_files/a.c
design1/c_files/b.c
design1/c_files/c.c
design1/c_files/d.c
design1/c_files/e.c
design1/c_files/f.c
Your tgz only contains a directory symlink and it only extracts a symlink. If the target files do not exist in the target location, it will attempt to re-extract for every file.
EDIT2:
When actual files are not included in the tgz file, but are available through a symlink, there is a chance that they are older than the tgz file itself. In such situation make checks the file timestamp through a symlink and discovers that it needs to be remade - but the file is not actually remade (since it's not included in the tgz). This check is done for every single file, so for every file the archive is extracted:
$ make FL1 -d
...
Finished prerequisites of target file 'design1/c_files/a.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/a.c'.
Must remake target 'design1/c_files/a.c'.
tar zxvfmhC design1.tgz .
Putting child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286 on the chain.
Live child 0x7fffe11ae730 (design1/c_files/a.c) PID 6286
design1/
design1/c_files
Reaping winning child 0x7fffe11ae730 PID 6286
Removing child 0x7fffe11ae730 PID 6286 from chain.
Successfully remade target file 'design1/c_files/a.c'.
Considering target file 'design1/c_files/b.c'.
Pruning file 'design1.tgz'.
Finished prerequisites of target file 'design1/c_files/b.c'.
Prerequisite 'design1.tgz' is newer than target 'design1/c_files/b.c'.
Must remake target 'design1/c_files/b.c'.
tar zxvfmhC design1.tgz .
...
In general extracting archive in this way is not reliable anyway (image running in parallel with -j), so it would be better to use a group target rule to explicitly point out that this rule generates all the files at the same time:
$(DESIGN_FILES1) &: $(DESIGN_TAR_FILE1)
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
This however is a new feature introduced in GNU make 4.3, so to make it work with previous versions, we could use a phony target that is only used for synchronization, e.g.:
$(DESIGN_FILES1): extract-$(DESIGN_TAR_FILE1)
.PHONY: extract-$(DESIGN_TAR_FILE1)
extract-$(DESIGN_TAR_FILE1):
tar zxvfmhC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
Output:
$ make FL1
tar zxvfmhC design1.tgz .
design1/
design1/c_files
Note that the archive is now extracted only once.
I followed your advice and changed the dependency and it resolves the tar issue. The file function in make resolves the length of the variable but I have 2 undesired effect due to it. I have added one more thing to the mix here and I cannot explain the behavior:
On linux:
% mkdir -p design3/c_files
% cd design3/c_files
% touch x.c y.c z.c
% cd ../..
% ls ./design3/c_files/*.c > design3/c_proj3.f
Here's what I have done to the Makefile:
DIR_LIST := design3
IP_EXTRACT1: $(DESIGN_TAR_FILE1)
gtar zxvfmC $(DESIGN_TAR_FILE1) $(DESIGN_EXTRACT_DIR)
$(DESIGN_FILES1):
$(MAKE) IP_EXTRACT1
$(FILE_LIST1): $(DESIGN_FILES1)
$(file > $(FILE_LIST1))
#$(if $(strip $(DIR_LIST)), cat $(foreach dir, $(DIR_LIST), $(wildcard $(dir)/*.f)) >> $(FILE_LIST1))
#$(foreach file,$(DESIGN_FILES1),$(file >>$(FILE_LIST1),$(file)))
This resolves the tar issue but now I have added DIR_LIST and I want to add the list of files in each directory to the file list. It does the right thing but it has a couple of undesired effect:
1) I expect the content of design3/c_proj3.f at the top but it appears in the end.
%make FL1
%more c_proj1.f
./design1/c_files/a.c
./design1/c_files/b.c
./design1/c_files/c.c
./design1/c_files/d.c
./design1/c_files/e.c
./design1/c_files/f.c
./design3/c_files/x.c
./design3/c_files/y.c
./design3/c_files/z.c
I expected x, y, z at the top as there is order dependence but its at the bottom. Also, if I do make FL1 --dry-run after removing c_proj1.f the files a through f gets into it but not x,y,z because that is from a cat shell command. I would prefer no files get created. Is there a way to resolve this?. This is a partial solution so I am not sure if this needs to go back into the question.
I want to iterate through a list of files, execute cd in their respective directories before executing a command.
So I would like to use $(dir $(FILES)) function to get the directory, but it seems not to work properly in the for loop.
FILES=../dir1/file1 ../dir2/file2
.PHONY: all
all:
#for f in $(FILES); do \
echo $$f in $(dir $${f}) ; \
done
outputs
../dir1/file1 in ./
../dir2/file2 in ./
The $(dir $${f}) gets expanded to ./ instead of ../dirN.
Note :
Writing only echo $(dir $(FILES)) outside of the for loop outputs ../dir1/ ../dir2/ as expected.
using $(abspath ...) doesn't work either.
What am I missing ?
You are mixing make constructs (dir) and shell constructs. Unfortunately they are not evaluated at the same time. Make evaluates its constructs before passing the recipe to the shell, once for all, not during the execution of the recipe by the shell. When expanding your recipe (before passing it to the shell) it transforms it into:
for f in ../dir1/file1 ../dir2/file2; do echo $f in ./; done
because when make expands $(dir $${f}) it first expands the parameter:
$(dir ${f})
and then, as there is no / in the string ${f}, the expansion of the dir function replaces it with ./. From the GNU make manual:
$(dir names…)
Extracts the directory-part of each file name in names. The directory-part of the file name is everything up through (and
including) the last slash in it. If the file name contains no slash,
the directory part is the string ./. For example,
$(dir src/foo.c hacks)
produces the result src/ ./.
Anyway, mixing make and shell constructs is usually not what you want to do (there are exceptions). You could use only shell constructs:
all:
#for f in $(FILES); do \
echo $$f in $$(dirname $$f) ; \
done
Or you could use a static pattern rule to get completely rid of the shell loop and let make iterate over a list of similar targets:
FILES = ../dir1/file1 ../dir2/file2
FOOS = $(addsuffix .foo,$(FILES))
.PHONY: all $(FOOS)
all: $(FOOS)
$(FOOS): %.foo:
#echo $* in $(dir $*)
I am trying to use a nested for loop for searching and copying some files inside the recipe of one of the targets inside a makefile:
DIR = $(DIR_A) $(DIR_B)
install:
for dirs in $(DIR); do \
for file in $(shell find $(dirs) -type f -and -not -path "*/.svn*" | sed -e "s|$(dirs)||"); do \
folder=$${file%/*}; \
$(INSTALL) -d $(DEST_DIR)$$folder/log; \
$(INSTALL) $(dirs)/$$file $(DEST_DIR)$$folder/log; \
done \
done
However $(dirs) variable always evaluates to empty inside the second for loop and the current working directory gets passed to "find" instead of first directory path from $(DIR).
Can someone please suggest if I am missing something ?
Thanks.
The $(dirs) in the find command is being expanded by make to the make variable dirs which is unset and thus the empty string. To reference the shell variable, you need to escape the $:
for file in $$(find $${dirs} -type f -and \
-not -path "*/.svn*" | sed -e "s|$${dirs}||"); do
But don't do this. It is much cleaner to explicitly list the files you intend to install. If there are many, it is fine to write a script to generate the Makefile. What you are doing is a fragile mess.
You have made several errors, and you will find it almost impossible to solve them as l;ong as you insist on trying to solve them all at once.
Let's try this in stages. Suppose your DIR_A and DIR_B are north and south. On the command line, if you try this:
for dirs in north south; do echo $dirs; done
you will get the correct output:
north
south
If you try it as a makefile rule:
install:
for dirs in $(DIR); do echo $$dirs; done
again, it works correctly.
If you try your makefile recipe:
install:
for dirs in $(DIR); do \ for file in $(shell ls $$dirs); do \ echo "file is $$file"; \ done \ done
it fails, because Make expands the $(shell ...) command before passing the entire for command to the shell, when dirs has not yet been assigned a value. One way to construct a sensible shell command is to use backticks:
for dirs in north south; do for file in `ls $dirs`; do echo "file is $file"; done done
This works on the command line. A makefile rule built around it:
install:
for dirs in $(DIR); do for file in `ls $$dirs`; do echo "file is $$file"; done done
also works.
That should be enough to allow you to rewrite your makefile.
I want to produce a.txt from a.tar.gz in the directory. I tried writing the following goal in Makefile:
a.txt: a.tar.gz
tar xvf a.tar.gz
This works, but after running it once a.tar.gz is uncompressed and no longer exists. And subsequent make fails. How can I write a goal that checks if a.txt does not exist, and if so run tar, but otherwise don't do anything?
wildcard will only expand to something if it can find at least one file:
a.txt: $(wildcard a.tar.gz)
tar xvmf $<
Adding the m flag will avoid unnecessary remakes.
The rule will fail if neither file exists however, but without knowing what exactly you're trying to do it's hard to provide a comprehensive solution.
The 'test' or ' [ ' and ' ] ' can do this.
a.txt:
if [ -e a.tar.gz ]; then \
tar xvf a.tar.gz \
fi
Have documents stored in a file system which includes "daily" directories, e.g. 20050610. In a bash script I want to list the files in a months worth of these directories. So I'm running a find command find <path>/200506* -type f >> jun2005.lst. Would like to check that this set of directories is not a null set before executing the find command. However, if I use if[ -d 200506* ] I get a "too many arguements error. How can I get around this?
Your "too many arguments" error does not come from there being a huge number of files and exceeding the command line argument limit. It comes from having more than one or two directories that match the glob. Your glob "200506*" expands to something like "20050601 20050602 20050603..." and the -d test only expects one argument.
$ mkdir test
$ cd test
$ mkdir a1
$ [ -d a* ] # no error
$ mkdir a2
$ [ -d a* ]
-bash: [: a1: binary operator expected
$ mkdir a3
$ [ -d a* ]
-bash: [: too many arguments
The answer by zed_0xff is on the right track, but I'd use a different approach:
shopt -s nullglob
path='/path/to/dirs'
glob='200506*/'
outfile='jun2005.lst'
dirs=("$path"/$glob) # dirs is an array available to be iterated over if needed
if (( ${#dirs[#]} > 0 ))
then
echo "directories found"
# append may not be necessary here
find "$path"/$glob -type f >> "$outfile"
fi
The position of the quotes in "$path"/$glob versus "$path/$glob" is essential to this working.
Edit:
Corrections made to exclude files that match the glob (so only directories are included) and to handle the very unusual case of a directory named literally like the glob ("200506*").
prefix="/tmp/path"
glob="200611*"
n_dirs=$(find $prefix -maxdepth 1 -type d -wholename "$prefix/$glob" |wc -l)
if [[ $n_dirs -gt 0 ]];then
find $prefix -maxdepth 2 -type f -wholename "$prefix/$glob"
fi
S=200506*
if [ ${#S} -gt 6 ]; then
echo haz filez!
else
echo no filez
fi
not a very elegant one, but w/o any external tools/commands (if don't think of "[" as an external one)
the clue is if there is some files matched, then "S" variable will contain their names delimited with space. Otherwise it will contain a "200506*" string itself.
You could us ls like this:
if [ -n "$(ls -d | grep 200506)" ]; then
# There are directories with this pattern
fi
Because there is a limit on command line length in most shells: anything like "$(ls -d | grep 200506)" or /path/200506* will run the risk of overflowing the limit. I'm not sure if substitutions and glob expansions count towards it in BASH, but I assume so. You would have to test it and check the bash docs and source to be sure.
The answer is in simplifying your question.
find <path>/200506* -type f -exec somescript '{}' \;
Where somescript is a shell script that does the test. Something like this perhaps:
#!/bin/sh
[ -d "$#" ] && echo "$#" >> june2005.lst
Passing the june2005.lst to the script (advice: use an environment variable), and dealing with any possibility that 200506* may expand to tooo huge a file path, being left as an exercise for the OP ;)
Integrating the whole thing into a pipe line or adapting a more general scripting language would yield performance boosts, by minimizing the number of shells spawned. Now that would be fun. Here is a hint for that, use -exec and another program (awk, perl, etc) to do the directory test as part of a one line filter, and keep the >>june2005.lst on the find command.