There are four folders in my src directory: dir1, dir2, dir3, dir4.
PARSE_MODULES = $$(MODULES=$$(ls -l $1 | grep '^d' | awk '{print $$9}'); echo $$MODULES);
MODULES:=$(call PARSE_MODULES,src)
all:
#echo $(MODULES)
Hence, this little piece of code outputs
dir1 dir2 dir3 dir4
Now, I add the following two lines before the all rule:
X:=dir1 dir3
MODULES:=$(filter-out $(X),$(MODULES))
The output remains unchanged, i.e., filter-out is not working as expected. However, if I manually define MODULES, I get the expected output:
MODULES:=dir1 dir2 dir3 dir4
X:=dir1 dir3
MODULES:=$(filter-out $(X),$(MODULES))
Output: dir2 dir4
Why is it impossible to process the output of awk in filter-out? Is there any way to do so?
Your problem is that you have deferred expansion of the contents of PARSE_MODULES by using $$, so that it's being run by the shell in the recipe, not by make.
This:
PARSE_MODULES = $$(MODULES=$$(ls -l $1 | grep '^d' | awk '{print $$9}'); echo $$MODULES);
MODULES:=$(call PARSE_MODULES,src)
leaves the MODULES variable containing this text:
$(MODULES=$(ls -l src | grep '^d' | awk '{print $9}'); echo $MODULES);
Then when you use $(MODULES) in the recipe, it expands like this:
all:
#echo $(MODULES=$(ls -l src | grep '^d' | awk '{print $9}'); echo $MODULES);
then the shell will run the command inside $(...) and echo the results.
When you try to filter out names like dir1 etc. from the value of the make variable MODULES, nothing happens because the make variable MODULES contains the shell script itself, as text, it doesn't contain the results of executing the shell script.
I'm not sure why you are trying to do things in such an incredibly convoluted way, with so much escaping and using difficult to understand shell scripting. Maybe there's a reason that your full environment needs something so complicated but you can FAR more easily implement the example above just using some basic make functions:
PARSE_MODULES = $(sort $(notdir $(patsubst %/.,%,$(wildcard $1/*/.))))
MODULES := $(call PARSE_MODULES,src)
Now the PARSE_MODULES is expanded by make, not by the recipe, and filter-out etc. will work fine on it. Plus it's a lot simpler then doing a shell with grep and awk.
As far as I understand the shell list $(MODULES=$(ls -l src | grep '^d' | awk '{print $9}'); echo $MODULES) is evaluated by the shell when the recipe of all is executed. This is too late. Try this, maybe:
PARSE_MODULES = $(shell ls -l $1 | grep '^d' | awk '{print $$9}')
MODULES := $(call PARSE_MODULES,src)
X := dir1 dir3
MODULES := $(filter-out $(X),$(MODULES))
all:
#echo $(MODULES)
Minor simplification from OP version:
# possibly: find -maxdepth 1
PARSE_MODULES = $(shell find $1 -type d -printf "%P\n")
SKIP := dir1 dir3
MODS := $(call PARSE_MODULES,src)
FLTR := $(filter-out $(SKIP), $(MODS))
all:
#echo MODS: $(MODS)
#echo SKIP: $(SKIP)
#echo FLTR: $(FLTR)
Output:
MODS: dir1 dir2 dir3 dir4
SKIP: dir1 dir3
FLTR: dir2 dir4
Related
I am trying the below :
find /dir1/dir2/dir3 -name '*.txt' -type f
What I want to do is I want to search for files recursively in the dir3 folder. That means under dir3 there are dir4 and dir5 folders and I want that the files matching with *txt extension should be returned from dir4 and dir5 directories. Also, how can I get the files which are only created on today's date?
Assuming you use bash, I came up with this:
#!/bin/bash
# This depends on your systems locale I think.
# for me, `date` returns `So 25. Jul 12:32:27 CEST 2021`.
# Therefore I want the $2 for DAY and $3 for MONTH, yours might be different.
DAY="$(date | awk '{print $2}')"
MONTH="$(date | awk '{print $3}')"
FINDIN="/dir1/dir2/dir3"
# This prints only `.txt` files which were created today along with all their details.
# If you only want the path, you could pipe it into
# `awk '{print $11}'`
find "$FINDIN" -type f -ls | grep -E "*.txt$" | grep "$MONTH $DAY"
Using bashisms you could technically make this a (lengthy) one-liner but if you put it in a script, you can substitute your path ($FINDIN) for a dynamic value passed as an argument ($1, $2, …) or the caller directory (implied/parsed from $0).
How do I list files that exist, but not present in the list? More specifically, I'd like to remove *.cpp files not listed in Build. Something like this lists files that are present in both the current directory and the Build file:
ls *.cpp | xargs -I % bash -c 'grep % Build'
However, the following line is incorrect of course:
ls *.cpp | xargs -I % bash -c 'grep -v % Build'
Thus the question: how does one list the *.cpp files that are not present in the Build file using shell commands? I can do something like this, bug this is ugly:
ls *.cpp | perl -e 'while(<>){chomp;my $l=`grep $_ Build`;chomp $l;if(length $l==0){print("rm $_\n");}}'
More specifically, I'd like to remove *.cpp files not listed in Build
You want comm or join to join two sorted lists together. I always mix comm arguments, but I think:
comm -23 <(find . -type f -name '*.cpp' | sort) <(sort Build) |
xargs -n '\n' echo rm
or if you want to depend on filename expansion:
shopt -s nullglob # at least
comm -23 <(printf "%s\n" *.cpp | sort) <(sort Build) | ...
Do not parse `ls.
The <(...) is bash specific process substitution. In non-bash shell just create a temporary file with the output of processes.
GNU grep already offers you this possibility with the -f switch:
printf '%s\n' *.cpp | grep -F -x -v -f Build
-F: no regex
-x: full-line match
-v: invert (not match)
-f: any of the line in Build
In other words: filter out any line in Build
I can easily print all the files inside some directory from bash:
$ cat go.sh
BASEDIR=~/Downloads
MYDIR=${BASEDIR}/ddd
for f in $(ls ${MYDIR}); do echo $f; done
$ ./go.sh
m.txt
d.txt
When I try to do a similar thing from makefile it doesn't work well:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
for f in $(ls ${MYDIR}); do echo ${f}; done
$ make
for f in ; do echo ; done
And here is another trial that doesn't work:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
for f in $(shell ls ${MYDIR}); do echo ${f}; done
$ make
for f in d.txt m.txt; do echo ; done
Maybe you can do it purely Makefile way?
MYDIR = .
list: $(MYDIR)/*
#echo $^
You can still run command from Makefile like this
MYDIR = .
list: $(MYDIR)/*
for file in $^ ; do \
echo "Hello" $${file} ; \
done
If I were you, I'd rather not mix Makefile and bash loops based on $(shell ...). I'd rather pass dir name to some script and run loop there - inside script.
Also almost "true way" from documentation
TEMPLATES_DIR = ./somedir
list:
$(foreach file, $(wildcard $(TEMPLATES_DIR)/*), echo $(file);)
Here is the edited answer based on #Oo.oO:
$ cat makefile
BASEDIR = ${HOME}/Downloads
MYDIR = ${BASEDIR}/ddd
all:
#for f in $(shell ls ${MYDIR}); do echo $${f}; done
$ make
d.txt
m.txt
There is a little problem with #Oo.oO's answer.
If there is any file/folder has the same name with a target in makefile, and that target has some prerequisites, and you want to loop through that folder, you will get that target recipe being executed.
For example: if you have a folder named build, and you have a rule like:
build: clean server client
clean:
#echo project cleaned!
server:
#echo server built!
client:
#echo client built!
To loop through the folder contains that special build folder, let's says you have the following rules:
MYDIR = .
ls: $(MYDIR)/*
#echo $^
The result will be:
$ make ls
project cleaned!
server built!
client built!
build Makefile
I would suggest to use #Mike Pylypyshyn's solution. According to the make documentation, the foreach function is more suitable in this case.
Suppose we have a directories two foo and bar. Where foo contains files example.S, run.c, exec.S and bar contains example.S
when I pass foo and bar as dependencies.
I want to compile only example.S in the directory bar by ignoring example.S in directory foo
DEP := foo bar
DIR := $(foreach $dirs, $(DEP), $(shell $(HOME) -find -type d -name $(dirs))),
would provide me the absolute path of the foo and bar directories.
FILES := $(foreach file, $(DIR), $(wildcard $(file)/*)), would provide me the list of files foo and bar as
foo/example.S foo/run.c foo/exec.S bar/example.S
I would like to check for multiple occurance of file with same name. and only compile the latest, i.e, instead of foo/example.S I like to compile bar/example.S...
I have no idea how to do it, As I am very new to creation of Makefile.
This is actually more of a shell question than a makefile one. What you want to do is create a shell command that generates two column list for each file -- the first column being the full filename, and the second being just the filename. Then sort it, and remove the ones with duplicate filenames, and then output only the first column.
Basically you would use:
DIRS = ./A ./B
files=`find $(DIRS) -type f -exec sh -c 'echo {} $$(basename {})' \; | sort -u --stable -k2,2 | awk '{print $$1}'`
You would have to replace the ./B ./A with a sorted list of directories you wanted to search in. Notice the $$'s -- Make resolves these to $ before running the shell command. The $(DIRS) only has a single $, so it is expanded before the command is executed.
Explanation:
find ./B ./A -type f -exec sh -c 'echo {} $(basename {})' \; : This searches the directories B and then A (in that order)
type -f specifies files only -- it won't return directories.
-exec sh -c 'echo {} $(basename {})' \; : for each file found, it will run the echo command where {} resolves to the file it found. It will therefore print the full file path, followed by just the filename.
sort -u --stable -k2,2: Sort the list of files based on column 2 (the filename). The -u means unique, so if there are matching files, it only prints the first. The --stable means that if there are two matching lines, it will always output the first one.
awk '{print $1}': print the first column of the output (the full path name).
You then have a list of unique filenames, including their directories in $(files)
There's a (relatively) simple way to get what you want, starting with reversing a list (with thanks to #simona). After you have calculated DIR your way,
reverse = $(if $(1),$(call reverse,$(wordlist 2,$(words $(1)),$(1)))) $(firstword $(1))
DIR := $(call reverse,$(DIR))
Then use VPATH:
VPATH = $(DIR)
Now you can tell Make to search for a file (such as example.S), and it will find the last instance (in this case .../bar/example.S):
example: example.S
#echo building $# from $^
I have a file called file1.txt:
dir1
dir2
dir3
...
I wanted to use xargs to check if some files exist farther into the file system like this:
cat file1.txt | xargs -i ls /projects/analysis7/{}/meta_bwa/hg19a/*varFilter 2>/dev/null
But xargs never seems smart enough to expand the *. ie, it never finds the files even when they would match the pattern (if the * was expanded).
Any ideas?
You just need to add sh -c:
cat file1.txt | xargs -i sh -c 'ls /projects/analysis7/{}/meta_bwa/hg19a/*varFilter' 2>/dev/null