I'm trying to use GNU make to automate my analysis pipeline. I have a script that reads files with the pattern data/sub001/sub001_input.txt and writes to data/sub001/sub001_output.txt. How could I write a rule that matches this pattern for each subject? Here is my attempt so far:
# List of all the subjects
SUBJECTS ?= sub001 sub002 sub003
/data/%/%_output.txt : process.py data/%/%_input.txt
python process.py $*
# for each $SUBJECT in $SUBJECTS
all : /data/$(SUBJECT)/$(SUBJECT)_output.txt
#echo 'Data analysis complete!'
I would like the all target to call:
python process.py sub001
python process.py sub002
python process.py sub003
And I would like a single subject to be re-processed if the corresponding sub###_input.txt file changes, and I would like all subjects to be re-processed if the process.py file changes.
You cannot use multiple pattern characters in pattern rules.
You can use a single pattern which will stand for the entire middle portion, then strip out the part you want like this:
/data/%_output.txt : process.py data/%_input.txt
python process.py $(*F)
Related
I have a Makefile, trying to loop over a series of strings in a recipe and make them lower case.
My goal: I have a series of commands I would like to run on different files with the appropriate suffix.
# files in directory: test_file1.txt test_file2.txt test_file3.txt
MYLIST = file1 file2 file3
recipe:
for name in $(MYLIST) ; do \
$(eval FILENAME=`echo $($name) | tr A-Z a-z`) \
echo "Final name : test_${FILENAME}.txt" ; \
done
My problem, FILENAME always resolves to blank:
File name: test_.txt
I hope to see:
File name: test_file1.txt
File name: test_file2.txt
File name: test_file3.txt
You cannot mix make functions and shell commands like this. Make works like this: when it decides that your target is out of date and it wants to run the recipe, first it will expand the entire recipe string. Second it sends that expanded string to the shell to run.
So in your case, the $(eval ...) (which is a make operation) is expanded one time, then the resulting string is passed to the shell to run. The shell runs the for loop and all that stuff.
You have to use shell variables here to store values obtained by running your shell for loop. You cannot use make variables or make functions.
In general if you ever think about using $(shell ...) or $(eval ...) inside a recipe, you are probably going down the wrong road.
If I have a list of things such as this:
nodes = A B C
How would I generate dynamic targets that have dynamic prerequisites. For example (this isn't working but might help explain what I want).
# node.csr is a file that already exists, like a template
# this would create outputs like node-A-csr.json, node-B-csr.json
# I am basically guessing at the syntax here
node-%-csr.json: node-csr.json
sed 's/$${node}/$*' node-csr.json > $#
# this would create outputs like node-A-key.pem node-A.pem and would require node-A-csr.json
node-%-key.pem node-%.pem: node-%-csr.json
# some command that generates node-NAME-key.pem and node-NAME-csr.pem
$(nodes): node-%-key.pem node-%.pem
Id basically like to be able to run make all and have those targets run for everything in my list.
I am very new to Makefiles and I just dont see how something like this would work and the documentation and syntax of Make is extremely confusing to me.
I am willing to use any tool to do this but seems like Make is very standard.
You can use Make's Substitution References to generate the 'all' target. This will be enough to start processing of all rules.
Notice minor change: the 'node-csr.json' should have the token NODE where the actual node name is to be inserted
# First rule is the default
default: all
nodes = A B C
# use node-csr as a template, replacing NODE with current ID: A, B, C
node-%-csr.json: node-csr.json
sed 's/__NODE__/$*/' node-csr.json > $#
# this would create node-A-key.pem node-A.pem from node-A-csr.json
# Put real commands here
node-%-key.pem node-%.pem: node-%-csr.json
ls -l node-$*-csr.json > node-$*-key.csr
ls -l node-$*-csr.json > node-$*.pem
# all will be node-A-key.pem, node-B-key.pem, ... node-C.pem
all: $(nodes:%=node-%-key.pem) $(nodes:%=node-%.pem)
echo "done"
# Using pathsubst: all: $(patsubst %,node-%-key.pem,$(nodes)) $(pathsubst
Please pay attention to tab/spaces, some version are sensitive. You will have to put back tabs before all commands (sed, ls, ...)
Substitution References man: https://www.gnu.org/software/make/manual/html_node/Substitution-Refs.html
pathsubst function: https://www.gnu.org/software/make/manual/html_node/Text-Functions.html
I read this question: Makefile: $subst in dependency list, but I still can't make my shell script work correctly.
I have a makefile with a line with the contents:
##public_detailed#|test_create|Syntax: commoncmdsyntax test_create test_name=<test-name>
A target runs a multiline bash script, where the commoncmdsyntax must be replaced by a string containing words and spaces.
In the script, I use cut to assign to a variable desc the following string:
Syntax: commoncmdsyntax test_create test_name=<test-name>
The problem is that commoncmdsyntax is not replaced by new text here:
$(subst commoncmdsyntax,new text,$$desc)
I also tried to replace it by a single word, like XX, but it also does not work.
The subst function (as in $(subst commoncmdsyntax,new text,$$desc)) is a Make function, so Make will perform the substitution before running any rule and therefore before your script assigns a value to desc. So even if secondary expansion worked the way you seem to think it will, this approach would still fail.
If you want to perform a substitution within something made by a shell script (in a recipe), the sensible way is to do so within the recipe:
echo $dest | sed 's/commoncmdsyntax/new text/'
We can give you a more detailed solution if you give us a minimal complete example of the problem.
I have a programme that is generating files like this "Incoming11781Arp", and there is always Incoming, and there is always 5 numbers, but there are 3 letters/upper-case/lower-case/numbers/special case _ in any way. Like Incoming11781_pi, or Incoming11781rKD.
How can I delete them using a script run from a cron job please? I've tried -
#!/bin/bash
file=~/Mail/Incoming******
rm "$file";
but it failed saying that there was no matching file or directory.
You mustn't double-quote the variable reference for pathname expansion to occur - if you do, the wildcard characters are treated as literals.
Thus:
rm $file
Caveat: ~/Mail/Incoming****** doesn't work the way you think it does and will potentially match more files than intended, as it is equivalent to ~/Mail/Incoming*, meaning that any file that starts with Incoming will match.
To only match files starting with Incoming that are followed by exactly 6 characters, use ~/Mail/Incoming??????, as #Jidder suggests in a comment.
Note that you could make your glob (pattern) even more specific:
file=~/Mail/Incoming[0-9][0-9][0-9][0-9][0-9][[:alpha:]_][[:alpha:]_][[:alpha:]_]
See the bash manual for a description of pathname expansion and pattern syntax: http://www.gnu.org/software/bash/manual/bashref.html#index-pathname-expansion.
You can achieve the same effect with the find command...
$ directory='~/Mail/'
$ file_pattern='Incoming*'
$ find "${directory}" -name "${file_pattern}" -delete
The first two lines define the directory and the file pattern separately, the find command will then proceed to delete any matching files inside that directory.
egrep -r --include "*.[chS]" "myregularexpression" .
What does [chS] mean in the shell command above?
That is part of the shell globbing that selects multiple files.
The expression [chS] matches a single character containing the value c, h, or S.
So, the glob "*.[chS]" is looking for all files that have the extension .c, .h, or .s
[chS] is a character class and is equivalent to the expression c|h|S. It matches any one of the listed characters. In this case, *.[chS] is matching files (*.c, or *.h or *.S), i.e., C source and headers, and assembly files.