Question 1
If you define step
22 clean:
23 rm $(OBJECTS)
If there any way to gracefully "do nothing, if there is nothing to delete"?
Question 2
Assume the following line, again, is there a way to gracefully exit with a warhing when no files are found when processing line
6 SOURCES = $(shell echo src/*.cpp)
Question 3
How can one perform the final post processing on the final product, like mv $(PRODUCT) someDir? Where would this instruction be?
1) Just use rm -f, which is telling rm to ignore it if the files are missing.
3) That can just be the last step of the target that actually builds the product, or you can create a target named install (for example) that depends on your build target, and then contains this mv command.
Answering question #2:
SOURCES := $(or $(wildcard src/*.cpp), $(warning No source found in 'src'))
This will emit a warning when there are no files matching src/*.cpp pattern. SOURCES variable remains empty.
See the corresponding chapter in GNU Make manual.
Related
all: ./data/for_analysis.csv ./data/tables/*.docx
./data/for_analysis.csv : ./src/convert-xls-to-gold-standard.py ./data/ED-TRAUMA-DELTA-STUDY_3_2019_total.xlsx
python3 $< --rawDataPath $(word 2,$^) --fieldCodesPath ./data/excel_field_codes.json --processedDataPath ./data/for_analysis.csv --logDir ./logs
./data/tables/%.docx : ./src/make-%.py ./data/for_analysis.csv
python3 $< --fieldCodesPath ./data/excel_field_codes.json --processedDataPath ./data/for_analysis.csv --logDir ./logs --tablesDir ./data/tables
When I update ./src/make-table-2.py, the second target isn't updated. This behavior doesn't depend on whether ./data/table/table-2.docx exists or not.
When I run make or make all even after updating the py file, I get the message make: Nothing to be done for 'all'.
It's not exactly clear from your question what the state of your targets is before you run make. But:
all: ./data/for_analysis.csv ./data/tables/*.docx
this can't really work, in general. This tells make, "go find all the files that exist with the filename matching the wildcard ./data/tables/*.docx". E.g., that's the same thing you'd get if you run ls ./data/tables/*.docx before you started make.
But of course, if you haven't built anything yet then there are no files matching that pattern, because that's what you're asking make to build. So this expands to nothing and make won't do anything with them.
You have to list the targets that you want to build explicitly, or else convert them from the source files you want them to be built from, so you can tell make what it should be building.
For example, maybe:
all: ./data/for_analysis.csv $(patsubst ./src/make-%.py,./data/tables/%.docx,$(wildcard ./src/make-*.py))
I am working on a writing project and would like to use make for running pandoc on files. So far I've tried to pass arguments to make like I do with a bash script.
For example:
$ make chapter 2
In the make file chapter is the target and 2 would be the argument.
I don't know if makefiles have the facility to take cli arguments. I haven't been able to find what I'm looking for in the documentation.
So far I have tried to run make with this recipe.
chapter:
#pandoc -s -o "$1.epub" "$1.md"
I get this error back
pandoc: .md: openBinaryFile: does not exist (No such file or directory)
make: *** [Makefile:2: chapter] Error 1
This is for turning sections of a book I'm working on into epubs. I'm open to other ways to do this with make seeing as tokens don't to work.
In the make file chapter is the target and 2 would be the argument
$ make chapter num=2
The assignment to variables on the make command-line overrides any definition inside the makefile
(yep, such variables effectively become read-only).
This suggests a makefile something like:
num = $(error You must set $$num to the chapter number you want (make chapter num=4))
.PHONY: chapter
chapter:
pandoc -s -o "${num}.epub" "${num}.md"
What's going on here?
Well, if you forget to set num,
when make expands the recipe for chapter the
$(error) will cause make to stop.
$ make
Makefile:5: *** You must set $num to the chapter number you want (make chapter num=4). Stop.
And your original example?
$ make chapter num=2
pandoc -s -o "2.epub" "2.md"
Tips
I rarely recommend using the # prefix — Users can use make's -s if they don't want to see the shell commands
Don't lie to make — In particular, your rule does not produce a file called chapter, so please tell make that by marking the target .PHONY
The natural way to say this in Make is to enumerate all the chapters as targets, typically as dependencies for make all.
So basically
src := $(wildcard *.md)
epubs := $(patsubst %.md,%.epub,$(src))
.PHONY: all
all: $(epubs)
%.epub: %.md
pandoc -s -o $# $<
You can say make ch4.epub if you have a chapter whose source is ch4.md. You can't really pass in an argument which isn't a file name or a target name, and these cannot contain spaces.
I suppose you could add a phony like
.PHONY: 2
2: ch2.epub
to be able to say make 2 and have it mean make ch2.epub. If file names are systematically named like this, you could generalize to
short := $(patsubst ch%.md,%,$(src))
.PHONY: $(short)
$(short): %: ch%.epub
Don't use # in front, it just makes things harder. You can use make -s if you don't want to see the output and not wreck your Makefile.
I have a number of makefiles that build and run tests. I would like to create a script that makes each one and notes whether the tests passed or failed. Though I can determine test status within each make file, I am having trouble finding a way to communicate that status to the caller of the make command.
My first thought is to somehow affect the return value of the make command, though this does not seem possible. Can I do this? Is there some other form of communication I can use to express the test status to the bash script that will be calling make? Perhaps by using environment variables?
Thanks
Edit: It seems that I cannot set the return code for make, so for the time being I will have to make the tests, run them in the calling script instead of the makefile, note the results, and then manually run a make clean. I appreciate everyone's assistance.
Make will only return one of the following according to the source
#define MAKE_SUCCESS 0
#define MAKE_TROUBLE 1
#define MAKE_FAILURE 2
MAKE_SUCCESS and MAKE_FAILURE should be self-explanatory; MAKE_TROUBLE is only returned when running make with the -q option.
That's pretty much all you get from make, there doesn't seem to be any way to set the return code.
The default behavior of make is to return failure and abandon any remaining targets if something failed.
for directory in */; do
if ( cd "$directory" && make ); then
echo "$0: Make in $directory succeeded" >&2
else
echo "$0: Make in $directory failed" >&2
fi
done
Simply ensure each test leaves its result in a file unique to that test. Least friction will be to create test.pass if thes test passes, otherwise create test.fail. At the end of the test run gather up all the files and generate a report.
This scheme has two advantages that I can see:
You can run the tests in parallel (You do us the -jn flag, don't you? (hint: it's the whole point of make))
You can use the result files to record whether the test needs to be re-run (standard culling of work (hint: this is nearly the whole point of make))
Assuming the tests are called test-blah where blah is any string, and that you have a list of tests in ${tests} (after all, you have just built them, so it's not an unreasonable assumption).
A sketch:
fail = ${#:%.pass=%.fail}
test-passes := $(addsuffix .pass,${tests})
${test-passes}: test-%.pass: test-%
rm -f ${fail}
touch $#
$* || mv $# ${fail}
.PHONY: all
all: ${test-passes}
all:
# Count the .pass files, and the .fail files
echo '$(words $(wildcard *.pass)) passes'
echo '$(words $(wildcard *.fail)) failures'
In more detail:
test-passes := $(addsuffix .pass,${tests})
If ${tests} contains test-1 test-2 (say), then ${test-passes} will be test-1.pass test-2.pass
${test-passes}: test-%.pass: test-%
You've just gotta love static pattern rules.
This says that the file test-1.pass depends on the file test-1. Similarly for test-2.pass.
If test-1.pass does not exist, or is older than the executable test-1, then make will run the recipe.
rm -f ${fail}
${fail} expands to the target with pass replaced by fail, or test-1.fail in this case. The -f ensures the rm returns no error in the case that the file does not exist.
touch $# — create the .pass file
$< || mv $# ${fail}
Here we run the executable
If it returns success, our work is finished
If it fails, the output file is deleted, and test-1.fail is put in its place
Either way, make sees no error
.PHONY: all — The all target is symbolic and is not a file
all: ${test-passes}
Before we run the recipe for all, we build and run all the tests
echo '$(words $(wildcard *.pass)) passes'
Before passing the text to the shell, make expands $(wildcard) into a list of pass files, and then counts the files with $(words). The shell gets the command echo 4 passes (say)
You run this with
$ make -j9 all
Make will keep 9 jobs running at once — lovely if you have 8 CPUs.
Is there a way to let make determine the number of files to be recompiled before actually compiling? The problem is this: Consider having a quite big project with hundreds of source files. It would very convenient to have a rough idea of how long compilation will take, but to know that, one needs to know the number of files to be compiled.
The general answer is no, because your build could generate files which themselves are inputs to other rules which generate more files. And so on. However if a rough answer is good enough you can try the --dry-run flag. From the GNU make documentation...
“No-op”. Causes make to print the recipes that are needed to make the targets up to date, but not actually execute them. Note that some recipes are still executed, even with this flag (see How the MAKE Variable Works). Also any recipes needed to update included makefiles are still executed (see How Makefiles Are Remade).
As you can see, despite its name even the --dry-run flag will change the state of your build.
"make -n" will do the dry run. But you can't get the list of files to be rebuilt. May be you can write shell script to store the last modified time of files and get the list of files.
I think a found a decent solution for unix. Here SRC are your source files, HDR your headers and DEP the dependency files (something like DEP:=$(OBJ:.o=.d) )
isInDepFile+=$(shell grep -q $(modifiedFile) $(depFile) 1>&2 2> /dev/null && echo $(depFile))
COMPFILES=
checkDepFiles=$(foreach depFile,$(DEP), $(eval filesToCompile+=$(isInDepFile))) $(thinOutDepFiles)
thinOutDepFiles=$(foreach fileToCompile,$(filesToCompile),$(eval DEP=$(filter-out $(fileToCompile),$(DEP))))
countFilesToCompile: $(SRC) $(HDR)
$(eval modifiedFiles=$?)
$(foreach modifiedFile,$(modifiedFiles), $(call checkDepFiles))
$(eval numOfFilesToCompile = $(words $(filesToCompile)))
$(eval numDepFiles = $(words $(DEP)))
$(eval NumSRCFiles = $(words $(SRC)))
#echo $(NumSRCFiles) sources
#echo $(numDepFiles) files to leave
#echo $(numOfFilesToCompile) files to compile
#touch $#
This first generates a list of modified files within your source and header files lists. Then for each modified file it checks all dependency files for its filename. If a dependency file contains the current file we are looking at, it is added to the list of filesToCompile. It is also removed from the list of dependency files to avoid duplication.
This can be invoked in the main building rule of your project. The advantage of that over the dry run is that it gives you a simple number to work with.
I tried looking for answers to this question, so I apologize in advance if this is a duplicate of a question I didn't find. Also sorry that I cannot directly provide the code that I am working with (it would require a lot of environmental dependencies, anyway).
I have a sequence of actions, which all depend on the success of the previous actions, and also don't need to be repeated unless they are out of date. A make solution seemed like the proper one. I've come up with a solution that does almost all of it. Here is the sequence of steps I am trying to replicate, with the output of each step listed below its input:
ZIP file
extract to package/
package/directory/*.comp
execute uncomp.py to create a .uncomp file from a .comp file
Everything works fine up to this point
package/directory/*.uncomp
For *.uncomp files, execute script1 to produce a .html file
For *_ext.uncomp files, execute script2 to produce numbered *_ext.##.png file(s)
Multiple numbered files (_ext.0.png, _ext.1.png, _ext.2.png) are possible, and may not be present at the time make is run. However, make should know that they are the output of the previous step, and only run this recipe if these files (a) don't exist or (b) any are older than the *_ext.uncomp file.
I have put together a Makefile which does almost what I'm looking for, except that it delegates all of the last portion (numbered files) to a shell script which I could program to look at file times, but that defeats the purpose of using make in the first place, in my opinion.
Environment
Debian 8.8 (x86)
GNU Make 4.0
Built for x86_64-pc-linux-gnu
My Question
What rules and recipes can I use to inform GNU make of the relationship between the *_ext.uncomp files and the _ext.##.png files so that those recipes only get executed as necessary (and say 'Target is up-to-date' if all .png files are at least as new as the _ext.uncomp file), that won't also apply to the *.uncomp files, and that will still work of there are no .png files in the output?
I will also need to indicate the relationship between non-_ext files and their corresponding HTML counterparts. So that script1 only gets executed when the HTML file is out of date or doesn't exist. This recipe/rule should not pay attention to _ext.uncomp files.
Any other advice on my Makefile would also be appreciated, because I am not overly familiar with it.
Generalized contents of my current Makefile
.PHONY : all
all : package package/directory/*.uncomp
./process $^
%.comp.uncomp : %.comp package
python uncomp.py $<
package : *.zip
rm -rf package/
unzip *.zip -d package/
Contents of the process script
This script should no longer exist if all the goals of the question are met (make will handle everything). It works great, but it always processes .uncomp files no matter what, even if the output from them already exists and is newer than the source.
#!/bin/bash
if [ $# -lt 2 ]; then
echo "$0 expects at least 2 arguments"
exit 1
fi
# Discard the first agrument, it's always 'package'
shift
# Iterate over each of the remaining arguments
while [ $# -gt 0 ]; do
if [[ $1 == *_ext.uncomp ]] ; then
python script2 $1
elif [[ $1 == *.uncomp ]] ; then
python script1 $1
else
echo "Warning: Unknown file type: $1"
fi
shift
done
I learned a lot about GNU make trying to get this to work. I discovered that the solution to my problem was in not overthinking it.
The most important realization was that I didn't need make to track all of the numbered output files, but just the first one (if the first one is out of date or missing, they all will be, and they all get re-extracted by the script, so a 1:1 relationship was all I needed to indicate there).
I found out that GNU make 3.82 and later uses "shortest stem first" order instead of definition order when matching pattern rules. To make my file compatible with both versions, I made sure to define the most specific stems first.
After that it was a matter of setting up some implicit rules, and just telling make what to expect to be able to find—the concept is a little backwards to my way of thinking which is why I had some trouble at first (look for this file that doesn't exist yet; now, here's a way to make it from a file that does exist). The end result, fully functional:
PACKAGE := package
COMP := .comp
UNCOMP := .comp.uncomp
PNG0 := .comp.0.png
TXT := .comp.txt
SUFFIX := _ext
COMPFILES = $(wildcard $(PACKAGE)/subdir/*$(COMP))
UNCOMPFILES = $(COMPFILES:$(COMP)=$(UNCOMP))
SUFFIXFILES = $(filter %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
PNGFILES = $(SUFFIXFILES:$(UNCOMP)=$(PNG0))
NOSUFFIXFILES = $(filter-out %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
TXTFILES = $(NOSUFFIXFILES:$(UNCOMP)=$(TXT))
.PHONY : all
all : pngs txts htaccess
.PHONY : txts
txts : $(TXTFILES)
.PHONY : pngs
pngs : $(PNGFILES)
.PHONY : uncomp
uncomp : $(UNCOMPFILES)
make pngs
make txts
.PHONY : htaccess
htaccess : $(PACKAGE)/.htaccess
%$(SUFFIX)$(PNG0) : %$(SUFFIX)$(UNCOMP)
## Ignore failures when extracting PNG files
-python script1.py $<
%$(TXT) : %$(UNCOMP)
## Ignore failures when dumping TXT files
-python script2.py $< > $#
%$(UNCOMP) : %$(COMP)
## Ignore decompression failure
-python uncomp.py $<
$(PACKAGE)/.htaccess : .htaccess | $(PACKAGE)
cp .htaccess $(PACKAGE)/
$(PACKAGE) : *.zip
rm -rf $(PACKAGE)/
unzip *.zip -d $(PACKAGE)/
make uncomp
.PHONY : clean
clean :
rm -rf $(PACKAGE)/