I use make to create thumbnails from image files that will in turn be used to create web pages.
Sometimes I delete a source image and I would like the thumbnail to be removed automatically. The obvious way to do it would be to simply add a rule where the target and prerequisite are swapped but that causes a circular reference and messages like this:
make[2]: Circular image123.jpg <- thumbnails/image123.jpg dependency dropped.
Is there a way to do it or must I use a separate makefile? Actually tried but I can't seem to make that work either. I tried this:
# -*- mode: makefile -*-
images = $(wildcard *.jpg)
thumbnails = $(wildcard thumbnails/*.jpg)
.PHONY: all
all : $(images)
# Delete thumbnails if source is not present.
%.jpg: thumbnails/%.jpg
[ -f "$#" ] || rm "$<"
It didn't do anything except print one line referring to the image that is to be kept.
To summarize: is there a way to use make to remove targets that would not be built if they were missing?
First, let's set up a rule to build the thumbnails:
images := $(wildcard *.jpg)
TARGETS := $(addprefix thumbnails/, $(images))
.PHONY: all
all : $(TARGETS)
thumbnails/%.jpg: %.jpg
create the thumbnail somehow
Then add some code to find the obsolete thumbnails:
thumbnails := $(wildcard thumbnails/*.jpg)
OBSOLETE := $(filter-out $(TARGETS), $(thumbnails))
Then write a rule to remove them:
.PHONY: purge
purge:
rm $(OBSOLETE)
This will cause trouble if there are no obsolete thumbnails, so let's wrap the recipe in a conditional:
purge:
ifneq ($(OBSOLETE),)
rm $(OBSOLETE)
endif
And finally make purge a prerequisite of all:
all : $(TARGETS) purge
Related
My goal is the following: I have a directory src which contains markdown files (.md). I want to run a command on each of these files so that the comments are removed and the edited files are stored in a separate directory. For this I want to use make.
This is the Makefile I have:
.PHONY: clean all
BUILD_DIR := build
SRC_DIRS := src
SRCS := $(shell find $(SRC_DIRS) -name *.md)
DSTS := $(patsubst $(SRC_DIRS)/%.md,$(BUILD_DIR)/%.md,$(SRCS))
all: $(DSTS)
# The aim of this is to remove all my comments from the final documents
$(DSTS): $(SRCS)
pandoc --strip-comments -f markdown -i $< -t markdown -o $#
clean:
rm $(BUILD_DIR)/*.md
While this works in general, I noticed that the command is executed on all files, even though I changed only one single file.
Example: I have 3 Files src/a.md, src/b.md and src/c.md. Now I run make and all files are correctly generated in the build folder. Now I only edit c.md and run make again. I would expect that make only "compiles" src/c.md anew but instead all three files are compiled again. What am I doing wrong?
Your line
$(DSTS): $(SRCS)
is saying ‘All of the DSTS depend on all of the SRCS’, so whenever any one of the $(SRCS) is newer than any of the $(DSTS), this pandoc action will be run.
That's not what you want to express. What you want is something more like
$(BUILD_DIR)/%.md: $(SRC_DIRS)/%.md
pandoc --strip-comments -f markdown -i $< -t markdown -o $#
all: $(DSTS)
That says that all of the $(DSTS) should be up to date, and the pattern rule teaches Make what each one depends on, and how to build it, if it is out of date.
(As a general point, looking your original rule, it's rarely the right thing to do to have multiple targets in a rule, as you have with $(DSTS); also note that in your original, $< always refers only to the first of the dependencies in $(SRCS))
This is my first ever post to stackoverflow. It ought to be straightforward, but I have this problem whenever I try to write a makefile and I've never been able to figure out a satisfactory solution. Apologies if there is already a solution somewhere on the site. I couldn't find one.
What I'm trying to do is as follows:
Search my src directory for matching source files.
Compile the src code into a sandbox.
Here's my directory structure:
makefile
src1
file1.src
file2.src
src2
file3.src
subfolder
src3
file4.src
file5.src
And here's my makefile:
BUILDDIR := ./sandbox
SRC_DIRS := ./
SRCS := $(shell find $(SRC_DIRS) -name *.src)
OBJS := $(addprefix $(BUILDDIR)/o., $(notdir $(SRCS) ) )
# make print-X prints value of variable X
print-%: ; #echo $* = $($*)
.PHONY: help
help:
#echo "make <all|clean>"
.PHONY: all
all: $(OBJS)
#echo "compilation done"
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
$(BUILDDIR)/.create:
#echo "creating sandbox"
mkdir -p $(BUILDDIR) && cd $(BUILDDIR)
#touch $#
.PHONY: clean
clean:
#echo "deleting sandbox"
#rm -rf $(BUILDDIR)
If I type make all, the file works as expected. However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
And the process of creating objects of objects continues recursively as many times as I type make.
Any help would be appreciated.
Incidentally, please don't post solutions that rely on the build in compile functions of make. I'm looking for a general solution that can be used for any task. For example, in this instance, I'm trying to read the source files into a tool using its command line interface.
Well, first, by having your sandbox as a subdirectory of your source directory, then using find on the source directory, every time you run it you're going to find all the files in both the source directory and all its subdirectories, including the sandbox. If the built files in the sandbox have the same names as the files in the source directories, the find will find them all.
Maybe instead of:
SRCS := $(shell find $(SRC_DIRS) -name *.src)
you want something like:
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -o -name *.src -print)
Or, alternatively, don't make your sandbox a subdirectory of your source directory. Or, make sure that whatever name you give to the files in the sandbox directory won't match the *.src pattern you give to find.
But beyond that this is wrong:
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
Suppose SRCS is foo.src bar.src, which means OBJS is sandbox/o.foo.src sandbox/o.bar.src. Then the above expands to this:
sandbox/o.foo.src sandbox/o.bar.src : foo.src bar.src sandbox/.create
This is a common mistake; people seem to think that make will go through the targets and prerequisites and match them up, so the first target depends on the first prerequisite and the second target depends on the second one etc. but of course this cannot work correctly and that's not how make works. Make treats the above as if you'd written one rule for each target, with the same prerequisites; like this:
sandbox/o.foo.src : foo.src bar.src sandbox/.create
sandbox/o.bar.src : foo.src bar.src sandbox/.create
You can see this won't do what you want at all, since the $< will always be foo.src which is clearly wrong.
You need to write a single pattern rule that will build ONE target. Then make sure the pattern applies to all the targets.
You have made things hard for yourself by trying to "flatten" a directory structure of multiple source subdirectories, into a single level of target directory (by using the $(notdir $(SRCS))). Because of this, there's no pattern that will match the same target and directory, unless you write a separate rule for every subdirectory.
Luckily there is a solution for this: VPATH. This should work for you:
VPATH := $(sort $(dir $(SRCS))
$(BUILDDIR)/o.%.src : %.src $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
The VPATH tells make to go look in all the directories that it found any sources in, whenever it can't find one to build.
The basic problem is that your SRCS is all files in all subdirectories that match the pattern *.src (when you run make). That means that all your object files ($(OBJS)) also match, so they copied as well.
The solution is to change your SRCS pattern so it does not match the "object" files in the build directory. Possibilities:
SRCS := $(wildcard *.src)
or
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -false -o -name *.src)
or change the names of your "object" files so they don't end in .src
If I type make all, the file works as expected.
By which I take you to mean that directory ./sandbox exists and contains these files:
o.file1.src
o.file2.src
o.file3.src
o.file4.src
o.file5.src
However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
Of course you do, because everything is not up to date at that point, according to the target list you create. This line ...
SRCS := $(shell find $(SRC_DIRS) -name *.src)
... defines SRCS as a list of all the files under path ./ (which is the value of SRC_DIRS) that ends in .src. That includes any such files in ./sandbox, which include all the files placed there by the first make run. You generate a corresponding target file to build for each source, and those targets corresponding to sources built by the previous make run will not, in general, exist yet. So make builds them, just as you instructed it.
The best solution, short of abandoning that automatic source identification altogether, would probably be to change your naming scheme so that outputs cannot be mistaken so easily for sources. For example, if you want the output names to have the form foo.src, then have the corresponding input named something like foo.src.in. In that particular case, you could convert from source names to target names with
OBJS := $(addprefix $(BUILDDIR)/o., $(basename $(notdir $(SRCS) ) ) )
Alternatively, you could modify the find command to skip the sandbox directory, maybe by modifying SRC_DIRS:
SRC_DIRS = src1 src2 subfolder/src3
(These specific alternatives are not mutually exclusive.)
I am trying to statically build HTML files that requires a markdown file and a meta file called "whatlinkshere" for the HTML file to demonstrate its back links.
I believe it can be effeciently done by a Makefile, by first generating all the "whatlinkshere" files. I don't think this can be done in parallel, because the program that generates these files needs to append to the whatlinkshere files, and there could be race conditions that I am not quite sure how to solve.
Once the "whatlinkshere" files are generated then if a markdown file is edited, say foo.mdwn to point to bar.mdwn, only foo.mdwn needs to be analysed again for "whatlinkshere" changes. And finally only foo.html and bar.html need to be rebuilt.
I am struggling to accomplish this in my backlinks project.
INFILES = $(shell find . -name "*.mdwn")
OUTFILES = $(INFILES:.mdwn=.html)
LINKFILES = $(INFILES:.mdwn=.whatlinkshere)
all: $(OUTFILES)
# These need to be all made before the HTML is processed
$(LINKFILES): $(INFILES)
#echo Creating backlinks $#
#touch $#
#go run backlinks.go $<
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
Current problems here is that *.whatlinkshere** aren't being generated on first run. My workaround is for i in *.mdwn; do go run backlinks.go $i; done. Furthermore there are not rebuilding as I want after editing a file as described earlier. Something is horribly wrong. What am I missing?
I think I finally understood your problem. If I understood well:
You have a bunch of *.mdwn source files.
You generate *.whatlinkshere files from your *.mdwn source files using the backlinks.go utility. But this utility does not produce foo.whatlinkshere from foo.mdwn. It analyzes foo.mdwn, searches for links to other pages in it and, for each link to bar it finds, it appends a [foo](foo.html) reference to bar.whatlinkshere.
From each foo.mdwn source file you want to build a corresponding foo.html file with:
$ cmark foo.mdwn foo.whatlinkshere
Your rule:
$(LINKFILES): $(INFILES)
#echo Creating backlinks $#
#touch $#
#go run backlinks.go $<
contains one error and has several drawbacks. The error is the use of the $< automatic variable in the recipe. It expands as the first prerequisite, that is probably always pageA.mdwn in your case. Not what you want. $^ expands as all prerequisites but it is not the correct solution because:
your go utility takes only one source file name, but even if it was accepting several...
...make will run the recipe several times, one per link file, which is a waste, and...
...as your go utility appends to the link files it will even be worse than a waste: back links will be counted several times each, and...
...if make runs in parallel mode (note that you can prevent this with make -j1 or by adding the .NOTPARALLEL: special rule to your Makefile, but it is a pity) there is a risk of race conditions.
Important: the following works only with a flat organization where all source files and HTML files are in the same directory as the Makefile. Other organizations are possible, of course, but they would require some modifications.
First option using multi-targets pattern rules
One possibility is to use a special property of make pattern rules: when they have several targets make considers that one single execution of the recipe produces all targets. For instance:
pageA.w%e pageB.w%e pageC.w%e: pageA.mdwn pageB.mdwn pageC.mdwn
for m in $^; do go run backlinks.go $$m; done
tells make that pageA.whatlinkshere, pageB.whatlinkshere and pageC.whatlinkshere are all generated by one execution of:
for m in pageA.mdwn pageB.mdwn pageC.mdwn; do go run backlinks.go $m; done
(make expands $^ as all prerequisites and $$m as $m). Of course, we want to automate the computation of the pageA.w%e pageB.w%e pageC.w%e pattern targets list. This should make it:
INFILES := $(shell find . -name "*.mdwn")
OUTFILES := $(INFILES:.mdwn=.html)
LINKFILES := $(INFILES:.mdwn=.whatlinkshere)
LINKPATTERN := $(INFILES:.mdwn=.w%e)
.PHONY: all clean
.PRECIOUS: $(LINKFILES)
all: $(OUTFILES)
# These need to be all made before the HTML is processed
$(LINKPATTERN): $(INFILES)
#echo Creating backlinks
#rm -f $(LINKFILES)
#touch $(LINKFILES)
#for m in $^; do go run backlinks.go $$m; done
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
clean:
rm -f $(LINKFILES) $(OUTFILES)
Notes:
I declared all and clean as phony because... it is what they are.
I declared the whatlinkshere files as precious because (some of them) are considered by make as intermediates and without this declaration make would delete them after building the HTML files.
In the recipe for the whatlinkshere files I added rm -f $(LINKFILES) such that, if the recipe is executed, we restart from a clean state instead of concatenating new stuff to old (possibly outdated) references.
The pattern stem in the $(LINKPATTERN) can be anything but must match at least one character. I used w%e but whatlin%shere would work too. Use whatever is specific enough in your case. If you have a pageB.where file prefer whatlin%shere or what%here.
There is a drawback with this solution but it is due to your particular set-up: each time one single mdwn file changes it must be re-analyzed (which is normal) but any whatlinkshere file can be impacted. This is not predictable, it depends on the links that have been modified in this source file. But more problematic is the fact that the result of this analysis is appended to the impacted whatlinkshere files. They are not "edited" with the old content relative to this source file replaced by the new one. So, if you change just a comment in a source file, all its links will be appended again to the respective whatlinkshere files (while they are already there). This is probably not what you want.
This is why the solution above deletes all whatlinkshere files and re-analyzes all source files each time one single source file changes. And another negative consequence is that all HTML files must also be re-generated because all whatlinkshere files changed (even if their content did not really change, but make does not know this). If the analysis is super fast and you have a small number of mdwn files, it should be OK. Else it is sub-optimal but not easy to solve because of your particular set-up.
Second option using recursive make, separated back link files and marker files
There is a possibility, however, which consists in:
separating all back links references with one whatlinkshere file per from/to pair: foo.backlinks/bar.whatlinkshere contains all references to bar found in foo.mdwn,
using recursive make with one first invocation (when the STEP make variable is unset) to update all whatlinkshere files that need to be and a second invocation (STEP set to 2) to generate the HTML files that need to be,
using empty dummy files to mark that a foo.mdwn file has been analyzed: foo.backlinks/.done,
using the secondary expansion to be able to refer to the stem of a pattern rule in its list of prerequisites (and using $$ to escape the fist expansion).
But it is probably a bit more difficult to understand (and maintain).
INFILES := $(shell find . -name "*.mdwn")
OUTFILES := $(INFILES:.mdwn=.html)
DONEFILES := $(patsubst %.mdwn,%.backlinks/.done,$(INFILES))
.PHONY: all clean
ifeq ($(STEP),)
all $(OUTFILES): $(DONEFILES)
$(MAKE) STEP=2 $#
%.backlinks/.done: %.mdwn
rm -rf $(dir $#)
mkdir -p $(dir $#)
cp $< $(dir $#)
cd $(dir $#); go run ../backlinks.go $<; rm $<
touch $#
else
all: $(OUTFILES)
.SECONDEXPANSION:
%.html: %.mdwn $$(wildcard *.backlinks/$$*.whatlinkshere)
#echo Deps $^
#cmark $^ > $#
endif
clean:
rm -rf *.backlinks $(OUTFILES)
Even if it looks more complicated there are a few advantages with this version:
only outdated targets are rebuilt and only once each,
all whatlinkshere files are updated (if needed) before any HTML file is updated (if needed),
the whatlinkshere files can be built in parallel,
the HTML files can be built in parallel.
Third option using only recursive make and marker files
If you do not care about inaccurate results where back links persist in the results after they disappeared from the source files or where back links are uselessly replicated, we can reuse ideas from the previous solution but drop the separation in individual from/to whatlinkshere files.
INFILES := $(wildcard *.mdwn)
OUTFILES := $(patsubst %.mdwn,%.html,$(INFILES))
LINKFILES := $(patsubst %.mdwn,%.whatlinkshere,$(INFILES))
DONEFILES := $(patsubst %.mdwn,.%.done,$(INFILES))
.PHONY: all clean
.PRECIOUS: $(LINKFILES)
ifeq ($(STEP),)
.NOTPARALLEL:
all $(OUTFILES): $(DONEFILES)
$(MAKE) STEP=2 $#
.%.done: %.mdwn
go run backlinks.go $<
touch $#
else
all: $(OUTFILES)
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
%.whatlinkshere:
touch $#
endif
clean:
rm -f $(OUTFILES) $(LINKFILES) $(DONEFILES)
Notes:
As this works only for a flat organization I replaced the $(shell find...) by the make built-in $(wildcard ...).
I used patsubst instead of the old syntax but it's just a matter of taste.
The %.whatlinkshere: rule is a default rule to create the missing empty whatlinkshere files.
The NOTPARALLEL: special target prevents parallel execution when building the whatlinkshere files.
Goog evening,
I'm completely new to makefiles and worked out a file which fits our needs good but I'm not completely satisfied. We use bootstrap3 and have around 40 customers with differend color settings. That's why we need to compile 40 slightly different css files. Until now, we have the following file structure
less/customer1.less
css/customer1.css
color/customer.less contains bootstraps variables file
customer1.less contains
#variables: 'myCompany/color/customer1'; //this is forwarded to where bootrstrap loads the variables template
#import "bootstrap";
#import 'myCompany/modifications';
Our makefile
SOURCES = $(shell ls less/*.less)
# Files we don't want to be build
SOURCES := $(filter-out less/bootstrap.less, $(SOURCES))
SOURCES := $(filter-out less/a11y.less, $(SOURCES))
TARGETS = $(patsubst less/%.less,css/%.css,$(SOURCES))
DEPEND = $(patsubst less/%.less,make/%.d,$(SOURCES))
css/%.css: less/%.less
# First building dependency files
lessc -M $< $# > 'make/$*.d'
# Then building CSS and sourcemap
lessc -s $< > $# --source-map=map/$*.css.map --source-map-basepath=map --clean-css
-include $(DEPEND)
all: $(TARGETS)
Call
$ make all
Creates Makefiles in make/, CSS in css/ CSS source-maps in map/ and expects LESS being in less/.
This works but we need to create customerX.less for each customer manually even if the only difference is the assigned color scheme/variables file.
Make should look in the color folder if there is a file for this customerX and then create (but not overwrite!) customerX.less in less directory.
Any make guru out here know how to do this with make?
I believe you can do what you want here with an order-only prerequisite.
Something like:
less/customer%.less: | color/customer.less
[ -f '$#' ] || cp $^ $#
I don't think the -f test is strictly necessary there but it shouldn't hurt and is safer.
On a different topic $(shell ls less/*.less) can probably be done better with either $(shell echo less/*.less) (you don't care about what ls does you just want the shell glob expansion) or $(wildcard less/*.less). (Technically shelling out and wildcard are slightly different but I don't know that that will matter for you here.)
Also note that the all target will not create these missing less files for you (as SOURCES will not contain them as the file didn't exist) but make css/customer#.css will create them if necessary.
I'd like to learn how to re-use the % sign in my makefile target's prequisite, supposing that the target is X.pdf, and the prerequesite is in X/X.tex.
To elaborate, I currently have a makefile like so:
all: foo.pdf
%.pdf: %.tex
pdflatex $*.tex
I additionally have a file foo.tex, and when I type make it will make foo.pdf by running pdflatex foo.tex.
Now for various reasons I can't control, my directory structure has changed:
my_dir
|- Makefile
|- foo
|- foo.tex
I'd like to modify my Makefile such that when it tries to make X.pdf, it looks for the file X/X.tex.
I tried the following (I tried to put '%/%.tex' to tell it to look for foo/foo.tex):
all: foo.pdf
%.pdf: %/%.tex
pdflatex $*/$*.tex
However, this yields:
No rule to make target `foo.pdf', needed by `all'. Stop.
So does %.pdf: $*/$*.tex.
If I change the %/%.tex to foo/%.tex it works as expected, but I don't want to hard-code the foo in there, because in the future I'll do all: foo.pdf bar.pdf and it should look for foo/foo.tex and bar/bar.tex.
I'm fairly new to Makefiles (experience limited to modifying someone else's to my needs), and have never done a more-than-absolutely basic one, so if anyone could give me a pointer that would help (I don't really know what words to search for in the Makefile documentation - the only ones that looked promising were % and $*, which I couldn't get to work).
you can use VPATH to specify a list of directories that make should search.
exemplary makefile:
# Find all tex files
tex := $(shell find -iname '*.tex')
# Make targets out of them
PDFS := $(notdir $(tex:%.tex=%.pdf))
# specify search folder
VPATH := $(dir $(tex))
all : $(PDFS)
%.pdf : %.tex
pdflatex $<
or even better would be to use vpath (lowercase):
vpath %.tex $(dir $(tex))
it will look only for .tex files in those directories.