I'd like to use GNUmake to do image compression and some preparations for my websites.
I have a structure like
src
www.site1.org
images
foo.jpg
bar.png
css
style.scss
www.site2.org
images
baz.svg
css
design.scss
I want be able to recreate this structure in target directory while using some other tools to optimize/compile the sources like e.g. convert --strip src/www.site1.org/images/foo.jpg target/www.site1.org/images/foo.jpg
I can find all my jpegs using something like SRC:=$(shell find . -name '*.jpg') and even create a variable holding all the targets with TARGETS=$(SRC:src=target).
But now I don't find a way of writing a rule for which would (naively) look like:
$(TARGET): $(SRC)
convert --strip $< $#
I've searched the internet for quiet some time now but didn't find anything appropriate.
You could use:
SRC := $(shell find src -name '*.jpg')
TARGETS = $(patsubst src/%,target/%,$(SRC))
all: $(TARGETS)
$(TARGETS): target/%: src/%
convert --strip $< $#
which basically says: for each word in TARGETS, match it against the target/% pattern (and record the % sub-expression), consider that its single dependency is src/% and apply the recipe. This is a GNU make feature, documented in section 4.12 Static Pattern Rules of the manual.
By the way, $(SRC:src=target) does not do what you think because the pattern matches only at the end of each word of $(SRC). So, foo_src would be substituted with foo_target but src/foo would be left unmodified. patsubst is a more powerful substitution function.
Related
This is my first ever post to stackoverflow. It ought to be straightforward, but I have this problem whenever I try to write a makefile and I've never been able to figure out a satisfactory solution. Apologies if there is already a solution somewhere on the site. I couldn't find one.
What I'm trying to do is as follows:
Search my src directory for matching source files.
Compile the src code into a sandbox.
Here's my directory structure:
makefile
src1
file1.src
file2.src
src2
file3.src
subfolder
src3
file4.src
file5.src
And here's my makefile:
BUILDDIR := ./sandbox
SRC_DIRS := ./
SRCS := $(shell find $(SRC_DIRS) -name *.src)
OBJS := $(addprefix $(BUILDDIR)/o., $(notdir $(SRCS) ) )
# make print-X prints value of variable X
print-%: ; #echo $* = $($*)
.PHONY: help
help:
#echo "make <all|clean>"
.PHONY: all
all: $(OBJS)
#echo "compilation done"
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
$(BUILDDIR)/.create:
#echo "creating sandbox"
mkdir -p $(BUILDDIR) && cd $(BUILDDIR)
#touch $#
.PHONY: clean
clean:
#echo "deleting sandbox"
#rm -rf $(BUILDDIR)
If I type make all, the file works as expected. However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
And the process of creating objects of objects continues recursively as many times as I type make.
Any help would be appreciated.
Incidentally, please don't post solutions that rely on the build in compile functions of make. I'm looking for a general solution that can be used for any task. For example, in this instance, I'm trying to read the source files into a tool using its command line interface.
Well, first, by having your sandbox as a subdirectory of your source directory, then using find on the source directory, every time you run it you're going to find all the files in both the source directory and all its subdirectories, including the sandbox. If the built files in the sandbox have the same names as the files in the source directories, the find will find them all.
Maybe instead of:
SRCS := $(shell find $(SRC_DIRS) -name *.src)
you want something like:
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -o -name *.src -print)
Or, alternatively, don't make your sandbox a subdirectory of your source directory. Or, make sure that whatever name you give to the files in the sandbox directory won't match the *.src pattern you give to find.
But beyond that this is wrong:
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
Suppose SRCS is foo.src bar.src, which means OBJS is sandbox/o.foo.src sandbox/o.bar.src. Then the above expands to this:
sandbox/o.foo.src sandbox/o.bar.src : foo.src bar.src sandbox/.create
This is a common mistake; people seem to think that make will go through the targets and prerequisites and match them up, so the first target depends on the first prerequisite and the second target depends on the second one etc. but of course this cannot work correctly and that's not how make works. Make treats the above as if you'd written one rule for each target, with the same prerequisites; like this:
sandbox/o.foo.src : foo.src bar.src sandbox/.create
sandbox/o.bar.src : foo.src bar.src sandbox/.create
You can see this won't do what you want at all, since the $< will always be foo.src which is clearly wrong.
You need to write a single pattern rule that will build ONE target. Then make sure the pattern applies to all the targets.
You have made things hard for yourself by trying to "flatten" a directory structure of multiple source subdirectories, into a single level of target directory (by using the $(notdir $(SRCS))). Because of this, there's no pattern that will match the same target and directory, unless you write a separate rule for every subdirectory.
Luckily there is a solution for this: VPATH. This should work for you:
VPATH := $(sort $(dir $(SRCS))
$(BUILDDIR)/o.%.src : %.src $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
The VPATH tells make to go look in all the directories that it found any sources in, whenever it can't find one to build.
The basic problem is that your SRCS is all files in all subdirectories that match the pattern *.src (when you run make). That means that all your object files ($(OBJS)) also match, so they copied as well.
The solution is to change your SRCS pattern so it does not match the "object" files in the build directory. Possibilities:
SRCS := $(wildcard *.src)
or
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -false -o -name *.src)
or change the names of your "object" files so they don't end in .src
If I type make all, the file works as expected.
By which I take you to mean that directory ./sandbox exists and contains these files:
o.file1.src
o.file2.src
o.file3.src
o.file4.src
o.file5.src
However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
Of course you do, because everything is not up to date at that point, according to the target list you create. This line ...
SRCS := $(shell find $(SRC_DIRS) -name *.src)
... defines SRCS as a list of all the files under path ./ (which is the value of SRC_DIRS) that ends in .src. That includes any such files in ./sandbox, which include all the files placed there by the first make run. You generate a corresponding target file to build for each source, and those targets corresponding to sources built by the previous make run will not, in general, exist yet. So make builds them, just as you instructed it.
The best solution, short of abandoning that automatic source identification altogether, would probably be to change your naming scheme so that outputs cannot be mistaken so easily for sources. For example, if you want the output names to have the form foo.src, then have the corresponding input named something like foo.src.in. In that particular case, you could convert from source names to target names with
OBJS := $(addprefix $(BUILDDIR)/o., $(basename $(notdir $(SRCS) ) ) )
Alternatively, you could modify the find command to skip the sandbox directory, maybe by modifying SRC_DIRS:
SRC_DIRS = src1 src2 subfolder/src3
(These specific alternatives are not mutually exclusive.)
Goog evening,
I'm completely new to makefiles and worked out a file which fits our needs good but I'm not completely satisfied. We use bootstrap3 and have around 40 customers with differend color settings. That's why we need to compile 40 slightly different css files. Until now, we have the following file structure
less/customer1.less
css/customer1.css
color/customer.less contains bootstraps variables file
customer1.less contains
#variables: 'myCompany/color/customer1'; //this is forwarded to where bootrstrap loads the variables template
#import "bootstrap";
#import 'myCompany/modifications';
Our makefile
SOURCES = $(shell ls less/*.less)
# Files we don't want to be build
SOURCES := $(filter-out less/bootstrap.less, $(SOURCES))
SOURCES := $(filter-out less/a11y.less, $(SOURCES))
TARGETS = $(patsubst less/%.less,css/%.css,$(SOURCES))
DEPEND = $(patsubst less/%.less,make/%.d,$(SOURCES))
css/%.css: less/%.less
# First building dependency files
lessc -M $< $# > 'make/$*.d'
# Then building CSS and sourcemap
lessc -s $< > $# --source-map=map/$*.css.map --source-map-basepath=map --clean-css
-include $(DEPEND)
all: $(TARGETS)
Call
$ make all
Creates Makefiles in make/, CSS in css/ CSS source-maps in map/ and expects LESS being in less/.
This works but we need to create customerX.less for each customer manually even if the only difference is the assigned color scheme/variables file.
Make should look in the color folder if there is a file for this customerX and then create (but not overwrite!) customerX.less in less directory.
Any make guru out here know how to do this with make?
I believe you can do what you want here with an order-only prerequisite.
Something like:
less/customer%.less: | color/customer.less
[ -f '$#' ] || cp $^ $#
I don't think the -f test is strictly necessary there but it shouldn't hurt and is safer.
On a different topic $(shell ls less/*.less) can probably be done better with either $(shell echo less/*.less) (you don't care about what ls does you just want the shell glob expansion) or $(wildcard less/*.less). (Technically shelling out and wildcard are slightly different but I don't know that that will matter for you here.)
Also note that the all target will not create these missing less files for you (as SOURCES will not contain them as the file didn't exist) but make css/customer#.css will create them if necessary.
I'd like to learn how to re-use the % sign in my makefile target's prequisite, supposing that the target is X.pdf, and the prerequesite is in X/X.tex.
To elaborate, I currently have a makefile like so:
all: foo.pdf
%.pdf: %.tex
pdflatex $*.tex
I additionally have a file foo.tex, and when I type make it will make foo.pdf by running pdflatex foo.tex.
Now for various reasons I can't control, my directory structure has changed:
my_dir
|- Makefile
|- foo
|- foo.tex
I'd like to modify my Makefile such that when it tries to make X.pdf, it looks for the file X/X.tex.
I tried the following (I tried to put '%/%.tex' to tell it to look for foo/foo.tex):
all: foo.pdf
%.pdf: %/%.tex
pdflatex $*/$*.tex
However, this yields:
No rule to make target `foo.pdf', needed by `all'. Stop.
So does %.pdf: $*/$*.tex.
If I change the %/%.tex to foo/%.tex it works as expected, but I don't want to hard-code the foo in there, because in the future I'll do all: foo.pdf bar.pdf and it should look for foo/foo.tex and bar/bar.tex.
I'm fairly new to Makefiles (experience limited to modifying someone else's to my needs), and have never done a more-than-absolutely basic one, so if anyone could give me a pointer that would help (I don't really know what words to search for in the Makefile documentation - the only ones that looked promising were % and $*, which I couldn't get to work).
you can use VPATH to specify a list of directories that make should search.
exemplary makefile:
# Find all tex files
tex := $(shell find -iname '*.tex')
# Make targets out of them
PDFS := $(notdir $(tex:%.tex=%.pdf))
# specify search folder
VPATH := $(dir $(tex))
all : $(PDFS)
%.pdf : %.tex
pdflatex $<
or even better would be to use vpath (lowercase):
vpath %.tex $(dir $(tex))
it will look only for .tex files in those directories.
I dont know much makefile stuff I've been tending to learn bits as required.
The biggest failing of my makefiles is that I have been listing all the files manually, while this hasn't been a problem my current project is getting unwieldy. I have 4 directories each with sources files.
How can I get all the object file listing without having to list them manually.
This doesn't work, but it shows what I've been trying to do.
VPATH = Lib GameCode Moot/Moot Moot/Impl
OBJS = $(subst .cpp, .o, $(VPATH))
foobar: $(OBJS)
g++ -o $# $^
%.o: %.cpp
g++ -c $< -o $# -I Moot
clean:
rm main.o lib.o foo.o foobar
Personally, I never had any problem in listing all files manually. Listing a file to the makefile takes negligible time compared to adding filling it with useful content.
To get all files from different directories, one might suggest using wildcard function. So my_sources:=$(wildcard *.cpp dir1/*.cpp) will make the variable contain source files that match wildcard expression.
However, I find it less convenient than using usual Linux find command via shell:
# Find all sources
my_sources:=$(shell find -iname '*.cpp')
# Make targets out of them
OBJS=$(my_sources:%.cpp=%.o)
Find is more powerful than Make's builtin wildcard. You might also want to use other shell capabilities, such as pipelines, for example, to filter output of find (if Make's filter-out function is not enough). Or something like this, to avoid excessive variables:
OBJS:=$(shell find -iname '*.cpp' | sed 's/\.cpp$/.o/')
You name it!
Using VPATH or vpath will not work for your problem.. it provides a search path to find files but you still need to list the files. If you just need to compile all and any .c/.cpp files found in those directories then this should work:
foobar: $(shell ls Lib/*.cpp) $(shell ls GameCode/*.cpp) $(shell ls Moot/Moot/*.cpp) $(shell ls Moot/Impl/*cpp)
g++ -o $# $^
clean:
rm foobar $(shell ls Lib/*.o) $(shell ls GameCode/*.o) $(shell ls Moot/Moot/*.o) $(shell ls Moot/Impl/*o)
The VPATH info is not needed, the substitution of .o for .cpp can go as can the override of the implicit rule. Additionally, not the use of ls instead of find to look in, and only in, the specfified directory.
I have to integrate the generation of many HTML files in an existing Makefile.
The problem is that the HTML files need to reside in many different directories.
My idea is to write an implicit rule that converts the source file (*.st) to the corresponding html file
%.html: %.st
$(HPC) -o $# $<
and a rule that depends on all html files
all: $(html)
If the HTML file is not in the builddir, make doesn't find the implicit rule: *** No rule to make target.
If I change the implicit rule like so
$(rootdir)/build/doc/2009/06/01/%.html: %.st
$(HPC) -o $# $<
it's found, but then I have to have an implicit rule for nearly every file in the project.
According to Implicit Rule Search Algorithm in the GNU make manual, rule search works like this:
Split the entire target name t into a directory part, called d, and the rest, called n. For
example, if t is src/foo.o,
then d is src/,
and n is foo.o.
Make a list of all the pattern rules one of whose targets matches t or n.
If the target pattern contains a slash,
it is matched against t;
otherwise, against n.
Why is the implicit rule not found, and what would be the most elegant solution, assuming GNU make is used?
Here is a stripped down version of my Makefile:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html
%.html: %.st
$(HPC) -o $# $<
#This works, but requires a rule for every output dir
#$(rootdir)/build/doc/2009/06/01/%.html: %.st
# $(HPC) -o $# $<
.PHONY: all
all: $(html)
The best solution I found so far is to generate an implicit rule per target directory via foreach-eval-call, as explained in the GNU make manual. I have no idea how this scales to a few thousand target directories, but we will see...
If you have a better solution, please post it!
Here is the code:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html \
$(rootdir)/build/doc/2009/06/02/some.html
targetdirs = $(rootdir)/build/doc/2009/06/01 \
$(rootdir)/build/doc/2009/06/02
define generateHtml
$(1)/%.html: %.st
-mkdir -p $(1)
$(HPC) -o $$# $$<
endef
$(foreach targetdir, $(targetdirs), $(eval $(call generateHtml, $(targetdir))))
.PHONY: all
all: $(html)
Like Maria Shalnova I like recursive make (though I disagree with "Recursive Make Considered Harmful"), and in general it's better to make something HERE from a source THERE, not the reverse. But if you must, I suggest a slight improvement: have generateHtml generate only the RULE, not the COMMANDS.
Your active implicit rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on $(rootdir)/build/doc/2009/06/01/some.st. If $(rootdir)/build/doc/2009/06/01/some.st doesn't exist then the rule won't be used/found.
The commented out rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on some.st.
One solution is to make you're source layout match your destination/result layout.
Another option is to create the rules as required with eval. But that will be quite complicated:
define HTML_template
$(1) : $(basename $(1))
cp $< $#
endef
$(foreach htmlfile,$(html),$(eval $(call HTML_template,$(htmlfile))))
An other possibility is to have the commando make call itself recursively with the argument -C with every output directory.
Recursive make is somewhat the standard way to deal with subdirectories, but beware of the implications mentioned in the article "Recursive Make Considered Harmful"