Hi!
I started messing with makefiles a few days ago, so I gave myself a exercise to learn as well as making my life easier.
I basically want to read the directory the makefile is in (root) and use luamin to compress the file as much as possible before I deploy it to our server. But I would like to have it as flexible as possible, so depending on where said file is in the directory it should mirror it to the server.
So if it finds a file in a sub folder called home it should create a new folder with the same name with the compressed file within. I have gotten the compression of files in the root folder working as well as creation of the directories where the files should reside.
objs = $(wildcard *.lua)
dirs = $(wildcard */)
compress: $(objs)
mkdir -p .build
luamin -f $(objs) > .build/$(objs)
mkdir .build/$(dirs)
clean:
rm -rf ./.build
deploy: .build
cp ./.build/* ~
If you use GNU make, there are several features that really help to do what you want. Warning: this works if and only if your file names do not contain spaces:
srcfiles := $(shell find . -path .build -prune -o -type f -name '*.lua' -print)
dstfiles := $(addprefix .build/,$(srcfiles))
.PHONY: compress clean deploy
compress: $(dstfiles)
$(dstfiles): .build/%: %
mkdir -p $(dir $#)
luamin -f $< > $#
clean:
rm -rf ./.build
deploy: .build
cp ./.build/* ~
Explanation:
The shell make function is used to run the find command that searches all subdirectories, except .build, for *.lua files. The result is assigned to the srcfiles make variable.
The addprefix make function is used to add the .build/ prefix to all words of the srcfiles make variable and assign the result to the dstfiles make variable.
The compress target is the first (real) target in the Makefile. It is thus the default goal that gets run when invoking just make. It is the same as invoking make compress. The compress target is declared as phony. This tells make that it is not a real file, just like clean and deploy. The compress target depends on all destination files. If one is missing or older than its corresponding source file, it must be rebuilt.
The make static pattern rule $(dstfiles): .build/%: %... declares a generic rule where each destination file (.build/./foo/bar/baz.lua) depends on the corresponding source file (./foo/bar/baz.lua). The recipe creates the destination directory (./foo/bar/), computed thanks to the dir make function. Then, it applies the luamin command. The recipe makes use of the $# and $< automatic variables.
Related
This is my first ever post to stackoverflow. It ought to be straightforward, but I have this problem whenever I try to write a makefile and I've never been able to figure out a satisfactory solution. Apologies if there is already a solution somewhere on the site. I couldn't find one.
What I'm trying to do is as follows:
Search my src directory for matching source files.
Compile the src code into a sandbox.
Here's my directory structure:
makefile
src1
file1.src
file2.src
src2
file3.src
subfolder
src3
file4.src
file5.src
And here's my makefile:
BUILDDIR := ./sandbox
SRC_DIRS := ./
SRCS := $(shell find $(SRC_DIRS) -name *.src)
OBJS := $(addprefix $(BUILDDIR)/o., $(notdir $(SRCS) ) )
# make print-X prints value of variable X
print-%: ; #echo $* = $($*)
.PHONY: help
help:
#echo "make <all|clean>"
.PHONY: all
all: $(OBJS)
#echo "compilation done"
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
$(BUILDDIR)/.create:
#echo "creating sandbox"
mkdir -p $(BUILDDIR) && cd $(BUILDDIR)
#touch $#
.PHONY: clean
clean:
#echo "deleting sandbox"
#rm -rf $(BUILDDIR)
If I type make all, the file works as expected. However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
And the process of creating objects of objects continues recursively as many times as I type make.
Any help would be appreciated.
Incidentally, please don't post solutions that rely on the build in compile functions of make. I'm looking for a general solution that can be used for any task. For example, in this instance, I'm trying to read the source files into a tool using its command line interface.
Well, first, by having your sandbox as a subdirectory of your source directory, then using find on the source directory, every time you run it you're going to find all the files in both the source directory and all its subdirectories, including the sandbox. If the built files in the sandbox have the same names as the files in the source directories, the find will find them all.
Maybe instead of:
SRCS := $(shell find $(SRC_DIRS) -name *.src)
you want something like:
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -o -name *.src -print)
Or, alternatively, don't make your sandbox a subdirectory of your source directory. Or, make sure that whatever name you give to the files in the sandbox directory won't match the *.src pattern you give to find.
But beyond that this is wrong:
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
Suppose SRCS is foo.src bar.src, which means OBJS is sandbox/o.foo.src sandbox/o.bar.src. Then the above expands to this:
sandbox/o.foo.src sandbox/o.bar.src : foo.src bar.src sandbox/.create
This is a common mistake; people seem to think that make will go through the targets and prerequisites and match them up, so the first target depends on the first prerequisite and the second target depends on the second one etc. but of course this cannot work correctly and that's not how make works. Make treats the above as if you'd written one rule for each target, with the same prerequisites; like this:
sandbox/o.foo.src : foo.src bar.src sandbox/.create
sandbox/o.bar.src : foo.src bar.src sandbox/.create
You can see this won't do what you want at all, since the $< will always be foo.src which is clearly wrong.
You need to write a single pattern rule that will build ONE target. Then make sure the pattern applies to all the targets.
You have made things hard for yourself by trying to "flatten" a directory structure of multiple source subdirectories, into a single level of target directory (by using the $(notdir $(SRCS))). Because of this, there's no pattern that will match the same target and directory, unless you write a separate rule for every subdirectory.
Luckily there is a solution for this: VPATH. This should work for you:
VPATH := $(sort $(dir $(SRCS))
$(BUILDDIR)/o.%.src : %.src $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
The VPATH tells make to go look in all the directories that it found any sources in, whenever it can't find one to build.
The basic problem is that your SRCS is all files in all subdirectories that match the pattern *.src (when you run make). That means that all your object files ($(OBJS)) also match, so they copied as well.
The solution is to change your SRCS pattern so it does not match the "object" files in the build directory. Possibilities:
SRCS := $(wildcard *.src)
or
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -false -o -name *.src)
or change the names of your "object" files so they don't end in .src
If I type make all, the file works as expected.
By which I take you to mean that directory ./sandbox exists and contains these files:
o.file1.src
o.file2.src
o.file3.src
o.file4.src
o.file5.src
However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
Of course you do, because everything is not up to date at that point, according to the target list you create. This line ...
SRCS := $(shell find $(SRC_DIRS) -name *.src)
... defines SRCS as a list of all the files under path ./ (which is the value of SRC_DIRS) that ends in .src. That includes any such files in ./sandbox, which include all the files placed there by the first make run. You generate a corresponding target file to build for each source, and those targets corresponding to sources built by the previous make run will not, in general, exist yet. So make builds them, just as you instructed it.
The best solution, short of abandoning that automatic source identification altogether, would probably be to change your naming scheme so that outputs cannot be mistaken so easily for sources. For example, if you want the output names to have the form foo.src, then have the corresponding input named something like foo.src.in. In that particular case, you could convert from source names to target names with
OBJS := $(addprefix $(BUILDDIR)/o., $(basename $(notdir $(SRCS) ) ) )
Alternatively, you could modify the find command to skip the sandbox directory, maybe by modifying SRC_DIRS:
SRC_DIRS = src1 src2 subfolder/src3
(These specific alternatives are not mutually exclusive.)
I need a Makefile that allows me to enter make foo-program and, if any foo-program/**/*.hs file has changed since last build, build the target (output in foo-program/.stack-work).
Here is my directory tree:
project/
|-bar-other-program/
|-.stack-work/ # Generated output goes here
|-src/
|-BarOtherProgram.hs
|-test/
|-Tests.hs
|-foo-program/
|-.stack-work/ # Generated output goes here
|-src/
|-FooProgram.hs
|-test/
|-Tests.hs
|-notes/ # non-source, so no Make target for this
Here is what I have so far:
# evaluates to 'bar-other-program foo-program'
PROGS := $(shell find * -type f -name '*.hs' | cut -d'/' -f1 | uniq)
.SECONDEXPANSION:
$(PROGS): $$(wildcard $$#/src/*.hs) $$(wildcard $$#/test/*.hs)
# do-build $#
When I run make foo-program, whether the source has changed or not, I get:
make: Nothing to be done for 'foo-program'
UPDATE: My final (non-abstracted) Makefile can be found on GitHub. Note that my solution took a different turn than I intended when I wrote up this question. Looking at that Makefile also might also make it more clear as to my original goal.
I am not quite sure of the the purpose of cut -d'/' there.
But if you just want a list of *.hs files in the current directory (recursively found) and then build a target/executable based on whether they have changed, you can do something like this:
PROGS = $(subst ./,,$(shell find . -type f -name '*.hs'))
DEPS = $(addprefix stackwork/,$(addsuffix .dep,$(basename $(PROGS))))
DIRS = $(dir $(DEPS))
.PHONY: foo-program
foo-program: $(DEPS) $(DIRS)
stackwork/%.dep: %.hs | $(DIRS)
#echo making $#
#touch $#
$(DIRS):
#echo creating dir $#
#mkdir -p $#
clean:
#rm -rf $(DEPS) $(DIRS)
Where:
PROGS is your list of .hs files
DEPS is a list of generated dependency files (empty but date stamps will be used)
DIRS is a list of output directories that need to be created (I guess they don't exist by default since they are output folders?)
foo-program is a rule that you can call (PHONY because at the moment it does not create a real file)
%.dep: %.hs is a rule how to generate a .dep file (this could be a .o .obj or any other file type) which depends on its .hs file equivalent.
$(DIRS): is a rule to create your output directories if needed.
So if the .dep files don't exist, all of the .hs files will be "compiled". If all the .dep files exist and are up to date, then nothing will be compiled. If one or more file is out of date then just those files will be built. Here is the output of running this on my PC with a few test files:
admin#osboxes:~/sandbox$ make
creating dir stackwork/
creating dir stackwork/test/
creating dir stackwork/test/test2/
making stackwork/file.dep
making stackwork/test/file.dep
making stackwork/test/test2/file2.dep
admin#osboxes:~/sandbox$ make
make: Nothing to be done for 'foo-program'.
admin#osboxes:~/sandbox$ touch test/file.hs
admin#osboxes:~/sandbox$ make
making stackwork/test/file.dep
admin#osboxes:~/sandbox$ make
make: Nothing to be done for 'foo-program'.
I am writing a Makefile for a project with a large number of js files in a complex directory structure. When run it needs to perform some compilation on each file and save the results to a different directory with the same tree structure (right now that's simulated by a cp). When I run make js it builds as it should and when I run make js again it says that there is no work to do. However when I modify one of the files and make js it re-builds the entire tree instead of just the modified file.
SHELL := /bin/bash
BUILDDIR := build/gui/
RAWJSFILES := $(shell find app -name '*.js')
BUILT_RAWJSFILES := $(patsubst %, $(BUILDDIR)%,$(RAWJSFILES))
$(BUILDDIR):
mkdir -p $(BUILDDIR)
$(RAWJSFILES): $(BUILDDIR)
$(BUILT_RAWJSFILES): $(RAWJSFILES)
mkdir -p $(#D)
# compile step
cp $(shell python -c "print '$#'.lstrip('${BUILDDIR}')") $(#D)
.PHONY: js
js: $(BUILT_RAWJSFILES)
The line $(BUILT_RAWJSFILES): $(RAWJSFILES) is setting the prerequisites of each file in $(BUILT_RAWJSFILES) to all the files in $(RAWJSFILES).
To get the one-to-one mapping you want you need a pattern rule or a static pattern rule.
Also that embedded python snippet isn't at all doing what you intended for it to do there. It is removing the longest leading prefix from the value of $# that contains any of the characters in BUILDDIR namely build/gui/ (only not as a string as characters so that's equivalent to lstrip('/iugdlb')). (Oh, also you don't need $(shell) here you are already in a shell context so a normal command substitution would work just as well ($$(python ...).)
That being said if you use a pattern or static pattern rule you'll have easier ways to get the source filename then messing with python like that.
Something like this should work (untested):
$(BUILDDIR)%.js: %.js
mkdir -p $(#D)
cp $^ $(#D)
the makefile below processes files matching the patterncontent/%.md and outputs the targets in the html directory. Source files are named with the convention of putting a leading number in front of them, like content/01.index.md or content/O2.second-page.md and so on. I would like to remove the leading 0x. number sequence in the target file. For instance, content/01.index.html would generate html/index.html.
How can I do this?
Thanks
MD_FILES = $(shell find content/ -type f -name '*.md')
HTML_FILES = $(patsubst content/%.md, html/%.html, $(MD_FILES))
all: $(HTML_FILES) html/static
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
.PHONY: html/static
html/static :
rsync -rupE generator/static html/
.PHONY: clean
clean:
rm -fr html
Replace:
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
with:
html/%.html : content/%.md
mkdir -p $(#D)
file='$(#F)'; python generator/generate.py $< "$(#D)/${file#*.}"
Unfortunately, I can't think of a good way of doing that in make itself. I can think of one way but it isn't as simple as that escaping and it isn't safe for files with spaces (not that that matters much here since make already can't handle those).
IMHO, it is a bad idea to use find or wildcards to list files in makefiles. This is because developers have temporary or debugging files sometimes. It is best to list files explicitly. This way, it forces the developer to think about their intent.
If you agree to list files explicitly, then in this case it is best to list the target files, rather than source files, and here is your answer:
HTML_FILES := html/index.html html/second-page.html
.SECONDEXPANSION:
$(HTML_FILES): html/%.html : $$(wildcard content/*.$$*.md)
(put recipe here, using $# and $<)
I'm trying to create a target which matches multiple directories, but no files. Is there any way to flag the target explicitly as being a directory?
For example:
output/%:
mkdir -p $#
matches files as well, which is definitely not something I want.
Basically, I have many variables which contain directory names which need to be created if they don't exist, and the best way to do that, as far as I've found is this:
all: directories other_stuff
directories: $(dir1) $(dir2) ...
$(dir1):
mkdir -p $#
$(dir2):
mkdir -p $#
...
So, I'm OK with repeating the directory variables as dependencies for the directories target, but creating targets for each directory I add seems overkill and hard to maintain.
Here is one possibility:
DIR_LIST = $(dir1) $(dir2) $(dir3)
.PHONY: all
all: directories other_stuff
.PHONY: directories # Note: you should be marking this as phony
directories: $(DIR_LIST)
$(DIR_LIST):
mkdir -p $#
See http://www.gnu.org/software/make/manual/make.html#Multiple-Targets.