I'm trying to write a rule that would replace a set of .c files with .o files for a variable.
To make sure it works, I'm echoing their values.
the .c files are obtained from the current working directory via a shell wildcard
sources := *.c
However, nothing gets printed out for the .o files, as 'sources' doesn't seem to get expanded for this, whether I do with using patsubst or the handier $(src:%.c=%.o).
On the other hand, if I set the value of 'manually', everything works as expected, e.g.
sources := source1.c source2.c source3.c
The makefile:
sources := *.c
srcs = one.c two.c
objects := $(sources:.c=.o)
objs := $(patsubst %.c, %.o, $(srcs))
prints:
#echo sources
#echo $(sources)
#echo objects
#echo $(objects)
#echo $(objs)
Output:
sources
source1.c source2.c source3.c source4.c
objects
*.o
one.o two.o
As you can see, 'objects' (based on the wildcard-using 'sources') is a no-go, but 'objs' (based on the manally-set 'srcs' works as expected).
What am I missing. I know it must have to do with when or how the expansion takes place.
Make is not the shell. This line:
sources := *.c
sets the make variable sources to the literal string *.c. Then this line:
objects := $(sources:.c=.o)
sets objects to the literal string *.o.
Then this line:
#echo $(sources)
Sends echo *.c to the shell, and the SHELL expands the *.c to the list of files that match that glob.
Then this line:
#echo $(objects)
sends echo *.o to the shell, and since there are no files that match the *.o glob, it just prints the glob itself *.o.
If you want to get a list of files in make you have to use the wildcard function:
sources := $(wildcard *.c)
Also, you have fallen prey to two different common errors people make when they are trying to debug makefiles:
First, you should never, EVER use # in your recipe lines to hide the commands. This is like trying to debug your makefile with a blindfold on. Seeing what make is sending to the shell gives you absolutely vital information as to what is happening here. I personally don't use # even after my makefile is working, except for some trivial operations, and I definitely never use it while trying to debug.
Second, if you want to see the contents of make variables in recipes you should always use single quotes around them. If you leave them unquoted or use double quotes, then the shell will come along and mess with those values (depending on what the values are) and you won't be able to see what they really are. Either use '' around them or else use a make function like $(info ...) to see their value rather than a shell program like echo.
If you had written your test makefile like this:
prints:
echo sources
echo '$(sources)'
echo objects
echo '$(objects)'
echo '$(objs)'
it would have been more immediately obvious what the problem was.
Related
This is my first ever post to stackoverflow. It ought to be straightforward, but I have this problem whenever I try to write a makefile and I've never been able to figure out a satisfactory solution. Apologies if there is already a solution somewhere on the site. I couldn't find one.
What I'm trying to do is as follows:
Search my src directory for matching source files.
Compile the src code into a sandbox.
Here's my directory structure:
makefile
src1
file1.src
file2.src
src2
file3.src
subfolder
src3
file4.src
file5.src
And here's my makefile:
BUILDDIR := ./sandbox
SRC_DIRS := ./
SRCS := $(shell find $(SRC_DIRS) -name *.src)
OBJS := $(addprefix $(BUILDDIR)/o., $(notdir $(SRCS) ) )
# make print-X prints value of variable X
print-%: ; #echo $* = $($*)
.PHONY: help
help:
#echo "make <all|clean>"
.PHONY: all
all: $(OBJS)
#echo "compilation done"
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
$(BUILDDIR)/.create:
#echo "creating sandbox"
mkdir -p $(BUILDDIR) && cd $(BUILDDIR)
#touch $#
.PHONY: clean
clean:
#echo "deleting sandbox"
#rm -rf $(BUILDDIR)
If I type make all, the file works as expected. However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
And the process of creating objects of objects continues recursively as many times as I type make.
Any help would be appreciated.
Incidentally, please don't post solutions that rely on the build in compile functions of make. I'm looking for a general solution that can be used for any task. For example, in this instance, I'm trying to read the source files into a tool using its command line interface.
Well, first, by having your sandbox as a subdirectory of your source directory, then using find on the source directory, every time you run it you're going to find all the files in both the source directory and all its subdirectories, including the sandbox. If the built files in the sandbox have the same names as the files in the source directories, the find will find them all.
Maybe instead of:
SRCS := $(shell find $(SRC_DIRS) -name *.src)
you want something like:
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -o -name *.src -print)
Or, alternatively, don't make your sandbox a subdirectory of your source directory. Or, make sure that whatever name you give to the files in the sandbox directory won't match the *.src pattern you give to find.
But beyond that this is wrong:
$(OBJS) : $(SRCS) $(BUILDDIR)/.create
Suppose SRCS is foo.src bar.src, which means OBJS is sandbox/o.foo.src sandbox/o.bar.src. Then the above expands to this:
sandbox/o.foo.src sandbox/o.bar.src : foo.src bar.src sandbox/.create
This is a common mistake; people seem to think that make will go through the targets and prerequisites and match them up, so the first target depends on the first prerequisite and the second target depends on the second one etc. but of course this cannot work correctly and that's not how make works. Make treats the above as if you'd written one rule for each target, with the same prerequisites; like this:
sandbox/o.foo.src : foo.src bar.src sandbox/.create
sandbox/o.bar.src : foo.src bar.src sandbox/.create
You can see this won't do what you want at all, since the $< will always be foo.src which is clearly wrong.
You need to write a single pattern rule that will build ONE target. Then make sure the pattern applies to all the targets.
You have made things hard for yourself by trying to "flatten" a directory structure of multiple source subdirectories, into a single level of target directory (by using the $(notdir $(SRCS))). Because of this, there's no pattern that will match the same target and directory, unless you write a separate rule for every subdirectory.
Luckily there is a solution for this: VPATH. This should work for you:
VPATH := $(sort $(dir $(SRCS))
$(BUILDDIR)/o.%.src : %.src $(BUILDDIR)/.create
#echo "\"compiling\" $< to produce $#"
cp $< $#
The VPATH tells make to go look in all the directories that it found any sources in, whenever it can't find one to build.
The basic problem is that your SRCS is all files in all subdirectories that match the pattern *.src (when you run make). That means that all your object files ($(OBJS)) also match, so they copied as well.
The solution is to change your SRCS pattern so it does not match the "object" files in the build directory. Possibilities:
SRCS := $(wildcard *.src)
or
SRCS := $(shell find $(SRC_DIRS) -name $(notdir $(BUILDDIR)) -prune -false -o -name *.src)
or change the names of your "object" files so they don't end in .src
If I type make all, the file works as expected.
By which I take you to mean that directory ./sandbox exists and contains these files:
o.file1.src
o.file2.src
o.file3.src
o.file4.src
o.file5.src
However, if I type make all again, instead of saying everything is up to data, I end up with the following contents in the sandbox:
o.file1.src o.file2.src o.file3.src o.file4.src o.file5.src o.o.file1.src o.o.file2.src o.o.file3.src o.o.file4.src o.o.file5.src
Of course you do, because everything is not up to date at that point, according to the target list you create. This line ...
SRCS := $(shell find $(SRC_DIRS) -name *.src)
... defines SRCS as a list of all the files under path ./ (which is the value of SRC_DIRS) that ends in .src. That includes any such files in ./sandbox, which include all the files placed there by the first make run. You generate a corresponding target file to build for each source, and those targets corresponding to sources built by the previous make run will not, in general, exist yet. So make builds them, just as you instructed it.
The best solution, short of abandoning that automatic source identification altogether, would probably be to change your naming scheme so that outputs cannot be mistaken so easily for sources. For example, if you want the output names to have the form foo.src, then have the corresponding input named something like foo.src.in. In that particular case, you could convert from source names to target names with
OBJS := $(addprefix $(BUILDDIR)/o., $(basename $(notdir $(SRCS) ) ) )
Alternatively, you could modify the find command to skip the sandbox directory, maybe by modifying SRC_DIRS:
SRC_DIRS = src1 src2 subfolder/src3
(These specific alternatives are not mutually exclusive.)
I am trying to statically build HTML files that requires a markdown file and a meta file called "whatlinkshere" for the HTML file to demonstrate its back links.
I believe it can be effeciently done by a Makefile, by first generating all the "whatlinkshere" files. I don't think this can be done in parallel, because the program that generates these files needs to append to the whatlinkshere files, and there could be race conditions that I am not quite sure how to solve.
Once the "whatlinkshere" files are generated then if a markdown file is edited, say foo.mdwn to point to bar.mdwn, only foo.mdwn needs to be analysed again for "whatlinkshere" changes. And finally only foo.html and bar.html need to be rebuilt.
I am struggling to accomplish this in my backlinks project.
INFILES = $(shell find . -name "*.mdwn")
OUTFILES = $(INFILES:.mdwn=.html)
LINKFILES = $(INFILES:.mdwn=.whatlinkshere)
all: $(OUTFILES)
# These need to be all made before the HTML is processed
$(LINKFILES): $(INFILES)
#echo Creating backlinks $#
#touch $#
#go run backlinks.go $<
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
Current problems here is that *.whatlinkshere** aren't being generated on first run. My workaround is for i in *.mdwn; do go run backlinks.go $i; done. Furthermore there are not rebuilding as I want after editing a file as described earlier. Something is horribly wrong. What am I missing?
I think I finally understood your problem. If I understood well:
You have a bunch of *.mdwn source files.
You generate *.whatlinkshere files from your *.mdwn source files using the backlinks.go utility. But this utility does not produce foo.whatlinkshere from foo.mdwn. It analyzes foo.mdwn, searches for links to other pages in it and, for each link to bar it finds, it appends a [foo](foo.html) reference to bar.whatlinkshere.
From each foo.mdwn source file you want to build a corresponding foo.html file with:
$ cmark foo.mdwn foo.whatlinkshere
Your rule:
$(LINKFILES): $(INFILES)
#echo Creating backlinks $#
#touch $#
#go run backlinks.go $<
contains one error and has several drawbacks. The error is the use of the $< automatic variable in the recipe. It expands as the first prerequisite, that is probably always pageA.mdwn in your case. Not what you want. $^ expands as all prerequisites but it is not the correct solution because:
your go utility takes only one source file name, but even if it was accepting several...
...make will run the recipe several times, one per link file, which is a waste, and...
...as your go utility appends to the link files it will even be worse than a waste: back links will be counted several times each, and...
...if make runs in parallel mode (note that you can prevent this with make -j1 or by adding the .NOTPARALLEL: special rule to your Makefile, but it is a pity) there is a risk of race conditions.
Important: the following works only with a flat organization where all source files and HTML files are in the same directory as the Makefile. Other organizations are possible, of course, but they would require some modifications.
First option using multi-targets pattern rules
One possibility is to use a special property of make pattern rules: when they have several targets make considers that one single execution of the recipe produces all targets. For instance:
pageA.w%e pageB.w%e pageC.w%e: pageA.mdwn pageB.mdwn pageC.mdwn
for m in $^; do go run backlinks.go $$m; done
tells make that pageA.whatlinkshere, pageB.whatlinkshere and pageC.whatlinkshere are all generated by one execution of:
for m in pageA.mdwn pageB.mdwn pageC.mdwn; do go run backlinks.go $m; done
(make expands $^ as all prerequisites and $$m as $m). Of course, we want to automate the computation of the pageA.w%e pageB.w%e pageC.w%e pattern targets list. This should make it:
INFILES := $(shell find . -name "*.mdwn")
OUTFILES := $(INFILES:.mdwn=.html)
LINKFILES := $(INFILES:.mdwn=.whatlinkshere)
LINKPATTERN := $(INFILES:.mdwn=.w%e)
.PHONY: all clean
.PRECIOUS: $(LINKFILES)
all: $(OUTFILES)
# These need to be all made before the HTML is processed
$(LINKPATTERN): $(INFILES)
#echo Creating backlinks
#rm -f $(LINKFILES)
#touch $(LINKFILES)
#for m in $^; do go run backlinks.go $$m; done
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
clean:
rm -f $(LINKFILES) $(OUTFILES)
Notes:
I declared all and clean as phony because... it is what they are.
I declared the whatlinkshere files as precious because (some of them) are considered by make as intermediates and without this declaration make would delete them after building the HTML files.
In the recipe for the whatlinkshere files I added rm -f $(LINKFILES) such that, if the recipe is executed, we restart from a clean state instead of concatenating new stuff to old (possibly outdated) references.
The pattern stem in the $(LINKPATTERN) can be anything but must match at least one character. I used w%e but whatlin%shere would work too. Use whatever is specific enough in your case. If you have a pageB.where file prefer whatlin%shere or what%here.
There is a drawback with this solution but it is due to your particular set-up: each time one single mdwn file changes it must be re-analyzed (which is normal) but any whatlinkshere file can be impacted. This is not predictable, it depends on the links that have been modified in this source file. But more problematic is the fact that the result of this analysis is appended to the impacted whatlinkshere files. They are not "edited" with the old content relative to this source file replaced by the new one. So, if you change just a comment in a source file, all its links will be appended again to the respective whatlinkshere files (while they are already there). This is probably not what you want.
This is why the solution above deletes all whatlinkshere files and re-analyzes all source files each time one single source file changes. And another negative consequence is that all HTML files must also be re-generated because all whatlinkshere files changed (even if their content did not really change, but make does not know this). If the analysis is super fast and you have a small number of mdwn files, it should be OK. Else it is sub-optimal but not easy to solve because of your particular set-up.
Second option using recursive make, separated back link files and marker files
There is a possibility, however, which consists in:
separating all back links references with one whatlinkshere file per from/to pair: foo.backlinks/bar.whatlinkshere contains all references to bar found in foo.mdwn,
using recursive make with one first invocation (when the STEP make variable is unset) to update all whatlinkshere files that need to be and a second invocation (STEP set to 2) to generate the HTML files that need to be,
using empty dummy files to mark that a foo.mdwn file has been analyzed: foo.backlinks/.done,
using the secondary expansion to be able to refer to the stem of a pattern rule in its list of prerequisites (and using $$ to escape the fist expansion).
But it is probably a bit more difficult to understand (and maintain).
INFILES := $(shell find . -name "*.mdwn")
OUTFILES := $(INFILES:.mdwn=.html)
DONEFILES := $(patsubst %.mdwn,%.backlinks/.done,$(INFILES))
.PHONY: all clean
ifeq ($(STEP),)
all $(OUTFILES): $(DONEFILES)
$(MAKE) STEP=2 $#
%.backlinks/.done: %.mdwn
rm -rf $(dir $#)
mkdir -p $(dir $#)
cp $< $(dir $#)
cd $(dir $#); go run ../backlinks.go $<; rm $<
touch $#
else
all: $(OUTFILES)
.SECONDEXPANSION:
%.html: %.mdwn $$(wildcard *.backlinks/$$*.whatlinkshere)
#echo Deps $^
#cmark $^ > $#
endif
clean:
rm -rf *.backlinks $(OUTFILES)
Even if it looks more complicated there are a few advantages with this version:
only outdated targets are rebuilt and only once each,
all whatlinkshere files are updated (if needed) before any HTML file is updated (if needed),
the whatlinkshere files can be built in parallel,
the HTML files can be built in parallel.
Third option using only recursive make and marker files
If you do not care about inaccurate results where back links persist in the results after they disappeared from the source files or where back links are uselessly replicated, we can reuse ideas from the previous solution but drop the separation in individual from/to whatlinkshere files.
INFILES := $(wildcard *.mdwn)
OUTFILES := $(patsubst %.mdwn,%.html,$(INFILES))
LINKFILES := $(patsubst %.mdwn,%.whatlinkshere,$(INFILES))
DONEFILES := $(patsubst %.mdwn,.%.done,$(INFILES))
.PHONY: all clean
.PRECIOUS: $(LINKFILES)
ifeq ($(STEP),)
.NOTPARALLEL:
all $(OUTFILES): $(DONEFILES)
$(MAKE) STEP=2 $#
.%.done: %.mdwn
go run backlinks.go $<
touch $#
else
all: $(OUTFILES)
%.html: %.mdwn %.whatlinkshere
#echo Deps $^
#cmark $^ > $#
%.whatlinkshere:
touch $#
endif
clean:
rm -f $(OUTFILES) $(LINKFILES) $(DONEFILES)
Notes:
As this works only for a flat organization I replaced the $(shell find...) by the make built-in $(wildcard ...).
I used patsubst instead of the old syntax but it's just a matter of taste.
The %.whatlinkshere: rule is a default rule to create the missing empty whatlinkshere files.
The NOTPARALLEL: special target prevents parallel execution when building the whatlinkshere files.
I'm having a problem with conditional statements in make. Existing questions on SO only work on the highest level in the Makefile.
Here's what I have:
.PHONY: all
all: bar/*.o
bar/%.o: foo/%.cc
ifeq (,$(wildcard $(#D)))
#echo $(wildcard $(#D))
endif
There are two empty files under foo, a.cc and b.cc. The folder bar exists.
Here's the output:
PS C:\Users\cbrown2\Src> mingw32-make
bar
bar
It enters the ifeq statement, even though $(wildcard $(#D)) gives bar.
What gives?
You cannot use conditionals this way. When make evaluates your conditional automatic variables don't have a value yet. So make will see:
bar/%.o: foo/%.cc
#echo $(wildcard $(#D))
because $(#D) expands as the empty string. And when make will pass the recipe to the shell it will first expand $(#D) which, this time, has a value: bar.
One side note that has nothing to do with your problem: when bar does not contain all object files yet,
all: bar/*.o
will not do what you probably want. What you probably want is:
SRC := $(wildcard foo/*.cc)
OBJ := $(patsubst foo/%.cc,bar/%.o,$(SRC))
all: $(OBJ)
Is there a way to pick up the target name using automatic variable.
SOURCES = $(wildcard *.c)
dummytgt: $(OBJ)/tier.o
$(GCC) $(CFLAGS) -c $(filter $#,$(SOURCES)).c -o $#
I do not want to mention the filename as input but would want to use the filter function to get the .c file which is same as target name. make throws an error no input files
It's helpful to have a look at how make parses this:
SOURCES = $(wildcard *.c)
dummytgt: $(OBJ)/tier.o
$(GCC) $(CFLAGS) -c $(filter $#,$(SOURCES)).c -o $#
First off,
it will read the makefile, defining and expanding macros as it goes.
SOURCES = $(wildcard *.c) means that ${SOURCES} is a lazy variable with definition $(wildcard *.c).
Lazy? Yes, these recursive variables (as the make manual has it) only expand their right-hand side when they are themselves expanded.
Make needs the dependencies as it reads the file, so $(OBJ) is expanded.
Let's assume that the expansion of ${OBJ} is objs (say).
The shell command block remains as a single lazy variable.
It's important to note that these are not expanded until make decides that it wants to build dmmytgt.
You could have written this to exactly the same effect:
dummytgt: objs/tier.o
$(GCC) $(CFLAGS) -c $(filter $#,$(wildcard *.c)).c -o $#
For this fragment to work,
the file objs/tier.o must already exist.
Let's assume it does.
Make now has all it needs to build dummytgt (according to your makefile),
so now it expands the command block.
$(wildcard *.c) expands to 1.c 2.c (say).
$# is dummytgt
$(filter dummytgt,1.c 2.c) is of course empty (and always will be!)
${GCC} is gcc (say)
${CFLAGS} is empty (say)
Thus the shell gets
gcc -c .c -o dummytgt
Presumably gcc complains that there is no file called .c.
The resulting error stops make's execution.
A few thing not to like here:
$(wildcard ) is only for hacky one-liner makefiles IMHO.
dummytgt requires objs/tier.o, but its build instructions never reference it.
Your $(filter ) always expands to nothing.
$(filter $#.c,$(SOURCES))
But I don't see why you don't use
$#.c
Or better still, make it a prerequisite.
Goog evening,
I'm completely new to makefiles and worked out a file which fits our needs good but I'm not completely satisfied. We use bootstrap3 and have around 40 customers with differend color settings. That's why we need to compile 40 slightly different css files. Until now, we have the following file structure
less/customer1.less
css/customer1.css
color/customer.less contains bootstraps variables file
customer1.less contains
#variables: 'myCompany/color/customer1'; //this is forwarded to where bootrstrap loads the variables template
#import "bootstrap";
#import 'myCompany/modifications';
Our makefile
SOURCES = $(shell ls less/*.less)
# Files we don't want to be build
SOURCES := $(filter-out less/bootstrap.less, $(SOURCES))
SOURCES := $(filter-out less/a11y.less, $(SOURCES))
TARGETS = $(patsubst less/%.less,css/%.css,$(SOURCES))
DEPEND = $(patsubst less/%.less,make/%.d,$(SOURCES))
css/%.css: less/%.less
# First building dependency files
lessc -M $< $# > 'make/$*.d'
# Then building CSS and sourcemap
lessc -s $< > $# --source-map=map/$*.css.map --source-map-basepath=map --clean-css
-include $(DEPEND)
all: $(TARGETS)
Call
$ make all
Creates Makefiles in make/, CSS in css/ CSS source-maps in map/ and expects LESS being in less/.
This works but we need to create customerX.less for each customer manually even if the only difference is the assigned color scheme/variables file.
Make should look in the color folder if there is a file for this customerX and then create (but not overwrite!) customerX.less in less directory.
Any make guru out here know how to do this with make?
I believe you can do what you want here with an order-only prerequisite.
Something like:
less/customer%.less: | color/customer.less
[ -f '$#' ] || cp $^ $#
I don't think the -f test is strictly necessary there but it shouldn't hurt and is safer.
On a different topic $(shell ls less/*.less) can probably be done better with either $(shell echo less/*.less) (you don't care about what ls does you just want the shell glob expansion) or $(wildcard less/*.less). (Technically shelling out and wildcard are slightly different but I don't know that that will matter for you here.)
Also note that the all target will not create these missing less files for you (as SOURCES will not contain them as the file didn't exist) but make css/customer#.css will create them if necessary.