I'm trying to use a makefile to convert some markdown files to html files. I'm trying to accomplish in a few lines that which I previously had a long python script doing.
In the simple example below, I would like to see this code:
build: $(patsubst src/pages/%.md, output/%.html, $(wildcard src/pages/*.md))
%.html: %.md
#echo $< to $#
and this output
src/pages/index.md to output/index.html
src/pages/about.md to output/about.html
src/pages/contact.md to output/contact.html
src/page/foo/bar.md to output/foo/bar.html
Instead, it says:
$ make build
make: *** No rule to make target 'output/index.html', needed by 'build'. Stop.
I'm missing something very basic here.
Consider the target output/index.html. The dependency...
%.html: %.md
will effectively expand to...
output/index.html: output/index.md
with $* equal to output/index. So make looks for output/index.md but can't find it -- hence the error message.
To get the correct pattern stem ($* == index) you need to add the base directories to the pattern rule...
output/%.html: src/pages/%.md
#echo $< to $#
EDIT 1:
If you're concerned about repeated hard-coded strings such as output and src/pages then you can always assign them to parameters...
OUTPUT_DIR := output
SOURCE_DIR := src/pages
build: $(patsubst $(SOURCE_DIR)/%.md, $(OUTPUT_DIR)/%.html, $(wildcard $(SOURCE_DIR)/*.md))
$(OUTPUT_DIR)/%.html: $(SOURCE_DIR)/%.md
#echo $< to $#
(Assuming that's what you meant by `optimization' in your comment.)
Related
I have a series of directories organized like this:
foo/
foo.file1 foo.file2
bar/
bar.file1 bar.file2
baz/
baz.file1 baz.file2
Right now I'm processing these files using a script that does all the checking for file existence etc but I thought that perhaps I could use a Makefile for it (since said script is very fragile), to avoid reprocessing files that did not change.
The problem is that each directory is independent, and I'd need to do, for example:
foo.file1.processed: foo.file1
run_random_program foo.file1 -o foo.file1.processed
for each of the 71 directories that are in total in that path. This looks like being extremely tedious and I wonder if there's something that would prevent me from writing all of this by hand.
Is such a thing possible?
EDIT: Some examples that show what I have in mind, had I a single Makefile for each directory:
file1.cds.callable: file1.callable
long_script_name -i $< -o $#
file1.rds: file1.cds.callable
another_long_script_name $< additional_file_in_folder $#
file1.csv: file1.rds
yet_another_script $< $#
Seems like pattern rules are exactly what you need:
# These are the original source files (based on the example)
CALLABLE := $(wildcard */*.callable)
# These are the final targets
TARGETS := $(CALLABLE:%.callable=%.csv)
all: $(TARGETS)
%.csv : %.rds
yet_another_script $< $#
%.rds: %.cds.callable
another_long_script_name $< additional_file_in_folder $#
%.cds.callable: %.callable
long_script_name -i $< -o $#
I am looking for Makefile macro to get list of all files in a directory that were generated as rule1 processing and using this list for rule2 processing.
Here's what I am trying to achieve :
Rule 1: Generate source .c files (using xml files) and place them in $(MYDIR) directory.
Rule 2: Get the list of all files in $(MYDIR) and create object files and place them in $(OBJDIR).
Problem is, I want to update list of files in Rule2 after Rule 1 has been processed, else list of files in $(MYDIR) will be empty.
all : rule_1 rule_2
rule1 : $(MYDIR)/generated_source1.c $(MYDIR)/generated_source2.c
$(MYDIR)/generated_source1.c:
xsltproc generator1.xml style_generator.xsl -o $(MYDIR)/generated_source_1.c
$(MYDIR)/generated_source2.c:
xsltproc generator2.xml style_generator.xsl -o $(MYDIR)generated_source_2.c
#Get list of all $(MYDIR).*c , create corresponding $(OBJDIR)/*.o list.
SOURCES := $(wildcard $(MYDIR)/*.c)
OBJECTS := $(notdir ${SOURCES})
GENERATED_OBJS := $(patsubst %.c,$(OBJDIR)/%.o,$(OBJECTS))
#This rule is compiling of all .c generated in rule1.
rule2 : $(GENERATED_OBJS)
ld -r -o $(OBJDIR)/generated_lib.o $(GENERATED_OBJS)
$(OBJDIR)/%.o: $(MYDIR)/%.c
gcc $(CFLAGS) -c -o $# $<
$(SOURCES) is shown empty, but actually it should contain generated_source1.c and generated_source2.c
I am not sure how .SECONDEXPANSION rule will work for my case.
You can't really (and don't really want to) play around with getting make to re-evaluate file existence during the running of the make process.
What you want to do is track the files from start to finish in make and then you have all your lists.
You can start at either direction but starting with the initial source tends to be easier.
So start with
MYDIR:=dir
OBJDIR:=obj
XML_SOURCES := $(wildcard $(MYDIR)/*.xml)
then translate from there to the generated source files
SOURCES := $(subst generator,generated_source,$(XML_SOURCES:.xml=.c))
and from there to the generated object files
GENERATED_OBJS := $(patsubst $(MYDIR)/%.c,$(OBJDIR)/%.o,$(SOURCES))
At which point you can define the default target
all: $(OBJDIR)/generated_lib.o
and then define the rules for each step
$(MYDIR)/%.c:
cat $^ > $#
$(OBJDIR)/%.o: $(MYDIR)/%.c
cat $^ > $#
$(OBJDIR)/generated_lib.o: $(GENERATED_OBJS)
ld -r -o $# $^
The $(MYDIR)/%.c rule needs a bit of extra magic to actually work correctly. You need to define the specific input/output pairs so that they are used correctly by that rule.
$(foreach xml,$(XML_SOURCES),$(eval $(subst generator,generated_source,$(xml:.xml=.c)): $(xml)))
This .xml to .c step would be easier if the input and output files shared a basename as you could then just use this and be done.
%.c: %.xml
cat $^ > $#
I am VERY new to makefiles. I have discovered a flaw in a make file that causes files in a list to be copied from a single source file instead of each file in the list.
First, there is a sub model variable SUB_MODEL_LIST that contains 0 1 2 3 separated by white space.
Here is the segment that does the copy:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
TARGET_BIN_LIST_NEW contains new file names separated by white space and is composed of something like this:
file001.200 file001.201 file001.202 file001.203
and TARGET_BIN_LIST contains the existing file names and is composed of something like this:
file001c.200 file001c.201 file001c.202 file001c.203
The last digit in the file extension is the model number.
As I read this, the makefile runs:
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
four times, however, it always use the first file name in the TARGET_BIN_LIST due to the firstword function. This results in file001.200, file001.201, file001.202, file001.203 being created, but they are all copies of file001c.200 when they should be copies of their respective files in the list. Each file relates to a sub model version of the code.
My thought to solve this was to use the word function. Something like this:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(word $(sub), $(TARGET_BIN_LIST)), $#)
where sub is an element of SUB_MODEL_LIST, but I am not sure how that will work. Does the above roll out into 4 separate calls, or can it be looked at as a loop that can have an increment value for sub??
I also thought about using a foreach loop:
$(foreach sub,$(SUB_MODEL_LIST),$(call COPY, $(word $(sub), $(TARGET_BIN_LIST)), $(word $(sub), $(TARGET_BIN_LIST_NEW)))
But I get the error:
*** first argument to `word' function must be greater than 0. Stop.
Ok, so I tried:
$(foreach sub,$(SUB_MODEL_LIST),$(call COPY, $(word $(sub)+1, $(TARGET_BIN_LIST)), $(word $(sub)+1, $(TARGET_BIN_LIST_NEW)))
But then I got the error:
*** non-numeric first argument to `word' function. Stop.
Now I'm stuck. I would like to keep the existing implementation in tact at much as possible, but can adopt a loop method if needed.
Thanks for the help!
You have to step back. You're misunderstanding how this works. In make an explicit rule with multiple targets is EXACTLY THE SAME as writing the same rule multiple times, once for each target. So this:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $< to $#
$(call COPY, $(firstword $(TARGET_BIN_LIST)), $#)
If TARGET_BIN_LIST_NEW is file001.200 file001.201 file001.202 file001.203 and TARGET_BIN_LIST is file001c.200 file001c.201 file001c.202 file001c.203, is identical to writing this:
file001.200 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.201 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.202 : file001c.200 file001c.201 file001c.202 file001c.203
...
file001.203 : file001c.200 file001c.201 file001c.202 file001c.203
...
So you can clearly see that when each rule is run, the value of $< and $(firstword $(TARGET_BIN_LIST)) will be the same thing (file001c.200).
Is it really the case that whenever ANY of the fileXXXc.YYY files change, you want to rebuild ALL the fileXXX.YYY files? That's what your rule does, but based on the recipe it doesn't seem like that's what you want.
Make is mostly about writing one rule to build one target from zero or more prerequisites. If you use a pattern rule you can do this pretty easily:
all: $(TARGET_BIN_LIST_NEW)
file001.% : file001c.%
#echo copying from $< to $#
$(call COPY,$<,$#)
If your filenames may have a more complex naming convention then you'll need something more complicated.
ETA:
Since your naming convention doesn't fit into make's pattern rule capabilities you'll have to do something fancier. You can use eval to generate the rules, like this:
all: $(TARGET_BIN_LIST_NEW)
define TARGET_BIN_COPY
$(1) : $(basename $(1))c$(suffix $(1))
#echo copying from $$< to $$#
$$(call COPY,$$<,$$#)
endef
$(foreach T,$(TARGET_BIN_LIST_NEW),$(eval $(call TARGET_BIN_COPY,$T)))
# uncomment this for debugging
#$(foreach T,$(TARGET_BIN_LIST_NEW),$(info $(call TARGET_BIN_COPY,$T)))
First off, thank you to MadScientist for your help in clarifying how this works.
This implementation worked for me:
$(TARGET_BIN_LIST_NEW) : $(TARGET_BIN_LIST)
#echo copying from $(filter %$(suffix $#), $(TARGET_BIN_LIST)) to $#
$(call COPY, $(filter %$(suffix $#), $(TARGET_BIN_LIST)), $#)
I have a.csv,b.csv, ... in a my docs/csv directory, I need convert each of this file to a json file.
I follow this question to write a Makefile like this.
SRCS = $(wildcard docs/csv/*.csv)
DESTS = $(patsubst docs/csv/%.csv, scripts/data/%.lua, $(SRCS))
all: $(DESTS)
$(DESTS): $(SRCS)
echo $#
echo $<
but every time I ran make all, the echo $# show every file as expected, but echo $< always show the single file, called items.csv in my csv folder.
The trouble is that in this rule:
$(DESTS): $(SRCS)
...
every lua file depends on all csv files, which is not what I think you intend. And since $< expands to the first prerequisite, you get the same one (items.csv) for every target.
Try this:
all: $(DESTS)
scripts/data/%.lua: docs/csv/%.csv
echo $#
echo $<
$<
is the name of the FIRST dependency. Use $^ for all the dependencies
$#
is the name of the current target
The GNU make man page on Automatic Variables is extremely useful. Here's what it says:
$#
The file name of the target of the rule. If the target is an archive member, then ‘$#’ is the name of the archive file. In a
pattern rule that has multiple targets (see Introduction to Pattern
Rules), ‘$#’ is the name of whichever target caused the rule's recipe
to be run.
$<
The name of the first prerequisite. If the target got its recipe from an implicit rule, this will be the first prerequisite added by
the implicit rule (see Implicit Rules).
Incidentally, you probably want to write your make rule as a pattern rule instead:
%.lua : %.csv
<rules for making a lua from a csv>
I have to integrate the generation of many HTML files in an existing Makefile.
The problem is that the HTML files need to reside in many different directories.
My idea is to write an implicit rule that converts the source file (*.st) to the corresponding html file
%.html: %.st
$(HPC) -o $# $<
and a rule that depends on all html files
all: $(html)
If the HTML file is not in the builddir, make doesn't find the implicit rule: *** No rule to make target.
If I change the implicit rule like so
$(rootdir)/build/doc/2009/06/01/%.html: %.st
$(HPC) -o $# $<
it's found, but then I have to have an implicit rule for nearly every file in the project.
According to Implicit Rule Search Algorithm in the GNU make manual, rule search works like this:
Split the entire target name t into a directory part, called d, and the rest, called n. For
example, if t is src/foo.o,
then d is src/,
and n is foo.o.
Make a list of all the pattern rules one of whose targets matches t or n.
If the target pattern contains a slash,
it is matched against t;
otherwise, against n.
Why is the implicit rule not found, and what would be the most elegant solution, assuming GNU make is used?
Here is a stripped down version of my Makefile:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html
%.html: %.st
$(HPC) -o $# $<
#This works, but requires a rule for every output dir
#$(rootdir)/build/doc/2009/06/01/%.html: %.st
# $(HPC) -o $# $<
.PHONY: all
all: $(html)
The best solution I found so far is to generate an implicit rule per target directory via foreach-eval-call, as explained in the GNU make manual. I have no idea how this scales to a few thousand target directories, but we will see...
If you have a better solution, please post it!
Here is the code:
rootdir = /home/user/project/doc
HPC = /usr/local/bin/hpc
html = $(rootdir)/build/doc/2009/06/01/some.html \
$(rootdir)/build/doc/2009/06/02/some.html
targetdirs = $(rootdir)/build/doc/2009/06/01 \
$(rootdir)/build/doc/2009/06/02
define generateHtml
$(1)/%.html: %.st
-mkdir -p $(1)
$(HPC) -o $$# $$<
endef
$(foreach targetdir, $(targetdirs), $(eval $(call generateHtml, $(targetdir))))
.PHONY: all
all: $(html)
Like Maria Shalnova I like recursive make (though I disagree with "Recursive Make Considered Harmful"), and in general it's better to make something HERE from a source THERE, not the reverse. But if you must, I suggest a slight improvement: have generateHtml generate only the RULE, not the COMMANDS.
Your active implicit rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on $(rootdir)/build/doc/2009/06/01/some.st. If $(rootdir)/build/doc/2009/06/01/some.st doesn't exist then the rule won't be used/found.
The commented out rule makes $(rootdir)/build/doc/2009/06/01/some.html depend on some.st.
One solution is to make you're source layout match your destination/result layout.
Another option is to create the rules as required with eval. But that will be quite complicated:
define HTML_template
$(1) : $(basename $(1))
cp $< $#
endef
$(foreach htmlfile,$(html),$(eval $(call HTML_template,$(htmlfile))))
An other possibility is to have the commando make call itself recursively with the argument -C with every output directory.
Recursive make is somewhat the standard way to deal with subdirectories, but beware of the implications mentioned in the article "Recursive Make Considered Harmful"