I have the following Makefile, which is supposed to fetch a bunch of soundfiles in mp3 format, based on a list of sound ids, and then convert them into .wav files. I am not so familiar with GNU make so please bear with me.
What I need as a result is a bunch of .wav files, so, I have made an "all" target, which expects these .wav files. This Makefile, when run, will download normally the .mp3 files and then convert them into .wav files. Afterwards, it will delete all .mp3 files (since I have flagged them as .INTERMEDIATE). My problem is that if I re-run this Makefile, it will try to re-download the mp3 files and re-convert them into .wav files, even though, the .wav files are already present.
I can't understand why the "all" target gets re-triggered even though it should be satisfied since all its requirements have already been met.
How can I fix that?
BASE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
BASE_DIR := $(BASE_DIR:/=)
BUILD := $(BASE_DIR)/build
AUDIO := $(BUILD)/audio
SOUND_IDS_FILE := $(BASE_DIR)/sound_ids.txt
SOUND_IDS:=$(shell cat ${SOUND_IDS_FILE})
MP3_FILES:=$(patsubst %, $(AUDIO)/%.mp3, $(SOUND_IDS))
WAV_FILES:=$(patsubst %, $(AUDIO)/%.wav, $(SOUND_IDS))
.PHONY: all clean
.INTERMEDIATE: $(MP3_FILES)
# Main Rules:
all: $(WAV_FILES)
clean:
rm -rf $(BUILD)
# Dir Rules:
$(BUILD):
mkdir -p $#
$(AUDIO):
mkdir -p $#
$(WAV_FILES): $(MP3_FILES)
#echo "Converting $< into $# ..."
ffmpeg -i $< -acodec pcm_s16le -ac 1 -ar 16000 $#
$(MP3_FILES): $(AUDIO)
#echo "Downloading file: $# ..."
sound_fetcher -i $(patsubst %.mp3,%,$(#F)) -o $(AUDIO) -format mp3
Related
I'd like to turn a bunch of images into a video (video video1.mp4 is made from the files video1/*.mjpg), using a Makefile. I tried:
SOURCES=video1.mp4 video2.mp4
all: $(SOURCES)
%.mp4: $(shell find % -type f)
ffmpeg -framerate 24 -pattern_type glob -i '$*/*.jpg' -c:v libx264 -pix_fmt yuv420p $#
but it does not work, the % is not turned into the video1-like prefix. Any idea how to make it work?
You have to add the dependencies in another way as % is indeed not handled as you would like. Something like this (replacing the cat by an adequate ffmpeg call) should do the trick:
DESTS=video1.mp4 video2.mp4
all: $(DESTS)
define add_dependencies
$(1): $(wildcard $(1:.mp4=)/*.jpg)
endef
$(foreach dst,$(DESTS),$(eval $(call add_dependencies,$(dst))))
%.mp4:
cat $^ > $#
Note:
I'm using the built-in wildcard instead of find. The way I'm using it, I don't recurse in sub-directories, but your call to ffmpeg hints to me that wasn't desired.
I'd suggest if possible using $^ instead of doing the file matching a second time in ffmpeg.
This is for GNU make, but I don't think other makes accept the $(shell) syntax if they aren't targeting a wide GNU make compatibility.
My goal is the following: I have a directory src which contains markdown files (.md). I want to run a command on each of these files so that the comments are removed and the edited files are stored in a separate directory. For this I want to use make.
This is the Makefile I have:
.PHONY: clean all
BUILD_DIR := build
SRC_DIRS := src
SRCS := $(shell find $(SRC_DIRS) -name *.md)
DSTS := $(patsubst $(SRC_DIRS)/%.md,$(BUILD_DIR)/%.md,$(SRCS))
all: $(DSTS)
# The aim of this is to remove all my comments from the final documents
$(DSTS): $(SRCS)
pandoc --strip-comments -f markdown -i $< -t markdown -o $#
clean:
rm $(BUILD_DIR)/*.md
While this works in general, I noticed that the command is executed on all files, even though I changed only one single file.
Example: I have 3 Files src/a.md, src/b.md and src/c.md. Now I run make and all files are correctly generated in the build folder. Now I only edit c.md and run make again. I would expect that make only "compiles" src/c.md anew but instead all three files are compiled again. What am I doing wrong?
Your line
$(DSTS): $(SRCS)
is saying ‘All of the DSTS depend on all of the SRCS’, so whenever any one of the $(SRCS) is newer than any of the $(DSTS), this pandoc action will be run.
That's not what you want to express. What you want is something more like
$(BUILD_DIR)/%.md: $(SRC_DIRS)/%.md
pandoc --strip-comments -f markdown -i $< -t markdown -o $#
all: $(DSTS)
That says that all of the $(DSTS) should be up to date, and the pattern rule teaches Make what each one depends on, and how to build it, if it is out of date.
(As a general point, looking your original rule, it's rarely the right thing to do to have multiple targets in a rule, as you have with $(DSTS); also note that in your original, $< always refers only to the first of the dependencies in $(SRCS))
I would like to copy pdf files from several directories into a build directory, then use pdfunite to compile them into one pdf. The following make recipe works, but I have to run it twice because the first time through, I get an error from pdfunite - no files are found in the build directory (the PDFS variable is empty) even though they were just copied in the previous lines. How can I fix this so it works in one pass? I have simplified the recipe for clarity; I am actually pulling from various folders and making some pdfs on the fly as well, so I can't easily concatenate a full list of files from various subfolders (folder1 and folder2 in the example) to pass to pdfunite.
notebook:
mkdir -p $(out)
mkdir -p $(build)/notebook
$(eval PR := $(sort $(wildcard $(data)/folder1/*.pdf)) )
cp $(PR) $(build)/notebook
$(eval SR := $(sort $(wildcard $(data)/folder2/*.pdf)) )
cp $(SR) $(build)/notebook
$(eval PDFS := $(sort $(wildcard $(build)/notebook/*.pdf)) )
pdfunite $(PDFS) $(out)/notebook.pdf
Your Makefile is not in line with make's philosophy. You are using make as another scripting language, while make is more than this. It compares targets and prerequisites dates, based on this decides which must be built or re-built, and passes the recipes to the shell. So, for your particular problem, you should rather try something like:
PR := $(wildcard $(data)/folder1/*.pdf)
SR := $(wildcard $(data)/folder2/*.pdf)
PDFS1 := $(patsubst $(data)/folder1/%.pdf,$(build)/notebook/%.pdf,$(PR))
PDFS2 := $(patsubst $(data)/folder2/%.pdf,$(build)/notebook/%.pdf,$(SR))
PDFS := $(sort $(PDFS1) $(PDFS2))
.PHONY: notebook
notebook: $(out)/notebook.pdf
$(PDFS1): $(build)/notebook/%.pdf: $(data)/folder1/%.pdf | $(build)/notebook
cp $< $#
$(PDFS2): $(build)/notebook/%.pdf: $(data)/folder2/%.pdf | $(build)/notebook
cp $< $#
$(build)/notebook $(out):
mkdir -p $#
$(out)/notebook.pdf: $(PDFS) | $(out)
pdfunite $(PDFS) $#
The variables definitions are reasonably straightforward: patsubst, as its name says, substitutes strings. The target: pattern: prerequisites is a static pattern rule. And the prerequisites after | are order-only prerequisites.
What this makefile says, basically, is that $(out)/notebook.pdf depends on a set of pdf files in $(build)/notebook/ and that these pdf files depend on source pdf files with the same basenames in $(data)/folder1/ and $(data)/folder2/. It also says that directories must be created before being populated. Thanks to all this only what needs to be done will be done, no more, no less. And it is more in line with make's philosophy.
If you have many source folders and do not want to replicate the copying rules, you can use more advanced features like:
FOLDERS := folder1 folder2
.PHONY: notebook
notebook: $(out)/notebook.pdf
define MY_rule
$(1)_SRCS := $$(wildcard $$(data)/$(1)/*.pdf)
$(1)_DSTS := $$(patsubst $$(data)/$(1)/%.pdf,$$(build)/notebook/%.pdf,$$($(1)_SRCS))
PDFS += $$($(1)_DSTS)
$(1)_DSTS: $$(build)/notebook/%.pdf: $$(data)/$(1)/%.pdf | $$(build)/notebook
cp $$< $$#
endef
$(foreach f,$(FOLDERS),$(eval $(call MY_rule,$(f))))
$(build)/notebook $(out):
mkdir -p $#
$(out)/notebook.pdf: $(PDFS) | $(out)
pdfunite $(PDFS) $#
I would do another way.
PR:=$(sort $(wildcard $(data)/folder1/*.pdf))
SR:=$(sort $(wildcard $(data)/folder2/*.pdf))
PDFS=$(sort $(wildcard $(build)/notebook/*.pdf))
all: copy
pdfunite $(PDFS) $(out)/notebook.pdf
copy:
mkdir -p $(out)
mkdir -p $(build)/notebook
cp $(PR) $(build)/notebook
cp $(SR) $(build)/notebook
.PHONY: all copy
Please check: PDFS= and not PDFS:=. If you use simple = the variable's value will calculate when it needed (not sooner!).
When you run make it want to build all. The all's requirement is copy - so make does some mkdir and cp. After it return the all: the value of PDFS is needed so will evalute now - we have many-many pdf in $(build)/notebook :)
the makefile below processes files matching the patterncontent/%.md and outputs the targets in the html directory. Source files are named with the convention of putting a leading number in front of them, like content/01.index.md or content/O2.second-page.md and so on. I would like to remove the leading 0x. number sequence in the target file. For instance, content/01.index.html would generate html/index.html.
How can I do this?
Thanks
MD_FILES = $(shell find content/ -type f -name '*.md')
HTML_FILES = $(patsubst content/%.md, html/%.html, $(MD_FILES))
all: $(HTML_FILES) html/static
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
.PHONY: html/static
html/static :
rsync -rupE generator/static html/
.PHONY: clean
clean:
rm -fr html
Replace:
html/%.html : content/%.md
mkdir -p $(#D)
python generator/generate.py $< $#
with:
html/%.html : content/%.md
mkdir -p $(#D)
file='$(#F)'; python generator/generate.py $< "$(#D)/${file#*.}"
Unfortunately, I can't think of a good way of doing that in make itself. I can think of one way but it isn't as simple as that escaping and it isn't safe for files with spaces (not that that matters much here since make already can't handle those).
IMHO, it is a bad idea to use find or wildcards to list files in makefiles. This is because developers have temporary or debugging files sometimes. It is best to list files explicitly. This way, it forces the developer to think about their intent.
If you agree to list files explicitly, then in this case it is best to list the target files, rather than source files, and here is your answer:
HTML_FILES := html/index.html html/second-page.html
.SECONDEXPANSION:
$(HTML_FILES): html/%.html : $$(wildcard content/*.$$*.md)
(put recipe here, using $# and $<)
I would like make to copy files from the source directory into a target directory. And i would like to avoid copying unchanged files. Therefore, I am trying to utilize make function of checking for newer files with a %.:%. rule.
Now, in this instance, the source and target EXTENSION are the same. There are object files elsewhere but not for the graphical assets.
I use make to copy images.
TARGET := target
SOURCE := source
GRAPHICS := $(foreach dir,$(SOURCE), $(wildcard $(dir)/*.jpg ) $(wildcard $(dir)/**/*.jpg ) $(wildcard $(dir)/*.png ) $(wildcard $(dir)/**/*.png ) $(wildcard $(dir)/*.gif ) $(wildcard $(dir)/**/*.gif ) )
JPG = $(GRAPHICS:.jpg=.tmp)
PNG = $(GRAPHICS:.png=.tmp)
GIF = $(GRAPHICS:.gif=.tmp)
And then use the following rule to copy files into target directory:
%.tmp:%.jpg
find $< | cpio -p -d -v $(TARGET)
Questions
Is it possible to string replace the TARGET directory here
and thereby use the make newer capability?
I tried
JPG = $(GRAPHICS:$(TARGET).jpg=.tmp)
But that fails with No rule to make target. Is it only possible to compare source and object in the same directory?
Can one make a rule such that the source and object are the same extension?
%.jpg:%.jpg
The closest I can come up with is:
$(TARGET)%.jpg:%.jpg
but that never runs. Even after a clean.
Here is the solution.
This 'setup' remains the same.
TARGET := target
SOURCE := source GRAPHICS := $(foreach dir,$(SOURCE), $(wildcard $(dir)/*.jpg )
$(wildcard $(dir)/**/*.jpg ) $(wildcard $(dir)/*.png ) $(wildcard $(dir)/**/*.png )
$(wildcard $(dir)/*.gif ) $(wildcard $(dir)/**/*.gif ) )
Now add a prefix to ALL (space separated) values in the $GRAPHICS string
TARGET_GRAPHICS := $(addprefix $(TARGET)/, $(GRAPHICS) )
Note that patsubst aka -- TARGET_GRAPHICS := $(patsubst %, $(TARGET)/%, $(GRAPHICS) ) -- only worked for the first entry. Perhaps I was doing it incorrectly.
Provide a pattern match for each file type. Here do whatever you wish. In my case, copy the input into the target directory. cpio will make the needed directories. cp -p might also work as needed.
$(TARGET)/%.jpg : %.jpg
#echo "--- .jpg copying " $< " to " $# " into dir " $(<D)
#find $< | cpio -p -d -v $(TARGET)
$(TARGET)/%.png : %.png
#echo "--- .png copying " $< " to " $#
#find $< | cpio -p -d -v $(TARGET)
$(TARGET)/%.gif : %.gif
#echo "--- .gif Copying " $< " to " $#
#find $< | cpio -p -d -v $(TARGET)
So you want the copy of d1/d2/f.png into out/d1/d2/f.png to make use of make's dependency checking?
out/d1/d2/f.png: d1/d2/f.png
out/d1/d2/f.png:
cp $< $#
Adding a jpg file d3/g.jpg say,
out/d3/g.jpg: d3/g.jpg
out/d1/d2/f.png: d1/d2/f.png
out/d1/d2/f.png out/d3/g.jpg:
cp $< $#
We can express this more cleanly with a static pattern rule.
out/d3/g.jpg out/d1/d2/f.png: out/% %
cp $< $#
Nice. So adding your ${GRAPHICS} and fleshing things out a bit
outdir := out/
targets := $(addprefix ${outdir},${GRAPHICS})
.PHONY: all
all: ${targets} ; : $# Success
${targets}: ${outdir}%: %
echo $< | cpio -d -p ${outdir}
Parallel safe too, so make -j9 will exercise your 8 CPUs nicely.