I have a rule in a makefile which uses sox to convert wav files to another format.
Makefile:
INPUTFILE?=file1.wav
OUTFILE=$(INPUTFILE:.wav=.s32)
wav_to_s32:
#sox $(INPUTFILE) -r 16000 $(OUTFILE)
How would I make this work for multiple files specified at input (with multiple outputs)?
INPUTFILE?=file1.wav file2.wav file3.wav
As explained in this other answer, the key here is pattern rules. If you want to automate all conversions for all files that you list in your INPUTFILE variable you can complete your initial version and that other answer like this:
INPUTFILE?=file1.wav file2.wav file3.wav
OUTFILE=$(INPUTFILE:.wav=.s32)
.PHONY: wav_to_s32
wav_to_s32: $(OUTFILE)
%.s32: %.wav
sox $< -r 16000 $#
This tells make that:
wav_to_s32 is not a real file, it's just a short name for something else (it is a "phony" target).
When invoked with make wav_to_s32 it shall build all files listed in $(OUTFILE).
And if you prefer make to discover automatically the input files, you can further improve all this with:
INPUTFILE := $(wildcard *.wav)
OUTFILE := $(patsubst %.wav,%.s32,$(INPUTFILE))
.PHONY: wav_to_s32
wav_to_s32: $(OUTFILE)
%.s32: %.wav
sox $< -r 16000 $#
Note that make is smart enough to not reconvert a file if it has been converted already and did not change since. It is even the main purpose of make: track inter-files dependencies.
I would use the following pattern rule in your makefile:
%.s32: %.wav
sox $< -r 16000 $#
The rule above will build .s32 files from .wav files.
With that rule in your makefile you could call make for multiple targets in the following way:
$ make foo.s32 bar.s32
sox foo.wav -r 16000 foo.s32
sox bar.wav -r 16000 bar.s32
It generates foo.s32 and bar.s32 from foo.wav and bar.wav, respectively.
Related
I'd like to use make to process a large number of inputs to outputs using a script (python, say.) The problem is that the script takes an incredibly short amount of time to run per input, but the initialization takes a while (python engine + library initialization.) So, a naive makefile that just has an input->output rule ends up being dominated by this initialization time. Parallelism doesn't help with that.
The python script can accept multiple inputs and outputs, as so:
python my_process -i in1 -o out1 -i in2 -o out2 ...
and this is the recommended way to use the script.
How can I make a Makefile rule that best uses my_process, by sending in out of date input-output pairs in batches? Something like parallel but aware of which outputs are out of date.
I would prefer to avoid recursive make, if at all possible.
I don't completely grasp your problem: do you really want make to operate in batches or do you want a kind of perpetual make process checking the file system on the fly and feeding to the Python process whenever it finds necessary? If the latter, this is quite the opposite of a batch mode and rather a pipeline.
For the batch mode there is a work-around which needs a dummy file recording the last runnning time. In this case we are abusing make for because the makefile is in this part a one-trick pony which is unintuitive and against the good rules:
SOURCES := $(wildcard in*)
lastrun : $(SOURCES)
python my_process $(foreach src,$?,-i $(src) -o $(patsubst in%,out%,$(src)))
touch lastrun
PS: please note that this solution has a substantial flaw in that it doesn't detect the update of in-files when they happen during the run of the makefile. All in all it is more advisable to simply collect the filenames of the in-files which were updated by the update process itself and avoid make althogether.
This is what I ended up going with, a makefile with one layer of recursion.
I tried using $? both with grouped and ungrouped targets, but couldn't get the exact behavior needed. If one of the output targets was deleted, the rule would be re-run but $? wouldn't necessarily have some input files but not the correct corresponding input file, very strange.
Makefile:
all:
INDIR=in
OUTDIR=out
INFILES=$(wildcard in/*)
OUTFILES=$(patsubst in/%, out/%, $(INFILES))
ifdef FIRST_PASS
#Discover which input-output pairs are out of date
$(shell mkdir -p $(OUTDIR); echo -n > $(OUTDIR)/.needs_rebuild)
$(OUTFILES) : out/% : in/%
#echo $# $^ >> $(OUTDIR)/.needs_rebuild
all: $(OUTFILES)
#echo -n
else
#Recurse to run FIRST_PASS, builds .needs_rebuild:
$(shell $(MAKE) -f $(CURDIR)/$(firstword $(MAKEFILE_LIST)) FIRST_PASS=1)
#Convert .needs_rebuild into batches, creates all_batches phony target for convenience
$(shell cat $(OUTDIR)/.needs_rebuild | ./make_batches.sh 32 > $(OUTDIR)/.batches)
-include $(OUTDIR)/.batches
batch%:
#In this rule, $^ is all inputs needing rebuild.
#The corresponding utputs can be computed using a patsubst:
targets="$(patsubst in/%, out/%, $^)"; touch $$targets
clean:
rm -rf $(OUTDIR)
all: all_batches
endif
make_batches.sh:
#!/bin/bash
set -beEu -o pipefail
batch_size=$1
function _make_batches {
batch_num=$1
shift 1
#echo ".PHONY: batch$batch_num"
echo "all_batches: batch$batch_num"
while (( $# >= 1 )); do
read out in <<< $1
shift 1
echo "batch$batch_num: $in"
echo "$out: batch$batch_num"
done
}
export -f _make_batches
echo ".PHONY: all_batches"
parallel -N$batch_size -- _make_batches {#} {} \;
Unfortunately, the makefile is a one trick pony and there's quite a bit of boilerplate to pull this recipe off.
I've a list of files in a Makefile variable. I use it to call a couple of implicit rules.
In shorts:
CHAPTERS_FOLDER = chapters
CHAPTERS := $(CHAPTERS_FOLDER)/lesson1.Md $(CHAPTERS_FOLDER)/lesson2.Md [...] $(CHAPTERS_FOLDER)/lesson23.Md
$(CHAPTERS_FOLDER)/%.Md : %.xml
mkdir -p $(CHAPTERS_FOLDER)
pandoc -f docbook -t markdown_strict $< -o $#
$(CHAPTERS_FOLDER)/%.Md : %.odt
mkdir -p $(CHAPTERS_FOLDER)
pandoc -t markdown_strict $< -o $#
Is there a way to not have to manually add CHAPTERS_FOLDER to each item in CHAPTER? Something like a cartesian product of strings or something...
Is there a better way to design these rules to build the files?
Why not just use addprefix?
CHAPTERS = lesson1.Md lesson2.Md lesson3.Md ...
CHAPTER_TARGETS = $(addprefix $(CHAPTERS_FOLDER)/,$(CHAPTERS))
Read the first few paragraphs of the manual on file name functions to see that they operate on lists of file names, not single file names.
I need to do a makefile for run some programs. Every time I run that script all the file are processed also if the file are not changed. I'm sure there is a problem on my code but I don't understand where I made the mistakes.
RDIR=RAW
OUTDIR=Fusion_res/kallisto
RFILES:=$(wildcard $(RDIR)/*_R1_001.fastq.gz)
DATABASE=/home/sbsuser/databases/Kallsto_hg38_87
OUTFILE=$(patsubst %_R1_001.fastq.gz,%_R2_001.fastq.gz,$(RFILES))
OUTKAL=$(patsubst $(RDIR)/%_R1_001.fastq.gz,$(OUTDIR)/%,$(RFILES))
.PHONY: clean all
all: $(OUTFILE) $(RFILES) $(OUTDIR) $(OUTKAL)
#$(OUTKAL) $(OUTFILE): $(RDIR)/%._R1_001.fastq.gz
# echo "kallisto quant -i" $(DATABASE)/transcripts.idx -b 100 -o $# --fusion $< $(OUTFILE)
$(OUTDIR)/%: $(RDIR)/%_R1_001.fastq.gz $(OUTFILE)
kallisto quant -i $(DATABASE)/transcripts.idx -b 100 --fusion --rf-stranded -o $# $(RDIR)/$*_R1_00
1.fastq.gz $(RDIR)/$*_R2_001.fastq.gz
$(OUTDIR):
mkdir -p $(OUTDIR)
clean::
$(RM) -rf $(OUTDIR)
I suppose if the found some change on the input file and on the output execute the command. I don't know why every time force re-run. In some case Is that I want but I wan to also if there is some new input execute only that.
Thanks so much
A couple of things:
1) $(OUTDIR)/% is dependent on $(OUTFILE) (which is a list of all outfiles). Therefore if you change any one of the OUTFILEs, you make everything in $(OUTDIR)/% obsolete. I believe what you want is this:
$(OUTDIR)/%_R1_001.fastq.gz: $(RDIR)/%_R2_001.fastq.gz
.... (rules to make out/R1 from raw/R2
$(RDIR)/%_R2_001.fastq.gz: $(RDIR)/%_R1_001.fastq.gz
.... (rules to make R2 from R1
This makes each file dependent only on the files that effect it.
2) you have the target all dependent on $(OUTDIR) which is a directory. If you use parallel make, it may generate the $(OUTDIR) after it generates the other dependencies of all: (some of which would depend on $(OUTDIR) being created). What you want there is to remove all's dependency on $(OUTDIR), and add the line:
$(OUTFILE) : | $(OUTDIR)
Notice the |, which means order only (don't consider $(OUTFILE) out of date if $(OUTDIR) is newer. This is important, as a directory's timestamp is updated each time a file in the directory is changed, and so it tends to be newer than its contents.
I wrote the following makefile:
include config.mk
DATA_FILES=$(wildcard ../*.dat)
all : plot output_data
.PHONY : plot
plot : output_data $(PLOT_SRC)
$(PLOT_EXE)
.PHONY : output_data
output_data : $(SIM_SRC) $(DATA_FILES)
$(SIM_EXE)
.PHONY : clean
clean :
rm -f output_data
rm -f plot
As i expected make clean is not working, the problem is that i don't know the name or the number of files that are going to be created during make. I know their extensions but I'd rather don't use wildcards+extension to avoid delete other files. Moreover, I know I could make the code print out the name of the files it gives as an output, but 1) the code is not mine, so I' prefer not changing it, and 2) the code is not just creating files, but a number of folders.
Any other options to delete all the files created that I didn't consider?
If you are using GNU make, the filter-out function may be a solution. Assuming you have a list of files and directories to keep in a make variable KEEP, you can:
KEEP = <list-of-files-and-directories-to-keep>
clean:
#echo rm -rf $(filter-out $(KEEP),$(wildcard *))
Try to run this and, if satisfied, remove echo. Or, instead of rm, move the files and directories you do not want to keep in a temporary directory:
clean:
#mkdir -p ../backups; \
mv $(filter-out $(KEEP),$(wildcard *)) ../backups
and manually check the content of ../backups.
I have the following folder structure
1st-grade-math-class/
common/
mystyle.sty
mysubstyle.sty
fonts/
font1.ttf
font2.ttf
font3.ttf
week01/
handout.tex
image1.pdf
image2.pdf
week02/
handout.tex
image1.pdf
image2.pdf
...
week13/
handout.tex
output/
[empty]
And I would like to create a Makefile - in the best way - to do the following:
make sure I include the common directory properly in TEXINPUTS
compile the handout.tex into a PDF (using either pdflatex or xelatex) and have it in the output directory as week01-handout-student.pdf
compile the handout.tex with a line of LaTeX prepended to the beginning of the file (that sets a flag) into a PDF and have it in the output directory as week01-handout-teacher.pdf
clean everything up (the log, aux, etc. files)
I am not sure I know how to this in any other way than manually duplicating an elementary Makefile/bash script in every subdirectory, then calling each of them one by one with a for loop.
I would appreciate help on how to build this process, ideally with a single Makefile in the root directory. Thanks.
UPDATE: I purposefully did not want to give any details about how I compile LaTeX, in case somebody has a better suggestion than my current usage. Right now I am using Latexmk (which is already a make-like wrapper of LaTeX):
latexmk -pdf file.tex generates file.pdf
to add the line of code, I do a simple echo "line of code" > temp.tex and cat handout.tex >> temp.tex, then the same latexmk command
latexmk -c file.tex in a directory cleans all temporary files used to compile file.tex
TEXINPUTS is the TeX path variable, to let TeX find (in its path) the style files: I do TEXINPUTS=full-path-to/common and then export TEXINPUTS before compiling anything.
If anybody has a better suggestion, I am a willing taker.
Something like this should do what you want I believe:
OUTDIR := output
# Tell make to export this environment variable to all recipe lines it runs.
export TEXINPUTS := $(abspath common)
# Get the list of all of our week directories.
weekdirs := $(wildcard week*)
#$(info weekdirs:$(weekdirs))
# Create student output filenames from week directory names.
STUDENT_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-student.pdf,$(weekdirs))
#$(info STUDENT_HANDOUTS:$(STUDENT_HANDOUTS))
# Create teacher output filenames from week directory names.
TEACHER_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-teacher.pdf,$(weekdirs))
#$(info TEACHER_HANDOUTS:$(TEACHER_HANDOUTS))
# Default target depends on all output files.
all: $(STUDENT_HANDOUTS) $(TEACHER_HANDOUTS)
# Pattern rule for building pdf files.
%.pdf:
#echo + Making $# from $^
#echo cd $(#D) && echo latexmx -pdf $(abspath $<)
#echo cd $(#D) && echo latexmk -c $(abspath $<)
# Static pattern rule mapping student output files to input files.
$(STUDENT_HANDOUTS) : $(OUTDIR)/%-handout-student.pdf : %/handout.tex
# Pattern rule to generate temporary input files from original input files.
%/handout-tmp.tex: %/handout.tex
#echo echo 'line of code' '>' $#
#echo cat $^ '>>' $#
# Static pattern rule mapping teacher output files to (temporary) input files.
$(TEACHER_HANDOUTS) : $(OUTDIR)/%-handout-teacher.pdf : %/handout-tmp.tex
Uncomment the $(info) lines to see a bit of how the variables are put together.
This uses latexmk -c to clean up auxiliary files after creating the output files.
Technique 1. The echo foo >$#; cat $< >>$# thing is something I've done before; I think it's reasonable.
Technique 2. Another technique is to have your .tex document, or a package file it uses, include a line like:
\InputIfFileExists{lecturenotes.config}{}{}
That allows you to make certain adjustments in a makefile, as in for example:
# any bits of configuration that should be in all the .config files
SWITCHES=\makeindex
%-notes.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{sidenotes}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
%-single.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{oneside}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
Technique 3. A third technique for controlling LaTeX from outside is to include in your document (or in a package file):
\newif\ifdemonstrator
\expandafter\ifx\csname demonstrator\endcsname\relax
\demonstratorfalse
\else
\demonstratortrue
\fi
...
\ifdemonstrator
\typeout{Demonstrator is TRUE}
\else
\typeout{Demonstrator is FALSE}
\fi
Then call latex with:
%-plain.pdf: %.tex
latex $<
mv ${<:.tex=.pdf} $#
%-demo.pdf: %.tex
latex --jobname ${<:.tex=} '\def\demonstrator{x}\input{$<}`
mv ${<:.tex=.pdf} $#
Technique 1 is a bit of a blunt instrument, but if that's all that's needed, it's pretty lightweight.
Technique 2 is probably the most neatly engineered, but it is slightly more effort.
Technique 3 is probably the one I use most in this sort of circumstance.