I need to do a makefile for run some programs. Every time I run that script all the file are processed also if the file are not changed. I'm sure there is a problem on my code but I don't understand where I made the mistakes.
RDIR=RAW
OUTDIR=Fusion_res/kallisto
RFILES:=$(wildcard $(RDIR)/*_R1_001.fastq.gz)
DATABASE=/home/sbsuser/databases/Kallsto_hg38_87
OUTFILE=$(patsubst %_R1_001.fastq.gz,%_R2_001.fastq.gz,$(RFILES))
OUTKAL=$(patsubst $(RDIR)/%_R1_001.fastq.gz,$(OUTDIR)/%,$(RFILES))
.PHONY: clean all
all: $(OUTFILE) $(RFILES) $(OUTDIR) $(OUTKAL)
#$(OUTKAL) $(OUTFILE): $(RDIR)/%._R1_001.fastq.gz
# echo "kallisto quant -i" $(DATABASE)/transcripts.idx -b 100 -o $# --fusion $< $(OUTFILE)
$(OUTDIR)/%: $(RDIR)/%_R1_001.fastq.gz $(OUTFILE)
kallisto quant -i $(DATABASE)/transcripts.idx -b 100 --fusion --rf-stranded -o $# $(RDIR)/$*_R1_00
1.fastq.gz $(RDIR)/$*_R2_001.fastq.gz
$(OUTDIR):
mkdir -p $(OUTDIR)
clean::
$(RM) -rf $(OUTDIR)
I suppose if the found some change on the input file and on the output execute the command. I don't know why every time force re-run. In some case Is that I want but I wan to also if there is some new input execute only that.
Thanks so much
A couple of things:
1) $(OUTDIR)/% is dependent on $(OUTFILE) (which is a list of all outfiles). Therefore if you change any one of the OUTFILEs, you make everything in $(OUTDIR)/% obsolete. I believe what you want is this:
$(OUTDIR)/%_R1_001.fastq.gz: $(RDIR)/%_R2_001.fastq.gz
.... (rules to make out/R1 from raw/R2
$(RDIR)/%_R2_001.fastq.gz: $(RDIR)/%_R1_001.fastq.gz
.... (rules to make R2 from R1
This makes each file dependent only on the files that effect it.
2) you have the target all dependent on $(OUTDIR) which is a directory. If you use parallel make, it may generate the $(OUTDIR) after it generates the other dependencies of all: (some of which would depend on $(OUTDIR) being created). What you want there is to remove all's dependency on $(OUTDIR), and add the line:
$(OUTFILE) : | $(OUTDIR)
Notice the |, which means order only (don't consider $(OUTFILE) out of date if $(OUTDIR) is newer. This is important, as a directory's timestamp is updated each time a file in the directory is changed, and so it tends to be newer than its contents.
Related
I have a simple Makefile that will produce a file
all: build/foo.bin
build/foo.bin: foo.c
gcc $< -o $#
Works great and produces build/foo.bin as expected. If I then do a another make it will say make: Nothing to be done for 'all'. That's expected.
I then do rm build/foo.bin && make and it rebuilds the file. But if I do a echo "Modified" > build/foo.bin make doesn't think that anything has changed make: Nothing to be done for 'all'.
How can I write the rules of the Makefile to re-create the build/foo.bin if the binary ever gets modified outside of the Makefile?
Make compares timestamps between two files. If the target file exists and its timestamp is newer than all of the prerequisites' timestamps, then make decides the target is up to date and it doesn't need to do anything.
Make doesn't maintain some kind of database of timestamps on its own: it relies on the filesystem for that. So make cannot detect when a file changes from what it previously contained. It can only detect when some other file changed after the target file was last updated.
In short, make cannot do what you want it to do, using its standard methods.
If you want to do that you'll have to get complicated and create a way to turn the behavior you want to detect into a file with a timestamp, that make can compare.
One way to do this would be to keep the md5sum of the file in another file, then compare it and update the file only if it's changed. You can try this (I didn't test it):
build/foo.bin: foo.c checksum.out
gcc $< -o $#
md5sum $# > checksum.out
touch $#
checksum.out: FORCE
md5sum build/foo.bin > checksum.tmp; cmp $# checksum.tmp || cp checksum.tmp $#
FORCE: ;
Basically, the FORCE is there to require the md5sum check to always run, but then if the checksum doesn't actually change it doesn't update the output file which means that build/foo.bin won't be rebuilt (at least not because checksum.out is updated).
I have my project binary located at my repository's root, along with a Makefile used to build it.
This binary uses many of my self-made libraries, located in my lib/ folder
For the purpose of building (and cleaning) my repository's binary, I want to implement the following execution :
Instead of hardcoding the following lines,
clean_binaries:
make -C clean lib/folder1 -s
make -C clean lib/folder2 -s
make -C clean lib/another_folder -s
I created the BIN_PATH variable, containing the previous paths.
BIN_PATHS = lib/folder1 \
lib/folder2 \
lib/another_folder
And made a simple rule like this one :
clean_binaries: $(BIN_PATHS)
make -C clean $< -s
BUT it only executes the line for the first field of the variable (lib/folder1), which is not what I want to do.
I thought about using implicit rules(?), just like I compile the .c files, but I couldn't get it right.
In the end, I simply wonder how to execute a rule for every field of a given variable, and this inside a Makefile, if there is any way to do so.
Thank you for your answers :]
The way you get GNU make to generate a sequence of commands that vary by the
fields in a variable is to use the foreach function, e.g.
Makefile
BIN_PATHS := lib/folder1 lib/folder2 lib/another_folder
.PHONY: clean_binaries
clean_binaries:
$(foreach path,$(BIN_PATHS),make -C $(path) clean ;)
which runs like:
$ make
make -C lib/folder1 clean -s; make -C lib/folder2 clean -s; make -C lib/another_folder clean -s;
not requiring a shell-loop.
Note also that you need to correct:
make -C clean <directory>
to:
make -C <directory> clean
I have an odd project were a few files are generated at build time. they are among other files but have a special sulffix. (as opposed to a normal project, where all files of some type means they were auto generated)
for example:
src/fileA.js
src/fileB.js.tpl
src/fileC.css
src/fileD.css.tpl
...
then i have a pattern rule:
DATA=$(get string from template.txt)
%: %.tpl
sed 's/__TEMPLATE__/$(DATA)/g' $< > $#
templates: src/fileB.js src/fileD.css
And all is fine. Until the next build... now src/fileB.js will not get updated because there is one there already, and src/fileB.js.tpl was not changed, though the other file template.txt that i use as a data source to update it might. Which brings me to the clean step.
right now my clean step is rming each file. it is ugly.
.PHONY: clean
clean:
rm src/fileB.js
rm src/fileD.css
...
You can see how it gets ugly.
In a regular project my clean would be just rm *.o but here i can't do rm *.js as half the files are not auto-generated.
is there any way to make the rule %: %.tpl be a PHONY?
if not, is there any way to feed the file list from template into clean?
What about this?
TEMPLATES=$(wildcard src/*.tpl)
GENERATED=$(TEMPLATES:%.tpl=%)
clean:
rm -f $(GENERATED)
Well, I would backup before testing this...
I wrote the following makefile:
include config.mk
DATA_FILES=$(wildcard ../*.dat)
all : plot output_data
.PHONY : plot
plot : output_data $(PLOT_SRC)
$(PLOT_EXE)
.PHONY : output_data
output_data : $(SIM_SRC) $(DATA_FILES)
$(SIM_EXE)
.PHONY : clean
clean :
rm -f output_data
rm -f plot
As i expected make clean is not working, the problem is that i don't know the name or the number of files that are going to be created during make. I know their extensions but I'd rather don't use wildcards+extension to avoid delete other files. Moreover, I know I could make the code print out the name of the files it gives as an output, but 1) the code is not mine, so I' prefer not changing it, and 2) the code is not just creating files, but a number of folders.
Any other options to delete all the files created that I didn't consider?
If you are using GNU make, the filter-out function may be a solution. Assuming you have a list of files and directories to keep in a make variable KEEP, you can:
KEEP = <list-of-files-and-directories-to-keep>
clean:
#echo rm -rf $(filter-out $(KEEP),$(wildcard *))
Try to run this and, if satisfied, remove echo. Or, instead of rm, move the files and directories you do not want to keep in a temporary directory:
clean:
#mkdir -p ../backups; \
mv $(filter-out $(KEEP),$(wildcard *)) ../backups
and manually check the content of ../backups.
I have the following folder structure
1st-grade-math-class/
common/
mystyle.sty
mysubstyle.sty
fonts/
font1.ttf
font2.ttf
font3.ttf
week01/
handout.tex
image1.pdf
image2.pdf
week02/
handout.tex
image1.pdf
image2.pdf
...
week13/
handout.tex
output/
[empty]
And I would like to create a Makefile - in the best way - to do the following:
make sure I include the common directory properly in TEXINPUTS
compile the handout.tex into a PDF (using either pdflatex or xelatex) and have it in the output directory as week01-handout-student.pdf
compile the handout.tex with a line of LaTeX prepended to the beginning of the file (that sets a flag) into a PDF and have it in the output directory as week01-handout-teacher.pdf
clean everything up (the log, aux, etc. files)
I am not sure I know how to this in any other way than manually duplicating an elementary Makefile/bash script in every subdirectory, then calling each of them one by one with a for loop.
I would appreciate help on how to build this process, ideally with a single Makefile in the root directory. Thanks.
UPDATE: I purposefully did not want to give any details about how I compile LaTeX, in case somebody has a better suggestion than my current usage. Right now I am using Latexmk (which is already a make-like wrapper of LaTeX):
latexmk -pdf file.tex generates file.pdf
to add the line of code, I do a simple echo "line of code" > temp.tex and cat handout.tex >> temp.tex, then the same latexmk command
latexmk -c file.tex in a directory cleans all temporary files used to compile file.tex
TEXINPUTS is the TeX path variable, to let TeX find (in its path) the style files: I do TEXINPUTS=full-path-to/common and then export TEXINPUTS before compiling anything.
If anybody has a better suggestion, I am a willing taker.
Something like this should do what you want I believe:
OUTDIR := output
# Tell make to export this environment variable to all recipe lines it runs.
export TEXINPUTS := $(abspath common)
# Get the list of all of our week directories.
weekdirs := $(wildcard week*)
#$(info weekdirs:$(weekdirs))
# Create student output filenames from week directory names.
STUDENT_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-student.pdf,$(weekdirs))
#$(info STUDENT_HANDOUTS:$(STUDENT_HANDOUTS))
# Create teacher output filenames from week directory names.
TEACHER_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-teacher.pdf,$(weekdirs))
#$(info TEACHER_HANDOUTS:$(TEACHER_HANDOUTS))
# Default target depends on all output files.
all: $(STUDENT_HANDOUTS) $(TEACHER_HANDOUTS)
# Pattern rule for building pdf files.
%.pdf:
#echo + Making $# from $^
#echo cd $(#D) && echo latexmx -pdf $(abspath $<)
#echo cd $(#D) && echo latexmk -c $(abspath $<)
# Static pattern rule mapping student output files to input files.
$(STUDENT_HANDOUTS) : $(OUTDIR)/%-handout-student.pdf : %/handout.tex
# Pattern rule to generate temporary input files from original input files.
%/handout-tmp.tex: %/handout.tex
#echo echo 'line of code' '>' $#
#echo cat $^ '>>' $#
# Static pattern rule mapping teacher output files to (temporary) input files.
$(TEACHER_HANDOUTS) : $(OUTDIR)/%-handout-teacher.pdf : %/handout-tmp.tex
Uncomment the $(info) lines to see a bit of how the variables are put together.
This uses latexmk -c to clean up auxiliary files after creating the output files.
Technique 1. The echo foo >$#; cat $< >>$# thing is something I've done before; I think it's reasonable.
Technique 2. Another technique is to have your .tex document, or a package file it uses, include a line like:
\InputIfFileExists{lecturenotes.config}{}{}
That allows you to make certain adjustments in a makefile, as in for example:
# any bits of configuration that should be in all the .config files
SWITCHES=\makeindex
%-notes.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{sidenotes}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
%-single.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{oneside}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
Technique 3. A third technique for controlling LaTeX from outside is to include in your document (or in a package file):
\newif\ifdemonstrator
\expandafter\ifx\csname demonstrator\endcsname\relax
\demonstratorfalse
\else
\demonstratortrue
\fi
...
\ifdemonstrator
\typeout{Demonstrator is TRUE}
\else
\typeout{Demonstrator is FALSE}
\fi
Then call latex with:
%-plain.pdf: %.tex
latex $<
mv ${<:.tex=.pdf} $#
%-demo.pdf: %.tex
latex --jobname ${<:.tex=} '\def\demonstrator{x}\input{$<}`
mv ${<:.tex=.pdf} $#
Technique 1 is a bit of a blunt instrument, but if that's all that's needed, it's pretty lightweight.
Technique 2 is probably the most neatly engineered, but it is slightly more effort.
Technique 3 is probably the one I use most in this sort of circumstance.