I would like to have data processing performed when launching make and one or more files in the input directory have changed. With my current Makefile processing is not triggered.
Let's say, I'm setting up an exemplary project: two directories, one with data files. Use this script:
#!/bin/bash
mkdir -p proj/input
mkdir -p proj/output
cd proj/input
echo "a2018" > 2018a.txt
echo "b2018" > 2018b.txt
echo "a2019" > 2019a.txt
echo "b2019" > 2019b.txt
My Makefile:
#proj/Makefile
RAW := $(shell find input -name "*.txt")
OUT := ./output
.PHONY: all
all: $(OUT)/*.txt
echo "Running processing of raw files"
$(OUT)/2018merged.txt: $(RAW)
./merge.sh 2018
$(OUT)/2019merged.txt: $(RAW)
./merge.sh 2019
The merge script does basic concatenation of files by year from the filename and saves the result in the file in the output directory.
#!/bin/bash
# proj/merge.sh
echo "-- merging files for $1"
cat input/$1*.txt > "./output/$1merged.txt"
I believed that providing all files in the input directory as a prerequisite will be sufficient but apparently I'm doing something wrong.
I found few questions around similar poblems and partial solutions might be there:
Processing multiple files generated from single input,
Make: How to process many input files in one invocation of a tool?,
Make dummy target that checks the age of an existing file?.
Consider this rule:
all: $(OUT)/*.txt
echo "Running processing of raw files"
This says, all depends on all the files in the $(OUT) directory that match the pattern *.txt.
Well, of course before you've run your makefile there are no files: that's the whole point of your makefile to create them. So when you first run make that pattern expands to nothing, and thus there are no prerequisites for all, and thus nothing is done.
If you want to construct the list of targets to build you have to do it based on the source files, which will always exist, or in this case where they don't match up exactly you have to list them explicitly:
YEARS = 2018 2019
all: $(patsubst %,$(OUT)/%.txt,$(YEARS))
echo "Running processing of raw files"
Related
Thanks for all your time and response -
Currently, we are using the nested build, multiple Makefiles, and individual subdirectories having their own Makefile, all are connected with a top-level Makefile. We are running
make xxxxx_yyyy_defconfig
make
this builds and creates an output file which is xxxxx.elf file. --- Till here everything works fine.
Now we are having multiple def-configs(around 50), I want to build all configurations using one "make all" command. is that possible?
This is not a simple case where we can put all "all: prog01 prog02 prog03" as every program needs to have a different configuration. Configuration can be achieved by using "make xxxxx_yyyy_defconfig". The output of "make config" is the .config file, which is used during the "make" command.
Based on .config file many variables are exported which is used at the subdirectory level.
So How can I build multiple configurations using a single "make all" command?
Environment - Ubuntu, Cross compile for ARM, output file xxxx.elf.
With the use of script and make file I am able to solve, But I have to solve only using Makefile.
in Makefile add one PHONY target
all:
./build_all.sh #shell script calling.
Created one shell script like this
#! /usr/bin/bash
echo "Make All"
for entry in `ls conf`; do
make $entry
wait
make
if [ $? -eq 0 ] ; then
for xxxfile in `ls xxx*_*` ; do
xxxdir=$(echo $xxxfile | cut -b yy-zz)
mkdir -p $xxxdir
mv $xxxfile $xxxdir/
done
else
break
fi
done
If you want to build several configurations you must do this out of tree in separate build directories (make O=/tmp/builds/foo foo_defconfig; make -C /tmp/builds/foo) to avoid conflicts. A shell script could do this as well as a Makefile but if you insist on using a Makefile you could try the following that assumes your source tree is in /src/kernel and you want to build configuration foo in /tmp/builds/foo; adapt to your needs:
$ pwd
/tmp/builds
$ cat Makefile
CONFIGS := uuuu_vvvv xxxx_yyyy ...
BUILD := /tmp/build
KERNEL := /src/kernel
.PHONY: $(CONFIGS) all
all: $(CONFIGS)
$(CONFIGS):
rm -rf $#
mkdir -p $(BUILD)/$#
$(MAKE) -C $(KERNEL) O=$(BUILD)/$# $#_defconfig
$(MAKE) -C $(BUILD)/$#
$ make
I have a metadata dotfile which I store alongside my code (in each dir of a large project). This metadata file has a list of files in each directory that satisfies a particular constraint (not being auto-generated). This metadata is sincluded in Makefile.
I want to dep other targets (the auto-generated files) on this metadata file. If ever the list of "real" code files in a dir changes, update this metadata file, which will in turn cause the auto-generated files (plural) to be re-made.
I have a rule for the metadata file, and when that rule fires make restarts correctly. But I can't quite figure how to describe what I want. I want the rule to run but only consider $# as having been changed iff I actually touch the file. I can't use the timestamp of the dir because the act of auto-generating file A causes the timestamp to change, which triggers the need to re-generate file B, which cause the timestamp to change ...
I feel like I am missing something obvious, but I can not put my finger on it...
all: prep my-bin
# For demonstration purposes.
prep: real-code.foobar
real-code.foobar:
#touch real-code.foobar
my-bin: meta real-code.foobar genfile-A.foobar genfile-B.foobar genfile-C.foobar
touch $#
meta: .
F=$$(ls *.foobar | grep -v genfile); \
echo "FILES := $$F" > $#
sinclude meta
genfile-A.foobar: meta
touch $#
genfile-B.foobar: meta
touch $#
genfile-C.foobar: meta
touch $#
clean:
rm -f *.foobar my-bin meta
You could update the meta in a shell script at the beginning of the Makefile:
$(shell ls *.foobar | grep -v genfile > meta.tmp; \
diff -q meta.tmp meta && mv meta.tmp meta || rm meta.tmp)
This way the timestamp of meta is only updated if it changes, and it is updated before make has decided which rules to run (meaning dependencies of meta will not automatically rerun).
John
I have the following folder structure
1st-grade-math-class/
common/
mystyle.sty
mysubstyle.sty
fonts/
font1.ttf
font2.ttf
font3.ttf
week01/
handout.tex
image1.pdf
image2.pdf
week02/
handout.tex
image1.pdf
image2.pdf
...
week13/
handout.tex
output/
[empty]
And I would like to create a Makefile - in the best way - to do the following:
make sure I include the common directory properly in TEXINPUTS
compile the handout.tex into a PDF (using either pdflatex or xelatex) and have it in the output directory as week01-handout-student.pdf
compile the handout.tex with a line of LaTeX prepended to the beginning of the file (that sets a flag) into a PDF and have it in the output directory as week01-handout-teacher.pdf
clean everything up (the log, aux, etc. files)
I am not sure I know how to this in any other way than manually duplicating an elementary Makefile/bash script in every subdirectory, then calling each of them one by one with a for loop.
I would appreciate help on how to build this process, ideally with a single Makefile in the root directory. Thanks.
UPDATE: I purposefully did not want to give any details about how I compile LaTeX, in case somebody has a better suggestion than my current usage. Right now I am using Latexmk (which is already a make-like wrapper of LaTeX):
latexmk -pdf file.tex generates file.pdf
to add the line of code, I do a simple echo "line of code" > temp.tex and cat handout.tex >> temp.tex, then the same latexmk command
latexmk -c file.tex in a directory cleans all temporary files used to compile file.tex
TEXINPUTS is the TeX path variable, to let TeX find (in its path) the style files: I do TEXINPUTS=full-path-to/common and then export TEXINPUTS before compiling anything.
If anybody has a better suggestion, I am a willing taker.
Something like this should do what you want I believe:
OUTDIR := output
# Tell make to export this environment variable to all recipe lines it runs.
export TEXINPUTS := $(abspath common)
# Get the list of all of our week directories.
weekdirs := $(wildcard week*)
#$(info weekdirs:$(weekdirs))
# Create student output filenames from week directory names.
STUDENT_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-student.pdf,$(weekdirs))
#$(info STUDENT_HANDOUTS:$(STUDENT_HANDOUTS))
# Create teacher output filenames from week directory names.
TEACHER_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-teacher.pdf,$(weekdirs))
#$(info TEACHER_HANDOUTS:$(TEACHER_HANDOUTS))
# Default target depends on all output files.
all: $(STUDENT_HANDOUTS) $(TEACHER_HANDOUTS)
# Pattern rule for building pdf files.
%.pdf:
#echo + Making $# from $^
#echo cd $(#D) && echo latexmx -pdf $(abspath $<)
#echo cd $(#D) && echo latexmk -c $(abspath $<)
# Static pattern rule mapping student output files to input files.
$(STUDENT_HANDOUTS) : $(OUTDIR)/%-handout-student.pdf : %/handout.tex
# Pattern rule to generate temporary input files from original input files.
%/handout-tmp.tex: %/handout.tex
#echo echo 'line of code' '>' $#
#echo cat $^ '>>' $#
# Static pattern rule mapping teacher output files to (temporary) input files.
$(TEACHER_HANDOUTS) : $(OUTDIR)/%-handout-teacher.pdf : %/handout-tmp.tex
Uncomment the $(info) lines to see a bit of how the variables are put together.
This uses latexmk -c to clean up auxiliary files after creating the output files.
Technique 1. The echo foo >$#; cat $< >>$# thing is something I've done before; I think it's reasonable.
Technique 2. Another technique is to have your .tex document, or a package file it uses, include a line like:
\InputIfFileExists{lecturenotes.config}{}{}
That allows you to make certain adjustments in a makefile, as in for example:
# any bits of configuration that should be in all the .config files
SWITCHES=\makeindex
%-notes.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{sidenotes}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
%-single.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{oneside}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
Technique 3. A third technique for controlling LaTeX from outside is to include in your document (or in a package file):
\newif\ifdemonstrator
\expandafter\ifx\csname demonstrator\endcsname\relax
\demonstratorfalse
\else
\demonstratortrue
\fi
...
\ifdemonstrator
\typeout{Demonstrator is TRUE}
\else
\typeout{Demonstrator is FALSE}
\fi
Then call latex with:
%-plain.pdf: %.tex
latex $<
mv ${<:.tex=.pdf} $#
%-demo.pdf: %.tex
latex --jobname ${<:.tex=} '\def\demonstrator{x}\input{$<}`
mv ${<:.tex=.pdf} $#
Technique 1 is a bit of a blunt instrument, but if that's all that's needed, it's pretty lightweight.
Technique 2 is probably the most neatly engineered, but it is slightly more effort.
Technique 3 is probably the one I use most in this sort of circumstance.
I'm trying to create a Makefile that will download and process file a file to generate targets, this is a simplified version:
default: all
.PHONY: all clean filelist.d
clean:
#rm -fv *.date *.d
#The actual list comes from a FTP file, but let's simplify things a bit
filelist.d:
#echo "Getting updated filelist..."
#echo "LIST=$(shell date +\%M)1.date $(shell date +\%M)2.date" > $#
#echo 'all: $$(LIST)' >> $#
%.date:
touch $#
-include filelist.d
Unfortunately the target all doesn't get updated properly on the first run, it needs to be run again to get the files. This is the output I get from it:
$ make
Getting updated filelist...
make: Nothing to be done for `default'.
$ make
Getting updated filelist...
touch 141.date
touch 142.date
touch 143.date
I'm using GNU Make 3.81 whose documentation states that it reloads the whole thing if the included files get changed. What is going wrong?
You have specified filelist.d as a .PHONY target, so make believes making that target doesn't actually update the specified file. However, it does, and the new contents are used on the next run. For the first run, the missing file isn't an error because include is prefixed with the dash.
Remove filelist.d from .PHONY. However, remember it won't be regenerated again until you delete it (as it doesn't depend on anything).
By the same token, you should include "default" in .PHONY.
I wrote a shell script rather than lump all this in the makefile:
#!/bin/bash
# Check whether file $1 is less than $2 days old.
[ $# -eq 2 ] || {
echo "Usage: $0 FILE DAYS" >&2
exit 2
}
FILE="$1"
DAYS="$2"
[ -f "$FILE" ] || exit 1 # doesn't exist or not a file
TODAY=$(date +%s)
TARGET=$(($TODAY - ($DAYS * 24 * 60 * 60)))
MODIFIED=$(date -r "$FILE" +%s)
(($TARGET < $MODIFIED))
Replace X with the max number of days that can pass before filelist.d is downloaded again:
filelist.d: force-make
./less-than-days $# X || command-to-update
.PHONY: force-make
force-make:
Now filelist.d depends on a .PHONY target, without being a phony itself. This means filelist.d is always out of date (phony targets are always "new"), but its recipe only updates the file periodically.
Unfortunately, this requires you to write the update command as a single command, and space may be a problem if it is long. In that case, I would put it in a separate script as well.
In my GNUmakefile, I would like to have a rule that uses a temporary directory. For example:
out.tar: TMP := $(shell mktemp -d)
echo hi $(TMP)/hi.txt
tar -C $(TMP) cf $# .
rm -rf $(TMP)
As written, the above rule creates the temporary directory at the time that the rule is parsed. This means that, even I don't make out.tar all the time, many temporary directories get created. I would like to avoid my /tmp being littered with unused temporary directories.
Is there a way to cause the variable to only be defined when the rule is fired, as opposed to whenever it is defined?
My main thought is to dump the mktemp and tar into a shell script but that seems somewhat unsightly.
In your example, the TMP variable is set (and the temporary directory created) whenever the rules for out.tar are evaluated. In order to create the directory only when out.tar is actually fired, you need to move the directory creation down into the steps:
out.tar :
$(eval TMP := $(shell mktemp -d))
#echo hi $(TMP)/hi.txt
tar -C $(TMP) cf $# .
rm -rf $(TMP)
The eval function evaluates a string as if it had been typed into the makefile manually. In this case, it sets the TMP variable to the result of the shell function call.
edit (in response to comments):
To create a unique variable, you could do the following:
out.tar :
$(eval $#_TMP := $(shell mktemp -d))
#echo hi $($#_TMP)/hi.txt
tar -C $($#_TMP) cf $# .
rm -rf $($#_TMP)
This would prepend the name of the target (out.tar, in this case) to the variable, producing a variable with the name out.tar_TMP. Hopefully, that is enough to prevent conflicts.
A relatively easy way of doing this is to write the entire sequence as a shell script.
out.tar:
set -e ;\
TMP=$$(mktemp -d) ;\
echo hi $$TMP/hi.txt ;\
tar -C $$TMP cf $# . ;\
rm -rf $$TMP ;\
I have consolidated some related tips here: Multi-line bash commands in makefile
Another possibility is to use separate lines to set up Make variables when a rule fires.
For example, here is a makefile with two rules. If a rule fires, it creates a temp dir and sets TMP to the temp dir name.
PHONY = ruleA ruleB display
all: ruleA
ruleA: TMP = $(shell mktemp -d testruleA_XXXX)
ruleA: display
ruleB: TMP = $(shell mktemp -d testruleB_XXXX)
ruleB: display
display:
echo ${TMP}
Running the code produces the expected result:
$ ls
Makefile
$ make ruleB
echo testruleB_Y4Ow
testruleB_Y4Ow
$ ls
Makefile testruleB_Y4Ow
I dislike "Don't" answers, but... don't.
make's variables are global and are supposed to be evaluated during makefile's "parsing" stage, not during execution stage.
In this case, as long as the variable local to a single target, follow #nobar's answer and make it a shell variable.
Target-specific variables, too, are considered harmful by other make implementations: kati, Mozilla pymake. Because of them, a target can be built differently depending on if it's built standalone, or as a dependency of a parent target with a target-specific variable. And you won't know which way it was, because you don't know what is already built.