I am looking for an alternative for an existing make-based build system that contains multiple tasks for copyiing, packaging, zipping/unzipping actions for various consumers.
Make has 'phony' tasks with a name, that can depend on multiple files that are generated according to rules given in the makefile.
pseudo-code Example in makefile:
file123: input1 input2
zip input1 input2
filexyz: input11 input12
zip input11 input12
.PHONY: command1
command1: file123 filexyz
copy file123 targetdir
copy filexyz targetdir
executing "make command1", make would generate file123 and filexyz according to the rules given.
command1 is a phony target: there is no real file command1.
How could I define such tasks in waf? I would like to see commands that have files as inputs.
Related
I need to copy YUV sequence from 3 different locations from the server and put at common input folder in my file structure using make file.
I wrote a target in make file which works good for copying input from one server location only.
VID_INPUT_YUV_PATH = ../input
VID_INPUT_YUV_FILES = \
input0.yuv
VID_INPUT_YUV = $(addprefix $(VID_INPUT_YUV_PATH)/, $(VID_INPUT_YUV_FILES))
VID_INPUT_NOCLEAN = $(addprefix -e , $(VID_INPUT_YUV))
.PRECIOUS: $(VID_INPUT_YUV_PATH)/%.
$(VID_INPUT_YUV_PATH)/%:
mkdir -p $(VID_INPUT_YUV_PATH)
scp server:/stdDataset/airbus/$(notdir $#) $#
Expectation:
Copy input0 from server:/stdDataset/airbus to input folder
Copy input1 from server:/stdDataset/train to input folder
Copy input2 from server:/stdDataset/car to input folder
Query:
Is there any optimized way instead of writing separate target (3 make target) for copying all 3 at one location.
Since there is no algorithmic way to convert the file names into the directory names, you'll have to tell make about that mapping at least.
One way to do it is via variable names. You can do something like this:
input0_DIR = airbus
input1_DIR = train
input2_DIR = car
$(VID_INPUT_YUV_PATH)/%:
mkdir -p $(#D)
scp server:/stdDataset/$($(#F)_DIR)/$(#F) $#
I don't know what that .PRECIOUS line is for; doesn't seem useful to me.
I'm trying to make a make file for a static page generator, and I'm using jinja and pandoc so far, and the idea is to have a file structure as such
.
|-content
|-public
|-templates
|-Makefile
VPATH=public
TARGETS=$(find content -regex ".*(htm|md)" | sed -e "s/md$/htm/g;s/^content/public/g")
all: $(TARGETS)
#echo fullbuild
public/%: content/%
content/%.md:
# Pandoc script
pandoc -i $# -o ${${#:.md=.htm}:content=public}
content/%.htm:
# Jinja Script
The problem I'm having (At least I think that's it) is that according to me the syntax is
# For a final result
target: dependency
commands
# A rule for dependency
dependency:
commands
My dependencies are in the content dir and my targets are in the public dir that may or may not exist yet, and almost all the files I generate will be htm files, and in that case if the target is public/some/route/to/file.htm the dependency will be any of this two content/some/route/to/file.(htm|md).
I can easily generate by walking the content dir, and changing extensions.
How should I write the rules properly, so
- Make knows where to "watch" for changes every time I do a make, because right now it points that every file is up to date
- How do I indicate properly the dependency of a file to it's content file.
This rule:
public/%: content/%
does nothing because pattern rules without recipes delete existing pattern rules, they don't define new pattern rules.
It's very simple, you should write two rules like this:
public/%.htm: content/%.md:
# Pandoc script
pandoc -i $< -o $#
public/%.htm: content/%.htm
# Jinja Script
Here's a hint: whenever you're writing a makefile recipe and you discover that you need to create a target which is different than exactly $#, unmodified, immediately stop what you're doing and back up: you've taken a wrong turn.
I'm working with a makefile system using mcs as its build command. Briefly, mcs is invoked with:
source files: Passed directly;
resources: with -resource:;
a target: with -target: and can be exe, winexe, library or module.
And produces an *.exe, *.dll or *.netmodule. The -target: parameter is certain in the last two cases but an *.exe can be built with either exe or winexe.
Currently, each directory's makefile includes what boils down to this:
RESOLVED_SOURCES := $(addprefix $(srcdir)/,$(SOURCES))
RESOLVED_RESOURCES := $(addprefix $(srcdir)/,$(RESOURCES))
S_PARAM := $(RESOLVED_SOURCES)
R_PARAM := $(addprefix -resource:,$(RESOLVED_RESOURCES))
all: $(OUTPUT)
$(OUTPUT): $(RESOLVED_SOURCES) $(RESOLVED_RESOURCES)
$(MCS) $(MCSARGS) $(S_PARAM) $(R_PARAM) -target:$(TARGET) -out:$(OUTPUT)
And in each project directory the variables OUTPUT, TARGET, SOURCES and RESOURCES are defined before inclusion.
Even with everything I've left out it's fairly complicated and suffers from only being able to build a single concrete target where it's included, without duplicating the build rule.
To avoid the duplication and increase flexibility I'd like to define some pattern rules for this complex build command but it's difficult:
each compilation unit generally has many inputs;
there's no way to incrementally add each input;
some inputs need different handling.
e.g. For mcs these patterns seem possible:
%.exe: %.cs
$(MCS) $(MCSARGS) -target:$(target) -out:$# $<
%.dll: %.cs
$(MCS) $(MCSARGS) -target:library -out:$# $<
%.netmodule: %.cs
$(MCS) $(MCSARGS) -target:module -out:$# $<
As pattern rules go they aren't too bad but they aren't useful for most applications of mcs as there's only a single source file being passed.
If the user adds source file prerequisites for a target then the above $< could be swapped for $^ to pass them all but that won't work for resource files. The order only list could be used for passing resource file prerequisites but would prevent rebuilds when only resource files were updated.
Is there a general technique for turning this sort of complex command's invocation into a pattern rule or set of rules?
There are a number of problems:
the build command distinguishes more than one prerequisite type;
they must all be passed at once;
updates to any of them must cause a rebuild, they are all regular prerequisites.
Ideas:
delegating the inputs you want to separate to input files and then reading them back in allows you to have a simple target invoking the build;
using secondary expansion you can define those input files based on the contents of "pattern variables";
general search should work.
For example:
multi_A := foo
multi_B := bar
all: multi.foo
%.foo: .a.% .b.%
#echo $*: $(foreach _,$^,$(file < $_))
#touch $#
.SECONDEXPANSION:
.a.%: $$($$*_A)
$(file > $#,$^)
.b.%: $$($$*_B)
$(file > $#,$(addprefix -b:,$^))
To start with using secondary expansion seems dubious. Apart from that the immediate issue with this is that the user has to know that any regular prerequisites will be read and passed to the build command.
On the other hand, knowing that, the user can supplement a target with extra input parameters by depending on a target that writes them to another input file.
Pretty new to GNU Make. This is a less complex example of something more general I have been trying to get to work.
I have many input files that have similar name format .txt, and I have a shell script that will take the input file and generate an output of the same name but with a different extension .wc. I have written the following Make file.
# name of dependencies
SRC = $(wildcard *.txt)
# get name of targets (substitute .wc for .txt)
TAR = $(SRC:.txt=.wc)
all: $(TAR)
%.wc: %.txt
sh word_count.sh $<
This runs fine, and will generate all the .wc output files. However, if I modify one of the input(dependency) files, they are all rebuilt. So the question is; what is the best way to get GNU Make to only process the modified .txt files in the directory?
I'm trying to use the linux utility make in order to
Run a script to generate the data
Take all of the output files (data1.txt to data79.txt) and run a script to plot them each
Take all those images and make a movie from them
Yes, I realize that doing this in a shell/python script would be downright simple but I'm trying to learn how to use make in this context to do the work more intelligently.
My current make file looks something like this but is significantly flawed:
movie: data *.png
ffmpeg data_%d.png output.mp4
%.png: %.txt
python plot.py $< $#
data:
python make_data.py
You have several problems, so let's take them in order. (Caveats: I use GNUMake, so I can't promise my solution will work with other flavors, and I am not familiar with ffmpeg.)
The data rule looks correct, but you might want to warn Make that this rule does not actually produce a file called "data":
.PHONY: data
You can test this rule by itself: "make data".
The %.png rule looks correct. You can test it: "make data26.png" (after making sure that data26.txt exists).
The movie rule. This is a problem, because you're using "*.png" to indicate all png files, but at the time you run the rule there are no such files, so this evaluates to nothing. So we must look at all the data files that exist, and translate that list into a list of images to be made:
dfiles = $(wildcard *.txt)
images = $(dfiles:txt=png)
This will work if the data files already exist (and you can test it after "make data"), but when we first run make, the data files don't exist. There are several ways to address this; the simplest is to run Make a second time from within a rule, after the data files have been made:
$(MAKE) output.mp4
Putting it all together, we get something like this:
.PHONY: movie
movie: data
#$(MAKE) -s output.mp4 # I added the "#" and "-s" to make it quieter.
dfiles = $(wildcard *.txt)
images = $(dfiles:txt=png)
output.mp4: $(images)
ffmpeg data_%d.png $#
%.png: %.txt
python plot.py $< $#
.PHONY: data
data:
python make_data.py
(Note that some people like to put all the PHONY declarations together: ".PHONY: movie data". I prefer to do it as above.)