This is a GNU Make dependency resolution issue.
My problem is that I copy the source files from a remote file server to a scratch disk (which speeds up the build process by 50%). If the file copy fails, I want to use the source files from the file server, else I want to read them from the scratch disk.
I have tried to use the vpath mechanism, but the problem is that, as far as I understand, make will by default start looking for the source files in the current directory and only if it fails to find the files there, look in the directories listed with vpath.
Is it possible to have make first look in the vpath directories before looking in the current directory? Or perhaps only look in the vpath directories (and explicitly and dynamically add the current directory to vpath)?
Only way I can think of is to temporary change directory so that make always will fail to find the source files.
Look at the path to the source directory on the server. Suppose it's "/server/someplace/src/". And suppose you don't have a "src" directory in the current directory (if you do, we just have to tweak this method). Just make sure that the path to the source directory on the scratch disk ends in "/src/", such as "/scratch/wednesday/src/". Then you can do this:
SCRATCH_PATH = /scratch/wednesday/
SERVER_PATH = /server/someplace/
VPATH = $(SCRATCH_PATH) $(SERVER_PATH)
%.o: src/%.cc
$(CC) blah blah blah
Related
In my project, I have a set of sub-directories that contain package.yaml files, for e.g.:
A/package.yaml
B/package.yaml
C/package.yaml
If I run hpack A/package.yaml, the file A/A.cabal is (re-)generated. The list of such directories can change over time, so I want to use GNU make to find all immediate sub-directories containing package.yaml files and generate the corresponding .cabal files using hpack.
I tried this based on another question, but it didn't work:
HPACK_FILES := $(wildcard */package.yaml)
PKG_DIRS := $(subst /,,$(dir $(HPACK_FILES)))
CABAL_FILES := $(addsuffix .cabal,$(join $(dir $(HPACK_FILES)),$(PKG_DIRS)))
test:
#echo $(CABAL_FILES)
update-cabal: $(CABAL_FILES)
%.cabal: package.yaml
hpack $<
However, make update-cabal says there's nothing to be done. make test however does output the right cabal files. How can I fix this?
Cheers!
The problem is this:
%.cabal: package.yaml
There is no file package.yaml. The files are named things like A/package.yaml. That is not the same thing.
Because the prerequisite doesn't exist, make decides that this pattern rule cannot match and so it goes looking for another rule that might be able to build the target. It doesn't find any rule that can build the target, so make says there's nothing to do because all the output files already exist.
Unfortunately what you want to do is not at all easy with make, because make is most comfortable with input and output files that are tied together by the filename with extensions, or similar. And in particular, it has a really hard time with relationships where the variable part is repeated more than once (as in, A/A.cabal where the A is repeated). There's no easy way to do that in make.
You'll have to use an advanced feature such as eval to do this. Something like:
# How to build a cabal file
%.cabal:
hpack $<
# Declare the prerequisites
$(foreach D,$(dir $(HPACK_FILES)),$(eval $D/$D.cabal: $D/package.yml))
I am trying to understand a makefile from a project I am working on. It's using automake/autotools and contains this simple rule:
DEPS_SRC = $(shell cd $(srcdir); find . -name '*.ez')
DEPS = $(basename $(DEPS_SRC))
all : $(DEPS)
$(DEPS) : % : %.ez
$(UNZIP) -o $<
Say the directory structure is:
my-app/deps/
build/
When executing make my-app in the build folder the rule will basically unpack *.ez files located in the my-app/deps/ folder into build/my-app/deps/ folder.
I don't know if that's enough information to solve the problem that I am going to explain as I don't know enough about automake/autotools. Please ask if any additional information is needed.
The problem is that I noticed that having the unpacked directory in the source folder prevents make from unpacking the archive in the target folder. For example, given the following structure in the source folder:
my-app/deps/archive1.ez
my-app/deps/archive2.ez
my-app/deps/archive2/
make will only unpack archive1.ez in the build folder:
build/my-app/deps/archive1/
I would like to know if this is a bug in my makefile or a feature of automake. If the later, is there any workaround or setting or variable available to disable it?
This is primarily a GNU make question, not particularly specific to the Autotools. However, since the target system's make is of GNU's flavor (else none of this works), we can assume that the Makefile generated by configure uses GNU make's VPATH feature as part of its support for out-of-source (a.k.a. VPATH) builds such as the one you are performing.
The value of the VPATH variable that configure will have inserted into the Makefile is used as a search path for prerequisites that are not found relative to the build directory. The key point, however, is that it is also used as a search path for rule targets. That makes a certain amount of sense, especially for targets that are prerequisites of other rules.
In your case, however, that leads directly to the behavior you describe:
the default target depends on ./my-app/deps/archive2
resolving that name against the build directory does not produce a valid file name
before attempting to build that target, make looks in the directories listed in the VPATH, which, in your example case, will contain .. or an equivalent
make finds .././my-app/deps/archive2 in this VPATH search, and therefore determines that the specified target already exists, and does not need to be built
Thus, the behavior you observe is normal for GNU make, supposing the Makefile is constructed by Autoconf from an Automake-generated template.
is there any workaround or setting or variable available to disable it?
Do you really need one? If the archive file has already been unpacked in the source tree, then you can expect make to find its contents, too, via the VPATH. At least if the Makefile is well-prepared overall for for out-of-source builds.
But if you want to be certain to get the archive files unpacked in the build directory then you can specify that explicitly. This ought to do it:
DEPS_SRC = $(shell cd $(srcdir); find . -name '*.ez')
DEPS = $(basename $(DEPS_SRC))
LOCAL_DEPS = $(addprefix $(abs_builddir)/,$(DEPS))
all : $(LOCAL_DEPS)
$(LOCAL_DEPS) : $(abs_builddir)/% : %.ez
$(UNZIP) -o $<
That prefixes the name of each dependency with the absolute path to the build directory, and updates the rule for unzipping the archives to accommodate it. Targets with absolute paths such as that cannot be located in the VPATH.
GNU Make 3.8.1
I am working on what is basically a plugin for a software product. The plugin is built in a subdirectory of the main product, so the directory structure looks something like:
product
product/src
product/plugin
product/plugin/myPlugin
where "myPlugin" is the name of the plugin I'm working on. What I would like to do, is be able to build myPlugin as well as another version of myPlugin, call it myPlugin-lite. This plugin would have the same sources as myPlugin, but use different flags and defines in the makefiles. The idea was to create a duplicate of the myPlugin tree structure, containing only myPlugin's makefiles, and have it build using the sources from myPlugin. So the directory structure would look like:
product
product/src
product/plugin
product/plugin/myPlugin
product/plugin/myPlugin-lite
myPlugin would build and create all its targets within its subdirectory, and myPlugin-lite would build and create all its targets within its subdirectory. I found a few solutions here: http://make.mad-scientist.net/papers/multi-architecture-builds/ and out of these options it seems like the symbolic links would be best, but it still doesn't feel like the "right" way to do this.
My question is, is this the best/simplest/most maintainable way to do this? If not what are the alternatives?
If the only thing you need from myPlugin is source files then this is exactly what VPATH and The vpath Directive are for.
4.5.1 VPATH: Search Path for All Prerequisites
The value of the make variable VPATH specifies a list of directories that make should search. Most often, the directories are expected to contain prerequisite files that are not in the current directory; however, make uses VPATH as a search list for both prerequisites and targets of rules.
Thus, if a file that is listed as a target or prerequisite does not exist in the current directory, make searches the directories listed in VPATH for a file with that name. If a file is found in one of them, that file may become the prerequisite (see below). Rules may then specify the names of files in the prerequisite list as if they all existed in the current directory. See Writing Recipes with Directory Search.
In the VPATH variable, directory names are separated by colons or blanks. The order in which directories are listed is the order followed by make in its search. (On MS-DOS and MS-Windows, semi-colons are used as separators of directory names in VPATH, since the colon can be used in the pathname itself, after the drive letter.)
For example,
VPATH = src:../headers
specifies a path containing two directories, src and ../headers, which make searches in that order.
With this value of VPATH, the following rule,
foo.o : foo.c
is interpreted as if it were written like this:
foo.o : src/foo.c
assuming the file foo.c does not exist in the current directory but is found in the directory src.
Also see How Not to Use VPATH from MadScientist for more discussion about what they aren't for. Though that's largely just a build-up for the multi-architecture-builds paper you already read.
If I have some source files in another directory and I want to make some libraries using the source files, I want to have the *.o, *.po, and *.So files in the local directory. It isn't clear how to accomplish this. The transformation rules in bsd.lib.mk always point the .o into the source directory.
How do I get the results of the ${CC} -c to be in the local directory?
The file bsd.lib.mk inclused bsd.obj.mak, so you can use the MAKEOBJDIR environment variable.
Edit: If you cannot control how make is called, then don't use bsd.lib.mk and make explicit dependencies in your Makefile;
foo.o: ../bar/foo.c
One rule in my Makefile zips an entire directory (res/) into a ZIP file. Obviously, this rule needs to execute when any file under the res/ directory changes. Thus, I want the rule to have as a prerequisite all files underneath that directory. How can I implement this rule?
In Bash with the globstar option enabled, you can obtain a list of all the files in that directory using the wildcard pattern res/**/*. However, it doesn't seem to work if you specify it as a prerequisite in the Makefile:
filename.jar: res/**/*
Even after touching a file in res/, Make still reports
make: `filename.jar' is up to date.
so clearly it is not recognizing the pattern.
If I declare the directory itself as a prerequisite:
filename.jar: res
then Make will not re-execute when a file is modified (I think make only looks at the modified date of the directory itself, which only changes when immediate children are added, removed, or renamed).
This:
filename.jar: $(wildcard res/**/*)
seems to work, at least on some platforms.
EDIT:
Or better, just cut the knot:
filename.jar: $(shell find res -type f)