I am having a problem creating Makefile build that has dependency of some other file content.
I know that I can write something like that in makefile to generate prerequisites on-the-fly:
result/nuclease.stat : $(shell cat config/nuclease.sample.list)
#bash
make result/nuclease.stat
But the problem is that the file config/nuclease.sample.list might not exist and I have a target for it.
If I could force make to require config/nuclease.sample.list before $(shell cat config/nuclease.sample.list) gets evaluated - problem is solved.
What kind of dependency I should make?
One more thing, I would like keep all gnu make's good qualities: download files only once, process them only once and etc..
Please check
The most straightforward answer is to use constructed include files. In this method you'd include a makefile that defined the dependency relationships you wanted, and provide a make rule that generated that included file with the information you want. That's all you have to do: make knows how to do the rest. Based on your question it might be something like:
include nuclease-sample-list.mk
nuclease-sample-list.mk: config/nuclease.sample.list
echo "result/nuclease.stat : $$(cat $<)" > $#
That's all you have to do. If you're using an older version of GNU make you might need to use -include instead of include to avoid seeing warning messages.
Related
In my program, I have a somewhat complicated build process. Currently, in one directory I use include in Makefile.am with a file, that does not exist but has to be build on its own. The reason is that this include file is quite long. Further in the real program it is not just only one file but several and the generation process for this file can change from time to time.
The Makefile.am looks something like this
noinst_LIBRARIES = libtest.a
nodist_libtest_a_SOURCES = file.c
CLEANFILES = file.c Make_file.mk
$(builddir)/Make_test.mk: $(srcdir)/Perl/generate_mk_files.pl
perl $(srcdir)/Perl/generate_mk_files.pl file
include $(builddir)/Make_file.mk
After creation of Make_file.mk it looks something like
$(builddir)/file.c: $(srcdir)/file.template $(srcdir)/Perl/generate_c.pl
perl $(srcdir)/Perl/generate_c.pl $(srcdir)/file.template
Automake works and the final build process as well. The output to make is something like (I have shorted it somewhat):
Makefile:721: Make_file.mk: Datei oder Verzeichnis nicht gefunden (file not found)
perl ../../../../src/components/test/Perl/generate_mk_files.pl test
perl ../../../../src/components/test/Perl/generate_c.pl ../../../../src/components/test/file.template
Therefore, make first complains that the include file is not found, then creates it and then also follows the rules of the included file.
Although I am happy that it works I wonder why. First, I thought that make loads the Makefile. During this step, Make_file.mk does not exists. Therefore it seems the Makefile is loaded more than once.
Further, the manual of Automake for include states:
Note that these fragments are read and interpreted by automake, not by
make.
Which is not what I see, since the included fragment does not exist during the execution of Automake.
My questions basically are:
Why does it work?
Is this the correct way to do this or should I use another approach, e.g. starting new instances of make within Makefile.
I don't really know Automake, but, from the GNU make manual :
If an included makefile cannot be found in any of these directories {standard includes directories} , a
warning message is generated, but it is not an immediately fatal
error; processing of the makefile containing the include continues.
Once it has finished reading makefiles, make will try to remake any
that are out of date or don’t exist. See How Makefiles Are Remade.
Only after it has tried to find a way to remake a makefile and failed,
will make diagnose the missing makefile as a fatal error.
If you want make to simply ignore a makefile which does not exist or
cannot be remade, with no error message, use the -include directive
instead of include, like this:
-include filenames…
This acts like include in every way except that there is no error (not even a warning) if any of the filenames (or any
prerequisites of any of the filenames) do not exist or cannot be
remade.
So basically, make cannot execute the recipe for remaking the include file before he has finished to parse the main Makefile. So it raises a warning, continue to read the Makefile, find the rule for remaking the included file, remake it, and then restart itself (that is explained in details in the How Makefiles Are Remade section).
Going back to the manual, it states there's two forms for automakes include mechanism:
include $(srcdir)/file
and
include $(top_srcdir)/file
neither of which match your include. So I'd imagine the include is actually run by the underlying make whatever that might be (e.g. GNU Make, though of course other make programs have this functionality as well).
Now for the questions:
Why does it work?
As explained in another answer, GNU Make will attempt to make a missing included makefile, before failing.
Is this the correct way to do this or should I use another approach, e.g. starting new instances of make within Makefile
Generating makefiles is one of the tasks autotools do, either through autoconf or automake. Going through multiple stages of "making makefiles" seems prone to error (and hard to maintain). Recursive make has similar problems.
The reason is that this include file is quite long.
automake include statements will happily paste together a large makefile out of smaller components.
Further in the real program it is not just only one file but several and the generation process for this file can change from time to time.
It's hard to recommend what to suggest to do in autotools based on how the "changes" are determined. Since it seems you're also using libtool adding/removing sources to libs (or entire libs) there can be effected by conditionals, variables, etc.
For a project of mine I am automatically generating makefiles and including them, like this:
all:
#echo 'SUCCESS is $(SUCCESS)'
clean:
rm depend.mk
depend.mk:
#echo 'Creating $#'
#echo 'SUCCESS := 1' > $#
.PHONY: all clean
include depend.mk
This works, but the include line generates a warning message:
$ make
Makefile:13: depend.mk: No such file or directory
Creating depend.mk
SUCCESS is 1
I would like to silence that first warning line saying that depend.mk doesn't exist. I know it doesn't exist since I have a rule written to generate it, so the warning is unnecessary (unless of course there isn't a rule for it). I do NOT want make to ignore the error where the included file doesn't exist and there is no rule for it, so prefixing include with a - to ignore the error will not work for me. I'd like something similar to bash's convention of piping stderr to /dev/null like some_cmd 2>/dev/null but for including in make.
The sample above is a very simplified example of this case. In my actual project there are a lot of automatically generated makefiles (via clang's automatic dependency generation) being included, meaning a fresh run of make will flood my screen with these warning messages.
Is anything like this possible, or am I just going to have to deal with the annoying warning messages?
I've encountered and (re-re-re-re-)solved this problem a number of times myself. Really, the problem is in the thinking surrounding when the dependency files are generated and used.
This link has the detailed description of the "resolution": http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
Basically it comes down to the fact that dependency files are really only necessary for rebuilding, not the initial building of your library/executable. Resultantly you don't need to have a rule for generating dependency files up front (which is in fact less efficient), you instead should generate them during the object file step as intermediate files marked precious (so they're created and tracked as side-effect files that should never be cleaned up automatically). Subsequent builds will then have the files available, which is exactly what you were trying to achieve overall. You can then make it a "-include" on the dependency files, with the foreknowledge that your object file build step will fail if the dependency file generation fails, giving an immediate error, as you've mentioned is preferred, rather than an obscure and indirect one much later.
I've actually done a couple rather large build systems implementing this method, and it does work quite well, including ones that used non-GNU toolchains. To an outside user it appears identical, but internally it performs more efficiently and isn't hiding potentially important errors.
I tried many (many!) things to see if I could prevent or redirect the error message. No luck.
But when I tried -include (include with a leading dash), it didn't give an error, and make with clean, all, depend.mk and 'default' all worked properly and as expected.
Is there a particular reason you didn't want to use the -include variant? Seems to do exactly what you're looking for, and doesn't alter how the Makefile works in any way, just doesn't show the error during the first pass through the Makefile.
My Makefile.am includes a file (with various defined variables), for example:
include make.config
...
The problem is that this file is in turn generated by a tool (i.e. config.generator.sh) based on some input file (i.e. input.dat). The straightforward and wrong idea would be to add a rule to generate make.config:
make.config : input.dat
config.generator.sh input.dat > make.config
include make.config
...
Although this content is perfectly working makefile on its own without automake, the idea is doomed with automake. The make.config file is included by automake before I even have a chance to execute make (and it fails as the file is not yet generated):
automake: cannot open < make.config: No such file or directory
Is there a way to postpone effect of include directive until make is run (possibly by using another directive)?
There is probably a way to simply run arbitrary commands before any makefile generation is done (i.e. AC_CONFIG_COMMANDS*). But the question is more complicated because the config.generator.sh is supposed to use executables which are in turn also generated during the same build process (so there is a dependency chain which logically has to be managed by makefiles from the same project). The documentation simply confirms the logic without providing alternatives.
The solution is described in this email of Automake's mailing list.
The idea is to use include directives inside small regular "wrapper" makefile and include Automake-generated Makefile into it (note the upper case M). Because makefile is not an Automake template, the include works as expected triggering builds for non-existing files.
Note that:
By default make utility will search for makefile first (not for Makefile) making this approach working seamlessly.
It is still recommended to specify all rules inside Makefile.am and keep the "wrapper" makefile simple. The rules for non-existing files will naturally come from the generated Makefile anyway.
I've come across the same annoying problem today when moving my OCaml project to Autotools. My solution is to use autoconf's substitution to go around automake. For the above example, I'd add a substitution to configure.ac:
AC_SUBST([include_make_config], ["include make.config"])
and adjust Makefile.am, replacing the include directive with the autoconf variable reference:
make.config : input.dat
config.generator.sh input.dat > make.config
#include_make_config#
...
automake doesn't touch the #include_make_config# line so it gets carried over into the generated Makefile.in. When autoconf takes over, it substitutes the variable with include make.config in the final Makefile.
Note: I use this with OCaml's ocamldep dependency generator.
In the event that a Makefile itself is changed, a safe bet would be to consider all targets out of date.
Is there a clever way to add this dependency? Are there any alternatives?
Make sure the object files depend on the makefile:
$(OBJFILES) : Makefile
Where Makefile is the name of the make file.
A safe bet, but a terrible idea. Example: you're using automake and update Makefile.am to add a single source file. The correct response is to compile just the new file and link it in. In your scheme everything would be rebuilt.
Moreover, adding the dependency isn't going to do anything unless you touch the file, something like:
$(SRCS): Makefile
touch $#
This will then trip up editors that use the mtime to detect concurrent modification (emacs is one example).
If you're doing something major, just run make clean all after doing the change.
Since GNU make version 4.3 it is now possible with the use of those two special variable:
.EXTRA_PREREQS
To add new prerequisite to every target
MAKEFILE_LIST
To get the path of the make file
To have every target depend on the current make file:
Put near the top of the file (before any include since it would affect the MAKEFILE_LIST) the following line:
.EXTRA_PREREQS:= $(abspath $(lastword $(MAKEFILE_LIST)))
To have every target depend on the current make file and also the make files which were included
Put the following line at the end of your file:
.EXTRA_PREREQS+=$(foreach mk, ${MAKEFILE_LIST},$(abspath ${mk}))
I'm using makefiles to convert an internal file format to an XML file which is sent to other colleagues. They would make changes to the XML file and send it back to us (Don't ask, this needs to be this way ;)). I'd like to use my makefile to update the internal files when this XML changes.
So I have these rules:
%.internal: $(DATAFILES)
# Read changes from XML if any
# Create internal representation here
%.xml: %.internal
# Convert to XML here
Now the XML could change because of the workflow described above. But since no data files have changed, make would tell me that file.internal is up-to-date. I would like to avoid making %.internal target phony and a circular dependency on %.xml obviously doesn't work.
Any other way I could force make to check for changes in the XML file and re-build %.internal?
You want to allow two different actions: making the xml file from the internal file, and making the internal file from the xml file. Since Make knows only the modification times, it knows which target is older but not whether it should be remade. So put in another file as a flag to record when either action was last taken, and make that your primary target; if either target is newer than the flag, it has been modified by something other than these actions, and make should rebuild the older target (and then touch the flag).
There are several ways to implement this. In some versions of Make (such as recent versions of GNUMake) you can write double-colon rules, so that Make will rebuild a target differently, based on which preq triggered it:
%.flag:: %.internal
# convert $*.internal to $*.xml
touch $#
%.flag:: %.xml
# rewrite $*.internal based on $*.xml
touch $#
A less elegant but more portable way is to look at $? and rebuild the other file:
%.flag: %.xml %.internal
ifeq ($?,$*.internal)
# convert $*.internal to $*.xml
else
# rewrite $*.internal based on $*.xml
endif
touch $#
I think you could do something like this:
all: .last-converted-xml .last-converted-internal
.last-converted-internal: *.internal
./internal-2-xml $?
touch $# .last-converted-xml
.last-converted-xml: *.xml
./xml-2-internal $?
touch $# .last-converted-internal
This runs "xml-convert" on any .xml files newer than an arbitrary marker file, ".last-converted". The $? should give you a list of all dependencies (*.xml) that are newer than the marker file.
Of course, the xml-convert program will have to be written to take a list of xml files and process each one.
I'm not sure from the question whether you actually need the .internal file, or if that was just an attempt to get the makefile working. So, either your "xml-convert" program can convert each .xml file in place, or it can also generate file.internal as well if you need it.
Use the -W option of make to have make think one of the data files has changed:
make -W somedatafile
This will cause make to think somedatafile has been modified without actually changing it's modification time.
Would it be possible to use different names for the XML file? The file you create from the internal format would have one name and the file your colleagues send you another? If they used different names there would be no circular dependency.