i know it is possible to execute standard rules on a makefile by simply including them like
hi:; echo hello
include hi
but there is some way, by doing this, to fetch an external input like a ftp server or, lets say, the aws s3 to fetch a "newborn" makefile and then include it to the project? in runtime?
like
include https://s3.bucket.domain.com/newmk.mk
As mentioned there's no way to include a URL. However, you can do something almost as good with GNU make's auto-re-make feature.
To do this, include a local file you download and create a rule that knows how to download it:
include newmk.mk
newmk.mk:
wget https://s3.bucket.domain.com/newmk.mk -O $#
In this situation, if (and only if) newmk.mk doesn't exist, then the recipe will be invoked, the file will be downloaded, then make will re-exec itself to read that file.
The make utility has no built-in download support (not even GNU's make which sometimes seems to have the kitchen sink thrown in), but because it can run arbitrary commands, you can fake it up
The simplest case is a fragment like
your_primary_target: newmk.mk
#...
WGET=wget
WGET_FLAGS=-q
newmk.mk:
$(WGET) $(WGET_FLAGS) https://s3.bucket.domain.com/newmk.mk
#...
which will attempt to download the file only if no such file exists.
The main difficulty here is that your users may need to edit the makefile to specify the command-line downloader they have (perhaps curl instead of wget), or it set any flags their network environment requires.
To make the download unconditional use a unbuildable, non-existent target
WGET=wget
WGET_FLAGS=-1
newmk.mk: ALWAYS
$(WGET) $(WGET_FLAGS) https://s3.bucket.domain.com/newmk.mk
# Here "ALWAYS" is *not* the name of a file existing in the file-system
ALWAYS:
(or there are games you can play with .PHONY in GNU-make).
Related
In my program, I have a somewhat complicated build process. Currently, in one directory I use include in Makefile.am with a file, that does not exist but has to be build on its own. The reason is that this include file is quite long. Further in the real program it is not just only one file but several and the generation process for this file can change from time to time.
The Makefile.am looks something like this
noinst_LIBRARIES = libtest.a
nodist_libtest_a_SOURCES = file.c
CLEANFILES = file.c Make_file.mk
$(builddir)/Make_test.mk: $(srcdir)/Perl/generate_mk_files.pl
perl $(srcdir)/Perl/generate_mk_files.pl file
include $(builddir)/Make_file.mk
After creation of Make_file.mk it looks something like
$(builddir)/file.c: $(srcdir)/file.template $(srcdir)/Perl/generate_c.pl
perl $(srcdir)/Perl/generate_c.pl $(srcdir)/file.template
Automake works and the final build process as well. The output to make is something like (I have shorted it somewhat):
Makefile:721: Make_file.mk: Datei oder Verzeichnis nicht gefunden (file not found)
perl ../../../../src/components/test/Perl/generate_mk_files.pl test
perl ../../../../src/components/test/Perl/generate_c.pl ../../../../src/components/test/file.template
Therefore, make first complains that the include file is not found, then creates it and then also follows the rules of the included file.
Although I am happy that it works I wonder why. First, I thought that make loads the Makefile. During this step, Make_file.mk does not exists. Therefore it seems the Makefile is loaded more than once.
Further, the manual of Automake for include states:
Note that these fragments are read and interpreted by automake, not by
make.
Which is not what I see, since the included fragment does not exist during the execution of Automake.
My questions basically are:
Why does it work?
Is this the correct way to do this or should I use another approach, e.g. starting new instances of make within Makefile.
I don't really know Automake, but, from the GNU make manual :
If an included makefile cannot be found in any of these directories {standard includes directories} , a
warning message is generated, but it is not an immediately fatal
error; processing of the makefile containing the include continues.
Once it has finished reading makefiles, make will try to remake any
that are out of date or don’t exist. See How Makefiles Are Remade.
Only after it has tried to find a way to remake a makefile and failed,
will make diagnose the missing makefile as a fatal error.
If you want make to simply ignore a makefile which does not exist or
cannot be remade, with no error message, use the -include directive
instead of include, like this:
-include filenames…
This acts like include in every way except that there is no error (not even a warning) if any of the filenames (or any
prerequisites of any of the filenames) do not exist or cannot be
remade.
So basically, make cannot execute the recipe for remaking the include file before he has finished to parse the main Makefile. So it raises a warning, continue to read the Makefile, find the rule for remaking the included file, remake it, and then restart itself (that is explained in details in the How Makefiles Are Remade section).
Going back to the manual, it states there's two forms for automakes include mechanism:
include $(srcdir)/file
and
include $(top_srcdir)/file
neither of which match your include. So I'd imagine the include is actually run by the underlying make whatever that might be (e.g. GNU Make, though of course other make programs have this functionality as well).
Now for the questions:
Why does it work?
As explained in another answer, GNU Make will attempt to make a missing included makefile, before failing.
Is this the correct way to do this or should I use another approach, e.g. starting new instances of make within Makefile
Generating makefiles is one of the tasks autotools do, either through autoconf or automake. Going through multiple stages of "making makefiles" seems prone to error (and hard to maintain). Recursive make has similar problems.
The reason is that this include file is quite long.
automake include statements will happily paste together a large makefile out of smaller components.
Further in the real program it is not just only one file but several and the generation process for this file can change from time to time.
It's hard to recommend what to suggest to do in autotools based on how the "changes" are determined. Since it seems you're also using libtool adding/removing sources to libs (or entire libs) there can be effected by conditionals, variables, etc.
I am having a problem creating Makefile build that has dependency of some other file content.
I know that I can write something like that in makefile to generate prerequisites on-the-fly:
result/nuclease.stat : $(shell cat config/nuclease.sample.list)
#bash
make result/nuclease.stat
But the problem is that the file config/nuclease.sample.list might not exist and I have a target for it.
If I could force make to require config/nuclease.sample.list before $(shell cat config/nuclease.sample.list) gets evaluated - problem is solved.
What kind of dependency I should make?
One more thing, I would like keep all gnu make's good qualities: download files only once, process them only once and etc..
Please check
The most straightforward answer is to use constructed include files. In this method you'd include a makefile that defined the dependency relationships you wanted, and provide a make rule that generated that included file with the information you want. That's all you have to do: make knows how to do the rest. Based on your question it might be something like:
include nuclease-sample-list.mk
nuclease-sample-list.mk: config/nuclease.sample.list
echo "result/nuclease.stat : $$(cat $<)" > $#
That's all you have to do. If you're using an older version of GNU make you might need to use -include instead of include to avoid seeing warning messages.
I am trying to port an existing code into a larger project. The larger project has a main Makefile with Makefiles in each sub-directory. I am sure the path below tells you all about how it is setup. I want to port my code to
/WORKING_DIRECTORY/Drivers/Char/example
And here is the content:
sansari#ubuntu:~/WORKING_DIRECTORY/drivers/char/examples$ ls
hello1.c Makefile
My question is 1- Should modify this local Makefile or the main one? I am setting up to modify this one, but I am not sure.
2- My other question is if I modify this local file, can I just run make from here and validate my configuration instead of running make for the entire project? I know that make only updates the files that are changed; however I feel better when I clean the build environment before each make. I have run into situations, which this alone fixed my issue.
Just as background, I did try to include the make file of my target project, the one I am trying to import here with -f command. What I did was: make -f Makefile -f ../mytarget/core/Makefile
But I ran into some issues with make not doing some of the normal things it does in the primary project. For instance, there was an include statement with a relative path to a header file, which make gave me an error about not seeing it. So now I am abandoning that strategy for the time being.
#Ahmad Masoud - Hey man, thanks. Here is the Makefile. Hey man, the link is exactly what I needed. I think it will address my other questions also. You see, I cross compiled this code, and when I flashed my phone, I get the following for
uname -r: 1|root#hltespr:/lib/modules # uname -r
uname -r
3.4.0-g7e6fbf7-dirty
And I have been wondering what "dirty" means and where it comes from. If you know please tell me. The link you sent, states that perhaps make would insert the Linux kernel version there? I ask this since, modprobe does not work when I try to load my module. Instead insmod works, and I can validate that my module is in. My main issue now is that I don't know how to execute the file to make sure it runs. I only know how to run the file using modprobe, and I can not use it. It gives me the following error:
1|root#hltespr:/ # modprobe /lib/modules/hello1.ko
modprobe /lib/modules/hello1.ko
modprobe: can't change directory to '3.4.0-g7e6fbf7-dirty': No such file or directory
Update as of 06/20/15 -I put in include /home/sansari/mytree2/tbt/makefile in my module's make file. I get the following error: makefile:3: *** missing separator. Stop.
#Ahmad - This is an update as of 062415. Thanks for the info. My goal is to get make to look into this external directory, collect all the source files and build them for me. What would you suggest? I am stuck because as it stands, I know make looks into my examples directory, but no other changes I make to that local make file in the examples directory shows up in make. For instance I tried adding ($warning ....) and #echo messages, but even they do not show up.
Update on 070215- Thanks for the previous comments and support. I feel I really should reopen this thread since I did not explain the goal in detail, and now I feel I can describe it better, and hopefully the resolution will help other. I issue the command:
TARGET=msm8974 PLATFORM=msm8974 make drivers/char/examples
But I get a message stating: Nothing to be done, while I have added a number of tasks to do. Below is my make file, and I'll elaborate on what I have added right after:
lib_tbt := ../../../m/shahin/tbt
lib_daemon := ../../../m/shahin/daemon
lib_lib := ../../../m/shahin/lib
lib_tasks := ../../../m/shahin/tasks
lib_tbt_driver := ../../../m/shahin/tbt_driver
lib_tbt_make := ../../../m/shahin/tbt/make
lib_tbt_msm_common := ../../../m/shahin/tbt/platform/msm8974/common
lib_tbt_msm8974 := ../../../m/shahin/tbt/platform/msm8974
lib_asm_generic = ../../../m/shahin/tbt/platform/msm8974/include/asm-generic
$(warning This is what is in lib_asm_generic $(lib_asm_generic))
#include $(lib_tbt_make)/macros.mk
.PHONY: all $(lib_tbt)
$(lib_tbt) $(lib_daemon) $(lib_lib) $(lib_tasks) $(lib_tbt_driver) $(lib_tbt_make) $(lib_tbt_msm_common) $(lib_tbt_msm8974) $(lib_asmgeneric) :
$(MAKE) --directory=$#
$(lib_*): $(MAKE) --directory=$#
obj-$(CONFIG_EXAMPLES) += hello1.o
_
Initially I only had the obj-$(CONFIG_EXAMPLES) += hello1.ostatement in my make file. I then proceeded to add the directory variables at the top of my makefile and added the $(lib_*): $(MAKE) --directory=$# directing make to compile what is in the directory. I believe that is what it does. Please let me know if I am mistaken. And although this same make file proceeds to create object files when I put it a different directory within my project, it won't do so when it is in a device driver directory. And I do not understand why. The other directory is the /external directory and it is at the top of the tree. But that should not matter right. What I have done was to first make sure I can compile a hello program in my device driver directory called /examples. I now want to add more source code to this section. I believe the correct term is module? I also want to know if I should copy of the source files to the /examples directory or referencing them via the path is ok. That is should I move the source code directory under /examples directory or not?
It is a LOT simpler than that if you are using a kernel that uses Kbuild.
Highly recommend reading
http://www.tldp.org/LDP/lkmpg/2.6/html/x181.html
Situation A - Your source is a sub-tree of the kernel source
You would NOT modify the top-level Makefile, just ensure that ~/WORKING_DIRECTORY/drivers/char/examples/Makefile and ~/WORKING_DIRECTORY/drivers/char/examples/KBuild are set up correctly/normally. THEN at the top-level of the kernel build directory (assuming you have a separate build directory) you would type:
foo#bar:~/build-dir$ make drivers/char/examples
The kernel top-level makefile then builds just that sub-tree. You can try it out on any part of the kernel, for example:
foo#bar:~/build-dir$ make fs
NOTE: build-dir can be the same as the kernel source directory
Situation B - You are building an external module
Then use the normal module KBuild / Makefile process.
P.S.
If you post your makefile / Kbuild then I may be able to help with the actual build processing.
I have a somewhat complicated Makefile which runs perl scripts and other tools and generates some 1000 files. I would like to edit/modify some of those generated files after all files are generated. So I thought I can simply add a new rule to do so like this:
(phony new rule): $LIST_OF_FILES_TO_EDIT
file_modifier ...
however, the point here is some of those generated files which I'd like to edit ($LIST_OF_FILES_TO_EDIT) are used in the same make process to generate a long list of files. So I have to wait to make sure those files are no longer needed in the make process before I can go ahead and edit them. But I don't know how to do that. Not to mention that it is really hard to find out what files are generated by the help of $LIST_OF_FILES_TO_EDIT.
If it was possible to mention in the Makefile that this rule should be only run as the last rule, then my problem would be solved. but as far as I know this is not possible. So anyone has an idea?
Some points:
List of files to edit ($LIST_OF_FILES_TO_EDIT) is determined dynamically (not known before make process)
I am not sure I have picked a good title for this question. :)
1) If you're going to modify the files like that, it might behoove you to give the targets different names, like foo_unmodified and foo_modified, so that the Make's dependency handling will take care of this.
2) If your phony new rule is the one you invoke on the command line ("make phonyNewRule"), then Make will build whatever else it's going to build before executing the file_modifier command. If you want to build targets not on that list, you could do it this way:
(phony new rule): $(LIST_OF_FILES_TO_EDIT) $(OTHER_TARGETS)
file_modifier ...
3) If your dependencies are set up correctly, you can find out which targets depend on $(LIST_OF_FILES_TO_EDIT), but it's not very tidy. You could just touch one of the files, run make, see which targets it built, repeat for all files. You could save a little time by using Make arguments: "make -n -W foo1 -W foo2 -W foo3 ... -W foo99 all". This will print the commands Make would run-- I don't know of any way to get it to tell you which targets it would rebuild.
I'm using makefiles to convert an internal file format to an XML file which is sent to other colleagues. They would make changes to the XML file and send it back to us (Don't ask, this needs to be this way ;)). I'd like to use my makefile to update the internal files when this XML changes.
So I have these rules:
%.internal: $(DATAFILES)
# Read changes from XML if any
# Create internal representation here
%.xml: %.internal
# Convert to XML here
Now the XML could change because of the workflow described above. But since no data files have changed, make would tell me that file.internal is up-to-date. I would like to avoid making %.internal target phony and a circular dependency on %.xml obviously doesn't work.
Any other way I could force make to check for changes in the XML file and re-build %.internal?
You want to allow two different actions: making the xml file from the internal file, and making the internal file from the xml file. Since Make knows only the modification times, it knows which target is older but not whether it should be remade. So put in another file as a flag to record when either action was last taken, and make that your primary target; if either target is newer than the flag, it has been modified by something other than these actions, and make should rebuild the older target (and then touch the flag).
There are several ways to implement this. In some versions of Make (such as recent versions of GNUMake) you can write double-colon rules, so that Make will rebuild a target differently, based on which preq triggered it:
%.flag:: %.internal
# convert $*.internal to $*.xml
touch $#
%.flag:: %.xml
# rewrite $*.internal based on $*.xml
touch $#
A less elegant but more portable way is to look at $? and rebuild the other file:
%.flag: %.xml %.internal
ifeq ($?,$*.internal)
# convert $*.internal to $*.xml
else
# rewrite $*.internal based on $*.xml
endif
touch $#
I think you could do something like this:
all: .last-converted-xml .last-converted-internal
.last-converted-internal: *.internal
./internal-2-xml $?
touch $# .last-converted-xml
.last-converted-xml: *.xml
./xml-2-internal $?
touch $# .last-converted-internal
This runs "xml-convert" on any .xml files newer than an arbitrary marker file, ".last-converted". The $? should give you a list of all dependencies (*.xml) that are newer than the marker file.
Of course, the xml-convert program will have to be written to take a list of xml files and process each one.
I'm not sure from the question whether you actually need the .internal file, or if that was just an attempt to get the makefile working. So, either your "xml-convert" program can convert each .xml file in place, or it can also generate file.internal as well if you need it.
Use the -W option of make to have make think one of the data files has changed:
make -W somedatafile
This will cause make to think somedatafile has been modified without actually changing it's modification time.
Would it be possible to use different names for the XML file? The file you create from the internal format would have one name and the file your colleagues send you another? If they used different names there would be no circular dependency.