I'm using makefiles to convert an internal file format to an XML file which is sent to other colleagues. They would make changes to the XML file and send it back to us (Don't ask, this needs to be this way ;)). I'd like to use my makefile to update the internal files when this XML changes.
So I have these rules:
%.internal: $(DATAFILES)
# Read changes from XML if any
# Create internal representation here
%.xml: %.internal
# Convert to XML here
Now the XML could change because of the workflow described above. But since no data files have changed, make would tell me that file.internal is up-to-date. I would like to avoid making %.internal target phony and a circular dependency on %.xml obviously doesn't work.
Any other way I could force make to check for changes in the XML file and re-build %.internal?
You want to allow two different actions: making the xml file from the internal file, and making the internal file from the xml file. Since Make knows only the modification times, it knows which target is older but not whether it should be remade. So put in another file as a flag to record when either action was last taken, and make that your primary target; if either target is newer than the flag, it has been modified by something other than these actions, and make should rebuild the older target (and then touch the flag).
There are several ways to implement this. In some versions of Make (such as recent versions of GNUMake) you can write double-colon rules, so that Make will rebuild a target differently, based on which preq triggered it:
%.flag:: %.internal
# convert $*.internal to $*.xml
touch $#
%.flag:: %.xml
# rewrite $*.internal based on $*.xml
touch $#
A less elegant but more portable way is to look at $? and rebuild the other file:
%.flag: %.xml %.internal
ifeq ($?,$*.internal)
# convert $*.internal to $*.xml
else
# rewrite $*.internal based on $*.xml
endif
touch $#
I think you could do something like this:
all: .last-converted-xml .last-converted-internal
.last-converted-internal: *.internal
./internal-2-xml $?
touch $# .last-converted-xml
.last-converted-xml: *.xml
./xml-2-internal $?
touch $# .last-converted-internal
This runs "xml-convert" on any .xml files newer than an arbitrary marker file, ".last-converted". The $? should give you a list of all dependencies (*.xml) that are newer than the marker file.
Of course, the xml-convert program will have to be written to take a list of xml files and process each one.
I'm not sure from the question whether you actually need the .internal file, or if that was just an attempt to get the makefile working. So, either your "xml-convert" program can convert each .xml file in place, or it can also generate file.internal as well if you need it.
Use the -W option of make to have make think one of the data files has changed:
make -W somedatafile
This will cause make to think somedatafile has been modified without actually changing it's modification time.
Would it be possible to use different names for the XML file? The file you create from the internal format would have one name and the file your colleagues send you another? If they used different names there would be no circular dependency.
Related
At present, I have a makefile that has:
a target which links an executable image file from a bunch of object files
a pattern rule target that compiles the various object files the linker target depends on
I want to make the following changes.
Instead of compiling the object files outright, I want the pattern rule target mentioned above to create (for each object file that needs updating) an empty object_file_name.update file. Essentially, this target's job would be to take stock of all object files that actually need to be recompiled.
Write a new target that launches a Perl process which finds all these object_file_name.update files and, for each object file that must be recompiled, compiles it in this Perl process.
I know how to do 2) ... that part is not giving me any trouble. The part I'm worried about is 1). The reason is that that target would basically have to claim to update any needed object files while, in truth, merely creating an .update file for each such object file but not the object file itself.
I think I could trick GNU Make into not starting to try to link anything before all the object files have been built by declaring my dependencies accordingly (pseudo-code, not a valid GNU Make snippet):
# Phony target that reads the *.update files created by the pattern rule target below and then
# compiles each object file for which an *.update file exists.
COMPILE_OBJECTS :
...
# Pattern rule target to take stock of all object files that need updating. Creates an *.update file for
# each object file that needs recompiling.
%.o : %.c :
...
$(EXE_FILE_TO_LINK) : $(LIST_OF_OBJECT_FILE_PATHS) COMPILE_OBJECTS
...
but I still worry that this might result in undefined behavior because my pattern rule target would basically be lying to GNU Make about updating the needed object files. Is my worry justified?
Basically, I want to interject an intermediate layer between GNU Make and the compiler so that GNU Make doesn't compile each object file separately. Instead, the compiling would be done in a single Perl process that has access to the complete list of object files that need to be compiled, allowing me to do various fancy things that I couldn't do if GNU Make controlled compilation directly.
Yes, it's legal and I often use this pattern.
Consider the case where you only want to kick off a long build step if a file has changed.
target: config-file
target-creator $< -o $#
Now let's say we can't give make the dependencies for config-file (because the config file creation step lacks a dependency listing ability (BAH!)).
.PHONY: FORCE
FORCE: ;
config-file: FORCE
config-creator -o $#.tmp
cmp $#.tmp $# || mv $#.tmp $#
We ask make to build target
Make first has to build config-file
Make will always run the recipe for config-file,
as its dependency FORCE is out of date (being phony)
CRUCIALLY we only update config-file if config-creator decides something has actually changed
If cmp decides config-file.tmp and config-file are the same, and the last line of the recipe completes with no error
OTOH if cmp detects a mis-compare, it fails, and the shell goes on to execute the mv.
After running the recipe for config-file, make does actually check config-file's modification time. IF config-file has become younger than target, only then will target-creator be run.
The subtlety here is that even though config-file's recipe runs every time, config-file itself is not phony.
i know it is possible to execute standard rules on a makefile by simply including them like
hi:; echo hello
include hi
but there is some way, by doing this, to fetch an external input like a ftp server or, lets say, the aws s3 to fetch a "newborn" makefile and then include it to the project? in runtime?
like
include https://s3.bucket.domain.com/newmk.mk
As mentioned there's no way to include a URL. However, you can do something almost as good with GNU make's auto-re-make feature.
To do this, include a local file you download and create a rule that knows how to download it:
include newmk.mk
newmk.mk:
wget https://s3.bucket.domain.com/newmk.mk -O $#
In this situation, if (and only if) newmk.mk doesn't exist, then the recipe will be invoked, the file will be downloaded, then make will re-exec itself to read that file.
The make utility has no built-in download support (not even GNU's make which sometimes seems to have the kitchen sink thrown in), but because it can run arbitrary commands, you can fake it up
The simplest case is a fragment like
your_primary_target: newmk.mk
#...
WGET=wget
WGET_FLAGS=-q
newmk.mk:
$(WGET) $(WGET_FLAGS) https://s3.bucket.domain.com/newmk.mk
#...
which will attempt to download the file only if no such file exists.
The main difficulty here is that your users may need to edit the makefile to specify the command-line downloader they have (perhaps curl instead of wget), or it set any flags their network environment requires.
To make the download unconditional use a unbuildable, non-existent target
WGET=wget
WGET_FLAGS=-1
newmk.mk: ALWAYS
$(WGET) $(WGET_FLAGS) https://s3.bucket.domain.com/newmk.mk
# Here "ALWAYS" is *not* the name of a file existing in the file-system
ALWAYS:
(or there are games you can play with .PHONY in GNU-make).
I have a situation where I need to execute a rule in make depending on a remote file. This is an example of what I'm trying to do (the actual Makefile is a lot more complicated in ways that aren't relevant here):
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
stackoverflow.png: favico.ico modified | check_modified
convert favicon.ico $#
check_modified: modified
#echo Icon was modified. Downloading.
rm -f favicon.ico
wget $(URL)
.PHONY: check_modified
favico.ico: check_modified
modified:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" $#
The idea is:
The rule to build modified should always be run. The function of that rule is to change the modification time of the file to be the same as the modification time of what the URL points to.
After that, I want modified to behave normally in terms of dependencies. If modified is more recent than favicon.ico, I want to retrieve the new file and then let the dependency on it cause the target file to be remade.
As a wrinkle, in some applications, I have to retrieve the file manually. In those cases, I want to just have a rule that fires and tells me to manually download the file, but that doesn't otherwise affect building the target. As an example, if the source image that I'm converting were behind a site login, I'd need to manually login to the site and download it to a fixed location and then rerun make.
Everything I've tried either:
Fails to check the URL if the target is up to date, or
Always checks the URL and rebuilds the target, even if modified is not more recent than the target.
Any words of wisdom?
Your problem is probably that make builds the dependency graph before running any recipe. So, when your modified tag file is created / updated, it is too late for make to consider its last modification date and use it to decide whether other targets shall be built or not.
Your first observed behaviour (fails to check the URL if the target is up to date) probably corresponds to attempts where you did not use the .PHONY special target. As soon as modified has been created, make always considers it as up-to-date because it has no prerequisites and thus cannot be outdated.
The second behaviour (always checks the URL and rebuilds the target, even if modified is not more recent than the target) corresponds to what you posted: check_modified is a prerequisite of the .PHONY special target, which forces make to always consider it as outdated. stackoverflow.png and favico.ico, directly or not, depend on check_modified and are thus also always considered as outdated.
One option to solve your problem would be to use a recursive, two-passes, make invocation. On a first run make would build the all phony target (always), update a modified tag file, and then invoke itself again to build other targets that have the tag file as prerequisite and are outdated. As, for the second invocation, make rebuilds its dependency graph, it will take the last modification date of the tag file into account. Something like:
URL = http://cdn.sstatic.net/stackoverflow/img/favicon.ico
.PHONY: all
all:
touch -d "`wget --spider -S $(URL) 2>&1 | sed -n 's/.*Modified: //p'`" modified
$(MAKE) stackoverflow.ico
stackoverflow.ico: modified
#echo Icon was modified. Downloading.; \
rm -f favicon.ico; \
wget $(URL); \
convert favicon.ico $#
Explanations:
I replaced your conversion to png by a useless conversion to ico because the favicon.ico icon of SO is composite and its conversion to png creates two files instead of one, named stackoverflow-0.png and stackoverflow-1.png, which uselessly complicates things.
all is a true phony target and the default goal so, each time you invoke make (or make all), it is built. It first updates the modified tag file and then invokes make again to build stackoverflow.ico.
The second make invocation does nothing if stackoverflow.ico is more recent than modified, else it downloads and converts.
Your second question (get a message about required manual operations) is completely different and simpler to solve. Let's first define a message and echo it in the file's recipe:
define DIY_message
Dear user, you should first:
- do this
- and that.
Unless you know it is useless, of course.
endef
export DIY_message
the_remote_file:
#echo "$$DIY_message"
The message will be printed if make is invoked with this goal (make the_remote_file) or with a goal that somehow depends on the_remote_file and either:
the_remote_file does not exist,
or the_remote_file exists but is out-of-date with respect to its prerequisites (if you declared prerequisites for it),
or the_remote_file is a prerequisite of .PHONY.
Note: using an intermediate make variable assigned by a define-endef makes things easier for formatted multi-lines messages.
I have a somewhat complicated Makefile which runs perl scripts and other tools and generates some 1000 files. I would like to edit/modify some of those generated files after all files are generated. So I thought I can simply add a new rule to do so like this:
(phony new rule): $LIST_OF_FILES_TO_EDIT
file_modifier ...
however, the point here is some of those generated files which I'd like to edit ($LIST_OF_FILES_TO_EDIT) are used in the same make process to generate a long list of files. So I have to wait to make sure those files are no longer needed in the make process before I can go ahead and edit them. But I don't know how to do that. Not to mention that it is really hard to find out what files are generated by the help of $LIST_OF_FILES_TO_EDIT.
If it was possible to mention in the Makefile that this rule should be only run as the last rule, then my problem would be solved. but as far as I know this is not possible. So anyone has an idea?
Some points:
List of files to edit ($LIST_OF_FILES_TO_EDIT) is determined dynamically (not known before make process)
I am not sure I have picked a good title for this question. :)
1) If you're going to modify the files like that, it might behoove you to give the targets different names, like foo_unmodified and foo_modified, so that the Make's dependency handling will take care of this.
2) If your phony new rule is the one you invoke on the command line ("make phonyNewRule"), then Make will build whatever else it's going to build before executing the file_modifier command. If you want to build targets not on that list, you could do it this way:
(phony new rule): $(LIST_OF_FILES_TO_EDIT) $(OTHER_TARGETS)
file_modifier ...
3) If your dependencies are set up correctly, you can find out which targets depend on $(LIST_OF_FILES_TO_EDIT), but it's not very tidy. You could just touch one of the files, run make, see which targets it built, repeat for all files. You could save a little time by using Make arguments: "make -n -W foo1 -W foo2 -W foo3 ... -W foo99 all". This will print the commands Make would run-- I don't know of any way to get it to tell you which targets it would rebuild.
We have an ActionScript (Flex) project that we build using GNU make. We would like to add an M4 preprocessing step to the build process (e.g., so that we can create an ASSERT() macro that includes file and line numbers).
We are having remarkable difficulty.
Our current strategy is:
Create a directory "src/build" (assuming source code is in src/ and subdirectories).
Within src/build, create a Makefile.
Run make inside src/build.
The desired behavior is, make would then use the rules we write to send the *.as files src/ and its subdirs, creating new *.as files under build. For example:
src/bar.as -> m4 -> src/build/bar.as
src/a/foo.as -> m4 -> src/build/a/foo.as
The obvious make rule would be:
%.as : ../%.as
echo "m4 --args < $< > $#"
This works for bar.as but not a/foo.as, apparently because make is being "smart" about splitting and re-packing directories. make -d reveals:
Trying implicit prerequisite `a/../foo.as'.
Looking for a rule with intermediate file `a/../foo.as'.
but we want the prerequisite to be "../a/foo.as". This (what we don't want) is apparently documented behavior (http://www.gnu.org/software/make/manual/make.html#Pattern-Match).
Any suggestions? Is it possible to write a pattern rule that does what we want?
We've tried VPATH also and it does not work because the generated .as files are erroneously satisfying the dependency (because . is searched before the contents of VPATH).
Any help would be greatly appreciated.
One option is to use a different extension for files that haven't been preprocessed. Then you can have them in the same directory without conflict.
As Anon also said, your source code is no longer Flex - it is 'to be preprocessed Flex'. So, use an extension such as '.eas' (for Extended ActionScript) for the source code, and create a 'compiler' script that converts '.eas' into '.as' files, which can then be processed as before.
You may prefer to have the Extended ActionScript compiler do the whole compilation job - taking the '.eas' direct to the compiled form.
The main thing to be wary of is ensuring that '.eas' files are considered before the derived '.as' files. Otherwise, your changes in the '.eas' files will not be picked up, leading to hair-tearing and other undesirable behaviours (head banging, as in 'banging head against wall', for example) as you try to debug code that hasn't changed even though the source has.