Say there are a bunch of (e.g. nearly 200) modules that all depend on a core module. All are using Autotools.
The core module installs a core.m4 file which the dependents already use for various things.
All the dependents also have some lines in their install-data-local rule to generate some scripts and install them, e.g.
core_stuffdir=$(prefix)/share/core/stuff/
install-data-local:
generate-stuff stuff.xml
test -d $(DESTDIR)$(core_stuffdir) || mkdir $(DESTDIR)$(core_stuffdir)
stuff=`xmllint --xpath '//stuff[#install="yes"]/#name' stuff.xml`; \
$(INSTALL_DATA) $$stuff $(DESTDIR)$(core_stuffdir); \
rm $$stuff
…
$(INSTALL_DATA) other-module-specific-stuff …
…
I would like to remove those five lines which are currently redundantly duplicated over ~200 files, and instead define the lines in the core.m4. The dependents should be able to say something like SOMEVAR: somevalue (or something similarly short, at worst a one-line thing in install-data-local) and have those lines executed during make install.
Is there a nice way to do define these lines in core.m4 and make them available to Makefile.am? I can't find any similar examples on the net.
The only solution I can think of now is that the m4 spits out a shell script that I can call from install-data-local, but I'm not sure that's the best (or most autotoolish) way.
Alternatively, is there a simple way to distribute an automake fragment.am file from my core module? (This seems to be the way e.g. Python compilation rules are defined.)
After some digging, I found
http://thread.gmane.org/gmane.comp.sysutils.automake.general/5704/focus=5708
which has a solution to a similar problem (I couldn't get the
AC_CONFIG_FILES solution mentioned there to work).
So core.m4 now defines CORE_MKINCLUDE:
AC_DEFUN([CORE_MKINCLUDE],
[
AC_SUBST_FILE(core_include)
core_include=$srcdir/core_include.am
cat >$srcdir/core_include.am <<EOF
core_stuffdir=\$(prefix)/share/core/stuff/
install-stuff:
generate-stuff stuff.xml
test -d \$(DESTDIR)\$(core_stuffdir) || mkdir \$(DESTDIR)\$(core_stuffdir)
stuff=\`xmllint --xpath '//stuff#<:##install="yes"#:>#/#name' stuff.xml`; \\
\$(INSTALL_DATA) \$\$stuff \$(DESTDIR)\$(core_stuffdir); \\
rm \$\$stuff
EOF
])
Ie. the goal is now printed to the file core_include.am, and has its
own name. Each module's configure.ac calls CORE_MKINCLUDE, and each
Makefile.am just has
#core_include#
install-data-local: install-stuff
An improvement at least.
Related
Suppose I define several make PHONY commands:
auto-build-foo
auto-build-bar
auto-build-biz
...
auto-build-inf
And I can observe them all with the following keystrokes in bash with autocompletion configured:
$: make auto-build-<tab><tab>
auto-build-foo
auto-build-bar
auto-build-biz
...
auto-build-inf
Then my natural unix instinct is to write:
make auto-build-*
To build them all.
I understand this sort of thing needs to be implemented in make as a feature, a makefile as some sort of rule system, or some custom shell that integrates bash-completion history with some make-specialized interpretation.
But it would cool and useful to get this "out-of-the-box".
Is there such a mechanism that is -- or will be -- in GNU make?
No, GNU Make command line argument do not support wildcard.
But, you can easily do the job in bash:
One by one
$ for t in $(sed -e '/auto-build/!d' -e 's/:.*$//' -e 's/\n/ /' Makefile); do make "$t" ; done
At once
$ my_targets=$(sed -e '/auto-build/!d' -e 's/:.*$//' -e 's/\n/ /' Makefile)
$ make "$my_targets"
From a Makefile perspective, if you create your own, an efficient way could be to encapsulate the targets in a variable each time you write them:
TARGET:=auto-build-foo
$(TARGET):
#echo "$#"
ALL_TARGETS:=$(ALL_TARGETS) $(TARGET)
TARGET:=auto-build-bar
$(TARGET):
#echo "$#"
ALL_TARGETS:=$(ALL_TARGETS) $(TARGET)
auto-all: $(ALL_TARGETS)
.PHONY: auto-all
will give:
$ make auto-all
auto-build-foo
auto-build-bar
I think such a feature doesn't exist because many makefiles are written with an incomplete dependency graph; command line parameters - which tend to be recorded nowhere - are the main reason for this. Targets built for foo may introduce bugs when being reused as-is for bar - the usual procedure is to require an intermediate make clean to prevent this. The problem is that, given the current (and unlikely to change) execution logic of make, within one run there is no way to execute clean more than once, much less in an order which would not defy the purpose of make (partial updates) at all. I think the original use case for makefiles was more around single-purpose project building rather than a cross-over of dependency and batch processing. From a modern build system perspective this looks like a shortcoming, OTOH it is this focused simplicity which makes it the means of choice for many projects.
QUESTION
Using Make how do I run a command for every directory that contains a file matching *.csproj but does not including a file matching *.Test.csproj using pure make.
SCENARIO
I have previously used Fake and Rake extensively but this is my first time using Make to do anything over and above the simple use of dumb targets.
I have a simple makefile that compiles a .net core solution, runs some tests and then packages up a nuget package. Here is a simplified example.
build:
dotnet build ./src \
...
test:
dotnet test ./src/TestProject \
...
package:
dotnet pack ./src/PackageProject \
...
I now want to introduce additional projects which are also packaged but I do not want to specify each project to package individually. I want the make file to automagically pick up each project that can be packaged. This method has been tried in tested in various Fake builds.
I have been able to implement the following matching on the projects csproj extension and works fine but I have not been able to filter out the test project which also gets packaged.
package: ./src/**/%.csproj
%.csproj:
dotnet pack $(#D) \
...
I have been trying to understand the Make pattern rules and how to apply the filter-out function but have sadly failed. $(filter-out src/**/*.Test.csproj, src/**/%.csproj)
Would appreciate any help on figuring this one out.
EDIT
Based on the question from MadScientist if I run the following using make package using this dumbed down example:
package: ./src/*/%.csproj
%.csproj :
echo $(#)
I get the following output:
echo src/Namespace.Project1/%.csproj
src/Namespace.Project1/%.csproj
echo src/Namespace.Project2/%.csproj
src/Namespace.Project2/%.csproj
echo src/Namespace.Test/%.csproj
src/Namespace.Test/%.csproj
Additionally based on MadScientist's comments I have also been able to create a list of the directories I want to call the dotnet pack command against but I am now stuck on how to call the target for each match.
Note: I am trying to keep this pure Make and avoid using any bash specific syntax
projects := $(filter-out $(dir $(wildcard ./src/Bombora.Namespace*Test/.) ), $(dir $(wildcard ./src/Namespace.*/.) ) )
package:
echo $(projects)
Results in:
echo ./src/Namespace.Project1/ ./src/Namespace.Project2/
./src/Namespace.Project1/ ./src/Namespace.Project2/
EDIT
I have been able to make this work but I do not know if I have gone about it the correct way or if I am abusing something which will come back to bite me later.
This is what I am now doing which is working as expected:
PACKAGE_PROJECTS := $(filter-out $(wildcard ./src/Namespace*Test/*.csproj), $(wildcard ./src/Namespace*/*.csproj) )
package: $(PACKAGE_PROJECTS)
$(PACKAGE_PROJECTS): .
dotnet pack $(#D) \
...
Make implements standard globbing as defined by POSIX. It doesn't provide advanced globbing as implemented in some advanced shells like zsh (or bash if you enable it).
So, ** is identical *; there's no globbing character that means "search in all subdirectories". If you want to do that you need to use find.
Also, in make a pattern is a template that can match some target that you specifically want to build. It's not a way to find targets. And pattern rules only are pattern rules if the target contains the pattern character %; putting a % in a prerequisite of an explicit target doesn't do anything, make treats it as if it were just a % character.
so:
package: ./src/**/%.csproj
is identical to writing:
package: ./src/*/%.csproj
where it finds files matching the literal string %.csproj of which you probably don't have any.
I don't see how this package target does anything at all.
I don't understand what exactly you want to do: you need to make your question more explicit. Make works on targets and prerequisites. So, what is the target you want to build and what are the prerequisites used to create that target? What is the make command line you are invoking, what is the output you got, and what is the intended output you want?
ETA
You asked:
At the end of the day the question is about how do I run a command for every directory that contains a file matching *.csproj but not including *.Test.csproj using pure make.
This will get you that list:
TEST_PROJECTS := $(dir $(wildcard src/*/*.Test.csproj))
PROJECTS := $(filter-out $(TEST_PROJECTS),$(dir $(wildcard src/*/*.csproj)))
projects: $(PROJECTS)
$(PROJECTS):
...run commands...
.PHONY: $(PROJECTS)
You combine setting PROJECTS into one line if you prefer.
Please see below changes which will help you understand applying a filter with make targets and getting ahead.
Usage of filter and filter-out:
filter : Select words in text that match one of the pattern words.
Syntax : $(filter pattern...,text)
filter-out: Select words in text that do not match any of the pattern words.
Syntax : $(filter-out pattern...,text)
In the below example I will use filter to match pattern and if the conditions evaluates to true , then you can use your target with whatever execution you would like to do.
# List the project extensions that you would like to support
MYPROJECTS=.csproj .xyzproj
# MY_FILES would contain all files eg: abc.csproj def.csproj abc.xyzproj
# Below GETPATTERN will extract the suffix: eg. it would produce result .csproj .csproj .xyzproj
GETPATTERN:= $(suffix $(MY_FILES))
ifeq ($(filter $(GETPATTERN),$(MYPROJECTS)),)
package:
dotnet pack ./src/PackageProject
else
package: <default or any other dependency you want>
dotnet pack ./src/PackageDefaultProject
endif
I need to know the best way of dealing with this. Also you could answer this -- after reading the sample below --: Look at the makelib target in package.make and tell me if there is a way to force this be treated as not updated if the recipe (make -C ../lib/ -f lib.make) reports as nothing to be made (not using ordered prequisites)?
I need to explain this point using an example. I have inherited this and I need the best way to make this right.
A target which other targets will be depending on:
File lib.make
--------------
.DEFAULT_GOAL = thelib.dll
%.dll: file1.obj file2.obj
makelib file1.obj file2.obj -o thelib.dll
This by itself is pretty solid. You run it once (make -f lib.make) and it creates the lib. If you run it subsequently, having no modified files, then it will tell you it has nothing to do.
Now we're going to use this in a special way somewhere else:
File: package.make
------------------
.DEFAULT_GOAL: all
all: package
makelib:
#make -C ../lib/ -f lib.make
package: makelib file3 file4
#package_files file3 file4 ../lib/out/*.dll -o package
This is how lib.make is referenced inside package.make.
The issue is even though the package gets created when you call make -f package.make all make assumes that package target needs to be rebuilt every time since one of its dependencies -- makelib -- had to be remade
Make considers makelib out of date despite what happens after entering lib.make.
To correct this I thought of a few choices:
moving makelib to the ordered prerequisites ( after the |) but that's not quite right because in case of a newly built library my package wont' be updated
adding the dll (thelib.dll) as the dependency to the makelib target a second time but this would almost duplicate the logic and break the encapsulation.
removing makelib target and moving the line #make -C ../lib/ -f lib.make to inside the package recipe. There is a problem with this and that is that I have removed the dependency between the package and lib. If lib requires update, the package won't know about it and won't get updated.
using include lib.make and then rewriting package rule to something like: package: thelib.dll file3 file4. There are problems with this also and the least of them is for a make file to be included, it must be written as such. Otherwise a lot of overwriting/conflicting targets and definitions will be introduced.
Are there any suggestions other than and directly listing the dll as the dependency?
There are two main ways this works:
First, if you use recursive make (please remember to always invoke a sub-make using $(MAKE), never make directly) then you should make the target in the parent makefile be the actual file generated by the sub-make:
package: lib/thelib.dll ...
...
lib/thelib.dll: FORCE
$(MAKE) -f lib
FORCE:
Second, you can use non-recursive make which means you include the sub-makefile into the parent make and write it so that it expects that. You can play tricks with variables etc. to make this more generic, so it can be invoked from either the parent or subdirectory, if you want.
I'm trying to write a Makefile that automatically calls BibTeX on files that match a specific wildcard but don't exist when I first run Make. Specifically, I have the following:
.FORCE:
all: pdf
foo=something
lat: *.tex
pdflatex $(foo).tex
pdf: lat
open $(foo).pdf &
%.aux: .FORCE
bibtex $#
bib: lat $(foo)?.aux
pdflatex $(foo).tex
pdflatex $(foo).tex
open $(foo).pdf &
What I want to happen is that the following will occur when I run make bib:
pdflatex will be called on $(foo).tex, generating files $(foo)1.aux, $(foo)2.aux, etc.
bibtex will be called on $(foo)1.aux, then $(foo)2.aux, etc.
pdflatex will be called twice on $(foo).tex
open will be called on $(foo).pdf
However, this doesn't happen: in particular, it looks as if Make evaluates the prerequisites $(foo)?.aux up-front, at a point where the files $(foo)1.aux, $(foo)2.aux, etc. don't exist. As a result, BibTeX is never called on them. If I rerun make bib, however, things work, because the files now exist after being created on the previous run.
Question: Is forcing Make to re-evaluate prerequisites for a target the right way to fix this? If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat? If not, how can I achieve what I want please?
What I do in my Maiefile for LaTeX files is rename the targets.
That way, you can have different target names, depending on which phase has been used to create them. This is according to the spirit of make's pattern rules, which assume that files with different contents also have different extensions. So I have rules like this:
%.aux1 : %.tex
rm -f $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%.bbl : %.aux1
cp -pf $< $*.aux
bibtex $* || : > $#
%.aux2 : %.bbl
cp -pf $*.aux1 $*.aux
pdflatex -draftmode $*
mv -f $*.aux $#
%-tex.pdf: %.aux2
cp -pf $< $*.aux
pdflatex -jobname $*-tex $*
You can't do this in a completely straightforward way, since make fundamentally assumes that one run through a target's commands will update the target. That is, there's no way in principle that you can tell make that ‘you need to run these commands twice’.
You can try to get round this with (admirably clever) tricks such as #reinerpost suggests, but a problem with that general approach is that sometimes/often a single run of BibTeX (or makeindex, or whatever) is sufficient.
After having tried various types of tricks in the past, what I generally do here is to make a command list which explicitly includes two BibTeX calls where necessary:
%.bbl: %.aux
bibtex $(#:.bbl=)
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then \
bibtex $(#:.bbl=); \
fi
That command list re-runs BibTeX if the log file includes the ‘Label(s) may have changed. Rerun to get cross-references right’ message.
To be honest, what I actually do is just the single line bibtex $(#:.bbl=). When I'm writing a document, I inevitably re-run make so many times that the list of references comes out correct very quickly. This means that this target doesn't work for the ‘recreate the final version from a clean directory’ case, but that's sufficiently rare that I tend not to obsess about it.
Whenever I catch myself re-solving this problem, I now recognise that I'm trying to push water up-hill because I'm bored writing this document, so I go and do something else.
I just wanted to share an alternative solution: Using submake processes:
If so, how can I get it to re-evaluate the prerequisites for bib after running pdflatex as part of lat?
You can somewhat achieve that, by adding make lat to the recipe for bib. This will start a new make process targetting at bib. The sub-make doesn't know anything aboutits parents targets/prerequisites. (Such a concept is usually used, when some huge project is built from different smaller projekts each of which have different makefiles.)
This can be done in multiple layers (although it will be confusing):
bib: $(foo)?.aux lat check_for_bib
check_for_bib:
if grep -q Rerun $(#:.bbl=.log) >/dev/null; then make bib; fi
pdf: lat check_for_bib
open $(foo).pdf &
Note that I had to change some orders of prerequisites. The pseud-code would be something like:
latex compilation
while log suggests update:
update aux
latex compilation
Each iteration of the while loop will take place in a separate make process.
Consider having a makefile, which can generate some files using generating lines listed in a file. For example file 001 using first line, 002 using second line, etc.
This file can be changed (it has it's own dependences, but this doesn't metter).
If some lines in this file changed, appropriate files should be remade. But other files shouldn't.
The solution I found is this one: for every file there is flag-file which content is generating line it was made last time. After remaking file with generating lines, i check all this file, and remove them if line changed. So files which have dependences on removed files will be remade, and other files won't. But this works too slow if use msys-make.
Can you suggest any other solution, which doesn't need many extra calls to file system and executable runs.
If I understand your description correctly, what you're describing is a Makefile which depends on another file that is functionally a Makefile but for unknown reasons uses a different format.
Turn that file into Makefile format and include it into the original Makefile. (If you're using GNU Make.)
We have what may be similar to your problem. We have xml files, say foobar.xml which contains the dependencies for foobar.out
<files>
<file>a</file>
<file>b</file>
<file>c</file>
</files>
We decided to adhere to this simple layout, so we don't need to parse xml. We use makepp (because we got fed up with gmake not noticing dependencies like changed commands). This gives us the builtin &sed command (the expression is actually Perl programming, but as you see, you don't need to get into it much). Here's what we do with three simple substitutions for the three kinds of lines:
%.d: %.xml
&sed 's!<files>!$(stem).out: \\! || s!<file>(.+)</file>!$$1 \\! || s!</files>!!' \
$(input) -o $(output)
include foobar.d
This produces foobar.d which we then include:
foobar.out: \
a \
b \
c \
Stackoverflow is swallowing the last empty line here, which avoids having to worry about the trailing backslash.