Problem
Here is a short Makefile with a circular dependency introduced by inadvertence.
egg : chicken
chicken : egg
egg chicken foo :
#touch $#
When launching make egg, Makefile detects the circular dependency, drops it and executes the recipe. I would like to stop the execution of the recipe.
Clumsy solution
So far, the best solution I found is the following:
egg : chicken
chicken : egg
egg chicken foo :
#mqe=`make --question $# 2>&1 > /dev/null`; \
if [ -z "$$mqe" ]; then \
touch $# ; \
else \
echo "*** ERROR *** $$mqe" ; false ; \
fi
Is there a more elegant way to stop Makefile when a circular dependency is detected ?
Context
The example I gave is really simplified. The real project is a huge Fortran code where circular dependencies on modules are introduced by inadvertence and are not detected by sfmakedepend.
Related
I've tweak "a bit" Make so I can use it as a "kind of" cli for some tasks.
MAKEFLAGS += --no-builtin-rules
MAKEFLAGS += --no-builtin-variables
MAKEFLAGS += --no-print-directory
SHELL := /bin/bash
.ONESHELL:
.PHONY: project_list
project_list: all_projects_info.json
echo "Filtering project list with:" >&2
echo " PROJECT_FILTER: $(PROJECT_FILTER)" >&2
jq -r -S '.[] | select(
(.projectId | test("$(PROJECT_FILTER)"))
) | .projectId' $^ > $#
.PHONY: get_storage_info
get_storage_info: project_list
PROJECT_LIST=$$(cat $<)
$(MAKE) -f $(MKFILE) -j storage_info.json PROJECT_LIST="$$PROJECT_LIST"
all_projects_info.json:
curl -X GET https://toto/get_all_my_projects_info >$#
# here it's PHONY because we want to always rebuild it
.PHONY: storage_info.json
storage_info.json: $(STORAGE_INFO_JSON_FILES)
jq -s -S '[.[]?.items?[]?]' $(STORAGE_INFO_JSON_FILES) > $#
storage_info/:
mkdir -p $#
STORAGE_INFO_JSON_FILES=$(foreach project_name,$(PROJECT_LIST),storage_info/$(project_name).json)
$(STORAGE_INFO_JSON_FILES): storage_info/%.json: | storage_info/
curl \
-X GET \
"https://storage_api/list_s3?project=$*" \
2> /dev/null > $#
As you can see here, I've got 2 "command":
project_list witch list all project I can access too,
get_storage_info witch list all bucket in projects.
The trick here is because I've got a lot of projects and buckets, I may want to filter like this:
make get_storage_info PROJECT="foo"
And it will print ONLY bucket in project with foo in their name.
It's quit handy and fast (only the first time it may be slow, the time to get all informations).
What is bothering me, I've not found a better way than to call a sub make command (with the exact list of project to take into account).
Is it possible to express dynamic dependencies of a target ?
But something that can result from another target ?
Thanks.
I don't see anything wrong with invoking a submake. That's IMO the best way to do it, especially if you want to add -j to it.
It's not really possible to get rid of this easily. It's not the fact that you want to express a dynamic dependency: that can be done. The problem is you want the list of dependencies to be extracted from the results of running another rule. But that's not how make works: make always starts with the final target and works its way backwards. So, by the time you get around to building the prerequisite file, the target that depended on it has already been processed (not its recipe of course, but all the prerequisites).
The working example below always runs touch fileA with each run of make. How can I stop this?
all: analysis.txt
analysis.txt: countries
touch $#
countries: usa.txt mexico.txt
$(countries): %.txt
%.txt:
echo "bash ./cityGenerator $*" > $#
fileA depends on workflowB. So as long as workflowB is out of date, fileA will be out of date and the touch will be run.
workflowB, as shown here, is marked .PHONY so it will ALWAYS be considered out of date by make, so fileA will ALWAYS be rebuilt.
Even if it weren't .PHONY, there is no rule to create a file named workflowB so it would still always be out of date.
As for how you can stop it, that depends entirely on what these things do which you haven't told us, and why you structured workflowB that way which you also haven't told us.
One way to solve it would be to remove the .PHONY and give workflowB a recipe that touches a file, then it would be up to date.:
workflowB: workflowB_files
touch $#
But of course, that might defeat the purpose of this (which, again, you haven't told us).
Your analysis.txt depends on countries which is considered as a file, but such file is never created, hence the rule is always run. You can find out by the debug output:
$ make -dr
...
Finished prerequisites of target file 'analysis.txt'.
Prerequisite 'countries' of target 'analysis.txt' does not exist.
Must remake target 'analysis.txt'.
touch analysis.txt
...
It seems like you are treating countries as a variable, so maybe this could be reworked:
$ cat Makefile
all: analysis.txt
countries := usa.txt mexico.txt
analysis.txt: $(countries)
touch $#
%.txt:
echo "bash ./cityGenerator $*" > $#
Now there is no countries file considered and make behaves as expected:
$ make
echo "bash ./cityGenerator usa" > usa.txt
echo "bash ./cityGenerator mexico" > mexico.txt
touch analysis.txt
$ make
make: Nothing to be done for 'all'.
I need to restart the make process in case some intermediate target gets (re)build.
This is the case when a PIP requirements file gets (re)compiled, because the checksum of the resulting file is used in the path to the virtualenv that is used in the Makefile.
If the requirements file gets updated, the same $(MAKECMDGOALS) should get rebuild from the beginning.
I have come up with the following, but this requires to make the outer/inital make process fail (exit 1), which I would like to avoid.
foo:
echo "foo"
# Need to restart make if this has been used as intermediate target.
if [ $(MAKECMDGOALS) != "$#" ]; then \
echo "Restarting make..."; \
touch $#; \
$(MAKE) $(MAKECMDGOALS); \
exit 1; \
fi
bar: foo
echo "bar"
.PHONY: bar
A makefile on the lines below might fit the bill. bar and baz are goals
that will be sabotaged if they intermediately make foo (a.k.a tricky goals).
There's no restarting involved: instead, it is arranged that the tricky
goals are only made by a fresh $(MAKE), after the top make has ensured foo is
up-to-date.
.phony: all clean
tricky = bar baz
all:
$(MAKE) $(tricky)
foo:
echo "foo"
touch $#
clean:
rm -f foo
ifndef simple
tricky_goals = $(filter $(tricky),$(MAKECMDGOALS))
ifneq ($(tricky_goals),)
$(tricky_goals): foo
$(MAKE) simple=y $#
endif
else # ifndef simple
# All tricky goals follow...
bar: foo
echo "bar"
baz: foo
echo "baz"
endif # ifndef simple
The behaviour depends on whether the variable simple is defined, which
by default it is not.
If simple is defined then the goals, whatever they are, will be made
the obvious way - the way they'd be made if it weren't for the foo pitfall.
If simple is undefined then tricky_goals becomes the tricky members $(MAKECMDGOALS).
If $(tricky_goals) is empty there is nothing tricky to do. Otherwise the target list
$(tricky_goals) becomes dependent on foo. So if foo is out-of-date, it gets made now.
Finally each tricky goal is made by calling a simple $(MAKE) for it,
in which foo will be up-todate. Any non-tricky goals will be made the simple way.
I've included all and clean goals for completeness. Note that all
needs coded to go the tricky way.
I have a Makefile that does performs a task if it hasn't happened in the last hour. It does so like this:
HOUR_FROM_NOW = $(shell perl -e '($$s,$$m,$$h,$$d,$$M)=localtime(time()+3600); printf("%02d%02d%02d%02d\n",$$M+1,$$d,$$h,$$m);')
NOW_FILE = $(shell mkdir -p .make; touch .make/now; echo .make/now )
.PHONY: externals
externals: $(PROJECTS:%=.make/proj_%)
.make/proj_%: $(NOW_FILE)
$(MAKE) -s $(*F)
touch -t $(HOUR_FROM_NOW) $#
.PHONY: $(PROJECTS)
$(PROJECTS):
# do stuff, specifically, clone git-repo if not exists, else pull latest
That part works great, except that I now get warnings:
make: Warning: File `.make/proj' has modification time 3.5e+03 s in the future
make: Nothing to be done for `externals'.
make: warning: Clock skew detected. Your build may be incomplete.
Anyone know how to suppress those warnings? (Or to do a periodic task in a makefile)
Most versions of touch I have come across can do some date time maths which allows for setting the timestamp of a file directly via the --date option.
That and the fact that variables assigned with := are only "evaluated once" makes this a bit easier to read.
HOUR_AGO := .make/hour_ago
__UGLY := $(shell mkdir -p .make && touch --date='1hour ago' $(HOUR_AGO))
# The preceding line will be executed once
.make/proj_%: .make/hour_ago | .make
$(MAKE) -s $(*F)
#touch $#
.make:
mkdir -p $#
I'm using something very similar to this to periodically refresh login tokens.
Never would have thought of it if it wasn't for Dave's answer though.
The directory is created by specifying it as a order-only-prerequisite
I suspect that the + 3600 is at fault. What happens if you remove it?
I thought and thought, and then the stupid-obvious solution hit me ...
Instead of setting timestamps in the future with HOUR_FROM_NOW, I use the real time and compare with HOUR_AGO_FILE ...
HOUR_AGO = $(shell perl -e '($$s,$$m,$$h,$$d,$$M)=localtime(time()-3600); printf("%02d%02d%02d%02d\n",$$M+1,$$d,$$h,$$m);')
HOUR_AGO_FILE = $(shell mkdir -p .make; touch -t $(HOUR_AGO) .make/hour_ago; echo .make/hour_ago )
.PHONY: externals
externals: $(PROJECTS:%=.make/proj_%)
.make/proj_%: $(HOUR_AGO_FILE)
$(MAKE) -s $(*F)
#touch $#
Criteria: Makefile is a GNU Make Makefile - I'm not interested in makepp, qmake, cmake, etc. They're all nice (especially cmake), but this is for work and at work we use GNU Make. The optimal solution is a pure Makefile solution rather than a shell script that parses make for you.
I also don't want to do a 'continue on failure' solution - if it's broken, it's broken and needs to be fixed.
The situation is this, I've got a makefile that builds several directories in parallel - if one of them fails, of course the whole build fails, but not until all the running makes run to completion (or failure). This means that the reason why make actually failed is buried somewhere arbitrarily far from the end of make's output.
Here's an example of what I've got:
all: $(SUBDIRS)
SUBDIRS = \
apple \
orange \
banana \
pineapple \
lemon \
watermelon \
grapefruit
$(SUBDIRS):
cd $# && $(MAKE) $(MFLAGS) 2>&1 | sed -e "s/^/$(notdir $(#)): /g"
If I run 'make -j 5' and 'orange' happens to fail - I'd like to see a table like this at the end
of the make process
apple - passed
orange - FAILED
banana - passed
pineapple - passed
lemon - passed
I've considered having an && echo "passed" >.result || echo "FAILED" >.result, but make still needs some sort of TRAP or __onexit() cleanup command to print at them on exit.
Any Makefile ninjas out there have a pure-makefile solution for this?
un-edit - my solution wasn't actually working the way I had hoped.. STYMIED!
When you want make to abort at the first failure, end immediately and kill all in-flight jobs instead of waiting for them to finish, you need to patch GNU Make like this
http://lists.gnu.org/archive/html/bug-make/2009-01/msg00035.html
Then you need to set a trap for every shell that make invokes (as well as set -o pipefail if you use a pipe), as described in this post http://lists.gnu.org/archive/html/help-make/2009-02/msg00011.html
In a nutshell:
target1:
trap 'kill $$(jobs -p)'; command && something || something-else
target2:
trap 'kill $$(jobs -p)'; set -o pipefail; command | sed '...'
The only way I see is self-execution with a sub-make:
all : subdirs
subdirs :
$(MAKE) -f $(lastword $(MAKEFILE_LIST)) subdirs-recursive || cat log
subdirs-recursive: $(SUBDIRS)