I have a Makefile that does performs a task if it hasn't happened in the last hour. It does so like this:
HOUR_FROM_NOW = $(shell perl -e '($$s,$$m,$$h,$$d,$$M)=localtime(time()+3600); printf("%02d%02d%02d%02d\n",$$M+1,$$d,$$h,$$m);')
NOW_FILE = $(shell mkdir -p .make; touch .make/now; echo .make/now )
.PHONY: externals
externals: $(PROJECTS:%=.make/proj_%)
.make/proj_%: $(NOW_FILE)
$(MAKE) -s $(*F)
touch -t $(HOUR_FROM_NOW) $#
.PHONY: $(PROJECTS)
$(PROJECTS):
# do stuff, specifically, clone git-repo if not exists, else pull latest
That part works great, except that I now get warnings:
make: Warning: File `.make/proj' has modification time 3.5e+03 s in the future
make: Nothing to be done for `externals'.
make: warning: Clock skew detected. Your build may be incomplete.
Anyone know how to suppress those warnings? (Or to do a periodic task in a makefile)
Most versions of touch I have come across can do some date time maths which allows for setting the timestamp of a file directly via the --date option.
That and the fact that variables assigned with := are only "evaluated once" makes this a bit easier to read.
HOUR_AGO := .make/hour_ago
__UGLY := $(shell mkdir -p .make && touch --date='1hour ago' $(HOUR_AGO))
# The preceding line will be executed once
.make/proj_%: .make/hour_ago | .make
$(MAKE) -s $(*F)
#touch $#
.make:
mkdir -p $#
I'm using something very similar to this to periodically refresh login tokens.
Never would have thought of it if it wasn't for Dave's answer though.
The directory is created by specifying it as a order-only-prerequisite
I suspect that the + 3600 is at fault. What happens if you remove it?
I thought and thought, and then the stupid-obvious solution hit me ...
Instead of setting timestamps in the future with HOUR_FROM_NOW, I use the real time and compare with HOUR_AGO_FILE ...
HOUR_AGO = $(shell perl -e '($$s,$$m,$$h,$$d,$$M)=localtime(time()-3600); printf("%02d%02d%02d%02d\n",$$M+1,$$d,$$h,$$m);')
HOUR_AGO_FILE = $(shell mkdir -p .make; touch -t $(HOUR_AGO) .make/hour_ago; echo .make/hour_ago )
.PHONY: externals
externals: $(PROJECTS:%=.make/proj_%)
.make/proj_%: $(HOUR_AGO_FILE)
$(MAKE) -s $(*F)
#touch $#
Related
I've tweak "a bit" Make so I can use it as a "kind of" cli for some tasks.
MAKEFLAGS += --no-builtin-rules
MAKEFLAGS += --no-builtin-variables
MAKEFLAGS += --no-print-directory
SHELL := /bin/bash
.ONESHELL:
.PHONY: project_list
project_list: all_projects_info.json
echo "Filtering project list with:" >&2
echo " PROJECT_FILTER: $(PROJECT_FILTER)" >&2
jq -r -S '.[] | select(
(.projectId | test("$(PROJECT_FILTER)"))
) | .projectId' $^ > $#
.PHONY: get_storage_info
get_storage_info: project_list
PROJECT_LIST=$$(cat $<)
$(MAKE) -f $(MKFILE) -j storage_info.json PROJECT_LIST="$$PROJECT_LIST"
all_projects_info.json:
curl -X GET https://toto/get_all_my_projects_info >$#
# here it's PHONY because we want to always rebuild it
.PHONY: storage_info.json
storage_info.json: $(STORAGE_INFO_JSON_FILES)
jq -s -S '[.[]?.items?[]?]' $(STORAGE_INFO_JSON_FILES) > $#
storage_info/:
mkdir -p $#
STORAGE_INFO_JSON_FILES=$(foreach project_name,$(PROJECT_LIST),storage_info/$(project_name).json)
$(STORAGE_INFO_JSON_FILES): storage_info/%.json: | storage_info/
curl \
-X GET \
"https://storage_api/list_s3?project=$*" \
2> /dev/null > $#
As you can see here, I've got 2 "command":
project_list witch list all project I can access too,
get_storage_info witch list all bucket in projects.
The trick here is because I've got a lot of projects and buckets, I may want to filter like this:
make get_storage_info PROJECT="foo"
And it will print ONLY bucket in project with foo in their name.
It's quit handy and fast (only the first time it may be slow, the time to get all informations).
What is bothering me, I've not found a better way than to call a sub make command (with the exact list of project to take into account).
Is it possible to express dynamic dependencies of a target ?
But something that can result from another target ?
Thanks.
I don't see anything wrong with invoking a submake. That's IMO the best way to do it, especially if you want to add -j to it.
It's not really possible to get rid of this easily. It's not the fact that you want to express a dynamic dependency: that can be done. The problem is you want the list of dependencies to be extracted from the results of running another rule. But that's not how make works: make always starts with the final target and works its way backwards. So, by the time you get around to building the prerequisite file, the target that depended on it has already been processed (not its recipe of course, but all the prerequisites).
I'd like to use make to process a large number of inputs to outputs using a script (python, say.) The problem is that the script takes an incredibly short amount of time to run per input, but the initialization takes a while (python engine + library initialization.) So, a naive makefile that just has an input->output rule ends up being dominated by this initialization time. Parallelism doesn't help with that.
The python script can accept multiple inputs and outputs, as so:
python my_process -i in1 -o out1 -i in2 -o out2 ...
and this is the recommended way to use the script.
How can I make a Makefile rule that best uses my_process, by sending in out of date input-output pairs in batches? Something like parallel but aware of which outputs are out of date.
I would prefer to avoid recursive make, if at all possible.
I don't completely grasp your problem: do you really want make to operate in batches or do you want a kind of perpetual make process checking the file system on the fly and feeding to the Python process whenever it finds necessary? If the latter, this is quite the opposite of a batch mode and rather a pipeline.
For the batch mode there is a work-around which needs a dummy file recording the last runnning time. In this case we are abusing make for because the makefile is in this part a one-trick pony which is unintuitive and against the good rules:
SOURCES := $(wildcard in*)
lastrun : $(SOURCES)
python my_process $(foreach src,$?,-i $(src) -o $(patsubst in%,out%,$(src)))
touch lastrun
PS: please note that this solution has a substantial flaw in that it doesn't detect the update of in-files when they happen during the run of the makefile. All in all it is more advisable to simply collect the filenames of the in-files which were updated by the update process itself and avoid make althogether.
This is what I ended up going with, a makefile with one layer of recursion.
I tried using $? both with grouped and ungrouped targets, but couldn't get the exact behavior needed. If one of the output targets was deleted, the rule would be re-run but $? wouldn't necessarily have some input files but not the correct corresponding input file, very strange.
Makefile:
all:
INDIR=in
OUTDIR=out
INFILES=$(wildcard in/*)
OUTFILES=$(patsubst in/%, out/%, $(INFILES))
ifdef FIRST_PASS
#Discover which input-output pairs are out of date
$(shell mkdir -p $(OUTDIR); echo -n > $(OUTDIR)/.needs_rebuild)
$(OUTFILES) : out/% : in/%
#echo $# $^ >> $(OUTDIR)/.needs_rebuild
all: $(OUTFILES)
#echo -n
else
#Recurse to run FIRST_PASS, builds .needs_rebuild:
$(shell $(MAKE) -f $(CURDIR)/$(firstword $(MAKEFILE_LIST)) FIRST_PASS=1)
#Convert .needs_rebuild into batches, creates all_batches phony target for convenience
$(shell cat $(OUTDIR)/.needs_rebuild | ./make_batches.sh 32 > $(OUTDIR)/.batches)
-include $(OUTDIR)/.batches
batch%:
#In this rule, $^ is all inputs needing rebuild.
#The corresponding utputs can be computed using a patsubst:
targets="$(patsubst in/%, out/%, $^)"; touch $$targets
clean:
rm -rf $(OUTDIR)
all: all_batches
endif
make_batches.sh:
#!/bin/bash
set -beEu -o pipefail
batch_size=$1
function _make_batches {
batch_num=$1
shift 1
#echo ".PHONY: batch$batch_num"
echo "all_batches: batch$batch_num"
while (( $# >= 1 )); do
read out in <<< $1
shift 1
echo "batch$batch_num: $in"
echo "$out: batch$batch_num"
done
}
export -f _make_batches
echo ".PHONY: all_batches"
parallel -N$batch_size -- _make_batches {#} {} \;
Unfortunately, the makefile is a one trick pony and there's quite a bit of boilerplate to pull this recipe off.
I'm using GNU Make to build a dynamic web site but I need to build two versions. As a net result currently I run my makefile using two command line incantations. This is inefficient and can result in errors (I don't always remember to run both versions or make a typing error and they fail) thus I want to make it one build.
The Command Line incantations are:
sudo make release build=test
sudo make release build=release
The two incantations activate ifeq blocks that set the path and modify some files.
Part of the much simplified (to help readability) top level makefile:
subs = today tomorrow
files = index.php foot.php
ifeq ($(build),test)
export path = /var/www/html/cgi-test
$(shell cp -f head-test.php head.php)
$(shell sed -i '/"DB"/c\ define("DB", "db_test");' subs.php)
else ifeq ($(build),release)
export path = /var/www/html/cgi-new
$(shell cp -f head-release.php head.php)
$(shell sed -i '/"DB"/c\ define("DB", "db_new");' subs.php)
endif
.PHONY: all go
all:
$(MAKE) go
#for ALL in $(subs);\
do $(MAKE) -C $$ALL all || exit $$?;\
done;
go:
cp $(files) $(path)/.
The sub-makefiles have a very similar structure but don't need the ifeq blocks because the files and paths have already been setup.
I think I can simply move the shell commands into the .PHONY rules but I can't do that with the exports because I get errors "export: : bad variable name".
I could do it with a batch file and call the makefile twice but that sidesteps the problem rather than cures it and I wish to learn from the process.
So can anybody show me the way to do this in a makefile?
Thanks to Tripleee here is the answer that finally worked back ported to match my starting post. The one major change is that I have gone back to 'all' as the rule I expect to start the build habits die hard! - Thanks
.PHONY: all both rel-test rel-release
cgi-test := cgi-test
db-test := db_test
cgi-release := cgi-new
db-release := db_new
subs = today tomorrow
files = index.php foot.php
all: both
both: rel-test rel-release
rel-test rel-release: rel-%:
cp -f head-$*.php head.php
sed -i '/"DB"/c\ define("DB", "$(db-$*)");' subs.php
$(MAKE) go path=/var/www/html/strutts/$(cgi-$*)
#for ALL in $(subs);\
do $(MAKE) build=$* path=/var/www/html/strutts/$(cgi-$*) -C $$ALL all || exit $$?;\
done;
Something like this?
.PHONY: both rel-test rel-release
both: rel-test rel-release
cgi-test := cgi-test
db-test := db_test
cgi-release := cgi-new
db-release := db_new
rel-%:
cp -f head-$*.php head.php
sed -i '/"DB"/c\ define("DB", "$(db-$*)")' subs.php
$(MAKE) release build=$* path=/var/www/html/$(cgi-$*)
The reason the export can't be moved into a recipe is that you are using the export command of make itself, not the shell's command with the same name.
You absolutely should not use sudo unless you specifically require the output files to be owned and only writable by root. Even then, running as much as possible as a regular user would be proper hygiene; maybe add a sudo command inside the Makefile to copy the files to their final location.
I'm trying to create a Makefile that will download and process file a file to generate targets, this is a simplified version:
default: all
.PHONY: all clean filelist.d
clean:
#rm -fv *.date *.d
#The actual list comes from a FTP file, but let's simplify things a bit
filelist.d:
#echo "Getting updated filelist..."
#echo "LIST=$(shell date +\%M)1.date $(shell date +\%M)2.date" > $#
#echo 'all: $$(LIST)' >> $#
%.date:
touch $#
-include filelist.d
Unfortunately the target all doesn't get updated properly on the first run, it needs to be run again to get the files. This is the output I get from it:
$ make
Getting updated filelist...
make: Nothing to be done for `default'.
$ make
Getting updated filelist...
touch 141.date
touch 142.date
touch 143.date
I'm using GNU Make 3.81 whose documentation states that it reloads the whole thing if the included files get changed. What is going wrong?
You have specified filelist.d as a .PHONY target, so make believes making that target doesn't actually update the specified file. However, it does, and the new contents are used on the next run. For the first run, the missing file isn't an error because include is prefixed with the dash.
Remove filelist.d from .PHONY. However, remember it won't be regenerated again until you delete it (as it doesn't depend on anything).
By the same token, you should include "default" in .PHONY.
I wrote a shell script rather than lump all this in the makefile:
#!/bin/bash
# Check whether file $1 is less than $2 days old.
[ $# -eq 2 ] || {
echo "Usage: $0 FILE DAYS" >&2
exit 2
}
FILE="$1"
DAYS="$2"
[ -f "$FILE" ] || exit 1 # doesn't exist or not a file
TODAY=$(date +%s)
TARGET=$(($TODAY - ($DAYS * 24 * 60 * 60)))
MODIFIED=$(date -r "$FILE" +%s)
(($TARGET < $MODIFIED))
Replace X with the max number of days that can pass before filelist.d is downloaded again:
filelist.d: force-make
./less-than-days $# X || command-to-update
.PHONY: force-make
force-make:
Now filelist.d depends on a .PHONY target, without being a phony itself. This means filelist.d is always out of date (phony targets are always "new"), but its recipe only updates the file periodically.
Unfortunately, this requires you to write the update command as a single command, and space may be a problem if it is long. In that case, I would put it in a separate script as well.
I'm wondering how I can avoid some echo in a Makefile :
clean:
rm -fr *.o
this rule will print:
$>make clean
rm -fr *.o
$>
How can I avoid that?
To start with: the actual command must be on the next line (or at least that is the case with GNU Make, it might be different with other Make's - I'm not sure of that)
clean:
rm -rf *.o
(note, you need a TAB before rm -rf *.o as in every rule)
Making it silent can be done by prefixing a #:
so your makefile becomes
clean:
#rm -rf *.o
If there are no *.o files to delete, you might still end up with an error message. To suppress these, add the following
clean:
-#rm -rf *.o 2>/dev/null || true
2>/dev/null pipes any error message to /dev/null - so you won't see any errors
the - in front of the command makes sure that make ignores a non-zero return code
In fact I was looking for something else, adding this line to the Makefile :
.SILENT:clean
while execute every step of the "clean" target silently.
Until someone point some drawback to this, I use this as my favourite solution!
I'm responding to this ancient topic because it comes up high in search and the answers are confusing. To do just what the user wants,all that is needed is:
clean:
#rm -f *.o
The # means that make will not echo that command.
The -f argument to rm tells rm to ignore any errors, like there being no *.o files, and to return success always.
I removed the -r from the OPs example, because it means recursive and here we are just rming .o files, nothing to recurse.
There's no need for the 2>&1 >/dev/null because with the -f there will be no errors printed.
.SILENT: clean
works in place of the #, but it isn't at the same place in the Makefile as the command that it affects, so someone maintaining the project later might be confused. That's why # is preferred. It is better locality of reference.
If you put an # in front of the command, it doesn't echo onto the shell. Try changing rm to #rm. (Reference)
From the manual: .SILENT is essentially obsolete since # is more flexible.
Much worse is that make prints far too much information. Warning/error/private messages are buried in the output. On the other hand -s (.SILENT) suppresses just anything. Especially the "nothing to be done" and "up to date" messages can be a pain. There is no option to suppress them. You have to filter them out actively or use something like colormake. Here is a solution for grep:
make | egrep -hiv 'nothing to be done|up to date'
But the output will have line numbers. The Perl solution is therefore better, because it suppresses line numbers and flushes stdout immediately:
make | perl -ne '$|=1; print unless /nothing to be done|up to date/i'
Make's a flawed tool. "What’s Wrong With GNU make?" explains this better than I can.
There's a great article on using .SILENT that explains how to conditionally activate it.
I have used that information to put this in my Makefile:
# Use `make V=1` to print commands.
$(V).SILENT:
# Example rule, only the #echo needs to be added to existing rules
*.o: %.c
#echo " [CC] $<"
gcc ...
What this does is if you run make normally, normal output is silenced and instead the echo commands work:
$ make
[CC] test.c
[CC] test2.c
But it allows you to debug problems by passing the V=1 parameter, which still shows the [CC] messages as it helps break up the output, but the traditional Makefile output is also visible:
$ make V=1
[CC] test.c
gcc ...
[CC] test2.c
gcc ...