I have few simple targets which create some files for me.
Example:
$(MAKE_INA):
#echo Building ASM compilation flags file $(notdir $(MAKE_INA))
#$(foreach i, $(sort $(ASMFLAGS) $(PFLAGS) $(ALL_INC_DIR) $(cppGetPreProcessorDefines)), $(shell echo $i >> $# ))
The target works fine, the file is being created and echo text displayed, but in that order (first the file is build then the echo is shown on cmd.exe console).
I guess that is related somehow with output buffering, but I was not able to find the way to flush the echos immediately.
Any hint? Is it even possible?
I am using Gnu Make 4.0
You are mixing up contexts here.
The first #echo line is a recipe line and is run by the shell when the target runs.
The second $(foreach) line is within the rule but is a make context line and is evaluated by make before running the recipe lines. Within that line $(shell) is also a make command and is run during the make expansion of the recipe instead of being run by the shell at recipe execution time.
To do what you want you can just use:
$(MAKE_INA):
#echo Building ASM compilation flags file $(notdir $(MAKE_INA))
#printf "%s\\n" $(sort $(ASMFLAGS) $(PFLAGS) $(ALL_INC_DIR) $(cppGetPreProcessorDefines)) >> $#
Which does the echoing at recipe execution time (so has the right order) and uses a single call to the printf built-in to output to the file instead of running N calls to echo.
Edit: For Windows cmd.exe compat you need to use echo $i >> $# & as the $(foreach) body so that cmd.exe runs multiple commands correctly.
If you did want to keep the N echo calls then you could use:
$(MAKE_INA):
#echo Building ASM compilation flags file $(notdir $(MAKE_INA))
#$(foreach i, $(sort $(ASMFLAGS) $(PFLAGS) $(ALL_INC_DIR) $(cppGetPreProcessorDefines)), echo $i >> $#; ))
Which has the $(foreach) output echo XXX >> $#; ....; echo ZZZ >> $#; as the recipe line to then execute during recipe execution.
Related
I'd like to use make to process a large number of inputs to outputs using a script (python, say.) The problem is that the script takes an incredibly short amount of time to run per input, but the initialization takes a while (python engine + library initialization.) So, a naive makefile that just has an input->output rule ends up being dominated by this initialization time. Parallelism doesn't help with that.
The python script can accept multiple inputs and outputs, as so:
python my_process -i in1 -o out1 -i in2 -o out2 ...
and this is the recommended way to use the script.
How can I make a Makefile rule that best uses my_process, by sending in out of date input-output pairs in batches? Something like parallel but aware of which outputs are out of date.
I would prefer to avoid recursive make, if at all possible.
I don't completely grasp your problem: do you really want make to operate in batches or do you want a kind of perpetual make process checking the file system on the fly and feeding to the Python process whenever it finds necessary? If the latter, this is quite the opposite of a batch mode and rather a pipeline.
For the batch mode there is a work-around which needs a dummy file recording the last runnning time. In this case we are abusing make for because the makefile is in this part a one-trick pony which is unintuitive and against the good rules:
SOURCES := $(wildcard in*)
lastrun : $(SOURCES)
python my_process $(foreach src,$?,-i $(src) -o $(patsubst in%,out%,$(src)))
touch lastrun
PS: please note that this solution has a substantial flaw in that it doesn't detect the update of in-files when they happen during the run of the makefile. All in all it is more advisable to simply collect the filenames of the in-files which were updated by the update process itself and avoid make althogether.
This is what I ended up going with, a makefile with one layer of recursion.
I tried using $? both with grouped and ungrouped targets, but couldn't get the exact behavior needed. If one of the output targets was deleted, the rule would be re-run but $? wouldn't necessarily have some input files but not the correct corresponding input file, very strange.
Makefile:
all:
INDIR=in
OUTDIR=out
INFILES=$(wildcard in/*)
OUTFILES=$(patsubst in/%, out/%, $(INFILES))
ifdef FIRST_PASS
#Discover which input-output pairs are out of date
$(shell mkdir -p $(OUTDIR); echo -n > $(OUTDIR)/.needs_rebuild)
$(OUTFILES) : out/% : in/%
#echo $# $^ >> $(OUTDIR)/.needs_rebuild
all: $(OUTFILES)
#echo -n
else
#Recurse to run FIRST_PASS, builds .needs_rebuild:
$(shell $(MAKE) -f $(CURDIR)/$(firstword $(MAKEFILE_LIST)) FIRST_PASS=1)
#Convert .needs_rebuild into batches, creates all_batches phony target for convenience
$(shell cat $(OUTDIR)/.needs_rebuild | ./make_batches.sh 32 > $(OUTDIR)/.batches)
-include $(OUTDIR)/.batches
batch%:
#In this rule, $^ is all inputs needing rebuild.
#The corresponding utputs can be computed using a patsubst:
targets="$(patsubst in/%, out/%, $^)"; touch $$targets
clean:
rm -rf $(OUTDIR)
all: all_batches
endif
make_batches.sh:
#!/bin/bash
set -beEu -o pipefail
batch_size=$1
function _make_batches {
batch_num=$1
shift 1
#echo ".PHONY: batch$batch_num"
echo "all_batches: batch$batch_num"
while (( $# >= 1 )); do
read out in <<< $1
shift 1
echo "batch$batch_num: $in"
echo "$out: batch$batch_num"
done
}
export -f _make_batches
echo ".PHONY: all_batches"
parallel -N$batch_size -- _make_batches {#} {} \;
Unfortunately, the makefile is a one trick pony and there's quite a bit of boilerplate to pull this recipe off.
I have the following lines in my makefile:
.PHONY : clean
clean:
#echo "Running Clean"
$(shell if [ -e exe ]; then rm exe; else echo "no files"; fi)
When I run:
make clean
I get the following output on the shell
Running Clean
no files
make: no: Command not found
Makefile:22: recipe for target 'clean' failed
make: *** [clean] Error 127
Any suggestions?
The problem is the use of $(shell ...). What you want is:
.PHONY : clean
clean:
#echo "Running Clean"
#if [ -e exe ]; then rm exe; else echo "no files"; fi
As far as an explanation of what's going wrong -- when you first run the clean target, make will expand all make variables and functions in the recipes before it starts running them -- because $(shell ...) only has one $, this is considered a make function. Make runs the command, which outputs no files to stdout, and replaces the call with that string, and then starts executing the recipes... So now make sees the following:
clean:
#echo "Running Clean"
no files
When it tries to run no files, due to the lack of a #, it echos the line to the screen, and then passes the command to the shell. Because the shell doesn't recognize the keyword no it outputs the error you're seeing. Make itself then fails because the shell returned an error.
Hey all I'm the same guy who asked this question but I found an answer right after I posted this, I think I'll leave this up (unless this is against stackoverflow etiquette) in case someone else has the same problems. My solution was echoing the string to stdout.
$(shell if [ -e exe ]; then rm exe; else echo "no files" >&2; fi)
I would like to run a command before and after each target in a makefile.
so something like this
pre:
#echo pre
#echo running | mailx -s "Start {target}" user#foo.com
post:
#echo post
#echo post | mailx -s "Finish {target}" user#foo.com
j:
long_running_command && echo $# > $#
k: j
long_running_command2 && echo $# > $#
I would like to run pre and post for j and k. Ideally, I would like to get an email for each task that starts and stops.
One way to do it is to modify all the recipes in your makefile to invoke some command. You can put it into a variable so it doesn't look too gross:
START = mailto --subject 'Started target $#' me#my.host
END = mailto --subject 'Finished target $#' me#my.host
j:
$(START)
long_running_command && echo $# > $#
$(END)
k: j
$(START)
long_running_command2 && echo $# > $#
$(END)
The nice thing about this is you can pick and choose which targets you want it for; maybe some of them don't need it. The disadvantage is if the command fails you won't get any "end" email at all.
If you really want to do it for every single target, then you can write a shell script that mimics the shell's behavior but also sends mail, while running the command.
$ cat mailshell
#!/bin/sh
# get rid of the -c flag
shift
mailto --subject "started $*" me#my.host
/bin/sh -c "$#"
r=$?
mailto --subject "ended $* with exit code $r" me#my.host
exit $r
(note this is totally untested but you get the idea I hope). Then in your makefile, set SHELL to that shell:
SHELL := mailshell
j:
long_running_command && echo $# > $#
k: j
long_running_command2 && echo $# > $#
I guess you could still pick and choose by setting SHELL as a target-specific variable only for those targets you wanted to use the shell.
One downside of this is that if you have recipes that have multiple lines you'll get an email for each line individually. You can work around this by enabling .ONESHELL: to force the entire recipe to be passed to a single shell. However, I believe that this may require your mailshell tool to be more sophisticated.
If you only have one command per recipe, you can do this by changing the configuration for the shell the commands are run in.
Have the config file run the pre commands directly and trap EXIT to run the after commands in.
For example:
$ cat Makefile
SHELL := BASH_ENV=/dev/fd/3 3<<<'echo before; trap "echo after" EXIT' /bin/bash
default:
echo default
other:
echo first
echo second
$ make default
echo default
before
default
after
However this may not be what you want if a recipe runs several commands, as the before and after code will run each time.
$ make other
echo first
before
first
after
echo second
before
second
after
And I don't know of a way (outside of recursive Makefiles) to have different recipes use different shells, so this won't work if you only want to set before/after for several recipes.
I'm stuck trying to figure out how to run a program, on a set of files, using GNU Make:
I have a variable that loads some filenames alike this:
FILES=$(shell ls *.pdf)
Now I'm wanting to run a program 'p' on each of the files in 'FILES', however I can't seem to figure how to do exactly that.
An example of the 'FILES' variable would be:
"a.pdf k.pdf omg.pdf"
I've tried the $(foreach,,) without any luck, and #!bin/bash like loops seem to fail.
You can do a shell loop within the command:
all:
for x in $(FILES) ; do \
p $$x ; \
done
(Note that only the first line of the command must start with a tab, the others can have any old whitespace.)
Here's a more Make-style approach:
TARGETS = $(FILES:=_target)
all: $(TARGETS)
#echo done
.PHONY: $(TARGETS)
$(TARGETS): %_target : %
p $*
In my Makefile, I need to test if the current directory is an SVN repo or not and if it is not I want to indicate an error using the $(error) directive in Makefile.
So I plan to use the return value of $(shell svn info .) but I'm not sure how to get this value from within the Makefile.
Note: I'm not trying to get the return value in a recipe, but rather in the middle of the Makefile.
Right now I'm doing something like this, which works just because stdout is blank when it is an error:
SVN_INFO := $(shell svn info . 2> /dev/null)
ifeq ($(SVN_INFO),)
$(error "Not an SVN repo...")
endif
I'd still like to find out if it is possible to get the return value instead within the Makefile.
How about using $? to echo the exit status of the last command?
SVN_INFO := $(shell svn info . 2> /dev/null; echo $$?)
ifeq ($(SVN_INFO),1)
$(error "Not an SVN repo...")
endif
If you want to preserve the original output then you need to do some tricks. If you are lucky enough to have GNU Make 4.2 (released on 2016-05-22) or later at your disposal you can use the .SHELLSTATUS variable as follows.
var := $(shell echo "blabla" ; false)
ifneq ($(.SHELLSTATUS),0)
$(error shell command failed! output was $(var))
endif
all:
#echo Never reached but output would have been $(var)
Alternatively you could use a temporary file or play with Make's eval to store the string and/or the exit code into a Make variable. The example below gets this done but I would certainly like to see a better implementation than this embarrassingly complicated version.
ret := $(shell echo "blabla"; false; echo " $$?")
rc := $(lastword $(ret))
# Remove the last word by calculating <word count - 1> and
# using it as the second parameter of wordlist.
string:=$(wordlist 1,$(shell echo $$(($(words $(ret))-1))),$(ret))
ifneq ($(rc),0)
$(error shell command failed with $(rc)! output was "$(string)")
endif
all:
#echo Never reached but output would have been \"$(string)\"
This worked fine for me - based on #eriktous' answer with a minor modification of redirecting stdout as well to skip the output from svn info on a valid svn repo.
SVN_INFO := $(shell svn info . 1>&2 2> /dev/null; echo $$?)
ifneq ($(SVN_INFO),0)
$(error "Not an SVN repo...")
endif
Maybe something like this?
IS_SVN_CHECKED_OUT := $(shell svn info . 1>/dev/null 2>&1 && echo "yes" || echo "no")
ifne ($(IS_SVN_CHECKED_OUT),yes)
$(error "The current directory must be checked out from SVN.")
endif
I use .NOTPARALLEL and a make function:
.NOTPARALLEL:
# This function works almost exactly like the builtin shell command, except it
# stops everything with an error if the shell command given as its argument
# returns non-zero when executed. The other difference is that the output
# is passed through the strip make function (the shell function strips only
# the last trailing newline). In practice this doesn't matter much since
# the output is usually collapsed by the surroundeing make context to the
# same result produced by strip.
SHELL_CHECKED = \
$(strip \
$(if $(shell (($1) 1>/tmp/SC_so) || echo nonempty), \
$(error shell command '$1' failed. Its stderr should be above \
somewhere. Its stdout is in '/tmp/SC_so'), \
$(shell cat /tmp/SC_so)))