Suppose I have the following situation:
.SECONDEXPANSION:
rule-%: etc/$$*
some other things
here
something-something -c "cat $< | envsubst | ..." \
> $#
And I'd like to rewrite this rule, as it is a common pattern across n items, without getting clever:
.SECONDEXPANSION:
rule-%: etc/$$*
some other things
here
make something-something-$< > $#
something-something-%:
something-something -c "cat $* | envsubst | ..."
But I can't find any guarantees that this will work if, suppose, I call the outer rule, rule-% as a submake from a different directory (make, in my experience, sometimes prints directory position into the pipeline -- which can be suppressed by executing make silently, but I would never assume a user to know that).
Will this work in all cases? Alternatively, I know I can:
etc...
make something-something arg1=... arg2=...
something-something:
something-something -c "cat ${arg1} | envsubst | ..." > ${arg2}
But I'd like to avoid this.
I don't think I understand your question, but: first you should never use raw make when invoking make recursively: you must always use the $(MAKE) variable.
Second, if you want the recursive make to not print directory info, or even to run silently, you can just add those options to the submake: it won't matter what options the person who invoked the outer make specified:
$(MAKE) -s --no-print-directory something-something-$< > $#
If you just want to keep your code DRY, you can just make use of a template, like so:
$ cat Makefile
.SECONDEXPANSION:
define something
bash -c "cat $(1) | envsubst"
endef
rule-%: etc/$$*
$(call something,$<) > $#
another-rule-%: etc/$$*
$(call something,$<) > $#
Output:
$ make rule-foo another-rule-foo rule-bar another-rule-bar
bash -c "cat etc/foo | envsubst" > rule-foo
bash -c "cat etc/foo | envsubst" > another-rule-foo
bash -c "cat etc/bar | envsubst" > rule-bar
bash -c "cat etc/bar | envsubst" > another-rule-bar
Related
I've tweak "a bit" Make so I can use it as a "kind of" cli for some tasks.
MAKEFLAGS += --no-builtin-rules
MAKEFLAGS += --no-builtin-variables
MAKEFLAGS += --no-print-directory
SHELL := /bin/bash
.ONESHELL:
.PHONY: project_list
project_list: all_projects_info.json
echo "Filtering project list with:" >&2
echo " PROJECT_FILTER: $(PROJECT_FILTER)" >&2
jq -r -S '.[] | select(
(.projectId | test("$(PROJECT_FILTER)"))
) | .projectId' $^ > $#
.PHONY: get_storage_info
get_storage_info: project_list
PROJECT_LIST=$$(cat $<)
$(MAKE) -f $(MKFILE) -j storage_info.json PROJECT_LIST="$$PROJECT_LIST"
all_projects_info.json:
curl -X GET https://toto/get_all_my_projects_info >$#
# here it's PHONY because we want to always rebuild it
.PHONY: storage_info.json
storage_info.json: $(STORAGE_INFO_JSON_FILES)
jq -s -S '[.[]?.items?[]?]' $(STORAGE_INFO_JSON_FILES) > $#
storage_info/:
mkdir -p $#
STORAGE_INFO_JSON_FILES=$(foreach project_name,$(PROJECT_LIST),storage_info/$(project_name).json)
$(STORAGE_INFO_JSON_FILES): storage_info/%.json: | storage_info/
curl \
-X GET \
"https://storage_api/list_s3?project=$*" \
2> /dev/null > $#
As you can see here, I've got 2 "command":
project_list witch list all project I can access too,
get_storage_info witch list all bucket in projects.
The trick here is because I've got a lot of projects and buckets, I may want to filter like this:
make get_storage_info PROJECT="foo"
And it will print ONLY bucket in project with foo in their name.
It's quit handy and fast (only the first time it may be slow, the time to get all informations).
What is bothering me, I've not found a better way than to call a sub make command (with the exact list of project to take into account).
Is it possible to express dynamic dependencies of a target ?
But something that can result from another target ?
Thanks.
I don't see anything wrong with invoking a submake. That's IMO the best way to do it, especially if you want to add -j to it.
It's not really possible to get rid of this easily. It's not the fact that you want to express a dynamic dependency: that can be done. The problem is you want the list of dependencies to be extracted from the results of running another rule. But that's not how make works: make always starts with the final target and works its way backwards. So, by the time you get around to building the prerequisite file, the target that depended on it has already been processed (not its recipe of course, but all the prerequisites).
I am trying to write a simple Makefile to build .expected files and compare them but I am failing.
APSSCHED=../../bin/apssched
BASE=.:../../base:../../examples
FLAGS=-DCOT
EXAMPLES=../../examples/
CASES=simple-binding1 simple-binding2
# skipping lines doesn't work ...
# run command and skip the first line
%.aps:
${APSSCHED} ${FLAGS} -p ${BASE} ${EXAMPLES}/$* | tail -n +2
# get all cases as an array to pipe it to different make targets
# maybe overcomplicating
cases:
echo ${CASES} | \
awk '{split($$0,numbers," ")} END {for(n in numbers){ print numbers[n] }}'
# create all .expected files from ${CASES}
build.expected:
$(MAKE) cases | xargs -n1 -I file /bin/bash -c '$(MAKE) file.build.expected'
# create single .expected file
%.build.expected:
$(MAKE) $*.aps > $*.expected
# compare result with
%.compare:
$(MAKE) $*.aps | diff $*.expected -
# run command for all cases and diff the result with corresponding expected
all:
$(MAKE) cases | xargs -n1 -I file /bin/bash -c '$(MAKE) file.compare'
clean.expected:
rm *.expected
Running make without any target and nothing happens.
echo simple-binding1 simple-binding2 | \
awk '{split($0,numbers," ")} END {for(n in numbers){ print numbers[n] }}'
simple-binding1
simple-binding2
I think the issue is with my cases target. I am not sure if I am on the right track.
I appreciate any help or hint.
I would avoid re-running make just to call a different target - it's a performance hit and may be unreliable (depending on rest of the Makefile) since separate calls may not be able to track dependencies correctly.
Moreover, I would avoid using | - every time a command is concatenated with pipe, exit code of piped command would be exit code of the last command. So a call like command | tail would return the exit code of tail (which would almost always succeed). Even if the command has failed, it would be covered with exit code 0 from tail and make will not detect the error and will not stop.
Thus said, I tried to rewrite your approach by just creating dependencies between the targets, like so:
$ cat Makefile
APSSCHED=../../bin/apssched
EXAMPLES=../../examples
BASE=.:../../base:$(EXAMPLES)
FLAGS=-DCOT
CASES=simple-binding1 simple-binding2
# Just for reproducing
$(EXAMPLES)/%.aps: ;
# Generate output and store it in a file
%.output: $(EXAMPLES)/%.aps
# echo is only for reproducing
echo $(APSSCHED) $(FLAGS) -p $(BASE) $< > $#
# Copy actual output as expected
%.expected: %.output
cp -f $< $#
# Compare actual output with expected
.PHONY: %.compare
%.compare: %.output | %.expected
diff $| $<
# Generate and verify all outputs
.PHONY: all
all: $(addsuffix .compare,$(CASES))
# Regenerate expected output
.PHONY: build.expected
build.expected: $(addsuffix .expected,$(CASES))
.PHONY: clean.expected
clean.expected:
-rm -f *.expected
Now the make build.expected will create expected output files, while make all or make will check the actual output against expected:
$ make build.expected
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding1.aps > simple-binding1.output
cp -f simple-binding1.output simple-binding1.expected
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding2.aps > simple-binding2.output
cp -f simple-binding2.output simple-binding2.expected
rm simple-binding1.output simple-binding2.output
$ make
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding1.aps > simple-binding1.output
diff simple-binding1.expected simple-binding1.output
echo ../../bin/apssched -DCOT -p .:../../base:../../examples ../../examples/simple-binding2.aps > simple-binding2.output
diff simple-binding2.expected simple-binding2.output
rm simple-binding1.output simple-binding2.output
Criteria: Makefile is a GNU Make Makefile - I'm not interested in makepp, qmake, cmake, etc. They're all nice (especially cmake), but this is for work and at work we use GNU Make. The optimal solution is a pure Makefile solution rather than a shell script that parses make for you.
I also don't want to do a 'continue on failure' solution - if it's broken, it's broken and needs to be fixed.
The situation is this, I've got a makefile that builds several directories in parallel - if one of them fails, of course the whole build fails, but not until all the running makes run to completion (or failure). This means that the reason why make actually failed is buried somewhere arbitrarily far from the end of make's output.
Here's an example of what I've got:
all: $(SUBDIRS)
SUBDIRS = \
apple \
orange \
banana \
pineapple \
lemon \
watermelon \
grapefruit
$(SUBDIRS):
cd $# && $(MAKE) $(MFLAGS) 2>&1 | sed -e "s/^/$(notdir $(#)): /g"
If I run 'make -j 5' and 'orange' happens to fail - I'd like to see a table like this at the end
of the make process
apple - passed
orange - FAILED
banana - passed
pineapple - passed
lemon - passed
I've considered having an && echo "passed" >.result || echo "FAILED" >.result, but make still needs some sort of TRAP or __onexit() cleanup command to print at them on exit.
Any Makefile ninjas out there have a pure-makefile solution for this?
un-edit - my solution wasn't actually working the way I had hoped.. STYMIED!
When you want make to abort at the first failure, end immediately and kill all in-flight jobs instead of waiting for them to finish, you need to patch GNU Make like this
http://lists.gnu.org/archive/html/bug-make/2009-01/msg00035.html
Then you need to set a trap for every shell that make invokes (as well as set -o pipefail if you use a pipe), as described in this post http://lists.gnu.org/archive/html/help-make/2009-02/msg00011.html
In a nutshell:
target1:
trap 'kill $$(jobs -p)'; command && something || something-else
target2:
trap 'kill $$(jobs -p)'; set -o pipefail; command | sed '...'
The only way I see is self-execution with a sub-make:
all : subdirs
subdirs :
$(MAKE) -f $(lastword $(MAKEFILE_LIST)) subdirs-recursive || cat log
subdirs-recursive: $(SUBDIRS)
How to get the name of the makefile in the makefile?
Thanks.
Note:
I would need that because I would like my makefile to call itself, but the makefile is not called Makefile, so I'd like to write something like this:
target:
($MAKE) -f ($MAKEFILENAME) other_target
location = $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
WHERE_ART_THOU := $(location)
$(warning $(WHERE_ART_THOU))
I also believe this is GNU make-specific, but I'm not too sure.
(Should you have any questions, refer to amazingly written GNU make manual. But remember, that, just like Makefile, this manual should be read completely before putting the concepts into practice).
I couldn't figure out how it is done easily. As far as I understand, you'll have to do some manual job.
Later I will describe how it could be done and show scripts that introduce current_makefile variable. But I would like to stress an important concept at the first place.
You should understand that if we had some kind of variable current_makefile, that expands to the current makefile name, then it will have to change during the process of reading makefiles. That means that it should be used withinin "immediate" expansion context -- i.e. within commands that are executed during reading the makefile. Most commands, however, are executed after makefiles are read. Therefore, some commands will print the correct value smoothly, while in certain places, where "deferred" expansion is used, it will always expand to the root makefile name.
If you would want to use this variable within rule text, for example, you'll have to do tricks, because rule text always has deferred expansion. So, if your have the rule
rule:
echo In makefile $(current_makefile):
echo Making target $#
it will always print the name of the root makefile. Instead, to force immediate expansion, you will have to create another variable with makefile-specific name (i.e. names of such variables should be different in each makefile):
this_makefile_unique_name := $(current_makefile)
rule:
echo In makefile $(current_makefile):
echo Making target $#
or use eval:.
define make_rule
rule:
echo In makefile $(1):
echo Making target $$#
$(eval $(call make_rule,$(current_makefile)))
If you want to use the name of current makefile for debug purpose only, consider special debugging functions, like warning or info:.
$(warning We're in makefile $(current_makefile))
These functions use "immediate" expansion and will print the correct value.
How to define such a $(current_makefile)?
You have to manually maintain stack of makefile inclusions. When you include a makefile, its name is placed to the top of the stack; when you return from included makefile to the outer one, the topmost name is popped out of stack. This is achieved by inserting special calls to the beginning and the end of makefile:
# Beginning of makefile
$(eval $(makefile_names_push))
#... makefile text
$(warning $(current_makefile))
#...
$(eval $(makefile_names_pop))
#End of file
Now define the functions at the beginning of your root makefile.
lastword=$(word $(words $(1)),$(1))
define makefile_names_push
current_makefile := $$(CURDIR)/$$(call lastword,$$(MAKEFILE_LIST))
makefile_stack :=$$(makefile_stack) $$(current_makefile)
endef
define makefile_names_pop
makefile_stack := $$(filter-out $$(current_makefile),$$(makefile_stack))
current_makefile := $$(call lastword,$$(makefile_stack))
endef
If you're sure your make is new enough (version 3.81+), replace lastword call with builtin function:.
#inctead of $$(call lastword,$$(MAKEFILE_LIST))
$$(lastword $$(MAKEFILE_LIST))
Is it useful?
Totally useless. An only use that might be useful here is to make 100 makefiles that are symlinks to one makefile, the rules in these makefiles depending on their names. But it can be achieved within one makefile and foreach-eval technique described in the manual. So my post was a complete waste of time, though I had some fun :-)
This returns the name of the first Makefile called, i.e. the one at the bottom of the call stack:
MAKEFILE_JUSTNAME := $(firstword $(MAKEFILE_LIST))
MAKEFILE_COMPLETE := $(CURDIR)/$(MAKEFILE_JUSTNAME)
When used in non-cross-recursive situations (e.g. for makedepend), it is just the name of the current makefile.
I wanted to do something similar (for echoing the contents of the Makefile) for when I use Make for managing simple repetitive tasks. I came across this page and found it was exactly what I was after and really useful for my limited understanding of make.
My result after reading this page:
# Makefile - 'make' and 'make help' now echo the makefile.
help:
cat $(lastword $(MAKEFILE_LIST))
start:
sudo -u www /path/to/webapp/myhttpd restart
stop:
sudo kill `cat /path/to/webapp/data/httpd.pid`
A quick excursion to Google suggests this site has the answer.
G'day,
If you make a copy of your original makefile, say makefile_test, and then enter the command:
make -np -f makefile_test 2>&1 | tee output
That will evaluate the makefile and your make environment but not execute any of the commands. Looking through the output file for references to makefile_test will show you what is set in make's environment and where that value is being set.
N.B. This can generate a lot of info! And don't add the -d (debug) switch which will generate tons of additional output about make's decision process but minimal additional info about make's env.
HTH
The solutions here addresses 1) POSIX make with 2) Invoked, non included, makefile in 3) A Unix alike platform.
What the OP asked for:
target:
#pid=$$$$; \
while test `ps -ocomm= $$pid` != make; do \
pid=`ps -oppid= $$pid`; \
done; \
MAKEFILENAME=`ps -oargs= $$pid|sed 's/^.* -f *\([^ ]*\).*$$/\1/'`; \
test -z "$$MAKEFILENAME" -a -f Makefile && MAKEFILENAME=Makefile; \
test -z "$$MAKEFILENAME" -a -f makefile && MAKEFILENAME=makefile; \
export MAKEFILENAME; \
$(MAKE) -e -f $$MAKEFILENAME other_target
The targets depends on the makefile, kind of bloated:
TARGET1_MAKEFILENAME = target1_preamble
all: target1 target2...
target1: $(TARGET1_MAKEFILENAME) other_dependencies...
#test $(TARGET1_MAKEFILENAME) == target1_preamble && exit 0; \
built_instructions_for_target1;
target1_preamble:
#pid=$$$$; \
while test `ps -ocomm= $$pid` != make; do \
pid=`ps -oppid= $$pid`; \
done; \
MAKEFILENAME=`ps -oargs= $$pid|sed 's/^.* -f *\([^ ]*\).*$$/\1/'`; \
test -z "$$MAKEFILENAME" -a -f Makefile && MAKEFILENAME=Makefile; \
test -z "$$MAKEFILENAME" -a -f makefile && MAKEFILENAME=makefile; \
export MAKEFILENAME; \
$(MAKE) -e -f $$MAKEFILENAME target1;
Can be a bit simplified if make is invoked only for all targets.
MAKEFILENAME = invoked_makefile_placeholder
all: target1 target2...
target1: $(MAKEFILENAME) other_dependencies...
#test $(MAKEFILENAME) == invoked_makefile_placeholder && exit 0; \
built_instructions_for_target1;
invoked_makefile_placeholder:
#pid=$$$$; \
while test `ps -ocomm= $$pid` != make; do \
pid=`ps -oppid= $$pid`; \
done; \
MAKEFILENAME=`ps -oargs= $$pid|sed 's/^.* -f *\([^ ]*\).*$$/\1/'`; \
test -z "$$MAKEFILENAME" -a -f Makefile && MAKEFILENAME=Makefile; \
test -z "$$MAKEFILENAME" -a -f makefile && MAKEFILENAME=makefile; \
export MAKEFILENAME; \
$(MAKE) -e -f $$MAKEFILENAME
With the previous approach is trivial to implement a solution for included makefiles based in grep and a unique pattern contained in the makefile.
I never answer when I feel the question got a proper solution.
I want to process each source code file after it has been preprocessed:
myprocess `gcc -E file1.c`
myprocess `gcc -E file2.c`
...
myprocess `gcc -E fileN.c`
This gets tedious so how do I make this a single command?
That is, something along the line:
myprocess SOMETHINGMAGIC(gcc -E file*.c)
Thanks in advance!
You mean like
for i in file*.c ; do
myprocess `gcc -E $i`
done
If this is part of an ongoing processes (as opposed to a one time thing), use make, it is good at automating work pipelines.
In particular use suffix rules with traditional make or gmake style implicit rules.
Here is an outline for a suffix rule implementation:
.c.cpre:
$(CC) -E $(CPPFLAGS) -o $# $<
.cpre.cmy:
$(MY_PROCESS) $<
# Or whatever syntax you support..
#
# You could
# $(RM) $<
# here, but I don't recommend it
.cmy.o:
$(CC) -c $(CFLAGS) $(CPPFLAGS) -o $# $<
# $(RM) $<
No magic nedded, just
my_process $(gcc -E *.c)
Note that I used the $(command) form because backticks are deprecated.
As an aside: are you sure you want to do that? You are putting the whole output of gcc -E as command line parameters of my_process. Not sure this is what you want. Maybe you want to use a simple pipe
gcc -E file.c | my_process
so that the output of gcc becomes the input of my_process.
In the latter case something like
for c_file in *.c ; do
gcc -E $c_file | myprocess > ${c_file}.processed
done
would do the trick.
Perhaps this is what you're looking for?:
for i in file*.c ; do
echo "Header for myprocess: $i"
gcc -E $i 2>&1
done | myprocess
2>&1 assumes you want to grab stderr, too
echo ... gives myprocess a starting point for each compilation
How about:
myprocess {$(gcc -E file1.c),$(gcc -E file2.c),$(gcc -E file3.c)}
Or if you pipe it:
{$(gcc -E file1.c),$(gcc -E file2.c),$(gcc -E file3.c)} | myprocess
It's been a while since I've used bash, so, please, point out my mistakes!
ls file*.c | xargs --replace myprocess $(gcc -E {})