I have a rule in my makefile:
$(OW_GROUP_ONE_C): $(OW_GROUP_ONE_PNG)
for file in $^; \
do \`enter code here`
grit $$file -ftc -fh\! -fa -gt -gz\! -gB4 -m\! -p -pzl -pu16 -o $#; \
done
It builds a single c file out of different images, those are iterated in a for loop (They are, I checked using an echo)
The rule which depends on that is
$(OW_GROUP_ONE_O): $(OW_GROUP_ONE_C)
$(CC) $(CFLAGS) -c -o $# $<
which is executed via
$(SPRITES_BINARY): $(NORMAL_PAL_OBJ) $(SHINY_PAL_OBJ) $(SPRITE_FRONT_OBJ) $(SPRITE_BACK_OBJ) $(NORMAL_CASTFORM_PAL_OBJ) $(SHINY_CASTFORM_PAL_OBJ) $(CASTFORM_FRONT_OBJ) $(CASTFORM_BACK_OBJ) $(OW_GROUP_ONE_O)
If I execute the rule by calling "make $(OW_GROUP_ONE_C)" everything works fine, but as soon as the rule is executed via dependency from another rule, the loop seems to just read the first file. I again used echo to check, but the loop accumulates all files in the list. I don't know what the deal i, the tool (GRIT - GBA raster image transmogrifier) should be able to handle that, but there must be a difference between calling the rule explicit if it works that way...
Thanks in advance for any hints!
Related
I've tweak "a bit" Make so I can use it as a "kind of" cli for some tasks.
MAKEFLAGS += --no-builtin-rules
MAKEFLAGS += --no-builtin-variables
MAKEFLAGS += --no-print-directory
SHELL := /bin/bash
.ONESHELL:
.PHONY: project_list
project_list: all_projects_info.json
echo "Filtering project list with:" >&2
echo " PROJECT_FILTER: $(PROJECT_FILTER)" >&2
jq -r -S '.[] | select(
(.projectId | test("$(PROJECT_FILTER)"))
) | .projectId' $^ > $#
.PHONY: get_storage_info
get_storage_info: project_list
PROJECT_LIST=$$(cat $<)
$(MAKE) -f $(MKFILE) -j storage_info.json PROJECT_LIST="$$PROJECT_LIST"
all_projects_info.json:
curl -X GET https://toto/get_all_my_projects_info >$#
# here it's PHONY because we want to always rebuild it
.PHONY: storage_info.json
storage_info.json: $(STORAGE_INFO_JSON_FILES)
jq -s -S '[.[]?.items?[]?]' $(STORAGE_INFO_JSON_FILES) > $#
storage_info/:
mkdir -p $#
STORAGE_INFO_JSON_FILES=$(foreach project_name,$(PROJECT_LIST),storage_info/$(project_name).json)
$(STORAGE_INFO_JSON_FILES): storage_info/%.json: | storage_info/
curl \
-X GET \
"https://storage_api/list_s3?project=$*" \
2> /dev/null > $#
As you can see here, I've got 2 "command":
project_list witch list all project I can access too,
get_storage_info witch list all bucket in projects.
The trick here is because I've got a lot of projects and buckets, I may want to filter like this:
make get_storage_info PROJECT="foo"
And it will print ONLY bucket in project with foo in their name.
It's quit handy and fast (only the first time it may be slow, the time to get all informations).
What is bothering me, I've not found a better way than to call a sub make command (with the exact list of project to take into account).
Is it possible to express dynamic dependencies of a target ?
But something that can result from another target ?
Thanks.
I don't see anything wrong with invoking a submake. That's IMO the best way to do it, especially if you want to add -j to it.
It's not really possible to get rid of this easily. It's not the fact that you want to express a dynamic dependency: that can be done. The problem is you want the list of dependencies to be extracted from the results of running another rule. But that's not how make works: make always starts with the final target and works its way backwards. So, by the time you get around to building the prerequisite file, the target that depended on it has already been processed (not its recipe of course, but all the prerequisites).
I'd like to use make to process a large number of inputs to outputs using a script (python, say.) The problem is that the script takes an incredibly short amount of time to run per input, but the initialization takes a while (python engine + library initialization.) So, a naive makefile that just has an input->output rule ends up being dominated by this initialization time. Parallelism doesn't help with that.
The python script can accept multiple inputs and outputs, as so:
python my_process -i in1 -o out1 -i in2 -o out2 ...
and this is the recommended way to use the script.
How can I make a Makefile rule that best uses my_process, by sending in out of date input-output pairs in batches? Something like parallel but aware of which outputs are out of date.
I would prefer to avoid recursive make, if at all possible.
I don't completely grasp your problem: do you really want make to operate in batches or do you want a kind of perpetual make process checking the file system on the fly and feeding to the Python process whenever it finds necessary? If the latter, this is quite the opposite of a batch mode and rather a pipeline.
For the batch mode there is a work-around which needs a dummy file recording the last runnning time. In this case we are abusing make for because the makefile is in this part a one-trick pony which is unintuitive and against the good rules:
SOURCES := $(wildcard in*)
lastrun : $(SOURCES)
python my_process $(foreach src,$?,-i $(src) -o $(patsubst in%,out%,$(src)))
touch lastrun
PS: please note that this solution has a substantial flaw in that it doesn't detect the update of in-files when they happen during the run of the makefile. All in all it is more advisable to simply collect the filenames of the in-files which were updated by the update process itself and avoid make althogether.
This is what I ended up going with, a makefile with one layer of recursion.
I tried using $? both with grouped and ungrouped targets, but couldn't get the exact behavior needed. If one of the output targets was deleted, the rule would be re-run but $? wouldn't necessarily have some input files but not the correct corresponding input file, very strange.
Makefile:
all:
INDIR=in
OUTDIR=out
INFILES=$(wildcard in/*)
OUTFILES=$(patsubst in/%, out/%, $(INFILES))
ifdef FIRST_PASS
#Discover which input-output pairs are out of date
$(shell mkdir -p $(OUTDIR); echo -n > $(OUTDIR)/.needs_rebuild)
$(OUTFILES) : out/% : in/%
#echo $# $^ >> $(OUTDIR)/.needs_rebuild
all: $(OUTFILES)
#echo -n
else
#Recurse to run FIRST_PASS, builds .needs_rebuild:
$(shell $(MAKE) -f $(CURDIR)/$(firstword $(MAKEFILE_LIST)) FIRST_PASS=1)
#Convert .needs_rebuild into batches, creates all_batches phony target for convenience
$(shell cat $(OUTDIR)/.needs_rebuild | ./make_batches.sh 32 > $(OUTDIR)/.batches)
-include $(OUTDIR)/.batches
batch%:
#In this rule, $^ is all inputs needing rebuild.
#The corresponding utputs can be computed using a patsubst:
targets="$(patsubst in/%, out/%, $^)"; touch $$targets
clean:
rm -rf $(OUTDIR)
all: all_batches
endif
make_batches.sh:
#!/bin/bash
set -beEu -o pipefail
batch_size=$1
function _make_batches {
batch_num=$1
shift 1
#echo ".PHONY: batch$batch_num"
echo "all_batches: batch$batch_num"
while (( $# >= 1 )); do
read out in <<< $1
shift 1
echo "batch$batch_num: $in"
echo "$out: batch$batch_num"
done
}
export -f _make_batches
echo ".PHONY: all_batches"
parallel -N$batch_size -- _make_batches {#} {} \;
Unfortunately, the makefile is a one trick pony and there's quite a bit of boilerplate to pull this recipe off.
I'm writing my first complex Makefile for a highly-modularized project.
I have various sub-directories, each one has its own Makefile which supports at least the all and the clean targets.
These sub-Makefiles work just fine, however I have a problem with the main Makefile, that should call all the sub-Makefiles automatically from the list contained in the variable COMPONENTS.
I tried with the following Makefile:
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY = all clean FORCE $(OUTFILE) $(COMPONENTS)
all: $(OUTFILE)
$(OUTFILE): $(COMPONENTS)
./make_image
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $component; \
done
FORCE:
clean:
for component in $(COMPONENTS); do \
make -C $component clean; \
done
This results in the following error message:
for component in bootloader; do \
make -C omponent; \
done
make: *** omponent: No such file or directory. Stop.
make: *** [bootloader] Error 2
as if the $component expression was only parsed as $c. I don't understand why that happens and how to fix it.
Just double the dollar sign:
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $$component; \
done
The trouble is that with your makefile, Make expands $component before executing the rule. And since $c has no value (there is no such variable), it expands to nothing, leaving "omponent", which it passes to she shell, which complains that there's no such directory. (If you had written $(component), Make would have expanded it to nothing, since Make knows of no such variable, and then the shell would have complained that you were not specifying a directory at all.)
With the double dollar sign, Make expands $$component to $component, which it then passes to the shell, which interprets it as the loop variable, and everything proceeds as planned.
You really should have played around with a simple loop in a command, before attempting to do actual work with one.
Several issues.
.PHONY should be written as a dependency, not a macro definition
Don't write shell loops, use make syntax instead
When you call make recursively, you must do it via the ${MAKE} macro invocation
Leading to
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY: all
all: ${OUTFILE}
.PHONY: ${OUTFILE}
${OUTFILE}: ${COMPONENTS}
./make_image
.PHONY: ${COMPONENTS}
${COMPONENTS}:
${MAKE} -C $#
The advantage of this formulation is that it is parallel make friendly.
Always a test of a good Makefile.
Here make -j5 all will cause make to keep 5 commands running at once,
across all invocations of make.
Nice if you have 4 CPUs.
What about clean?
(Personally I hate clean targets—it's a sign of dodgy dependencies,
and of unhygienic mixing of source and target folders.)
Just add -clean (say) to each of the component names,
and repeat the pattern above.
CLEANS := $(addsuxffix -clean,${COMPONENTS})
.PHONY: clean
clean: ${CLEANS} ; #echo Clean succesful
.PHONY: ${CLEANS}
${CLEANS}: %-clean:
${MAKE} -C $* clean
These two sections can tidied up and combined into one if you feel so inclined.
Tip
Always run make with --warn (or --warn-undefined-variables to give it its full name) to catch inadvertent expansion of $c in things like $component.
I have a makefile that uses a source file from the internet. There are two locations where the file resides, neither of which I consider very dependable, so I also keep a local copy. So the relevant lines of my makefile look like:
src.c:
wget -nv http://location.com/$# || wget -nv http://otherplace.com/$# || cp local/$# .
src.o: src.c
$(CC) -o $# $<
Is this the "right way" to do this? What if there are multiple steps in each different way of creating the target - how do I tell make "Try A. If A fails, try B. If B fails, ..."?
The right thing to do is this:
.PHONY: phony
src.c: phony
if (wget -nv http://location.com/$# -O $#.temp) && ! diff $#.temp $# >/dev/null; then \
mv $#.temp $#; \
fi
I shortened your command to a single wget but you can put whatever you want there, including a sequence of ||s to achieve "try this, if not, try that etc". Just make sure it outputs to a temporary file (and does not hang indefinitely !) .
It is in fact important to use phony here, and not only .PHONY. Can you see why?
Also, with this method, there is no longer a need to keep another "local" copy and/or use cp. Your target src.c is your "local copy" - the latest one you were able to successfully get from the Internet.
I'm not sure if it's gmake or gcc that I don't understand here.
I'm using the -MM and -MD options to generate dependency rules for the Unit Testing framework I'm using. Specifically:
$(TEST_OBJ_DIR)/%.d: $(TEST_SRC_DIR)/%.cpp
#$(CPPC) -MM -MD $< -o $#
#sed -i -e 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(TEST_OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1.o:|' $#
-include $(TEST_DEP_FILES)
When I run make, after all binaries are linked (properly), I see the following extra (unexplained) line before make exits
rm test/obj/dice.d test/obj/regex.o test/obj/inventoryContainer.d test/obj/color-string.d test/obj/dice.o test/obj/inventoryContainer.o test/obj/color-string.o test/obj/regex.d
From whence is that rm command coming? The only place - anywhere - that I have an rm command in my makefile is in the clean directive
test-clean:
rm -f $(TEST_BIN_FILES)
rm -f $(TEST_OBJ_DIR)/*.{a,d,o}
Any ideas?
make will automatically create intermediate files if necessary to chain two rules together, but it will delete them at the end of the build. You can use the .PRECIOUS special target to prevent it from removing them
One helpful option for debugging these kind of problems is the -n switch:
make -n {TARGET}
It will show you the commands it would run but won't actually run them. This lets you see what rules are firing but doesn't give you all the extra output that makes it difficult to diagnose the problem.
The -d debug flag can also be useful but be sure to run it in a context where you can scroll around easily, you'll be getting a lot of output. I usually use emacs shell mode as it has good searching functionality and saves the buffer.