I have seen this done in various installers, although unfortunately I cannot actually find much on how to do this or remember which programs had this feature so I can learn it from them.
I essentially am compiling a shared library (not with autoconf/libtool yet, just a standard makefile) and wish for make test to simply compile all files in the examples directory, linked to the library that was just built (i.e. ../bin/libfoo.so.1.0.1)
What does the makefile format provide to do this wildcard compile?
I know you're asking for a mechanism to automatically build all source files in a directory, but I dislike that approach; at some point in the future, an individual test or two will need to be disabled, and you're left doing something a little weird, like re-naming a file just so it won't automatically be compiled.
I much prefer listing every program that you intend on building in the Makefile, but it need not be horrible. With some suffix rules you can easily build all your .c files into .o files or directly into executables.
Here's a snippet from the AppArmor regression test suite, which is typical of what this sort of thing looks like. (I hope I copied out all the relevant bits, it has been a few years.)
SRC=access.c \
changeprofile.c \
changehat.c \
changehat_fork.c \
changehat_misc.c \
....
unlink.c \
xattrs.c
...
#only do the ioperm/iopl tests for x86 derived architectures
ifneq (,$(findstring $(shell uname -i),i386 i486 i586 i686 x86 x86_64))
SRC+=syscall_ioperm.c syscall_iopl.c
endif
...
LIBIMMUNIX:=$(shell if [ -f /usr/lib/libapparmor.so -o -f /usr/lib64/libapparmor.so ] ; then \
echo -lapparmor ; \
elif [ -f /lib/libimmunix.so.1 -o -f /lib64/libimmunix.so ] ; then \
echo -limmunix ; \
fi )
CFLAGS+=$(CHANGEHAT_FLAGS) -Wall -Wstrict-prototypes
LDLIBS+=$(LIBIMMUNIX)
EXEC=$(SRC:%.c=%)
...
all: $(EXEC) changehat.h
It isn't as easy as just dropping a new file into the directory; you do need to add it to the Makefile. But you only need to add the name, once, to one line, and it is there for good. If you want to disable it, then comment out the offending line. It's almost as easy and significantly more control over your build process.
Related
I want to generate a variable at compilation time in a Makefile and using it at run time in my executable.
Minimal example: In Makefile I generate variable COMMIT_ID, which contains the ID of the latest commit: COMMIT_ID=$(git rev-parse --verify HEAD).
Makefile also generates an executable, for instance executable.elf.
The C source code from which executable.elf is generated prints COMMIT_ID.
Which is the best way do to it?
Nerd sniper HardcoreHenry baited me for a better (or at least different) solution. Came up with this, having a file in the repository with its name signifying what commit the repo was last built with:
COMMIT_ID := $(shell git rev-parse --verify HEAD)
# target is made if it does not exist
$(COMMIT_ID).commit_id:
## clean other commits than current
rm -f *.commit_id
## placeholder for current commit
touch $#
commit_id.h: $(COMMIT_ID).commit_id
echo "#define COMMIT_ID ${COMMIT_ID}" > $#
It depends on what your makefile looks like. If you have a dedicated rule for the c file that accesses it, you could just add -DCOMMIT_ID=$(COMMIT_ID) to the recipe as so:
foo.o: foo.c
$(CC) $(CFLAGS) -DCOMMIT_ID=$(COMMIT_ID) $^ -o $#
or, if you're using pattern rules, you could just add it to CFLAGS, and then every .c file would have access to it (assuming of course that the pattern rule makes use of CFLAGS):
CFLAGS += -DCOMMIT_ID=$(COMMIT_ID)
After you've done this, COMMIT_ID will be a macro in the c file, so you could use the stringify operator # to access its value:
printf("commit id is " #COMMIT_ID "\n");
----- EDIT -----
Note that this does have a sharp stick involved with it -- specifically, if you build once, and then modify the git repository without modifying foo.c, then make will consider foo.o to be up to date, and will not rebuild it, meaning that foo.o will have the old commit id in it. I'll post a second answer on how you might get around this if this is a concern
My first answer would not rebuild executable.elf if the git repository changes, but your sources do not. If you need this to always be rebuilt when the repository changes, you can use some makefile trickery to do this: (There may be a better way to do this, and if anyone can think of one, I'd love to hear it):
First in your makefile do:
$(shell \
COMMIT_ID=$(git rev-parse --verify HEAD) && \
echo "#define COMMIT_ID $${COMMIT_ID}" > commitid.h.tmp; \
if [ ! -f commitid.h ] || ! cmp -s commitid.h.tmp commitid.h; then \
cp commitid.h.tmp commitid.h; \
fi; \
rm commitid.h.tmp \
)
And then in foo.c, you simply do a
#include commitid.h
So long as your makefile generates .d files then foo.c will be dependent on commitid.h, and will be rebuilt if this file is updated. (if not, add the line foo.c: commitid.h to your makefile). The file itself will only be updated if the commit id changes, so foo.o will only rebuilt if necessary.
You could certainly use a make variable, something like this:
printf("%s\n", COMMIT_ID);
where you'd have a Makefile with something like
executable.elf: CFLAGS+=-D'COMMIT_ID="$(shell git rev-parse --verify HEAD)"'
(This uses some GNU make features which are not portable.)
However, it's probably better if this is actually reproducible. Towards that end, perhaps write the value into a file, so that you can see what was used the last time this project was compiled.
executable.elf: commitid.c
.PHONY: commitid.c
commitid.c:
git rev-parse --verify HEAD \
| sed 's/.*/#define COMMIT_ID "&"/' >$#
and then obviously #include "commitid.c" from executable.c or whatever the C source file is called.
The .PHONY declaration to force commitid.c to be rebuilt every time is a bit of a blunt instrument; perhaps instead make commitid.c depend on every version-controlled file.
I need to do a makefile for run some programs. Every time I run that script all the file are processed also if the file are not changed. I'm sure there is a problem on my code but I don't understand where I made the mistakes.
RDIR=RAW
OUTDIR=Fusion_res/kallisto
RFILES:=$(wildcard $(RDIR)/*_R1_001.fastq.gz)
DATABASE=/home/sbsuser/databases/Kallsto_hg38_87
OUTFILE=$(patsubst %_R1_001.fastq.gz,%_R2_001.fastq.gz,$(RFILES))
OUTKAL=$(patsubst $(RDIR)/%_R1_001.fastq.gz,$(OUTDIR)/%,$(RFILES))
.PHONY: clean all
all: $(OUTFILE) $(RFILES) $(OUTDIR) $(OUTKAL)
#$(OUTKAL) $(OUTFILE): $(RDIR)/%._R1_001.fastq.gz
# echo "kallisto quant -i" $(DATABASE)/transcripts.idx -b 100 -o $# --fusion $< $(OUTFILE)
$(OUTDIR)/%: $(RDIR)/%_R1_001.fastq.gz $(OUTFILE)
kallisto quant -i $(DATABASE)/transcripts.idx -b 100 --fusion --rf-stranded -o $# $(RDIR)/$*_R1_00
1.fastq.gz $(RDIR)/$*_R2_001.fastq.gz
$(OUTDIR):
mkdir -p $(OUTDIR)
clean::
$(RM) -rf $(OUTDIR)
I suppose if the found some change on the input file and on the output execute the command. I don't know why every time force re-run. In some case Is that I want but I wan to also if there is some new input execute only that.
Thanks so much
A couple of things:
1) $(OUTDIR)/% is dependent on $(OUTFILE) (which is a list of all outfiles). Therefore if you change any one of the OUTFILEs, you make everything in $(OUTDIR)/% obsolete. I believe what you want is this:
$(OUTDIR)/%_R1_001.fastq.gz: $(RDIR)/%_R2_001.fastq.gz
.... (rules to make out/R1 from raw/R2
$(RDIR)/%_R2_001.fastq.gz: $(RDIR)/%_R1_001.fastq.gz
.... (rules to make R2 from R1
This makes each file dependent only on the files that effect it.
2) you have the target all dependent on $(OUTDIR) which is a directory. If you use parallel make, it may generate the $(OUTDIR) after it generates the other dependencies of all: (some of which would depend on $(OUTDIR) being created). What you want there is to remove all's dependency on $(OUTDIR), and add the line:
$(OUTFILE) : | $(OUTDIR)
Notice the |, which means order only (don't consider $(OUTFILE) out of date if $(OUTDIR) is newer. This is important, as a directory's timestamp is updated each time a file in the directory is changed, and so it tends to be newer than its contents.
I'm writing my first complex Makefile for a highly-modularized project.
I have various sub-directories, each one has its own Makefile which supports at least the all and the clean targets.
These sub-Makefiles work just fine, however I have a problem with the main Makefile, that should call all the sub-Makefiles automatically from the list contained in the variable COMPONENTS.
I tried with the following Makefile:
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY = all clean FORCE $(OUTFILE) $(COMPONENTS)
all: $(OUTFILE)
$(OUTFILE): $(COMPONENTS)
./make_image
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $component; \
done
FORCE:
clean:
for component in $(COMPONENTS); do \
make -C $component clean; \
done
This results in the following error message:
for component in bootloader; do \
make -C omponent; \
done
make: *** omponent: No such file or directory. Stop.
make: *** [bootloader] Error 2
as if the $component expression was only parsed as $c. I don't understand why that happens and how to fix it.
Just double the dollar sign:
$(COMPONENTS): FORCE
for component in $(COMPONENTS); do \
make -C $$component; \
done
The trouble is that with your makefile, Make expands $component before executing the rule. And since $c has no value (there is no such variable), it expands to nothing, leaving "omponent", which it passes to she shell, which complains that there's no such directory. (If you had written $(component), Make would have expanded it to nothing, since Make knows of no such variable, and then the shell would have complained that you were not specifying a directory at all.)
With the double dollar sign, Make expands $$component to $component, which it then passes to the shell, which interprets it as the loop variable, and everything proceeds as planned.
You really should have played around with a simple loop in a command, before attempting to do actual work with one.
Several issues.
.PHONY should be written as a dependency, not a macro definition
Don't write shell loops, use make syntax instead
When you call make recursively, you must do it via the ${MAKE} macro invocation
Leading to
OUTFILE = diskimage.bin
export NASM = nasm
COMPONENTS = bootloader
.PHONY: all
all: ${OUTFILE}
.PHONY: ${OUTFILE}
${OUTFILE}: ${COMPONENTS}
./make_image
.PHONY: ${COMPONENTS}
${COMPONENTS}:
${MAKE} -C $#
The advantage of this formulation is that it is parallel make friendly.
Always a test of a good Makefile.
Here make -j5 all will cause make to keep 5 commands running at once,
across all invocations of make.
Nice if you have 4 CPUs.
What about clean?
(Personally I hate clean targets—it's a sign of dodgy dependencies,
and of unhygienic mixing of source and target folders.)
Just add -clean (say) to each of the component names,
and repeat the pattern above.
CLEANS := $(addsuxffix -clean,${COMPONENTS})
.PHONY: clean
clean: ${CLEANS} ; #echo Clean succesful
.PHONY: ${CLEANS}
${CLEANS}: %-clean:
${MAKE} -C $* clean
These two sections can tidied up and combined into one if you feel so inclined.
Tip
Always run make with --warn (or --warn-undefined-variables to give it its full name) to catch inadvertent expansion of $c in things like $component.
I have a rule in my makefile:
$(OW_GROUP_ONE_C): $(OW_GROUP_ONE_PNG)
for file in $^; \
do \`enter code here`
grit $$file -ftc -fh\! -fa -gt -gz\! -gB4 -m\! -p -pzl -pu16 -o $#; \
done
It builds a single c file out of different images, those are iterated in a for loop (They are, I checked using an echo)
The rule which depends on that is
$(OW_GROUP_ONE_O): $(OW_GROUP_ONE_C)
$(CC) $(CFLAGS) -c -o $# $<
which is executed via
$(SPRITES_BINARY): $(NORMAL_PAL_OBJ) $(SHINY_PAL_OBJ) $(SPRITE_FRONT_OBJ) $(SPRITE_BACK_OBJ) $(NORMAL_CASTFORM_PAL_OBJ) $(SHINY_CASTFORM_PAL_OBJ) $(CASTFORM_FRONT_OBJ) $(CASTFORM_BACK_OBJ) $(OW_GROUP_ONE_O)
If I execute the rule by calling "make $(OW_GROUP_ONE_C)" everything works fine, but as soon as the rule is executed via dependency from another rule, the loop seems to just read the first file. I again used echo to check, but the loop accumulates all files in the list. I don't know what the deal i, the tool (GRIT - GBA raster image transmogrifier) should be able to handle that, but there must be a difference between calling the rule explicit if it works that way...
Thanks in advance for any hints!
I'm not sure if it's gmake or gcc that I don't understand here.
I'm using the -MM and -MD options to generate dependency rules for the Unit Testing framework I'm using. Specifically:
$(TEST_OBJ_DIR)/%.d: $(TEST_SRC_DIR)/%.cpp
#$(CPPC) -MM -MD $< -o $#
#sed -i -e 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(TEST_OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1.o:|' $#
-include $(TEST_DEP_FILES)
When I run make, after all binaries are linked (properly), I see the following extra (unexplained) line before make exits
rm test/obj/dice.d test/obj/regex.o test/obj/inventoryContainer.d test/obj/color-string.d test/obj/dice.o test/obj/inventoryContainer.o test/obj/color-string.o test/obj/regex.d
From whence is that rm command coming? The only place - anywhere - that I have an rm command in my makefile is in the clean directive
test-clean:
rm -f $(TEST_BIN_FILES)
rm -f $(TEST_OBJ_DIR)/*.{a,d,o}
Any ideas?
make will automatically create intermediate files if necessary to chain two rules together, but it will delete them at the end of the build. You can use the .PRECIOUS special target to prevent it from removing them
One helpful option for debugging these kind of problems is the -n switch:
make -n {TARGET}
It will show you the commands it would run but won't actually run them. This lets you see what rules are firing but doesn't give you all the extra output that makes it difficult to diagnose the problem.
The -d debug flag can also be useful but be sure to run it in a context where you can scroll around easily, you'll be getting a lot of output. I usually use emacs shell mode as it has good searching functionality and saves the buffer.