So I am writing a makefile that will take some files (*.in) as input to my C++ program and compare their output (results.out) to given correct output (*.out).
Specifically I have files t01.in, t02.in, t03.in, t04.in, and t05.in.
I have verified that $TESTIN = t01.in t02.in t03.in t04.in t05.in.
The problem is that it seems to run the %.in: %.out block only for three of these files, 1,3, and 4. Why is it doing this?
OUTPUT = chart
COMPILER = g++
SOURCES = chart.cpp
HEADERS =
OBJS = $(SOURCES:.cpp=.o)
TESTIN = tests/*.in
all: $(OUTPUT)
$(OUTPUT): $(OBJS)
$(COMPILER) *.o -o $(OUTPUT)
%.o: %.cpp
clear
$(COMPILER) -c $< -o $#
test: $(TESTIN)
%.in: %.out
./$(OUTPUT) < $# > tests/results.out
printf "\n"
ifeq ($(diff $< tests/results.out), )
printf "\tTest of "$#" succeeded for stdout.\n"
else
printf "\tTest of "$#" FAILED for stdout!\n"
endif
Additionally, if there is a better way of accomplishing what I am trying to do, or any other improvements I could make to this makefile (as I am rather new at this), suggestions would be greatly appreciated.
EDIT: If I add a second dependency to the block (%.in: %.out %.err), it runs the block for all five files. Still no idea why it works this way but not the way before.
First, I don't see how TESTIN can be correct. This line:
TESTIN = tests/*.in
is not a valid wildcard statement in Make; it should give the variable TESTIN the value tests/*.in. But let's suppose it has the value t01.in t02.in t03.in t04.in t05.in or tests/t01.in tests/t02.in tests/t03.in tests/t04.in tests/t05.in, or wherever these files actually are.
Second, as #OliCharlesworth points out, this rule:
%.in: %.out
...
is a rule for building *.in files, which is not what you intend. As for why it runs some tests and not others, here is my theory:
The timestamp of t01.out is later than that of t01.in, so Make decides that it must "rebuild" t01.in; likewise t03.in and t04.in. But the timestamp of t02.out is earlier than that of t02.in, so Make does not attempt to "rebuild" t02.in; likewise t05.in. The timestamps of t02.err and t05.err are later than those of t02.in and t05.in, respectively, so when you add the %.err prerequisite, Make runs all tests. You can test this theory by checking the timestamps and experimenting with touch.
Anyway, let's rewrite it. We need a new target for a new rule:
TESTS := $(patsubst %.in,test_%,$(TESTIN)) # test_t01 test_t02 ...
.PHONY: $(TESTS) # because there will be no files called test_t01, test_t02,...
$(TESTS): test_%: %.in %.out
./$(OUTPUT) < $< > tests/results.out
Now for the conditional. Your attempted conditional is in Make syntax; Make will evaluate it before executing any rule, so tests/result.out will not yet exist, and variables like $< will not yet be defined. We must put the conditional inside the command, in shell syntax:
$(TESTS): test_%: %.in %.out
./$(OUTPUT) < $< > tests/results.out
if diff $*.out tests/results.out >/dev/null; then \
echo Test of $* succeeded for stdout.; \
else echo Test of $* FAILED for stdout!; \
fi
(Note that only the first line of the conditional must begin with a TAB.)
Related
I have a makefile which produces an executable from several object files and I include a version number in each object file as it is compiled. However, I want the version number to be incremented only when a particular object file is created (ptarget below, the one containing main). I tried to do this using a conditional statement in the recipe:
ptarget:=$(addsuffix .obj,$(ouf))
%.obj : %.cpp
$(CXX) $(CXXFLAGS) $< -Fo$#
$(info $(ptarget))
$(info $#)
ifeq ($#, $(ptarget))
perl $(perlDir)versionBump/bump.pl -inc -f $(versionFile)
endif
I can see from the info commands that only when ptarget is built that $# == $(ptarget) -- I also tried using strip command to make sure no hidden whitespace, but the perl command to increment the version is never executed, it starts with a tab.
Just to add, this all works just fine without the conditional but the increment happens multiple times during a build, which is what I am trying to avoid. This example suggests it should work but...
This is a very common misunderstanding about make. ifeq is a make statement and is evaluated when make parses the Makefile. What you need is a conditional that gets evaluated when the recipe is executed by the shell, that is a shell if:
%.obj : %.cpp
$(CXX) $(CXXFLAGS) $< -Fo$#
$(info $(ptarget))
$(info $#)
if [ "$#" = "$(ptarget)" ]; then \
perl $(perlDir)versionBump/bump.pl -inc -f $(versionFile); \
fi
Do not forget the line continuations (the trailing \).
I have a makefile that for various reasons relies on a supporting python script to run every time and grab files from several external locations, copy into working directory, and run through a separate preprocessor before compiling.
This makefile must be able to be run in parallel (-j8) so the order of processing cannot be guaranteed.
In trying to explicitly specify prerequisites, I have created a situation where make skips all object files, goes straight to linking, and fails because the necessary objects do not exist. On a second run, all the objects already exist (the preprocess step skips the files that already exist) and all the files are compiled and linked properly.
When run without -j# everything works fine, but the moment I add -j2, the skipping begins.
Following is an example make file:
GEN_FILES := file1.cpp file2.cpp file3.cpp
CXX_FILES := bin_main.cpp $(GEN_FILES)
OBJ_FILES := $(patsubst %.cpp,%.o,$(CXX_FILES))
.PHONY : all clean prepare
all : bin_file
prepare :
# Copy and preprocess all source files
[ -f file1.cpp ] || cp d1/file1.cpp .
[ -f file2.cpp ] || cp d2/file2.cpp .
[ -f file3.cpp ] || cp d3/file3.cpp .
$(OBJ_FILES) : prepare
bin_file : $(OBJ_FILES)
[ -f file1.o ] && [ -f file2.o ] && [ -f file3.o ] && touch bin_file
%.o : %.cpp
#echo "Compiling $<..."
[ -f $< ] && touch $#
clean :
$(RM) *.o
$(RM) file*
$(RM) bin_file
How can I get this to build in one go, first running prepare to collect all files and then compiling and linking as necessary?
As code_fodder mentions the issue is the creation of the source files.
Basically what happens is, you have not told make how to create these source files, so as far as make knows they don't exist and there's no way to create them. So when make wants to build, for example, file1.o it looks at your pattern rule and finds it could build file1.o from file1.cpp. So then it looks for how to build file1.cpp. No file1.cpp exists, and there is no rule that make knows of that will build it, so make ignores that pattern rule as not matching.
Then make sees the target:
$(OBJ_FILES) : prepare
so it thinks there's no recipe needed to create the object files, and just runs the link line. The next time through, make sees the prepared source files (from the previous build) and then it can use your pattern rule.
If you change your pattern rule to a static pattern rule, where you explicitly tell make exactly what rule to use instead of providing it with a possible rule to use that it can ignore if it doesn't match (which is what a pattern rule is), you'll see the error:
$(OBJ_FILES): %.o : %.cpp
#echo "Compiling $<..."
sleep 1
[ -f $< ] && touch $#
will tell you:
make: *** No rule to make target 'file1.cpp', needed by 'file1.o'. Stop.
Remember, make is looking for a matching pattern rule BEFORE it actually builds anything: it doesn't want to build every possible prerequisite of every possible matching pattern rule, to decide whether or not at the end of it the rule can be used. The rule is matched based on the current state of the filesystem plus rules you have given make about changes it could make. Make has no idea that if it were to invoke the prepare target the source files it was looking for would magically come into existence.
Your basic problem is that this statement is the wrong dependency relationship:
$(OBJ_FILES) : prepare
It's not really true that the object files depend on prepare; what's true is that the PREPARED SOURCE FILES depend on prepare. The object files depend only the "prepared" source files, as your pattern rules shows. This rule should be written, instead:
$(GEN_FILES): prepare
If you do this with -j everything will wait as you want.
Yeah, this gets messy / difficult. The problem you have is that you can specify prerequisite lists - that can work in order, but as soon as you start to use -j then make can start processing prerequisites in any old order. So bin_file requires $(OBJ_FILES) which require prepare. Then %.o requires the same named %.cpp file - which it can do for main.o, but not the filex.o since they don't exist yet - but it tries anyway and fails - in the mean time make (in parallel) is potentially starting to generate the .cpp files, but by this time its too late...etc...
My Prerequisites Build Pattern
I use a very specific prerequisites pattern of my own design - some might frown upon - but I have carefully considered this over the years and found it to be optimal for me.
I create a rule called build or something - which requires build_prerequisites target and then calls make to do the actual build once this is complete:
.PHONY: build
build: build_prerequisites
build:
#echo "start_build"
#$(MAKE) bin_file
This means that build_prerequisites is always run first before the recipe runs. You cant seem to achieve the same forcing of order (at least not easily) using just dependencies. I.e. a list of dependencies can be run in any order with -j, but the rule recipe is always run last.
Now we have this pattern we can fill in the rest. First the build_prerequisites target which does your file generation - I am using echo in my example because I don't have your python script:
.PHONY: build_prerequisites
build_prerequisites:
#echo "build_prerequisites"
echo "create file1" > file1.cpp
echo "create file2" > file2.cpp
echo "create file3" > file3.cpp
Finally add in the c++ compile and link stages - these will be run with the single recursive make call from build - i.e. $(MAKE) bin_file (again I am using echo to create the files in my example):
%.o : %.cpp
#echo "compiling: $<"
##echo "$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $#"
#echo "touch" > $#
bin_file : $(OBJ_FILES)
#echo "linking: $<"
#echo $(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $#
#echo "touch" > $#
Output
Here is the output from my test program (using echo) and main.cpp already exists usingn -j10:
make -j10
build_prerequisites
echo "create file1" > file1.cpp
echo "create file2" > file2.cpp
echo "create file3" > file3.cpp
start_build
make[1]: Entering directory '/mnt/d/software/ubuntu/make'
compile: bin_main.cpp
compile: file1.cpp
compile: file2.cpp
compile: file3.cpp
link: bin_main.o
g++ bin_main.o file1.o file2.o file3.o -o bin_file
make[1]: Leaving directory '/mnt/d/software/ubuntu/make'
Note: if I put a sleep 1 in the "compile" rule - this still takes only 1 second for all 4 files to compile.
Put it all together
GEN_FILES := file1.cpp file2.cpp file3.cpp
CXX_FILES := bin_main.cpp $(GEN_FILES)
OBJ_FILES := $(patsubst %.cpp,%.o,$(CXX_FILES))
###### STAGE 1
.PHONY: build
build: build_prerequisites
build:
#echo "start_build"
#$(MAKE) bin_file
.PHONY: build_prerequisites
build_prerequisites:
#echo "build_prerequisites"
copy_and_pp_files.py $(CXX_FILES) $(SEARCH_DIRS) .
copy_and_pp_files.py $(CFG_FILES) $(SEARCH_DIRS) .
###### STAGE 2
%.o : %.cpp
#echo "compiling: $<"
#$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $#
bin_file : $(OBJ_FILES)
#echo "linking: $<"
#$(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $#
###### OTHER RULES
.PHONY: clean
clean :
#$(RM) *.o
#$(RM) file*
I have attempted to use your actual code, but I have no way to test this so there may be a bug in there. I split it up into 2 "stages" for clarity. Stage 1 is done in your makeor make build call, then state 2 is done in the recursive make call in the build recipe.
today I'm requesting your help about a Makefile that's driving me crazy. There it is:
# Executable name
NAME = libft.a
# Compiler and archive linker settings
CC = gcc
AR = ar
CFLAGS = -Wall -Wextra -Werror -O3 -g3
ARFLAGS = -rsc
IFLAGS = -I./includes/
# Project layout
SRC_DIR = ./src/
INC_DIR = ./inc/
OBJ_DIR = ./obj/
OBJ = $(shell grep -r .o ./obj | awk '{print $$3}' | tr '\n' ' ')
.PHONY: all clean fclean re
#------------------------------------------------------------------------------#
all: $(OBJ_DIR) $(NAME)
$(OBJ_DIR):
mkdir -p $(OBJ_DIR)
$(NAME): compile $(OBJ) $(INC_DIR)libft.h
#echo "Linking library $(NAME).\n"
#$(AR) $(ARFLAGS) $(NAME) $(OBJ)
#echo " ✧ $(AR) $(ARFLAGS) $(NAME) object files: OK! √\n"
compile:
make -C src/io
make -C src/lists
make -C src/memory
make -C src/strings
make -C src/tests
I've tried multiple combination of dependencies, rules, etc but I just don't get it. Sometimes I got it to stop relinking but in thoses cases it wouldn't re-compile object files because $(OBJ) was empty and wasn't updated after I ran compile.
This version is close to be good, but everytime I run make it executes the recipe $(NAME) and does the ar -rsc %(OBJ) .. How can I put them in dependencies to $(NAME) ?
Well, basically your entire approach here cannot succeed. Just for one example: you are trying to find the object files using grep (honestly I don't understand that shell command at all; what does printing the $3 word from the output of grep -r do??? Don't you just mean find $(OBJ_DIR) -name \*.o here?) This will expand to all the object files found in your subdirectories. But, that shell command runs when your top-level makefile is parsed, and that parsing happens before make runs any rules... so, no object files have been built yet! So, this target doesn't depend on anything. Even after some object files have been built, it only depends on object files that already exist, not on object files that are created during the build.
Really if I were you I'd do this completely differently. However, the simplest way to make your makefile work properly as written is to build $(NAME) using a recursive make as well; change your makefile like this:
all: compile
$(NAME): $(OBJ) $(INC_DIR)libft.h
#echo "Linking library $(NAME).\n"
#$(AR) $(ARFLAGS) $# $^
#echo " ✧ $(AR) $(ARFLAGS) $# object files: OK! √\n"
compile:
mkdir -p $(OBJ_DIR)
$(MAKE) -C src/io
$(MAKE) -C src/lists
$(MAKE) -C src/memory
$(MAKE) -C src/strings
$(MAKE) -C src/tests
$(MAKE) $(NAME)
Here all doesn't depend on $(NAME); instead, the compile step first builds everything then at the end it recursively invokes itself to build $(NAME); at this point we know everything is up to date and we can depend on the object files existing.
Other things: note I used the automatic variable $^ here not $(OBJ); that variable is a simple variable that runs a shell script: it's expensive! Every time you expand the $(OBJ) variable you pay that cost, so you only ever want to do it one time. Alternatively, you can use := to set OBJS instead so it's only invoked once per make instance. That's still one more time than you need but avoiding this will be painful.
I also moved the mkdir into the compile rule. It's cleaner there than as a prerequisite of all.
Finally, you should never invoke sub-makes using the make command directly. Always use the $(MAKE) variable, or various things will not work correctly.
The question was obvioulsy solved by the previous post.
You need to use the $(MAKE) variable to call recursively your make file with the $(NAME) rule instead of putting $(NAME) as a all dependency, after subsequent calls to your underlying Makefiles using the $(MAKE) variable again.
Is there any trick in GNU Makefiles to get all dependencies of a rule?
Example:
rule1: dep1_1 dep1_2 dep1_3
rule2: dep2_1 dep2_2 rule1
dump_deps:
echo "Dependencies of rule1: $(call do_the_trick, $(rule1))"
echo "Dependencies of rule2: $(call do_the_trick, $(rule2))"
install: $(prefix install-,$(call do_the_trick, $(rule1)))
I'd like to be able to call make dump_deps and see:
dep1_1 dep1_2 dep1_3
dep2_1 dep2_2 dep1_1 dep1_2 dep1_3
Or automatically install dependencies with make install and things like.
Is it possible?
EDIT:
I changed the example to show better that I want something automatic, and not having to hardcode dependency lists myself.
You can't show the transitive dependencies, only the direct ones, however, you can take the output generated below and feed it into the programs dot (part of graphviz) to make sense of the these transitive relationships.
Edit: I guess you could also post-process the results in other ways to just list the deps, but I think pretty pictures are better; feel free to downvote if you disagree ;)
Here's a sample makefile (watch for missing tabs when c&p!):
# Makefile that demonstrates how to dump dependencies.
# The macros we use for compiling stuff.
CC_OBJ=$(CC) -o $# -c $(CFLAGS) $<
CC_BIN=$(CC) -o $# $(CFLAGS) $(LDFLAGS) $^
# If we get "DUMP_DEPS=1 DUMP_DOT=1" on the command line, then instead of
# make-style dependencies, we'll output 'dot' syntax.
# Note: normally, DUMP_DOT_TAIL is undefined, so it doesn't generate any noise.
ifeq ($(DUMP_DOT),1)
DUMP_DOT_HEAD=digraph dependencies {
DUMP_DOT_TAIL=#echo "}"
$(info $(DUMP_DOT_HEAD))
list_dep=#for f in $^; do echo " \"$#\" -> \"$$f\";"; done
else
list_dep=#echo "$#: $^"
endif
# If we get "DUMP_DEPS=1" on the command line, then
# instead of building the code, just print the dependencies.
ifeq ($(DUMP_DEPS),1)
CC_OBJ=$(list_dep)
CC_BIN=$(list_dep)
endif
# An implicit rule that builds *.c -> *.o.
%.o:%.c
$(CC_OBJ)
# Variables for stuff we wanna build.
target=hello
objects=main.o
objects+=stuff.o
objects+=yeah.o
# The top-level 'all' target.
.PHONY: all
all: $(target)
$(DUMP_DOT_TAIL)
# Builds our final executable
$(target): $(objects)
$(CC_BIN)
# A standard clean target.
.PHONY: clean
clean:
-rm -f $(target) $(objects)
Now, you can do this:
make -B DUMP_DEPS=1
And it will go through and list all your pre-requisites is make style of "target: pre-requisite". Sample outputs:
Normal run:
cc -o main.o -c main.c
cc -o stuff.o -c stuff.c
cc -o yeah.o -c yeah.c
cc -o hello main.o stuff.o yeah.o
With make -B DUMP_DEPS=1:
main.o: main.c
stuff.o: stuff.c
yeah.o: yeah.c
hello: main.o stuff.o yeah.o
With make -B DUMP_DEPS=1 DUMP_DOT=1:
digraph dependencies {
"main.o" -> "main.c";
"stuff.o" -> "stuff.c";
"yeah.o" -> "yeah.c";
"hello" -> "main.o";
"hello" -> "stuff.o";
"hello" -> "yeah.o";
}
You could then run the following to output a pretty picture to an SVG image:
make -B DUMP_DEPS=1 DUMP_DOT=1 | dot -Tsvg > deps.svg
Here's what it looks like (this is actually a png, generated with -Tpng > deps.png):
I think this would need some extra work to produce accurate results in all situations, but the principle is sound (e.g. if you use gcc-generated dependency files, you'll need to create them first).
This should list all of the dependencies:
DEPEND = dep1 dep2 dep3
.PHONY: $(DEPEND)
dump_deps: $(DEPEND)
#printf "%s\n" $^
You should remove the line with .PHONY target for your needs. It is used for the sake of the example. And please note that there is a tab between the beginning of the line and printf.
use the buit in variables available with the make utility , $^ -- this represents all the dependencies of the rule.
also looking thru your code , i did make the rule1 into a variable / macro form to suit my purpose ......
rule1 := dep1 dep2 dep3
I am trying to exclude main.cpp file from the list of files to be compiled defined by the rule below:
$(TMPDIRPATH)%.o: %.cpp
#echo compile $<
ifneq ($(notdir $<), main.cpp)
#$(COMPILE.cpp) $(OUTPUT_OPTION) $<
endif
This 'ifneq' condition always evaluates to true, which is bizarre. What am I doing wrong? Is there a better way to exlude one file from an explicit rule?
Why don't you try using the filter-out text function if you're using GNU Make.
Returns all whitespace-separated words in text that do not match any of the pattern words, removing the words that do match one or more. This is the exact opposite of the filter function.
For example, given:
objects=main1.o foo.o main2.o bar.o
mains=main1.o main2.o
the following generates a list which contains all the object files not in ‘mains’:
$(filter-out $(mains),$(objects))
That isn't the best way to do it, but if you do it along these lines, write it as a shell condition, not using GNU make conditionals:
$(TMPDIRPATH)%.o: %.cpp
#echo compile $<
#if [ $(notdir $<) != main.cpp ]; \
then $(COMPILE.cpp) $(OUTPUT_OPTION) $<; \
fi
The continuation markers (backslashes) are needed. So are the semicolons. The values prefixed with $ will be expanded by make before the shell is invoked to interpret them. You probably don't want the echo where it is, either. You probably need:
$(TMPDIRPATH)%.o: %.cpp
#if [ $(notdir $<) != main.cpp ]; \
then echo compile $<; \
$(COMPILE.cpp) $(OUTPUT_OPTION) $<; \
fi
The way I would expect to do it is with a list of the files to be compiled. Using any wild card mechanism leads to problems when extra files are added - other tests, or stray files that aren't really part of the system.
The comment says "But the GNU Make Manual says ifneq should work".
The ifneq would work if it were positioned correctly, which means 'not indented as part of the commands associated with a rule'. You could, therefore, write something like (an appallingly bad example, but my brain's on the fritz):
ifneq (${CFLAGS}, -Wall)
CFLAGS += -Wall
endif
file1.o: file1.c
${CC} ${CFLAGS} -c $<
But when the ifneq is indented as in the question, it is just a command that actually isn't found on the system when the make runs the shell to process the command.
The ifneq line is evaluated only once, when make starts up and parses the makefile. In that context, $< is empty.
To get different behavior for each of the targets matched by your pattern rule, you could do something like
$(TMPDIRPATH)%.o: %.cpp
#echo compile $<
#$(if $(filter main.cpp,$<),$(COMPILE.cpp) $(OUTPUT_OPTION) $<)
It might help you to think of the difference between ifneq and $(if) in a makefile as like the difference between #if and if() in C code.
Taking a step back, though: If you don't want main.cpp to be compiled by this rule, then you probably want to provide an explicit rule with $(TMPDIRPATH)main.o as its target, which will be preferred to the pattern rule always. Or, if you don't want$(TMPDIRPATH)main.o to get made at all, you should be looking for rules that have it on the right sight of the :, and removing it from there.
Make doesn't really have a good way to handle conditionals within a rule. You could put the conditional in the command, but in this case there's a much cleaner way:
$(TMPDIRPATH)main.o:
#echo compile $< (but not really)
$(TMPDIRPATH)%.o: %.cpp
#echo compile $<
#$(COMPILE.cpp) $(OUTPUT_OPTION) $<
EDIT:
I didn't realize you didn't have a main.cpp. The solution is simple: remove main.cpp as the prerequisite of the main.o rule (I've removed it above). Now the makefile doesn't need it, and won't try to build it.
But you're still running the rule, which means that something is still trying to build main.o, as either an explicit target or a prerequisite of something else. That is a symptom of confusion, which this change to the makefile will not fix. If you tell us more about the situation, maybe we can propose a better solution. What calls for main.o? Do you have a main.o? What target do you specify when you call Make?