Getting Make to automatically detect changes in protocol buffers - makefile

I want Make to automatically compile protos when I update them, here is what I've got so far:
TARGET=main
BIN_DIR=bin
SRC_DIR=src
OBJ_DIR=obj
PROTO_DIR=protos/
PROTO_COMPILE_DIR=src/$(PROTO_DIR)
CC = g++
CFLAGS = -Wall -std=c++17 -ggdb -pipe -I.
LINKER = g++
LFLAGS = $(CFLAGS) -lprotobuf
SOURCES = $(wildcard src/*.cc) \
$(wildcard src/protos/*.cc) \
$(wildcard src/db_handler/*.cc)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.cc=$(OBJ_DIR)/%.o)
$(BIN_DIR)/$(TARGET): proto $(OBJECTS)
#mkdir -p $(BIN_DIR)/
$(LINKER) $(OBJECTS) $(LFLAGS) -o $#
$(OBJECTS): $(OBJ_DIR)/%.o : $(SRC_DIR)/%.cc
#mkdir -p obj/ obj/protos obj/db_handler
$(CC) $(CFLAGS) -c $< -o $#
.PHONY: proto
proto:
#printf "Compiling protos...\n"
#cd $(PROTO_DIR) && protoc * --grpc_out=../$(PROTO_COMPILE_DIR)\
--cpp_out=../$(PROTO_COMPILE_DIR)\
--plugin=protoc-gen-grpc="/usr/local/bin/grpc_cpp_plugin"\
&& cd ../
It successfully compiles protos; but, it does so every time, even if there are no changes in files. How can I prevent this and compile protos only if protos change?
Edit: Added project structure
├── LICENSE
├── makefile
├── protos
│   ├── client.proto
│   └── person.proto
├── README.md
└── src
├── db_handler
│   ├── db_handler.cc
│   └── db_handler.h
├── main.cc
└── protos
├── client.grpc.pb.cc
├── client.grpc.pb.h
├── client.pb.cc
├── client.pb.h
├── person.grpc.pb.cc
├── person.grpc.pb.h
├── person.pb.cc
└── person.pb.h

You want to add the source files of the protocol buffers to the right (the prerequisites) of the rule that cares about them. This is how Make understands and tracks their time stamps.
By adding the files to the prerequisites, Make will understand that this is the rule which cares about those source files.
PROTO_SOURCES := $(wildcard $(PROTO_DIR)/*.proto)
PROTOS := $(patsubst $(PROTO_DIR)/%.proto,$(PROTO_COMPILE_DIR)/%.cc,$(PROTO_SOURCES))
$(PROTOS): $(PROTO_SOURCES)
#printf "Compiling protos...\n"
#cd $(PROTO_DIR) && protoc * --grpc_out=../$(PROTO_COMPILE_DIR)\
--cpp_out=../$(PROTO_COMPILE_DIR)\
--plugin=protoc-gen-grpc="/usr/local/bin/grpc_cpp_plugin"
However, this $(PROTOS) : $(PROTO_SOURCES) is not good if you use parallel builds; because, Make will try to run the command once for each output file. So, make will run N instances of the command at the same time, which means they may clobber each other.
In order to know the fully correct solution you need to provide more information (for those not familiar with protoc). Is it a requirement that you invoke protoc once with all inputs? Or is it valid to run protoc individually on each input .proto file to get its output? Then you can write a pattern rule that generates one file at a time.
Note 1: As to your other attempt, if you use .PHONY to mark the rule then Make will rebuild the rule every time weather it needs to or not.
Note 2: You don't need the cd .. at the end of the second instruction, since it is run in a sub shell.

Thanks to #FiddlingBits I managed to figure out how to do it correctly
TARGET=main
BIN_DIR=bin
SRC_DIR=src
OBJ_DIR=obj
PROTO_DIR=protos/
PROTO_COMPILE_DIR=src/$(PROTO_DIR)
rm = rm -f
CC = g++
CFLAGS = -Wall -std=c++17 -ggdb -pipe -I.
LINKER = g++
LFLAGS = $(CFLAGS) -lprotobuf
SOURCES = $(wildcard src/*.cc) \
$(wildcard src/protos/*.cc) \
$(wildcard src/db_handler/*.cc)
OBJECTS := $(SOURCES:$(SRC_DIR)/%.cc=$(OBJ_DIR)/%.o)
PROTOS := $($(PROTO_DIR)/%.proto=$(PROTO_COMPILE_DIR)/%.cc)
$(BIN_DIR)/$(TARGET): $(PROTOS) $(OBJECTS)
echo $(PROTOS)
#mkdir -p $(BIN_DIR)/
$(LINKER) $(OBJECTS) $(LFLAGS) -o $#
$(OBJECTS): $(OBJ_DIR)/%.o : $(SRC_DIR)/%.cc
#mkdir -p obj/ obj/protos obj/db_handler
$(CC) $(CFLAGS) -c $< -o $#
$(PROTOS):
#printf "Compiling protos...\n"
#cd $(PROTO_DIR) && protoc * --grpc_out=../$(PROTO_COMPILE_DIR)\
--cpp_out=../$(PROTO_COMPILE_DIR)\
--plugin=protoc-gen-grpc="/usr/local/bin/grpc_cpp_plugin"\
&& cd ../
.PHONY: clean
clean:
#$(rm) -r $(OBJ_DIR)/*
#$(rm) -r $(BIN_DIR)/*
#printf "Cleanup complete!\n"

Related

Write generic Makefile rule for subdirectories

I have a project with a directory tree that looks like this:
.
├── modules
│   ├── mod1
│   │   └── mod1.f90
│   ├── mod2
│   │   └── mod2.f90
│   └── mod.f90
└── src
└── main.f90
├── bin
└── Makefile
main.f90 uses all the modules found in modules and in the 'submodules' that are mod1/ and mod2/.
A simple Makefile I can write to compile the project is something like:
F90 = gfortran
FLAGS = -g -I$(BINDIR)
MODFLAGS = -J$(BINDIR)
BINDIR = bin
SRCDIR = src
MODDIR = modules
SMODDIR = $(dir $(wildcard $(MODDIR)/*/.))
MODFILES = $(wildcard $(MODDIR)/*f90)
MODOBJ = $(join $(addsuffix $(MODFROMBIN)/, $(dir $(MODFILES))), \
$(notdir $(MODFILES:.f90=.o)))
SMODFILES = $(foreach smoddir, $(SMODDIR), $(wildcard $(smoddir)*f90))
SMODOBJ = $(join $(addsuffix $(SMODFROMBIN)/, $(dir $(SMODFILES))), \
$(notdir $(SMODFILES:.f90=.o)))
SRCFILES = $(wildcard $(SRCDIR)/*f90)
SRCOBJ = $(join $(addsuffix $(SRCFROMBIN)/, $(dir $(SRCFILES))), \
$(notdir $(SRCFILES:.f90=.o)))
ALLOBJ = $(MODOBJ) $(SMODOBJ) $(SRCOBJ)
BINOBJ = $(addprefix $(BINDIR)/, $(sort $(notdir $(ALLOBJ))))
all: main
$(MODDIR)/../bin/%.o: $(MODDIR)/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
modules/mod1/../../bin/%.o: modules/mod1/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
modules/mod2/../../bin/%.o: modules/mod2/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
$(SRCDIR)/../bin/%.o: $(SRCDIR)/%.f90
$(F90) $(FLAGS) -c $^ -o $#
main: $(ALLOBJ)
$(F90) $(FLAGS) -o ./bin/main $(BINOBJ)
clean:
#rm bin/*.o bin/*.mod
But now I want to write a generic rule to be able to compile all modules located in directories that are inside modules directory (the code I am working on have more than two submodules and I am not willing to write as many rules as I have subdirectories).
My first try was to write something like this:
$(SMODDIR)/../../%.o: $(SMODDIR)/%.f90
$(F90) $(FLAGS) -c $^ -o $# $(MODFLAGS)
but it fails; from what I understand, it will put all the subdirectories paths and I would actually end up with a rule that would looks like this:
modules/mod1/../../bin/ modules/mod2/../../bin/%.o: modules/mod1/ modules/mod2/%.f90
that indeed looks funny.
As shown in the original Makefile, it is possible to retrieve specificaly with
SMODDIR = $(dir $(wildcard $(MODDIR)/*/.))
SMODFILES = $(foreach smoddir, $(SMODDIR), $(wildcard $(smoddir)*f90))
so I guess it may be possible to use something similar to have more generic rules. I couldn't see however how to use such a syntax to write a rule that makes sense.
Any help would be appreciated!
You have two choices to avoid writing lots of rules.
One is, you can use VPATH and put all your source directories in it, something like this:
VPATH := $(MODDIR) $(SMODDIR) $(SRCDIR)
$(BINDIR)/%.o: %.f90
$(F90) $(FLAGS) -c $< -o $# $(MODFLAGS)
Or you can put all your object files into equivalent subdirectories of $(BINDIR), something like this:
MODOBJ = $(MODFILES:%.f90=$(BINDIR)/%.o)
SMODOBJ = $(SMODFILES:%.f90=$(BINDIR)/%.o)
SRCOBJ = $(SRCFILES:%.f90=$(BINDIR)/%.o)
$(BINDIR)/%.o: %.f90
#mkdir -p $(#D)
$(F90) $(FLAGS) -c $< -o $# $(MODFLAGS)

Failing to link to SFML libraries using Makefile even though it works as a one liner

Here is my directory structure :
.
├── a.out
├── assets
│   └── ...
├── build
│   ├── apps
│   └── objects
├── include
│   └── engine
│   └── Class.h
├── Makefile
└── src
├── engine
│   └── Class.cpp
└── program.cpp
I can compile program.cpp into a.out using the following command :
g++ src/engine/*.cpp src/program.cpp -Iinclude/ -L/usr/lib -lstdc++ -lm -lsfml-graphics -lsfml-window -lsfml-system -Wall
While this works, this project is likely to grow and obviously making a serious Makefile would be preferable to direct compiling with a one liner. So I've used a Makefile format that I've used many times before, and that works perfectly fine but I've never linked it to outside libraries in the past. Here is the Makefile I'm using :
CXX := -g++
CXXFLAGS := -std=gnu++0x -g -Wall
LDFLAGS := -L/usr/lib -lstdc++ -lm -lsfml-graphics -lsfml-window -lsfml-system
BUILD := ./build
OBJ_DIR := $(BUILD)/objects
APP_DIR := $(BUILD)/apps
TARGET := program
INCLUDE := -Iinclude/
SRC := $(wildcard src/engine/*.cpp) $(wildcard src/*.cpp)
OBJECTS := $(SRC:%.cpp=$(OBJ_DIR)/%.o)
all: build $(APP_DIR)/$(TARGET)
$(OBJ_DIR)/%.o: %.cpp
#mkdir -p $(#D)
$(CXX) $(CXXFLAGS) $(INCLUDE) -o $# -c $<
$(APP_DIR)/$(TARGET): $(OBJECTS)
#mkdir -p $(#D)
$(CXX) $(CXXFLAGS) $(INCLUDE) $(LDFLAGS) -o $(APP_DIR)/$(TARGET) $(OBJECTS)
.PHONY: all build clean debug release
build:
#mkdir -p $(APP_DIR)
#mkdir -p $(OBJ_DIR)
debug: CXXFLAGS += -DDEBUG -g
debug: all
release: CXXFLAGS += -O2
release: all
clean:
-#rm -rvf $(OBJ_DIR)/*
-#rm -rvf $(APP_DIR)/*
However, this leads to compilation errors, in the form of undefined references to SFML methods :
./build/objects/src/engine/Class.o: In function `Class::draw()':
/dir/Class.cpp:60: undefined reference to `sf::RenderTarget::getView() const'
I'm confused as to why this happens given that the one liner above compiles fine. The Makefile also works just fine if I remove all references to SFML from my code. Were the added LDFLAGS not enough even though they're all that's needed to make my one liner link to SFML ? If so, what else is needed ?
From gcc link options
-llibrary
It makes a difference where in the command you write this option; the linker searches and processes libraries and object files in the order they are specified. Thus, foo.o -lz bar.o searches library z after file foo.o but before bar.o. If bar.o refers to functions in z, those functions may not be loaded.
So put the libraries (-l<lib_1> ... -l<lib_x>) last on the line, after your object files.

Generic Makefile build directory error

I have the following directory structure for a dummy C project.
.
├── inc
│   ├── getmsg.c
│   └── getmsg.h
├── makefile
└── src
└── main.c
My current generic Makefile is below,
TARGET = main
# finds all .c files, inc/getmsg.c src/main.c
SOURCES := $(shell find * -name *.c)
# converts all .c files to .o files, inc/getmsg.o src/main.o
OBJECTS := $(SOURCES:.c=.o)
# directories that contain .h files for gcc -I flag, inc/
HEADERS := $(dir $(shell find * -name *.h))
CC = gcc
CFLAGS = -Wall -std=c99 -iquote "$(HEADERS)"
all: $(TARGET)
%.o: %.c
$(CC) $(CFLAGS) -c $< -o $#
$(TARGET): $(OBJECTS)
$(CC) -o $# $^
clean:
rm -rf $(shell find * -name *.o) $(TARGET)
This all compiles fine however it just dumps the .o files into the same directory as its corresponding .c file.
What I would like to do is have all object files put into a build directory. To do this I change the OBJECTS to OBJECTS := $(patsubst %,build/%,$(notdir $(SOURCES:.c=.o))) which lists the files build/getmsg.o build/main.o. Then I set the %.o target to build/%.o: %.c.
This however returns No rule to make target 'build/getmsg.o'. So the make file is unable to build the .o files. What am I missing here?
Try changing
%.o: %.c
to
build/%.o: %.c

Using GNU Make with subdirectories

I was wondering what different approaches of using Make in a project with subdirectories exist, and what are their advantages/drawbacks, but could never see a good summary or cookbook.
I have seen in my researches mainly the "recursive" and "single makefile" approaches, but are there others ?
I also assume that there is not only one "recursive" or "single makefile" approaches but several, so could somebody sum it up ?
For my particular case, I would like a directory architecture looking like this:
.
├── build
│   ├── *.d
│   ├── *.o
| ├── subdir1
| │ ├── *.d
| │ └── *.o
| └── subdir2
| ├── *.d
| ├── *.o
| └── subdir3
| ├── *.d
| └── *.o
├── include
│   ├── *.h
│   └── *.h
├── Makefile
└── src
├── *.c
├── *.h
├── subdir1
│   ├── *.c
│   └── *.h
└── subdir2
├── *.c
├── *.h
└── subdir3
├── *.c
└── *.h
Which solution should I choose ? Possibly one which would allow source files with the same name ?
Your project setup is really basic, so should be your Makefile:
SRC_DIR := src
BLD_DIR := build
SRC := $(shell find $(SRC_DIR) -name "*.c")
OBJ := $(SRC:$(SRC_DIR)/%.c=$(BLD_DIR)/%.o)
DEP := $(OBJ:.o=.d)
CPPFLAGS := -MMD -MP # enable auto-dependency generation
CFLAGS := -Wall -W -pedantic
.PHONY: all clean
all: $(OBJ)
clean:
$(RM) -r $(BLD_DIR)
.SECONDEXPANSION:
$(BLD_DIR)/%.o: $(SRC_DIR)/%.c | $$(#D)/ # First check that the destination directory exists
$(CC) $(CPPFLAGS) $(CFLAGS) -o $# -c $<
%/:
mkdir -p $* # -p flag necessary for recursive directory creation
ifeq "$(MAKECMDGOALS)" ""
-include $(DEP)
endif
The idea here is to list source files recursively using the find command, to supply make with the appropriate pattern rule to compile in the right place and pass the right preprocessor file to your compiler to enable auto-dependency generation.
Tested with GNU Make 4.1 under Windows 8.1 with the GIT Bash shell and the following directory structure:
.
├── Makefile
└── src
├── test.c
├── test1.c
└── subdir1
└── test.c
After reading Recursive Make Considered Harmful, I figured a quite simple and modular way to achieve this, by having files in all subdirectories that would include each other and be included in the main makefile:
CXX := gcc
SRCDIR := src
OBJDIR := build
# These lines are needed to set immediate evaluation for
# these variables, instead of deferred evaluation which is unsuitable.
SRCS :=
SUBDIRS :=
CFLAGS :=
LDFLAGS :=
include $(SRCDIR)/module.mk
OBJS := $(addprefix $(OBJDIR)/, $(SRCS:.c=.o))
SRCS := $(addprefix $(SRCDIR)/, $(SRCS))
DEPS := $(OBJS:.o=.d)
TMPS := $(OBJS) $(OBJS:.o=.d)
CFLAGS += -MD
debug: CFLAGS += -g -g3 -ggdb
CFLAGS += $(addprefix -I./$(SRCDIR)/, $(SUBDIRS))
LDFLAGS += -lsomelib
debug: LDFLAGS += -g -g3 -ggdb
NAME := yolo
all: $(NAME)
debug: re
-include $(DEPS)
$(OBJDIR)/%.o: $(SRCDIR)/%.c
$(COMPILE.c) $(OUTPUT_OPTION) $<
$(NAME): $(OBJS)
#$(CXX) $(OBJS) -o $(NAME) $(LDFLAGS)
$(OBJS): | $(OBJDIR)
$(OBJDIR):
#mkdir -p $(OBJDIR)
#for dir in $(SUBDIRS); \
do \
mkdir -p $(OBJDIR)/$$dir; \
done
clean:
rm -rf $(TMPS)
fclean: clean
rm -rf $(NAME)
rm -rf $(OBJDIR)
re: fclean all
.PHONY: all clean fclean re
And in every subdirectory, a module.mk file (I could have named it anything, but this seemde cool).
For src:
SRCS := main.c file1.c file2.c
SUBDIRS += subdir1 subdir2
include $(SRCDIR)/subdir1/module.mk
include $(SRCDIR)/subdir2/module.mk
For a level 1 subdirectory:
THIS_DIR_L0 := subdir1
MOD_SRC := file3.c file4.c
SRCS += $(addprefix $(THIS_DIR_L0)/, $(MOD_SRC))
SUBDIRS += $(THIS_DIR_L0)/subdir3
include $(SRCDIR)/$(THIS_DIR_L0)/subdir3/module.mk
And for a level 2 subdir (one deeper):
THIS_DIR_L1 := subdir3
MOD_SRC := file5.c file6.c
SRCS += $(addprefix $(THIS_DIR_L0)/$(THIS_DIR_L1)/, $(MOD_SRC))
And so on...
This is quite simple to set up, I find it very modular and it does not use recursive makefiles. It would not be complicated to make librairies and stuff inside your directory structure either.
Anybody having a better idea please tell me.

Makefile - move object files

After a bit of searching, I've managed to throw together the following Makefile:
CC = gcc
CFLAGS = -c -Wall
LDFLAGS =
SOURCEFILES = main.c
SOURCES = src/$(SOURCEFILES)
OBJECTS = $(SOURCES:.c=.o)
EXECUTABLE = netsim
all: $(SOURCES) $(EXECUTABLE)
$(EXECUTABLE): $(OBJECTS)
$(CC) $(LDFLAGS) $(OBJECTS) -o $#
.c.o:
$(CC) $(CFLAGS) $< -o $#
clean:
rm -rf netsim $(OBJECTS)
I would like to be able to move my object files into another directory, but have been struggling with getting that to work. What am I missing?
The trick is to not move your objects.
You should build it and use it from where they are built.
For example you have the following directory structure:
$ tree .
├── Makefile
├── include
│   └── common_head.h
├── obj
└── src
├── foo.c
└── main.c
Manual execution:
$ gcc -o ./obj/foo.o -c ./src/foo.c -I ./include # Build Object #
$ gcc -o ./obj/main.o -c ./src/main.c -I ./include
$ gcc -o exe ./obj/foo.o ./obj/main.o # Build Executable #
Makefile to simulate the above
C_FLAGS := -g -Wall -Wextra
CC := gcc
RM := rm
LINKFLAGS := -lanylibrary
.PHONY: $(TARGET)
.PHONY: clean
VPATH:= ./src/ ./obj/ ./include/
# Path for .c , .h and .o Files
SRC_PATH := ./src/
OBJ_PATH := ./obj/
INC_PATH := -I ./include
# Executable Name
TARGET := exe
# Files to compile
OBJ1 := foo.o \
main.o
OBJ := $(patsubst %,$(OBJ_PATH)%,$(OBJ1))
# Build .o first
$(OBJ_PATH)%.o: $(SRC_PATH)%.c
#echo [CC] $<
#$(CC) $(C_FLAGS) -o $# -c $< $(INC_PATH)
# Build final Binary
$(TARGET): $(OBJ)
#echo [INFO] Creating Binary Executable [$(TARGET)]
#$(CC) -o $# $^ $(LINKFLAGS)
# Clean all the object files and the binary
clean:
#echo "[Cleaning]"
#$(RM) -rfv $(OBJ_PATH)*
#$(RM) -rfv $(TARGET)
So when you do a Make
$ make -B
[CC] src/foo.c
[CC] src/main.c
[INFO] Creating Binary Executable [exe]
To see a dry-run use make -n
$ make clean ; make -n
g++ -g -Wall -Wextra -o obj/foo.o -c src/foo.c -I ./include
g++ -g -Wall -Wextra -o obj/main.o -c src/main.c -I ./include
g++ -o exe obj/foo.o obj/main.o -lanylibrary
So after building your directory structure should look like this.
$ tree .
├── Makefile
├── exe
├── include
│   └── common_head.h
├── obj
│   ├── foo.o
│   └── main.o
└── src
├── foo.c
└── main.c
So from my previous answer.
You don't have to use any PHONY move and also no objects are recreated unnecessarily.
Something like this?
SOURCES = src/main.c
OBJECTS = obj/main.o
...
obj/%.o: src/%.c
$(CC) $(CFLAGS) $< -o $#
Once that's working, you can add further tricks, like this:
OBJECTS = $(patsubst src/%.c, obj/%.o, $(SOURCES)

Resources