I am starting to work on a project that I would like to grow to be fairly large. I Want to create a makefile that will grow with the project without much maintenance. Here is the directory structure that I have right now.
.
+--src
| +--part1
| | +--part1.c
| | +--part1.h
| +--part2
| | +--part2.c
| | +--part2.h
. .
. .
. .
| +--partN
| | +--partN.c
| | +--partN.h
+--test
| +--part1_tests
| | +--part1_testX.c
| | +--part1_testY.c
. .
. .
. .
+--obj
| +--part1.o
| +--part2.o
. .
. .
. .
| +--partN.o
+--a.out
I have never had a project of this scale and never needed to make a make file for such a project. How would I design a makefile for this? Thanks!
There are different ways to do it, but i would start by not letting the makefile grow with the project. Instead I would use the regular structure to define rules to handle projects with one codefile in one directory, to big projects with thousend of files.
For example lets play a little bit with your structure (not tested might have some typos and I assume that you start make from project dir):
# we nead the header directory
INCDIRS = src
# collect every cpp file
CXXSRCS = $(shell find src/ -type f -name '*.cpp' 2>/dev/null)
# now generate the object file names
OBJSTMP = $(CXXSRCS:%.cpp=obj/%.o)
# compiled the source file
obj/%.o: src/%.cpp
$(CXX) ${CXXFLAGS} $(foreach bin,${INCDIRS},-I${bin}) -o "$#" "$<"
now lets say your main is in part1.cpp:
ifneq (,$(wildcard src/part1/part1.cpp))
# I just dont like a.out...
executable=${APPLICATION_NAME}
# now lets build the exe:
${executable}: ${OBJS}
$(LD) $(LDFLAGS) -o $# $^
endif
now the last thing is a little cosmetic:
.PHONY: clean all
all: install
compile: ${OBJS}
package: compile ${executable}
#now we can move the object files to were they should be
mv -f obj/part*/* obj/
install: package
no matter how big your project is gonna be this makefile will do some of your steps. Its just to give your an idea. I ignored the test_files. But thinking ahead you can collect the test sources in the same way like I did with the normal sources.
So my point is, there is absolutely no reason why your makefile has to grow with the size of your project, only with its complexity.
For more information, look in the Documentation here at stackoverflow, there is plenty information for makefiles...
Hope that helps,
Kai
Related
I'm exploring ways to improve code module integration and have found interest in using named soft links as opposed to include directories, mainly because that allows the integrator to set a directory name of any imported module to something guaranteed not to collide with the module built or any of its imported modules, as well as keeping imported modules from accidently intercepting eachother. However I'm not sure how to actually accomplish this in a makefile.
Here is the directory hierarchy and expected links:
+---proj1
| +---inc
| | head.h
| |
| \---src
| code.c
|
+---proj2
| +---inc
| | head.h
| |
| \---src
| code.c
|
\---proj3
| Makefile
|
\---src
| code.c
| head.h
|
+---A --> ../../proj1/inc
| head.h
|
\---B --> ../../proj2/inc
head.h
In proj3/src/code.c i would have:
#include "head.h"
#include "A/head.h"
#include "B/head.h"
I think I need some way to run ln -s for each {directory,name} tuple prior to compiling the source, putting the link in the same directory as the source being compiled. Parsing INC=dir1 dir2 is simple enough, but how to represent and run ln -s for each pair/tuple in such a list? Or, if having a whole command in each element (ln -s dir1 localname), how to exec all of them?
Other suggestions would be deeply appreciated.
Something like this?
linkdirs := A B
dir_A := ../proj1/src
dir_B := ../proj2/src
.PHONY: all symlinks
all: symlinks hello
symlinks:
$(eval $(patsubst %,ln -fs $$(dir_%) % &&,$(linkdirs)) true
I have a Makefile which downloads data from a biological database. Given a project number it should first download a file containing all the run information about that project, then extract accession numbers from the information, then download in parallel the FASTQ files associated with those accession numbers. My problem is that I cannot get the variable FASTQ to be deferred until after run.txt and sra.txt have been downloaded. I have tried combinations of order-only prerequisites and .SECONDEXPANSION but still cannot get it to work. Is it even possible?
# Project
PROJECT := PRJNA257197
# Download
.SECONDEXPANSION:
FASTQ = $(patsubst %, %.fastq, $(shell cat sra.txt))
download: $$(FASTQ) | run.txt sra.txt
%.fastq: sra.txt
# Download FASTQ files
fastq-dump $*
sra.txt: run.txt
# Extract SRA accession numbers
cat $^ | cut -f 1 -d ',' | grep SRR | tr '\n' ' ' > $#
run.txt:
# Download run information
esearch -db sra -query $(PROJECT) | efetch -format runinfo > $#
To do what you want you want something more like this (comments inline):
# Project
PROJECT := PRJNA257197
# Include the fastqs.mk makefile.
include fastqs.mk
# Default target is all the fastq files.
all: $(FASTQS)
%.fastq: sra.txt
# Download FASTQ files
fastq-dump $*
# Create the fastqs.mk file from sra.txt.
fastqs.mk: sra.txt
sed 's.*/FASTQS+=&.fastq/' $< > $#
sra.txt: run.txt
# Extract SRA accession numbers
cat $^ | cut -f 1 -d ',' | grep SRR | tr '\n' ' ' > $#
run.txt:
# Download run information
esearch -db sra -query $(PROJECT) | efetch -format runinfo > $#
Assuming each .fastq file has a matching bare file (i.e. foo.fastq -> foo) then you probably want this as the pattern target instead.
%.fastq: % sra.txt
The magic here is in that included makefile. Specifically that make is smart enough to notice when it needs to build an included makefile and restart processing after that has been done. See How Makefiles Are Remade in the manual for more details.
Check out my Makefile below. We have a set of graphs that are included in our paper. These graphs can be auto-generated using python scripts. But not all collaborators have the python tools (or need them). What I'd like to do is add rules that lead to this behavior:
An explicit "make graphs" is required to rebuild the graph PDFs (which are also tracked by git).
If the graph PDFs are out of date (relative to their sources), a simple "make" will NOT attempt to rebuild them. Only the main document may be rebuilt.
If any graph PDFs have been updated LATER than the main document (either due to a pull or a "make graphs"), then a "make" will notice that the dependencies have changed and rebuild the document.
If a graph PDF is missing (even if it can be generated by "make graphs"), it's okay for the "make" to just fail.
In searching for a solution, I have found out about "Order-only prerequisites." But this is the converse of what I want. With an order-only prereq, "make" would implicitly cause "make graphs" to happen, but it would not rebuild the document. I don't want a "make graphs" to happen, but I DO want to rebuild the document.
Is it possible to do what I'm trying to do?
Thanks.
MAIN = main
TEXFILES = $(shell find . -name '*.tex')
BIBFILES = $(shell find . -name '*.bib')
FIGURES = $(shell find FIGS/ -name '*.pdf')
GRAPHS = $(shell find GRAPHS/ -name '*.py' | sed -e 's/py/pdf/')
LATEX = pdflatex
BIBTEX = bibtex
default: $(MAIN).pdf
$(MAIN).pdf: $(FIGURES) $(TEXFILES) $(BIBFILES)
$(LATEX) -output-format=pdf $(MAIN)
$(BIBTEX) $(MAIN)
$(LATEX) -output-format=pdf $(MAIN)
$(LATEX) -output-format=pdf $(MAIN)
$(MAIN).ps: $(MAIN).pdf
pdf2ps $(MAIN).pdf $(MAIN).ps
graphs: $(GRAPHS)
GRAPHS/%.pdf: GRAPHS/%.py
cd GRAPHS; python2 $*.py; cd -
clean:
rm -f *~ *.aux *.log *.bbl *.blg *.brf $(MAIN).dvi $(MAIN).ps $(MAIN).pdf
I think all you have to do is ensure make doesn't know how to rebuild the graphs unless the graphs target was requested. Something like this should be enough:
.PHONY: graphs
graphs: $(GRAPHS)
ifeq ($(filter graphs,$(MAKECMDGOALS)),graphs)
GRAPHS/%.pdf: GRAPHS/%.py
cd GRAPHS; python2 $*.py; cd -
endif
I have a directory tree with several java files. Example:
top
|-- src1
| |--- folder A
| |--- folder B
|-- src2
| |--- folder A
| |--- folder B
...
I want to compile all the files in those folders and move the compiled files to folder A-binor folder B-bin accordingly in the respective src folder. I have read that I can do this with the xargs utility, but I can't make heads or tails from the manual entry.
Can some one point me a way?
Are you obliged to use xargs to compile these?
Why not take a look at java Makefiles?
They will make your life easier when building a project.
Also, one more advice, i recommend that you take look into Apache Maven. Easy to use, and very handful when your java project get bigger in time.
Here is a quick guide to Maven.
Basic Makefile:
JC=javac
JR=java
build: ref.java
$(JC) ref.java
run: ref.class
$(JR) ref
clean:
rm -f *.class
Another example: (taken from the guide above)
JFLAGS = -g
JC = javac
.SUFFIXES: .java .class
.java.class:
$(JC) $(JFLAGS) $*.java
CLASSES = \
Foo.java \
Blah.java \
Library.java \
Main.java
default: classes
classes: $(CLASSES:.java=.class)
clean:
$(RM) *.class
Another option if you want to stick with bash + javac is to use find to identify the .java files, store the results in a variable, and then check if the variable was not empty.
SRC=`find src -name "*.java"`
if [ ! -z $SRC ]; then
javac -classpath $CLASSPATH -d obj $SRC
# stop if compilation fails
if [ $? != 0 ]; then exit; fi
fi
I'd like to do a Makefile that runs either with gnumake or makepp that packs all the files under given directiories:
DIRS:=$(shell find . -mindepth 2 -maxdepth 2 -not -name mp3 -not -name ".*" -type d)
PACKAGES = $(DIRS:%=%.npk)
all: packages
packages: $(PACKAGES)
%.npk: %/*
npack c $# #^
.PHONY: all packages
the problem is that there's no such thing as %/* in the dependencies.
I need the targets (X.npk) to depend on every file in directory X, but I don't know what the files are when I write the Makefile, 'cause they're generated later.
An example:
./dirA/x
./dirA/y
./dirB/e
./dirB/f
I'd like to create ./dirA.npk (depending on x,y), ./dirB.npk (e,f)
There's nothing I know about the dirs or the files in advance except that the find used in the 1st line finds all the dirs.
Try using the wildcard directive:
DEPS := $(foreach dir, $(DIRS), $(wildcard $(dir)/*))
%.npk: $(DEPS)
npack c $# $^
EDIT:
The above is just an example of using wildcard and makes each .npk file dependent on the files in all of the other folders. Your usage would be slightly different.
I think there may be an easier way to go about this. Why are you wanting to have a dependency on all of the files in the folder? Is it just to use the $^ operator? Or do you need to rebuild the .npk if any of the files changed?
One alternate (and possibly cleaner) solution would be to use the find utility in your recipe instead of $^ and use the .FORCE directive to always force the .npk file to be rebuilt. The downside is that .npk files may be rebuilt unnecessarily.
EDIT 2:
If there's not a way to do this cleanly with make commands, you can work around it by using .FORCE to ensure that the recipe is always run and move the "should I rebuild this file" check into the body of the recipe:
%.npk: .FORCE
check_for_rebuild.sh $# && npack c $# $^
where check_for_rebuild.sh is a shell script that does something like this:
#!/bin/bash
# Returns non-zero if the archive needs to be rebuilt
if [ -e $1 ]; then
folder_name=$(basename $1 .npk)
[ -z "$(find $folder_name -newer $1 -not -type d)" ] && return 0
fi
return 1
I don't really like that solution because it works around the problem instead of solving it directly, but it may be able to get you going in the meantime. If you are going to go that route, it's probably cleaner and easier to do everything in the shell script and either have the makefile simply invoke the script or get rid of the makefile entirely.
This is the solution I found:
it is based on the makedepend idea, with some "meta" scripting. Not very nice, but works.
PACKAGES :=
all: packages
-include Makefile.depend
packages: Makefile.depend $(PACKAGES)
depend: clean Makefile.depend
Makefile.depend:
#(PACKAGES= ; \
for DIR in `find . -mindepth 2 -maxdepth 2 -not -name mp3 -not -name ".*" -type d` ; \
do \
PACKAGE=`basename $${DIR}.npk` ; \
PACKAGES="$${PACKAGES} $${PACKAGE}" ; \
DEPS=`find $${DIR} -not -type d | sed -e 's#\([: ]\)#\\\\\1#' -e 's#^\./\(.*\)# \1#' | tr -d "\n"` ; \
SUBDIR=`echo $${DIR} | sed -e 's#^\./\([^/]\+\)/.*#\1#'` ; \
FILES=`echo \ $${DEPS} | sed -e "s# $${SUBDIR}/# #g"` ; \
echo "$${PACKAGE}:$${DEPS}" ; \
echo " #cd $${SUBDIR} ; \\" ; \
echo " npack c ../\$$# $${FILES} ; \\" ; \
echo ; \
done ; \
echo "PACKAGES = $${PACKAGES}" \
)>> Makefile.depend ; \
cleanall: clean
rm -f *.npk
clean:
#rm -f Makefile.depend
.PHONY: all packages depend clean
With makepp you can do this in 2 steps, via the :foreach rule modifier:
$(foreach).txt: $(foreach)/*: foreach */
&echo $(inputs) -o $(output)
This provides a rule for every subdirectory, which reexecutes whenever there is a change in the list of files therein.