I'd like to use make to get a modular build in combination with continuous integration, automatic unit testing and multi-platform builds. Similar setups are common in Java and .NET, but I'm having a hard time putting this together for make and C/C++. How can it be achieved?
My requirements:
fast build; non-recursive make (Stack Overflow question What is your experience with non-recursive make?)
modular system (that is, minimal dependencies, makefile in subdirectory with components)
multiplatform (typically PC for unit testing, embedded target for system integration/release)
complete dependency checking
ability to perform (automatic) unit tests (Agile engineering)
hook into continuous integration system
easy to use
I've started with non-rec make. I still find it a great place to start.
Limitations so far:
no integration of unit tests
incompatibility of windows based ARM compilers with Cygwin paths
incompatibility of makefile with Windows \ paths
forward dependencies
My structure looks like:
project_root
/algorithm
/src
/algo1.c
/algo2.c
/unit_test
/algo1_test.c
/algo2_test.c
/out
algo1_test.exe
algo1_test.xml
algo2_test.exe
algo2_test.xml
headers.h
/embunit
/harnass
makefile
Rules.top
I'd like to keep things simple; here the unit tests (algo1_test.exe) depend on both the 'algorithm' component (ok) and the unit test framework (which may or may not be known at the time of building this). However, moving the build rules to the top make does not appeal to me as this would distribute local knowledge of components throughout the system.
As for the Cygwin paths: I'm working on making the build using relative paths. This resolves the /cygdrive/c issue (as compilers can generally handle / paths) without bringing in C: (which make dislikes). Any other ideas?
CMake together with the related tools CTest and CDash seem to answer your requirements. Worth giving it a look.
Bill Hoffman (A lead CMake developer) refers to the Recursive Make Considered Harmful paper in a post at the CMake mailing list:
... since cmake is creating the makefiles for you, many of the disadvantages
of recursive make are avoided, for example you should not have to debug
the makefiles or even think about how they work. There are other examples
of things in that paper that cmake fixes for you as well.
See also this answer for "Recursive Make - friend or foe?" here on stackoverflow.
-
Recursive Make - friend or foe?
Ok here is what I do:
I use one Makefile at the root and wildcard patterns to collect all files in a directory. Note that I assume that foo/*.c will make up foo.so for example. This makes the maintaining the Makefile minimal, since just adding a file to the directory automatically adds it to the build.
Since it is make you are using I am assuming (I do that for my projects) that a compiler is used that uses gcc (cc) compatible command line syntax. So MSC is out of order; but don't get frustrated, I do most of my development (unfortunately) on Windows and use MinGW with MSys; works like a charm. Produces native binaries, but was built with a Posix compliant build environment.
Dependency checking is done with the somewhat standard -MD switch. I then include all the *.d files into the Makefile. I build the patterns out of the automatically collected source files.
Finally unit tests are implemented with the "standard" check target. The check target is like the all target, except it depends on the unit test and executes that once everything is built. I do it this way so that you can just build the project or build the unit tests (and the rest of the project) separably. When I am not developing the project I want to just build it and be done with it.
Here is an example of how I do it: https://github.com/rioki/c9y/blob/master/Makefile
It also has the install, uninstall and dist targets.
As you can see everything is plain make, no recursive make calls and all is relatively simple. I used automake and autoconf and I will never do that again; also other build tools are out of the question, if I need to install foojam or barmake to build something, I normally ditch that project immediately.
Related
I happen to think (but maybe is a myth) that Cmake is greater than Autotools about making easy supporting Microsoft.
At the same time, I'm kind of sure that Autotools is even more straightforward than Cmake when it comes to important UNIX derivatives such as macOS and most popular Linux distros.
What if I can't choose?
Can a project support both Autotools and Cmake at the same time?
Bonus for: can a project support both Autotools and Cmake and even simply bare Make at the same time?
By "at the same time" I mean that ideally one should not necessarily run a clean script when changing from trying one of the build systems to another. But I guess it would be a reasonable configuration, if necessary.
Finally, do you know an example project that uses both Autotools and Cmake? One that uses both Autotools, Cmake and simply bare Make?
Yes, you can very easily support both CMake and Autotools at the same time, since they don't overlap (that is, the files you use to create those environments are different, so you can have both types of files in your project at the same time). One example of this is the GNU uCommon C++ framework.
No, you can't (easily) support bare make and either of the above systems at the same time. Neither Autotools nor CMake are actually build tools themselves. They're "build tool generators". So you don't run autotools or cmake and the result is your built project: instead you run autotools or cmake and they generate control files for a build tool. Then you run the build tool and the result is your built project.
Autotools generates makefiles, and cmake generates many different types of control files, where makefiles are one of the most common.
So, you can't have your OWN makefile in your project, because they'll conflict with the makefile generated by autotools or cmake.
Of course, you can do things like put your own makefiles in a subdirectory then invoke make with an argument like make -f rawmake/makefile or something like that. But there's no convenient way to support them all.
Realistically, I would never choose to support more than one of the above options. You will spend a lot of time getting it right, and every time you need to change your build environment it's two or three times as much work. People will find issues with whichever one of them you tend to use less often. It's a huge hassle for not that much benefit.
Which you choose depends a lot on your project. If your project runs only (or almost exclusively) on POSIX-type systems, you want it to be maximally portable even to much older systems even though it uses a lot of special OS features, or you want its installation and build options to be extremely flexible (straightforward support for cross-compilation, etc.) then autotools is a good choice. If your project runs on lots of different OS types (Windows in particular) and you want people to be able to develop with their choice of IDE (Visual Studio, Xcode, etc.) easily, then cmake is a good choice.
If your program is straightforward to build and needs hardly any configuration or customization, or you are already familiar with makefiles and don't feel like learning a whole new language just for builds, then raw makefiles may be a good choice.
How can I ensure that object files compiled for one hardware target will not be used for a different hardware target that needs to be compiled differently?
I am using the GNU ARM Embedded Toolchain while I am learning about embedded development. I already have a couple of development boards (with STM32F0 and STM32F4 processors), and plan to make my own boards in the future. I want to have several iterations of hardware using a common software repository.
Obviously I will have multiple targets in my Makefile, invoking the appropriate defines and compiler flags for each platform, and perhaps a make all to build for all platforms at once. As I understand it, make is an incremental build system that only re-compiles object code (*.o) files if the source file has been changed, it won't recompile if I have use different defines and options, and the wrong object code will be passed to the linker.
It seems that I could diligently make clean when switching between different targets, but that would rely on the human action and could produce bad builds if I forgot, and could not be used for a make all that produces multiple binaries for their respective hardware.
Edit Notes: Per feedback comments, I have shorted and rearranged to make the question more clear and objective. I'm not asking generically how to use Make, but rather how to prevent, say mylib.o being compiled for an STM32F0 and then later being re-used in a build for an STM32F4.
I am curious about alternative tools, and welcome discussion in the comments, but this question is specific to GNU Make.
To avoid the need for a clean build between targets, it is necessary for each target to have separate build directories in order that the target dependencies are independent and specifically generated using the appropriate tool chain and build switches etc.
I have a large project using autotools that contains some code that builds into a utility library. The project has quite a few dependencies and I would like to compile a specific subset of that utility library for mobile environments (Android/iOS). I expect a lot of dependencies to be unnecessary for that particular subset of functionality, and compiling the whole project for those architectures/platforms is impossible for technical reasons.
This mini version of the library would actually be useful not just to me but other people, as well. This is why the maintainer of the project suggested introducing a --enable-mini flag for the configure script. After experimenting a little (I have never done anything inside a configure.ac before) I actually got a build working that builds this extra mini library.
Now to the point: Is there a clean way to exclude all the other project executables and libraries from being built? What I want is a ./configure --enable-mini invocation that will result in only the libutilmini.a/libutilmini.la being built. Sure, there are some components that could be disabled via --disable-X options, but obviously the project was not set up in a way that makes all components optional. Apart from the fact that it does not seem necessary to build everything else just to build the mini library, the whole project will not build for, for instance, the iOS platform.
I really would like to avoid adding an if HAVE_MINI [...] to all the Makefile.ams in every subdirectory, especially since the mini library is not useful to most of the other developers, this does not seem like an elegant approach. Are there any recommended ways of achieving these goals?
I am aware I could just create a new project using the sources I need and build those, but as I said the mini library is useful to some other developers, too.
If you want to conditionally compile something with automake using a regular make command with no arguments, you have to use automake conditionals (the if HAVE_MINI thing you refer to); there is no other way. However, what you can do, alternatively, is to create an extra target (say, build_mini) in your toplevel Makefile.am which depends on everything needed to build your libmini. You could then tell people that if they want to build libmini (and nothing else), they don't run make, but they run make build_mini. This would look something like:
(toplevel Makefile.am)
SUBDIRS = foo bar baz
build_mini:
$(MAKE) -C foo libmini-depends
$(MAKE) -C bar libmini.la
or some such (the details would depend on what is needed to build libmini.la).
You would then have bar/Makefile.am look something like this:
if WANT_MINI
lib_LTLIBRARIES += libmini.la
endif
libmini_la_SOURCES = # ...
the only thing that really needs to be inside the conditional is adding the libmini.la to lib_LTLIBRARIES; everything else can be unconditional. So with this method, you should have only one if FOO...endif construct.
I have two projects called A and B that have complete working CMakeLists.txt projects, and each project can be built completely without errors. I would like to have a master project defined in CMake that will build both A and B (and maybe a hundred other things eventually).
My top level CMakeLists.txt project looks like
add_subdirectory(A build-A)
add_subdirectory(B build-B)
and CMake can parse all the files and make can start building just fine.
The problem is that project A is for one architecture (x86_64) and B is for a different architecture (k1om) and when CMake invokes various features like
find_package(Boost ....)
it caches the results of the library paths for the first architecture and reuses them (incorrectly!) for all subsequent architectures. We have Boost compiled for both x86_64 and k1om.
Is there a way to have CMake do what I want to do, by entirely invalidating the cache between the two projects? Something like this would be ok:
add_subdirectory(A build-A)
cmake_invalidate_cache_and_forget_everthing_that_just_happened()
add_subdirectory(B build-B)
cmake_invalidate_cache_and_forget_everthing_that_just_happened()
...
I am fully aware that I can just make a shell script that does this and just runs cmake multiple times in different output directories, but it would be really nice to have a uniform "entry" point for all projects written in CMake.
I'd recommend using a "super-build" setup whereby each subproject is included via ExternalProject_Add rather than add_subdirectory. This gives very clean separation between the subprojects' builds. I think you'd be fighting CMake very hard by trying to tinker with the generated CMakeCache.txt!
However, I've never tried actually doing this where the architecture differs between subprojects. So all I can do is suggest you try it - I think it should work.
(This article may help).
I think using ExternalProject, as Fraser suggests is the best practice for your setup, but I don't think it's going to solve the issue your having. Correct me if I'm wrong, but it sounds like you're using the same build tree for both platforms. Is that correct? If so, I can't see what can be gained from that.
If I'm wrong and you're just trying to prevent certain projects from configuring on certain architectures, then you should look into CMake's architecture blocks, like if(WIN32) ... if (CMAKE_SIZEOF_VOID_P 8) ... there are many other ways to limit code exposure base on compiler, 32 vs 64 Windows, *nix, MAC, etc ...
If I'm still not understanding, then my apologies, perhaps you can attempt a clear explanation. Perhaps all you need it the unset command for your cache variables that are incorrectly set in cache because of a different architecture. See: http://www.cmake.org/cmake/help/v3.0/command/unset.html
If that's the case though, you really should reconsider the design of your project because that approach sounds like an unmaintainable mess. Sorry.
I got it to work with ExternalProject_Add (Thank You Fraser). Here is what it looks like:
cmake_minimum_required(VERSION 2.8.4)
project(Demo1)
include(ExternalProject)
ExternalProject_Add(
A-build
SOURCE_DIR ${CMAKE_SOURCE_DIR}/A
INSTALL_COMMAND ""
)
ExternalProject_Add(
B-build
SOURCE_DIR ${CMAKE_SOURCE_DIR}/B
INSTALL_COMMAND ""
)
I've searched for the major differences between makefile and CMakeLists, but found weak differences such as CMake automates dependency resolution whereas Make is manual.
I'm seeking major differences, what are some pros and cons of me migrating to CMake?
You can compare CMake with Autotools. It makes more sense! If you do this then you can find out the shortcomings of make that form the reason for the creation of Autotools and the obvious advantages of CMake over make.
Autoconf solves an important problem—reliable discovery of system-specific build and runtime information—but this is only one piece of the puzzle for the development of portable software. To this end, the GNU project has developed a suite of integrated utilities to finish the job Autoconf started: the GNU build system, whose most important components are Autoconf, Automake, and Libtool.
Make can't do that. Not out of the box anyway. You can make it do it but it would take a lot of time maintaining it across platforms.
CMake solves the same problem (and more) but has a few advantages over GNU Build System.
The language used to write CMakeLists.txt files is readable and easier to understand.
It doesn't only rely on make to build the project. It supports multiple generators like Visual Studio, Xcode, Eclipse etc.
When comparing CMake with make there are several more advantages of using CMake:
Cross platform discovery of system libraries.
Automatic discovery and configuration of the toolchain.
Easier to compile your files into a shared library in a platform agnostic way, and in general easier to use than make.
Overall CMake is clearly the choice when compared to make but you should be aware of a few things.
CMake does more than just make so it can be more complex. In the long run it pays to learn how to use it but if you have just a small project on only one platform, then maybe make can do a better job.
The documentation of CMake can seem terse at first. There are tutorials out there but there are a lot of aspects to cover and they don't do a really good job at covering it all. So you'll find only introductory stuff mostly. You'll have to figure out the rest from the documentation and real life examples: there are a lot of open source projects using CMake, so you can study them.