Consider two software projects, proj_a and proj_b, with the latter depending on the former; and with both using CMake.
When reading about modern CMake, one gets the message that the "appropriate" way to express dependencies is via target dependencies; and one should arrange it so that dependent projects are represented as (imported) targets you can depend on. More specifically, in our example, proj_b will idiomatically have:
find_package(proj_a)
# etc etc.
target_link_library(bar proj_a::foo)
and proj_a will need to have been installed, utilizing the CMake installation-and-export-related commands, someplace where proj_b's CMake invocation will search for proj_a-config.cmake.
I like this approach and encourage others to adapt to it. It offers flexibility in the choice of your own version of proj_a vs the system version; and also allows for non-CMake proj_a's via a Findproj_a.cmake script (which again, can be system-level or part of proj_b).
So far so good, right? However, there are people who want to "take matters into their own hands" in terms of dependencies - and CMake officially condones this, with commands such as ExternalProject and more recently, FetchContent: This allows proj_b's configuration stage to actually download a (built, or in our case source-form) version of proj_a.
The puzzling part to me is that, after proj_a is downloaded, say to an external/proj_a directory, CMake's default behavior will be to
add_subdirectory(external/proj_a)
that is, to use proj_a as a subproject of proj_b and build them together. This, while the idiomatic use above allows the maintainer of proj_a to "do their own thing" in my CMakeFile, and only keep things neat and tidy for others via what I export/install.
My questions:
Why does it make sense to add_subdirectory(), rather than to build, install, and perform the equivalent of find_package() to meet the dependency? Or rather, why should the former, rather than the latter, be the default?
Should I really have to write my project-level CMakeLists.txt to be compatible with being add_subdirectory()'ed?
Note: Just to give some concrete examples of how this use constrains proj_a:
Must use unique option names which can't possibly clash with super-project names. So no more WITH_TESTS, BUILD_STATIC_LIB - it has to be: WITH_PROJ_A_TESTS and BUILD_PROJ_A_STATIC_LIB.
You have to account for the parent project having searched for other dependencies already, and perhaps differently than how you would like to search for them.
Following the discussion in comments, I decided to post a bug report about this:
#22904: Support FetchContent_MakeAvailable performing build+install+find_package rather than add_subdirectory
So maybe this will change and the question becomes moot.
Why does it make sense to add_subdirectory(), rather than to build, install, and perform the equivalent of find_package() to meet the dependency? Or rather, why should the former, rather than the latter, be the default?
FetchContent doesn't just have to be for project() dependencies. It can be used for fetching utility scripts too. I'm guessing it was designed with that kind of consideration in mind. If your utility script is just one file, you can just file(DOWNLOAD) and add_subdirectory() directly, but the utilities could be multiple files, such as is the case with aminaya/project_options. FetchContent() uses a lot of the same machinery as ExternalProject, so it can do a lot of the useful things that ExternalProject does. For example, you can use FetchContent to fetch aminaya/project_options as a remote git repo, or as its archive artifacts- ex. v0.20.0.zip
Should I really have to write my project-level CMakeLists.txt to be compatible with being add_subdirectory()'ed?
It's your choice! The reasoning here can be highly objective, or subjective. It's up to you. Some people just like to put in a lot of effort to support whatever their users might want. Some people have a lot of historical configuration baggage and are still catching up to newer CMake. And as you mentioned at the end of your question post, there are certain adjustments that need to be made to accomodate for cleanly allowing people to add_subdirectory() you as a dependency. One example of a project which chose "no" is glew (see issue #314 for explanation).
Just to give another reference to some related work mentioned in responses to the KitWare/CMake ticket your raised, here's the ticket which tracked work on "FetchContent and find_package() integration".
Related
I happen to think (but maybe is a myth) that Cmake is greater than Autotools about making easy supporting Microsoft.
At the same time, I'm kind of sure that Autotools is even more straightforward than Cmake when it comes to important UNIX derivatives such as macOS and most popular Linux distros.
What if I can't choose?
Can a project support both Autotools and Cmake at the same time?
Bonus for: can a project support both Autotools and Cmake and even simply bare Make at the same time?
By "at the same time" I mean that ideally one should not necessarily run a clean script when changing from trying one of the build systems to another. But I guess it would be a reasonable configuration, if necessary.
Finally, do you know an example project that uses both Autotools and Cmake? One that uses both Autotools, Cmake and simply bare Make?
Yes, you can very easily support both CMake and Autotools at the same time, since they don't overlap (that is, the files you use to create those environments are different, so you can have both types of files in your project at the same time). One example of this is the GNU uCommon C++ framework.
No, you can't (easily) support bare make and either of the above systems at the same time. Neither Autotools nor CMake are actually build tools themselves. They're "build tool generators". So you don't run autotools or cmake and the result is your built project: instead you run autotools or cmake and they generate control files for a build tool. Then you run the build tool and the result is your built project.
Autotools generates makefiles, and cmake generates many different types of control files, where makefiles are one of the most common.
So, you can't have your OWN makefile in your project, because they'll conflict with the makefile generated by autotools or cmake.
Of course, you can do things like put your own makefiles in a subdirectory then invoke make with an argument like make -f rawmake/makefile or something like that. But there's no convenient way to support them all.
Realistically, I would never choose to support more than one of the above options. You will spend a lot of time getting it right, and every time you need to change your build environment it's two or three times as much work. People will find issues with whichever one of them you tend to use less often. It's a huge hassle for not that much benefit.
Which you choose depends a lot on your project. If your project runs only (or almost exclusively) on POSIX-type systems, you want it to be maximally portable even to much older systems even though it uses a lot of special OS features, or you want its installation and build options to be extremely flexible (straightforward support for cross-compilation, etc.) then autotools is a good choice. If your project runs on lots of different OS types (Windows in particular) and you want people to be able to develop with their choice of IDE (Visual Studio, Xcode, etc.) easily, then cmake is a good choice.
If your program is straightforward to build and needs hardly any configuration or customization, or you are already familiar with makefiles and don't feel like learning a whole new language just for builds, then raw makefiles may be a good choice.
I have a large project written mostly in FORTRAN90 consisiting of a core and numerous add-on modules also written in FORTRAN90. At any given time I'd like to be able to:
package the core module together with any number of the modules
create a new config-file merging the core-config and module-configs
merge the various latex-files from the core and modules
The code+configs+documentation lives in SVN...
Can/shall MAVEN be used for this use-case?
******* UPDATE *******
#haraldkl:
Ok, I'll try to elaborate as it definitely is in my interest to gather as much info as possible on this - I really appreciate the comments I get!
My project contains a core module which is mandatory. In order to add additional functionality you may select an arbitrary number of add-on modules. The core and each module resides in their own directory and is under SVN-control. For a given delivery I would like to be able to select the "core" and an arbitrary number of modules and calculate the dependency chain in order to build the modules in the correct order as they sometimes, unfortunately, might have cross-dependencies. When the build order has been set I need to be able to merge property-files from the selected modules with the property-file for the "core" so I end-up with an assembled/aggregated property-file with the aggregated properties from the "core" and all the selected modules. The same goes for the latex-files: I'd like to get an assembled document based on the "core" + the selected modules latex-files, thus ending up with one latex-file.
So, the bottom line: a solution something like:
tick selected modules to go with the delivery (core is mandatory so no need to tick)
click "Assemble" (code is gathered from SVN)
The solution calculates correct build order
The solution merges property-files -> "package.property"
The solution merges latex-files -> "document.latex"
Currently we use make under UNIX but I'm a little uncertain as to what extent make is able to handle 4 and 5 above.
DONE!
Here is my take on it:
I believe steps 1 to 3 are completely achievable with commonly used configuration tools. Also steps 4 and 5 present, as far as I can see, just another build task, there is no reason why Make should not be able to do that. I regularly generate files for LaTeX processing via Make and then assemble them with latexmk mostly. The main point is how to select what to merge and how it has to be merged, you are a little bit unclear on the how, the what should be handled by the configuring system. Your choice probably also depends on what should be done at the end of step 3. Should the code actually be compiled, or do you need to have some written out version of the dependencies?
Traditional configure system on Unix is the autotools suite. However, as far as I know, it does not support the identification of Fortran dependencies out of the box, and you would need to enhance it in that direction.
A popular replacement for the autotools is CMake, which does include Fortran dependency resolution. It might best suite your needs as pointed out by casey, as it allows you to create various generators, so for example you could have it generating an appropriate Makefile for your selection of files.
Waf gives you great deal of flexibility to handle steps 4 and 5 in your list, it is also capable to identify Fortran dependencies, but I think, it is not as straight forward to generate for example Makefiles out of it as in CMake. The flexibility here is due to the fact, that your waf scripts are just ordinary Python scripts, so you could easily utilize any Python tools in your workflow and describe steps 4 and 5 in any complicated manner you desire.
Maven can compile Fortran code, though I do not have any experience with it, I would doubt that it also gives you automatic Fortran dependency resolution. In my understanding, it is not quite as well fit for your setup as the other tools.
The Fortranwiki has some more tools, for example you could come up with your own environment building Makefiles and use makedepf90 to generate the dependencies.
In Go, if you reference another package, e.g. something on GitHub, then Go always gets the latest version from the master branch. While this is great for development, I guess it's a problem in production: This way a build is not reproducible.
So, what is the correct way in Go to fix a version of a dependency, and how to handle this efficiently?
A friend pointed me to godep, and this seems fine, but I wanted to know what alternatives are there, and what's good / bad about godep?
Update 2018 with Go 1.11
Dependencies should now be referenced with modules (derived from the vgo project):
Go 1.11 adds preliminary support for a new concept called “modules,” an alternative to GOPATH with integrated support for versioning and package distribution.
Using modules, developers are no longer confined to working inside GOPATH, version dependency information is explicit yet lightweight, and builds are more reliable and reproducible.
See Defining a module. (and the original design proposal)
Update June 2015: first support for vendoring is making its way in Go 1.5!
See c/10923/:
When GO15VENDOREXPERIMENT=1 is in the environment, this CL changes the resolution of import paths according to the Go 1.5 vendor proposal:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Update March 2015: the go team is thinking about defining a go dependency management system integrated to the language: the debate is in this thread.
We think it’s time to start addressing the dependency & vendoring issue, especially before too many conflicting tools arise and fragment best practices in the Go ecosystem, unnecessarily complicating tooling. It would be nice if the community could converge on a standard way to vendor.
Our proposal is that the Go project,
officially recommends vendoring into an “internal” directory with import rewriting (not GOPATH modifications) as the canonical way to pin dependencies.
defines a common config file format for dependencies & vendoring
makes no code changes to cmd/go in Go 1.5. External tools such as “godep” or “nut” will implement 1) and 2). We can reevaluate including such a tool in Go 1.6+.
One possible downside of godep is that you can no longer use "go build" or "go test" directly.
You need to precede those commands with godep (or type godep save).
An alternative is glide, which remains compatible with classic go commands.
Manage project-specific GOPATHs
Ease dependency management
Support versioning packages
Support aliasing packages (e.g. for working with github forks)
Remove the need for "vendoring" or munging import statements
Work with all of the go tools
More generally, the article "Know your guarantees, Go edition" is interesting:
It’s also a deliberate choice, where the Go authors chose not to implement a feature when they felt that the trade-offs were no good.
One low-level reason they made this choice is to avoid slow compilation and bloated binaries (which are two sides of the same coin).
Remember, packages depend on other packages. So Foo might depend on Bar 2.1. Foo might also depend on Baz which in turn depends on Bar 1.9, and on down the tree. So that would mean compiling and linking several copies of nearly identical code.
Depending on several versions of the same package also means knowing which version one is calling, whereby the dependency mess seeps into your source code.
Which leads us to the high-level reasoning behind the Go platform punting on this feature: they did not have a logical solution they considered acceptable. It’s not that they don’t understand the problem; it’s that, at the moment, there is not a solution they like. So they choose no feature over over a regressive one.
You handle dependencies like you do handle dependencies in other languages too: You vendor. For Go there is no Nexus which does the vendoring so most just copy external libraries into a "vendor" folder, there are tools helping here. Personally I found all this "fix version" panic a bit exaggerated as it works pretty well without.
You might wanna take a look at http://labix.org/gopkg.in and search golang-nuts for dependency management. I think there is even a whole mailing list devoted to this.
What are the things I need in my install and uninstall targets in a Makefile for an OCaml library in order to make it play nicely with the rest of the installation, work seamlessly with ocamlfind and so on? Basically to be a "good citizen". I am not interested in GODI at the present time. Thanks!
META files for ocamlfind are easy to write (basically, look for a META in another ocaml project you know¹, copy it and make the corresponding changes), and they will give you ocamlfind integration, with in particular easy support for post-build installation and desinstallation (using ocamlfind install and ocamlfind remove). You should begin with that.
¹: for example I take inspiration from batteries's META.
The building part of the Makefile is more tricky, their are numerous solutions (OCamlMakefile, OMake, ocamlbuild, plain Makefile, etc.) with varying strenghts and weaknesses. If you project is simple enough I would recommend ocamlbuild that takes care of a lot of the dependency tracking by itself.
You may also use Oasis, which is a relatively new tools that builds on ocamlbuild and ocamlfind and seeks to provide a unified configuration file for pre-build configuration and various building and deployment (of your program, your software libraries if any, accompanying data or documentation...). It's not yet a mature project (and its little brother Oasis-DB isn't released yet), but I encourage you to give it a try if you have time. It's a bit more complex than META, as it does more in the end, so building the META first is still a good step.
Finally, you said you weren't interested in Godi (Godi is a very good system, and in some cases (eg. BSD etc.) it's a premium choice to have a good OCaml installation), but in case you may still be interested in Godiva, a tool to help the building of GODI packages. I have never used it myself, though.
I don't use makefiles but ocamlbuild and a shell script to install the software I distribute. Debian people did packages for my software with these scripts without problems. So you may want to check them out since they correspond to some of their requirements (e.g. separate targets for byte and native code).
You may also want to have a look
to their packaging policy, though I don't know if this document is still up to date.
Don't forget to add a META file for ocamlfind. And you may also want to include an _oasis file for the upcoming oasis-db project (not yet done in the software I distribute).
I've been looking for ways to learn about the right way to manage a software project, and I've stumbled upon the following blog post. I've learned some of the things mentioned the hard way, others make sense, and yet others are still unclear to me.
To sum up, the author lists a bunch of features of a project and how much those features contribute to a project's 'suckiness' for a lack of a better term. You can find the full article here: http://spot.livejournal.com/308370.html
In particular, I don't understand the author's stance on bundling dependencies with your project. These are:
== Bundling ==
Your source only comes with other code projects that it depends on [ +20 points of FAIL ]
Why is this a problem, especially given point 3, that you have modified your projects dependencies to fit your project's needs, doesn't it therefore make even greater sense that your code should be distributed with its dependencies?
If your source code cannot be built without first building the bundled code bits [ +10 points of FAIL ]
Doesn't this necessarily have to be the case for software built against 3rd party libs? Your code needs that other code to be compiled into its library before the linker can work?
If you have modified those other bundled code bits [ +40 points of FAIL ]
If this is necessary for your project, then it naturally follows that you've bundled said code with yours. If you want to customize a build of some lib,say WxWidgets, you'll have to edit that projects build scripts to bulid the library that you want. Subsequently, you'll have to publish those changes to people who wish to build your code, so why not use a high level make script with the params already written in, and distribute that? Furthermore, (especially in a windows env) if your code base is dependent on a particular version of a lib (that you also need to custom compile for your project) wouldn't it be easier to give the user the code yourself (because in this case, it is unlikely that the user will already have the correct version installed)?
So how would you respond to these comments, and what points may I be failing to take into consideration? Would you agree or disagree with the author's take (or mine), and why?
Edited for clarification.
Your source only comes with other code projects that it depends on.
My project requires project X.
However, since my project depends on secret inner mysteries of X, or a previous release of X, then my project includes a copy of X. Specifically release n.m of X. And no other.
Try to install the latest and greatest X and see what breaks in my project. Since upgrading X broke my project, forget my project. They will not struggle with something that spontaneously breaks after an update. They will find a better open source component.
Hence a FAIL score.
If your source code cannot be built without first building the bundled code bits.
My project doesn't rely on the API to X. It relies on deep, inner linking to specific parts of X, bypassing the API.
If my project on depended on the API to X, then -- for some languages like C or C++ -- my project could compile with only the C or C++ headers, not the binaries.
For Java this is less true, since there is not independent, non-binary header. And for dynamic languages (like Python) this makes no technical sense.
However, even Java and Python have ways to separate interface from implementation. If I rely on implementation (not interface), then I've still created the same essential problem.
If my project depends on C or C++ binaries, and they build things out of order, or upgrade another component without rebuilding mine, things may go badly for them. They may see weirdness, breakage, "instability". My product appears broken. They won't (and can't) debug it. They're done. They'll find something more stable.
Hence a FAIL score.
If you have modified those other bundled code bits.
I have two choices when I modify X.
Get it accepted as part of X.
Fix my program to work with unmodified X.
If my project depends on a modified X, no one can install X simply, correctly and independently. They can't upgrade X, they can't maintain X. They probably can't apply bug fixes or security patches to X.
I've essentially made their job impossible by modifying X.
Hence the FAIL score.
Subsequently, you'll have to publish those changes to people who wish to build your code.
Actually, they'll hate me for that. They don't want to know about the mysterious changes to X. They want to build X according to the rules, then build my stuff according to the rules. They don't want to read, think or be sure that the mystery update patch kit was applied correctly.
Rather than joke around with that, they'll download a competing package. FAIL.
if your code base is dependent on a particular version of a lib (that you also need to custom compile for your project)
That's really shabby. If I depend on a version with custom compiles, they're done looking at my package. They'll find something without version-specific inner mysteries and custom compiles before they'll struggle. FAIL.