I regularly (multiple times a day) build a project with make (generated with CMake, the compiler I use is clang 5.0).
Is there a way to (semi-)automatically count the number of targets, that make tried to build and how many of them were built successfully, apart from writing a script, that parses the output of the compiler?
Related
I am compiling my code using the MinGW toolchain. I have hundreds of files in my projects, that cannot be changed, so hundreds of object files are produced. During linking the g++.exe command failes because the argument list is too long.
Is there a way to fix this issue under Windows?
Given that it would be interesting to see how the tree of directories of the files is organized, you could divide the project into several libraries and compile them separately.
Later you could statically (or dynamically) to link the various libraries to compile the overall project.
What is purpose of no-codegen option when building crystal project ?
I have quite large codebase and when building without this option, it can take up to 20 seconds to build. When I use no-codegen option, it reduces build time to 6 seconds. But I don't understand can executable be debugged when that option is included (currently I debug using LLDB when building with --debug flag)?
--no-codegen means no code is generated. It does not produce any binary.
The purpose of this flag is to check the validity of a program, without actually building it. This is useful when you can't execute the result anyway, then the binary generation can be skipped. For example, this is used for the code examples in the Crystal repo: They are built in CI with --no-codegen just to check that the code is valid, but verifying their execution is out of scope for automated tests.
In short, it allows to check things are alright, but without generating a binary.
In long, there are four main phases within the Crystal compiler:
Typechecking and macro expansion
Code generation to LLVM
Optimization (by LLVM)
Binary generation (by LLVM)
Each process takes some time. In the first phase, the compiler just checks all the types and expand the macros in your program. In the second phase, the dead code is thrown away, some other language-specific optimizations take place, and the program is generated as LLVM. In the fourth phase (only with --release), Crystal calls LLVM to peform many optimization of the generated code. In the last phase LLVM creates the platform-specific binary.
In your case, it's taking 6s to do typechecking, and ~14s to do all the other phases (but 3).
I'm consulting a company on how to speed up their builds and I immediately pointed them to precompiled headers and unity builds - the 10 minute full build could easily drop to 2-3 minutes. Luckily CMake 3.16 was recently released and it supports both of those, so I told them to upgrade.
The problem is the following: once they switched from CMake 2.6 to 3.16 the time it took to run CMake jumped from about 20 seconds to more than 10 minutes. Most of the time is spent in the generate phase. It does complete successfully if you gave it enough time and the code compiled successfully with unity builds, but this CMake time is unacceptable.
Here is their setup:
CMake 2.6, old style CMake with global flags/defines/includes - not modern (target-based). Nothing too fancy - no custom commands or build rules & complicated dependencies.
the compiler used is GCC 7 and they generate Makefiles - the OS is CentOS 7, Kernel: Linux 3.10.0-862.14.4.el7.x86_64
around 2000 .cpp files spread across 100 libraries and 600 executables (most of which are test executables with a single .cpp in them)
most .cpp files are gathered/globbed with aux_source_directory - we know not explicitly listing the .cpp files is an anti-pattern, but that's besides the point - I think this is irrelevant since this should happen in the configuration step and not during the generation, correct?
Here is what we observed:
we did a binary search through the different CMake versions and concluded that the huge slowdown happened between 3.15 and 3.16 - exactly when the precompiled header and unity build support was added, but I don't think those features have anything to do with the slowdown - I can't think of a reason for them to have such an impact - it must be some other refactoring or change...
we disabled all tests (that means almost all 600 executables were removed) - slimming down the number of CMake targets from 700 to a bit more than 100 - and the time it took to run CMake dropped significantly, but was still a couple of times longer than with CMake 2.6 for all the 700 targets.
we observed what was happening using strace and there were mostly lstat and access calls along with some reads and writes - but this was endless - seemed like hundreds of operations per second in a very repeatable manner. Also there was constantly an attempt to find libgflags.so which was constantly failing. Unfortunately I don't have such logs right now.
we did a callgrind profile and here is what it looks like after a couple of minutes of running: https://i.imgur.com/Z9AObso.png (the callgrind output file can be found here) - seems like most of the time is spent in ComputeLinkLibs() and getting the full name and definitions of targets and whatnot... Is having 700 targets too much? Is this an exponential problem? Why isn't it an issue with CMake 3.15?
I couldn't find any reports of anyone else having the same problem on the internet... Any ideas what to try next? Perhaps profile using Perf? Try with Ninja as a backend (report of that being faster)?
Great analysis. This will be helpful for the developers of CMake. But I doubt you will find much help here. Please open an issue.
Bonus points if you can provide a minimal example showcasing your problem. You might get this with generating tons of files.
How can I ensure that object files compiled for one hardware target will not be used for a different hardware target that needs to be compiled differently?
I am using the GNU ARM Embedded Toolchain while I am learning about embedded development. I already have a couple of development boards (with STM32F0 and STM32F4 processors), and plan to make my own boards in the future. I want to have several iterations of hardware using a common software repository.
Obviously I will have multiple targets in my Makefile, invoking the appropriate defines and compiler flags for each platform, and perhaps a make all to build for all platforms at once. As I understand it, make is an incremental build system that only re-compiles object code (*.o) files if the source file has been changed, it won't recompile if I have use different defines and options, and the wrong object code will be passed to the linker.
It seems that I could diligently make clean when switching between different targets, but that would rely on the human action and could produce bad builds if I forgot, and could not be used for a make all that produces multiple binaries for their respective hardware.
Edit Notes: Per feedback comments, I have shorted and rearranged to make the question more clear and objective. I'm not asking generically how to use Make, but rather how to prevent, say mylib.o being compiled for an STM32F0 and then later being re-used in a build for an STM32F4.
I am curious about alternative tools, and welcome discussion in the comments, but this question is specific to GNU Make.
To avoid the need for a clean build between targets, it is necessary for each target to have separate build directories in order that the target dependencies are independent and specifically generated using the appropriate tool chain and build switches etc.
I don't understand, why do we need cmake to build libraries ? I am sorry if my question is stupid, but i need to use some libraries on Widnows, and what ever library i choose i need to build it and/or compile it with cmake.. What is it for ? Why cant i just #include "path" the things that i need into my project, and than it can be compiled/built at the same time as my project ?
And also, sometimes i needed to install Ruby, Perl, Python all of them some specific version so cmake can build libraries... Why do i need those programs, and will i need them only to build library or later in my project too ? (concrete can i uninstall those programs after building libraries ?)
Building things in c++ on different platforms is a mess currently.
There are several different build system out there and there is no standard way to do this. Just providing a visual studio solution wont help compilation on linux or mac.
If you add a makefile for linux or mac you need to repeat configurations between the solution and the makefiles. Which can result in a lot of maintenance overhead. Also makefiles are not really a good build tool compared to the new ones out there.
That you have only CMake libraries is mostly a coincidence. CMake is though a popular choice currently.
There are several solutions out there to unify builds. CMake is a build tool in a special way. It can create makefiles and build them but you can also tell cmake to create a visual studio solution if you like.
The same goes with external programs. They are the choice of the maintainer of the library you use and there are no standards for things like code generation.
While CMake may not be "the" solution (although the upcoming visual studio 2015 is integrating cmake support) but the trend for those build system which are cross-platform is going more and more in this direction.
To your question why you cannot only include the header:
Few libraries are header only and need to be compiled. Either you can get precompiled libs/dlls and just include the header + add the linker path. This is easier in linux because you can have -dev packages which just install a prebuild library and it's header via the package manager. Windows has no such thing natively.
Or you have to build it yourself with whatever buildtool the library uses.
The short answer is that you don't, but it would probably be difficult to build the project without it.
CMake does not build code, but is instead a build file generator. It was developed by KitWare (during the ITK project around 2000) to make building code across multiple platforms "simpler". It's not an easy language to use (which Kitware openly admits), but it unifies several things that Windows, Mac, and Linux do differently when building code.
On Linux, autoconf is typically used to make build files, which are then compiled by gcc/g++ (and/or clang)
On Windows, you would typically use the Visual Studio IDE and create what they call a "Solution" that is then compiled by msvc (the Microsoft Visual C++ compiler)
On Mac, I admit I am not familiar with the compiler used, but I believe it is something to do with XCode
CMake lets you write a single script you can use to build on multiple machines and specify different options for each.
Like C++, CMake has been divided between traditional/old-style CMake (version < 3.x) and modern CMake (version >= 3.0). Use modern CMake. The following are excellent tutorials:
Effective CMake, by Daniel Pfeifer, C++Now 2017*
Modern CMake Patterns, by Matheiu Ropert, CppCon 2017
Better CMake
CMake Tutorial
*Awarded the most useful talk at the C++Now 2017 Conference
Watch these in the order listed. You will learn what Modern CMake looks like (and old-style CMake) and gain understanding of how
CMake helps you specify build order and dependencies, and
Modern CMake helps prevent creating cyclic dependencies and common bugs while scaling to larger projects.
Additionally, the last video introduces package managers for C++ (useful when using external libraries, like Boost, where you would use the CMake find_package() command), of which the two most common are:
vcpkg, and
Conan
In general,
Think of targets as objects
a. There are two kinds, executables and libraries, which are "constructed" with
add_executable(myexe ...) # Creates an executable target "myexe"
add_library(mylib ...) # Creates a library target "mylib"
Each target has properties, which are variables for the target. However, they are specified with underscores, not dots, and (often) use capital letters
myexe_FOO_PROPERTY # Foo property for myexe target
Functions in CMake can also set some properties on target "objects" (under the hood) when run
target_compile_definitions()/features()/options()
target_sources()
target_include_directories()
target_link_libraries()
CMake is a command language, similar shell scripting, but there's no nesting or piping of commands. Instead
a. Each command (function) is on its own line and does one thing
b. The argument(s) to all commands (functions) are strings
c. Unless the name of a target is explicitly passed to the function, the command applies to the target that was last created
add_executable(myexe ...) # Create exe target
target_compile_definitions(...) # Applies to "myexe"
target_include_directories(...) # Applies to "myexe"
# ...etc.
add_library(mylib ...) # Create lib target
target_sources(...) # Applies to "mylib"
# ...etc.
d. Commands are executed in order, top-to-bottom, (NOTE: if a target needs another target, you must create the target first)
The scope of execution is the currently active CMakeLists.txt file. Additional files can be run (added to the scope) using the add_subdirectory() command
a. This operates much like the shell exec command; the current CMake environment (targets and properties, except PRIVATE properties) are "copied" over into a new scope ("shell"), where additional work is done.
b. However, the "environment" is not the shell environment (CMake target properties are not passed to the shell as environment variables like $PATH). Instead, the CMake language maintains all targets and properties in the top-level global scope CACHE
PRIVATE properties get used by the current module. INTERFACE properties get passed to subdirectory modules. PUBLIC is for the current module and submodules (the property is appropriate for the current module and applies to/should be used by modules that link against it).
target_link_libraries is for direct module dependencies, but it also resolves all transitive dependencies. This means when you link to a library, you gets all the PUBLIC properties of the parent modules as well.
a. If you want to link to a library that has a direct path, you can use target_link_libraries, and
b. if you want to link to a module with a project and take its interface, you also use target_link_libraries
You run CMake on CMakeLists.txt files to generate the build files you want for your system (ninja, Visual Studio solution, Linux make, etc.) and the run those to compile and link the code.