I need to pick up in the make file the number of processor used for the paralell compilation.
e.g.
make -j32 .....
I need to pick up the number 32 inside the Makefile.
I know this comes inside the variable MAKEFLAG so I could parse it, but is there some other variable that gives this information directly?
For example:
NUMCPU = 32
already solved in
GNU Make: Check number of parallel jobs
NUMPROC = $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS))
#DevSolar me too would like not mix make and ninja build But I need to do,the project is a big project involving several libraries and several teams so I cannot decide alone the build process.
in order to explain the build process I have a target and some libraries that use for the build system meson/ninja and other libraries that use make.
Now during the official release phase all the library must be recompiled, so first are compiled the ones with the legacy "make " and then the ones with meson and the final the binary/executable that will link al of the previous compiled libraries.
At the moment all is triggered by a make command and the production team wants to use the -j option both for make and ninja.
for this reason I am tring to provide the -j to the libraries and final binary/executable built with ninja.
Related
We have an HPC environment with multiple versions of most packages, causing us to have designed a home-rolled way to install packages in unique locations and use environment modules for programmers/researchers to be able to identify which library versions they are using when they build a program, run a program, or both. Is there a relatively painless way to be able to perform builds in this environment. In my case, we're using OpenBLAS, ARPACK, LAPACK and SuperLU when building armadillo. In my case, I'm shooting for armadillo-0.3.7. It would be real nice if the use of switches as was done in the ./configure and make days would work. But all I've found so far is CMake builds, and it appears to be pretty much non-trivial to do a build.
Oh yeah. And, by the way, there's a need for the output Armadillo library to be static.
Thanks in advance for your help. The initial question may be a little vague, but I can get as specific as you like. I just didn't want to write a novel for the initial question on this issue.
Tools exist nowadays to handle the complexity of the build of these scientific software. I would suggest you look at either Spack or EasyBuild. Such tool will help you to save a lot of time by automatically building all the required dependencies and also generating the modulefiles for your users to use the software built.
The CMakeList.txt file and other CMake-related files can be modified to meet your needs. The flags are defined on line 48+, for instance set(ARMA_USE_LAPACK false)
The variables related to the LAPACK library are then defined in the include(ARMA_FindLAPACK) on line 250. The flag in toggled on on line 347 if lapack is found.
The customized path where LAPACK is located can be specified in the file cmake_aux/ARMA_FindLAPACK.cmake. If your customized path is stored as an environment variable as export PATHLAPACKLIB=/usr/lib/openblas-base, you can use it in the ARMA_FindLAPACK.cmake file by modifying line 11 ( see How to retrieve a user environment variable in CMake (Windows) and FIND_LIBRARY()):
message("Searching Lapack in $ENV{PATHLAPACKLIB}")
FIND_LIBRARY(LAPACK_LIBRARY
NAMES ${LAPACK_NAMES}
HINTS $ENV{PATHLAPACKLIB}
NO_DEFAULT_PATH
)
It is not a beautiful modification of the CMakefile, because it makes it not portable as its outcome depends on an environment variable. But, it you intend to to build and install Armadillo once for all, it works. Remember to delete the CMakeCache.txt file every time you modify a CMakeFile.txt, otherwise it keeps some trace of previous runs of cmake . and it looks as though the modification is consequenceless.
To make the library static, add the keyword static to the command add_library() on line 514 of CMakeFile.txt, as shown in CMake - Creating a static library :
add_library( armadillo STATIC ...)
Running cmake . and then make creates a small armadillo.a file, since most of the source consists in cpp headers.
Finally, the exemple1 is compiled as :
g++ -O2 -std=c++11 example1.cpp -o example1 -larmadillo -L/home/...../softs/armadillo-9.800.3/armadillo-9.800.3 -I/home/...../softs/armadillo-9.800.3/armadillo-9.800.3/include -lopenblas
I'm a student doing research involving extending the TM capabilities of gcc. My goal is to make changes to gcc source, build gcc from the modified source, and, use the new executable the same way I'd use my distro's vanilla gcc.
I built and installed gcc in a different location (not /usr/bin/gcc), specifically because the modified gcc will be unstable, and because our project goal is to compare transactional programs compiled with the two different versions.
Our changes to gcc source impact both /gcc and /libitm. This means we are making a change to libitm.so, one of the shared libraries that get built.
My expectation:
when compiling myprogram.cpp with /usr/bin/g++, the version of libitm.so that will get linked should be the one that came with my distro;
when compiling it with ~/project/install-dir/bin/g++, the version of libitm.so that will get linked should be the one that just got built when I built my modified gcc.
But in reality it seems both native gcc and mine are using the same libitm, /usr/lib/x86_64-linux-gnu/libitm.so.1.
I only have a rough grasp of gcc internals as they apply to our project, but this is my understanding:
Our changes tell one compiler pass to conditionally insert our own "function builtin" instead of one it would normally use, and this is / becomes a "symbol" which needs to link to libitm.
When I use the new gcc to compile my program, that pass detects those conditions and successfully inserts the symbol, but then at runtime my program gives a "relocation error" indicating the symbol is not defined in the file it is searching in: ./test: relocation error: ./test: symbol _ITM_S1RU4, version LIBITM_1.0 not defined in file libitm.so.1 with link time reference
readelf shows me that /usr/lib/x86_64-linux-gnu/libitm.so.1 does not contain our new symbols while ~/project/install-dir/lib64/libitm.so.1 does; if I re-run my program after simply copying the latter libitm over the former (backing it up first, of course), it does not produce the relocation error anymore. But naturally this is not a permanent solution.
So I want the gcc I built to use the shared libs that were built along with it when linking. And I don't want to have to tell it where they are every time - my feeling is that it should know where to look for them since I deliberately built it somewhere else to behave differently.
This sounds like the kind of problem any amateur gcc developer would have when trying to make a dev environment and still be able to use both versions of gcc, but I had difficulty finding similar questions. I am thinking this is a matter of lacking certain config options when I configure gcc before building it. What is the right configuration to do this?
My small understanding of the instructions for building and installing gcc led me to do the following:
cd ~/project/
mkdir objdir
cd objdir
../source-dir/configure --enable-languages=c,c++ --prefix=/home/myusername/project/install-dir
make -j2
make install
I only have those config options because they seemed like the ones closest related to "only building the parts I need" and "not overwriting native gcc", but I could be wrong. After the initial config step I just re-run make -j2 and make install every time I change the code. All these steps do complete without errors, and they produce the ~/project/install-dir/bin/ folder, containing the gcc and g++ which behave as described.
I use ~/project/install-dir/bin/g++ -fgnu-tm -o myprogram myprogram.cpp to compile a transactional program, possibly with other options for programs with threads.
(I am using Xubuntu 16.04.3 (64 bit), within VirtualBox on Windows. The installed /usr/bin/gcc is version 5.4.0. Our source at ~/project/source-dir/ is a modified version of 5.3.0.)
You’re running into build- versus run-time linking differences. When you build with -fgnu-tm, the compiler knows where the library it needs is found, and it tells the linker where to find it; you can see this by adding -v to your g++ command. However when you run the resulting program, the dynamic linker doesn’t know it should look somewhere special for the ITM library, so it uses the default library in /usr/lib/x86_64-linux-gnu.
Things get even more confusing with ITM on Ubuntu because the library is installed system-wide, but the link script is installed in a GCC-private directory. This doesn’t happen with the default GCC build, so your own GCC build doesn’t do this, and you’ll see libitm.so in ~/project/install-dir/lib64.
To fix this at run-time, you need to tell the dynamic linker where to find the right library. You can do this either by setting LD_LIBRARY_PATH (to /home/.../project/install-dir/lib64), or by storing the path in the binary using -Wl,-rpath=/home/.../project/install-dir/lib64 when you build it.
1. cmake is a command from CMake software: preparation for build automation system; make and make install are commands from Make software: build automation system.
2. From reading this post, what I understand is that:
a. This "cmake and make" stuffs actually use g++ / gcc in its implementation. cmake and make stuffs are basically just tools in using g++ / gcc. Is that correct?
b. gcc / g++ are the compiler that do the actual work.
c. So I can just use gcc / g++ directly without using the make and CMake things?
3. According to this stackoverflow answer: CMake takes a CMakeList.txt file, and outputs it to a platform-specific build format, e.g., a Makefile, Visual Studio, etc.
However when I came across this openCV installation :
mkdir release
cd release
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
It executes cmake command in a directory where there is no CMakeLists.txt file. Can you explain and elaborate on this?
4. The usual steps that I've seen are: cmake, make, sudo make install.
I read this stackoverflow post, what I understand:
(i) make is for building the project.
(ii) make install is to copy the binary / executables to the installed directories.
a. So when we make, where are the result / binary files / executables stored at?
b. If we only run make without make install, does it mean that the files are not generated?
c. I came across this openCV tutorial on using openCV with GCC and CMake. It uses:
cd <DisplayImage_directory>
cmake .
make
Why doesn't it do make install as well?
5. In summary:
CMake takes CMakeList.txt file (which is cross platform) to generate a Makefile (which is specific to a platform).
I can just write Makefile manually and skip the CMake step. but it is better to do with the CMake step because it is cross platform, otherwise I have to rewrite the Makefile again if I change platform.
Make takes Makefile (which is generated by CMake or written manually) as a guide to compile and build. Make basically uses gcc / g++ or other compiler in its work. Make itself is just a tool for the compiler.
Make install put the result / executables into the install path
CMake generates files for other build systems. These can be Makefiles, Ninja files or projects files for IDEs like Visual Studio or Eclipse. The build files contain calls to compilers like GCC, Clang, or cl.exe. If you have several compilers installed, you can choose one.
All three parts are independent. The compiler, the build system and CMake.
It is easier to understand when you have the history. People used their compiler. Over time they added so many flags, that it was cumbersome to type them every time. So they put the calls in a script. From that the build systems (Make, Ninja) evolved.
The people wanted to support multiple platforms, compilers, scenarios and so on and the build system files became hard to maintain and their use was error-prone. That's the reason people invented meta build system that creates the files for the actual build system. Examples are Autotools or CMake.
Yes
CMake does not use your compiler, make does not implement it, but it calls (uses) the compiler.
The CMakeLists.txt file should be in the parent directory of release. The last argument of the CMake call indicates the path where the CMakeLists.txt file is located.
Right, make generates the file in the build directory. In your example from 3. release is the build directory. You can find all the generated files and use them. Installing is optional, especially if you want to develop the software, you are not installing it.
Try writing Makefiles for a large project and you will see how much work it is. But yes, everything in 5 is right.
I want to create a go executable which communicates with xen through it's native interface. There is a C shared library (actually 2) for this purpose and I created a simple go wrapper with cgo.
The problem is that I want to target 3 xen versions (3.2, 3.4, 4.0), each of which has a different shared library. The library itself provides basically the same API, but the sizes and shape of the structs defined in the C header are different, and thus the same compiled go binary cannot be used with these different shared libraries.
I want to have a go binary holding the 'main' and a go pkg which is the wrapper for xen.
I was thinking about 2 solutions:
I could build 3 different versions of the compiled pkg and also 3 different versions of the main binary each linked with the corresponding pkg version. This solution requires building manually the makefiles so that I can pass the correct paths etc
I could build a thin C wrapper as a shared library and build it in 3 versions against the 3 versions of xen C bindings. This C wrapper would then export a stable C interface which is then used by one single go pkg. Then I can deploy the correct C wrapper shared library to the hosts and have it resolve at runtime
Is there any other way to handle that ? I would prefer using pure (c)go code but I don't like the additional burden of maintaining complicated makefiles.
EDIT: More details about why I feel uncomfortable about handling that manually in the makefiles:
For example the name of the _obj dir is hardcoded in Make.inc and company, and these makefiles rely on some generated .c files containing special information about the name of the shared library, which I have to cleanup before building the next version of the pkg. A snipped of my makefile:
all:
rm -f _obj/_*
$(MAKE) -f Makefile.common XENVERSION=3.0
rm -f _obj/_*
$(MAKE) -f Makefile.common XENVERSION=3.4
rm -f _obj/_*
$(MAKE) -f Makefile.common XENVERSION=4.0
where Makefile.common basically is a normal pkg makefile which uses TARG=$(XENVERSION)/vimini/xen, as I cannot encode the version in the package name because I'd have to modify the imports in the source.
By encoding the version in the package directory I can use GCIMPORTS=-I../../pkg/xen/_obj/$(XENVERSION) to select the right une from the main cmd's Makefile.
Of course I could roll out my own makefile which invokes 6c and 6l, cgo etc, but I prefer to leverage the existing make infrastructure since it seems that there is some wisdom in it.
Have you tried this approach?
Architecture- and operating system-specific code
It could easily be adapted to use a $XENVER environment variable.
The last sentence in the article caught my eye
[F]or C/C++ developers and
students interested in learning to
program in C/C++ rather than users of
Linux. This is because the compiling
of source code is made simple in
GNU/Linux by the use of the 'make'
command.
I have always used gcc to compile my C/C++ programs, whereas javac to compile my Java programs. I have only used make to install programs to my computer by configure/make/make install.
It seems that you can compile apparently all your programs with the command make.
What is the difference between make and gcc?
Well ... gcc is a compiler, make is a tool to help build programs. The difference is huge. You can never build a program purely using make; it's not a compiler. What make does it introduce a separate file of "rules", that describes how to go from source code to finished program. It then interprets this file, figures out what needs to be compiled, and calls gcc for you. This is very useful for larger projects, with hundreds or thousands of source code files, and to keep track of things like compiler options, include paths, and so on.
gcc compiles and/or links a single file. It supports multiple languages, but does not knows how to combine several source files into a non-trivial, running program - you will usually need at least two invocations of gcc (compile and link) to create even the simplest of programs.
Wikipedia page on GCC describes it as a "compiler system":
The GNU Compiler Collection (usually shortened to GCC) is a compiler system produced by the GNU Project supporting various programming languages.
make is a "build tool" that invokes the compiler (which could be gcc) in a particular sequence to compile multiple sources and link them together. It also tracks dependencies between various source files and object files that result from compilation of sources and does only the operations on components that have changed since last build.
GNUmake is one popular implementation of make. The description from GNUmake is as follows:
Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files.
Make gets its knowledge of how to build your program from a file called the makefile, which lists each of the non-source files and how to compute it from other files.
gcc is a C compiler: it takes a C source file and creates machine code, either in the form of unlinked object files or as an actual executable program, which has been linked to all object modules and libraries.
make is useful for controlling the build process of a project. A typical C program consists of several modules (.c) and header files (.h). It would be time-consuming to always compile everything after you change anything, so make is designed to only compile the parts that need to be re-compiled after a change.
It does this by following rules created by the programmer. For example:
foo.o: foo.c foo.h
cc -c foo.c
This rule tells make that the file foo.o depends on the files foo.c and foo.h, and if either of them changes, it can be built by running the command on the second line. (The above is not actual syntax: make wants the commands indented by a TAB characters, which I can't do in this editing mode. Imagine it's there, though.)
make reads its rules from a file that is usually called a Makefile. Since these files are (traditionally) written by hand, make has a lot of magic to let you shorten the rules. For example, it knows that a foo.o can be built from a foo.c, and it knows what the command to do so is. Thus, the above rule could be shortened to this:
foo.o: foo.h
A small program consisting of three modules might have a Makefile like this:
mycmd: main.o foo.o bar.o
$(CC) $(LDFLAGS) -o mycmd main.o foo.o bar.o
foo.o: foo.h bar.h
bar.o: bar.h
make can do more than just compile programs. A typical Makefile will have a rule to clean out unwanted files:
clean:
rm -f *.o core myapp
Another rule might run tests:
check: myapp
./myapp < test.input > test.output
diff -u test.correct test.output
A Makefile might "build" documentation: run a tool to convert documentation from some markup language to HTML and PDF, for example.
A Makefile might have an install rule to copy the binary program it builds to wherever the user or system administrator wants it installed.
And so on. Since make is generic and powerful, it is typically used to automate the whole process from unpacking a source tarball to the point where the software is ready to be used by the user.
There is a whole lot of to learn about make if you want to learn it fully. The GNU version of make has particularly good documentation: http://www.gnu.org/software/make/manual/ has it in various forms.
Make often uses gcc to compile a multitude of C or C++ files.
Make is a tool for building any complex system where there are dependancies between the various system components, by doing the minimal amount of work necessary.
If you want to find out all the things make can be used for, the GNU make manual is excellent.
make uses a Makefile in the current directory to apply a set of rules to its input arguments. Make also knows some default rules so that it executes even if it doesn't find a Makefile (or similar) file in the current directory. The rule to execute for cpp files so happens to call gcc on many systems.
Notice that you don't call make with the input file names but rather with rule names which reflect the output. So calling make xyz will strive to execute rule xyz which by default builds a file xyz (for example based on a source code file xyz.cpp.
gcc is a compiler like javac. You give it source files, it gives you a program.
make is a build tool. It takes a file that describes how to build the files in your project based on dependencies between files, so when you change one source file, you don't have to rebuild everything (like if you used a build script). make usually uses gcc to actually compile source files.
make is essentially an expert system for building code. You set up rules for how things are built, and what they depend on. Make can then look at the timestamps on all your files and figure out exactly what needs to be rebuilt at any time.
gcc is the "gnu compiler collection". There are many languages it supports (C, C++, Ada, etc depending on your setup), but still it is just one tool out of many that make may use to build your system.
You can use make to compile your C and C++ programs by calling gcc or g++ in your makefile to do all the compilation and linking steps, allowing you to do all these steps with one simple command. It is not a replacement for the compiler.
'gcc' is the compiler - the program that actually turns the source code into an executable. You have to tell it where the source code is, what to output, and various other things like libraries and options.
'make' is more like a scripting language for compiling programs. It's a way to hide all the details of compiling your source (all those arguments you have to pass the compiler). You script all of the above details once in the Makefile, so you don't have to type it every time for every file. It will also do nifty things like only recompile source files that have been updated, and handle dependancies (if I recompile this file, I will then need to recompile THAT file.)
The biggest difference is that make is turing complete (Are makefiles Turing complete?) while gcc is not.
Let's take the gcc compiler for example.
It only knows how to compile the given .cpp file into .o file given the files needed for compilation to succeed (i.e. dependencies such as .h files).
However, those dependencies create a graph. e.g., b.o might require a.o in the compilation process which means it needs to be compiled independently beforehand.
Do you, as a programer want to keep track of all those dependencies and run them in order for your target .o file to build?
Of course not. You want something to do that task for you.
Those are build tools - tools that help making the build process (i.e. building the artifacts like .o files) easier. One such tool is make.
I hope that clarifies the difference :)