We have an HPC environment with multiple versions of most packages, causing us to have designed a home-rolled way to install packages in unique locations and use environment modules for programmers/researchers to be able to identify which library versions they are using when they build a program, run a program, or both. Is there a relatively painless way to be able to perform builds in this environment. In my case, we're using OpenBLAS, ARPACK, LAPACK and SuperLU when building armadillo. In my case, I'm shooting for armadillo-0.3.7. It would be real nice if the use of switches as was done in the ./configure and make days would work. But all I've found so far is CMake builds, and it appears to be pretty much non-trivial to do a build.
Oh yeah. And, by the way, there's a need for the output Armadillo library to be static.
Thanks in advance for your help. The initial question may be a little vague, but I can get as specific as you like. I just didn't want to write a novel for the initial question on this issue.
Tools exist nowadays to handle the complexity of the build of these scientific software. I would suggest you look at either Spack or EasyBuild. Such tool will help you to save a lot of time by automatically building all the required dependencies and also generating the modulefiles for your users to use the software built.
The CMakeList.txt file and other CMake-related files can be modified to meet your needs. The flags are defined on line 48+, for instance set(ARMA_USE_LAPACK false)
The variables related to the LAPACK library are then defined in the include(ARMA_FindLAPACK) on line 250. The flag in toggled on on line 347 if lapack is found.
The customized path where LAPACK is located can be specified in the file cmake_aux/ARMA_FindLAPACK.cmake. If your customized path is stored as an environment variable as export PATHLAPACKLIB=/usr/lib/openblas-base, you can use it in the ARMA_FindLAPACK.cmake file by modifying line 11 ( see How to retrieve a user environment variable in CMake (Windows) and FIND_LIBRARY()):
message("Searching Lapack in $ENV{PATHLAPACKLIB}")
FIND_LIBRARY(LAPACK_LIBRARY
NAMES ${LAPACK_NAMES}
HINTS $ENV{PATHLAPACKLIB}
NO_DEFAULT_PATH
)
It is not a beautiful modification of the CMakefile, because it makes it not portable as its outcome depends on an environment variable. But, it you intend to to build and install Armadillo once for all, it works. Remember to delete the CMakeCache.txt file every time you modify a CMakeFile.txt, otherwise it keeps some trace of previous runs of cmake . and it looks as though the modification is consequenceless.
To make the library static, add the keyword static to the command add_library() on line 514 of CMakeFile.txt, as shown in CMake - Creating a static library :
add_library( armadillo STATIC ...)
Running cmake . and then make creates a small armadillo.a file, since most of the source consists in cpp headers.
Finally, the exemple1 is compiled as :
g++ -O2 -std=c++11 example1.cpp -o example1 -larmadillo -L/home/...../softs/armadillo-9.800.3/armadillo-9.800.3 -I/home/...../softs/armadillo-9.800.3/armadillo-9.800.3/include -lopenblas
Related
This file: MoltenVK Runtime User Guide offers a nice description of how to link MoltenVK to a project that uses it, within XCode.
I prefer using makefiles, as it removes the need to boot up XCode to build (+ is a consistent workflow with the same project on Linux).
The problem is there's quite a bit of OSX/XCode specific terminology in the instructions, and I'm not quite sure how it translates to running clang via command line.
If you assume I'm looking to link only with the minimum specified to use MoltenVK, what would such a command line argument look like?
Here is my minimal attempt:
MOLTENVKDIR = /blah/vulkansdk-macos-1.1.108.0
a.out: my_headers.h my_src_files.cpp
clang -I$(MOLTENVKDIR)/MoltenVK/include -L$(MOLTENVKDIR)/MoltenVK/macOS/dynamic -lMoltenVK my_src_files.cpp
More specific questions:
How should I be setting LD_RUNPATH_SEARCH_PATHS? I assume this is a PATH-style variable that gets embedded in the executable with a list of places to look for the MoltenVK.dylib file?
In step 4 of the user guide, it says "drag (MoltenVK/macOS/dynamic/libMoltenVK.dylib) to the Link Binary With Libraries list"- is that essentially the -lMoltenVK line?
Where does step 5 come in? Should I append -framework Metal -framework Foundation -framework ...? What should I do with the .tbd file?
Is step 6 just ensuring that I copy the libMoltenVK.dylib file relative to a.out consistent with LD_RUNPATH_SEARCH_PATHS?
Is step 7 safe to ignore, as I'm not using XCode?
TLDR; I use g++, use the flags -lMoltenVK and -L$(MOLTENVK_DIR), and at the top define VK_ICD_FILENAMES using export VK_ICD_FILENAMES=$(ICD_DIR)
Header Search Paths is -I$(PATH)
Library Search Paths is -L$(PATH)
Runpath Search Paths is -rpath $(PATH) (with the space)
What I did was copy the .dylibs into a folder called lib in my project folder, set Library Search Path with -Llib, and link them with -lvulkan and -lMoltenVK. Since these dynamic libraries are hardwired to use #rpath when linking, I set -rpath lib.
You can see the vulkan tutorial for linux which uses a Makefile which shows a very basic version of making a Makefile for Vulkan on linux (no MoltenVK)
Most importantly, to not get a VkResult -9 when calling vkCreateInstance (MoltenVK not installed properly) make sure to define the VK_ICD_FILENAMES environment variable! I did this by writing export VK_ICD_FILENAMES=lib/MoltenVK_icd.json near the top of my Makefile, and copied the MoltenVK_icd.json file to my project! This is a file which points to where the dylib is, and is necesarry for MoltenVK to work. When installing MoltenVK, it was inside the same folder as libMoltenVK.dylib
Similarily, you must define VK_LAYER_PATH for validation layers to work.
Minimally, all you need for MoltenVK is:
export VK_ICD_FILENAMES=lib/MoltenVK_icd.json
main:
g++ test.cpp -o App -Llib -rpath lib -lMoltenVK
I'm not an expert in best practices and conventions, I do what works. Hope this helped
I'm a student doing research involving extending the TM capabilities of gcc. My goal is to make changes to gcc source, build gcc from the modified source, and, use the new executable the same way I'd use my distro's vanilla gcc.
I built and installed gcc in a different location (not /usr/bin/gcc), specifically because the modified gcc will be unstable, and because our project goal is to compare transactional programs compiled with the two different versions.
Our changes to gcc source impact both /gcc and /libitm. This means we are making a change to libitm.so, one of the shared libraries that get built.
My expectation:
when compiling myprogram.cpp with /usr/bin/g++, the version of libitm.so that will get linked should be the one that came with my distro;
when compiling it with ~/project/install-dir/bin/g++, the version of libitm.so that will get linked should be the one that just got built when I built my modified gcc.
But in reality it seems both native gcc and mine are using the same libitm, /usr/lib/x86_64-linux-gnu/libitm.so.1.
I only have a rough grasp of gcc internals as they apply to our project, but this is my understanding:
Our changes tell one compiler pass to conditionally insert our own "function builtin" instead of one it would normally use, and this is / becomes a "symbol" which needs to link to libitm.
When I use the new gcc to compile my program, that pass detects those conditions and successfully inserts the symbol, but then at runtime my program gives a "relocation error" indicating the symbol is not defined in the file it is searching in: ./test: relocation error: ./test: symbol _ITM_S1RU4, version LIBITM_1.0 not defined in file libitm.so.1 with link time reference
readelf shows me that /usr/lib/x86_64-linux-gnu/libitm.so.1 does not contain our new symbols while ~/project/install-dir/lib64/libitm.so.1 does; if I re-run my program after simply copying the latter libitm over the former (backing it up first, of course), it does not produce the relocation error anymore. But naturally this is not a permanent solution.
So I want the gcc I built to use the shared libs that were built along with it when linking. And I don't want to have to tell it where they are every time - my feeling is that it should know where to look for them since I deliberately built it somewhere else to behave differently.
This sounds like the kind of problem any amateur gcc developer would have when trying to make a dev environment and still be able to use both versions of gcc, but I had difficulty finding similar questions. I am thinking this is a matter of lacking certain config options when I configure gcc before building it. What is the right configuration to do this?
My small understanding of the instructions for building and installing gcc led me to do the following:
cd ~/project/
mkdir objdir
cd objdir
../source-dir/configure --enable-languages=c,c++ --prefix=/home/myusername/project/install-dir
make -j2
make install
I only have those config options because they seemed like the ones closest related to "only building the parts I need" and "not overwriting native gcc", but I could be wrong. After the initial config step I just re-run make -j2 and make install every time I change the code. All these steps do complete without errors, and they produce the ~/project/install-dir/bin/ folder, containing the gcc and g++ which behave as described.
I use ~/project/install-dir/bin/g++ -fgnu-tm -o myprogram myprogram.cpp to compile a transactional program, possibly with other options for programs with threads.
(I am using Xubuntu 16.04.3 (64 bit), within VirtualBox on Windows. The installed /usr/bin/gcc is version 5.4.0. Our source at ~/project/source-dir/ is a modified version of 5.3.0.)
You’re running into build- versus run-time linking differences. When you build with -fgnu-tm, the compiler knows where the library it needs is found, and it tells the linker where to find it; you can see this by adding -v to your g++ command. However when you run the resulting program, the dynamic linker doesn’t know it should look somewhere special for the ITM library, so it uses the default library in /usr/lib/x86_64-linux-gnu.
Things get even more confusing with ITM on Ubuntu because the library is installed system-wide, but the link script is installed in a GCC-private directory. This doesn’t happen with the default GCC build, so your own GCC build doesn’t do this, and you’ll see libitm.so in ~/project/install-dir/lib64.
To fix this at run-time, you need to tell the dynamic linker where to find the right library. You can do this either by setting LD_LIBRARY_PATH (to /home/.../project/install-dir/lib64), or by storing the path in the binary using -Wl,-rpath=/home/.../project/install-dir/lib64 when you build it.
I'm currently working on a project using Arduino 1.0.6 IDE and it does not seem to accept C++11 std::array. Is it possible to change the compiler flag to make this work?
Add custom compiler flags to platform.local.txt. Just create it in the same directory where platform.txt is. For example:
compiler.c.extra_flags=
compiler.c.elf.extra_flags=
compiler.S.extra_flags=
compiler.cpp.extra_flags=-mcall-prologues -fno-split-wide-types -finline-limit=3 -ffast-math
compiler.ar.extra_flags=
compiler.objcopy.eep.extra_flags=
compiler.elf2hex.extra_flags=
In this example C++ flags will make large sketch smaller. Of course, you can use your own flags instead. Since platform.local.txt does not overwrite standard files and is very short, it is very easy to experiment with compiler flags.
You can save platform.local.txt for each project in its directory. It will NOT have any effect in project's directory, but this way if you decide to work on your old project again you will be able to just copy it to the same directory where platform.txt is (typically ./hardware/arduino/avr/) and continue work on your project with project-specific compiler flags.
Obviously, using Makefile as ladislas suggests is more professional and more convenient if you have multiple projects and do not mind dealing with Makefile. But still, using platform.local.txt is better than messing with platform.txt directly and an easy way to play with compiler flags for people who are already familiar with Arduino IDE.
You can use #pragma inside the *.ino file so as not to have to create the local platforms file:
#pragma GCC diagnostic warning "-fpermissive"
#pragma GCC diagnostic ignored "-Wwrite-strings"
For other ones, see HERE.
Using the IDE is very difficult to do that.
I would advise you to go full command line by using Sudar's great Arduino Makefile.
This way you'll be able to customise the compiler flags to your liking.
I've also created the Bare Arduino Project to help you get started. The documentation covers a lot points, from installing the latest avr-gcc toolchain to how to use the repository, compile and upload your code.
If you find something missing, please, feel free to fill an issue on Github so that I can fix it :)
Hope this helps! :)
Yes, but not in 1.0.6, in 1.5.? the .\Arduino\hardware\arduino\avr\platform.txt specifies the command lines used for compiling.
One can either modify this file directly or copy it to your user .\arduino\hardware\... directory to create a custom platform. As not to alter the stock IDE. This will also then exist in other/updated IDEs that you can run. You can copy just the platform file and boards.txt. And have your boards.txt file link to the core: libraries as not to have a one-off. See
Reference: Change CPU speed, Mod New board
I wanted to add the -fpermissive flag.
Under Linux here what I have done with success
The idea is to replace the two compilers avr-gcc and avr-g++ by two bash scripts in which you add your flags (-fpermissive for me)
With root privilege:
rename the compiler avr-gcc (present in /usr/bin) avr-gcc-real
rename the compiler avr-g++ (present in /usr/bin) avr-gcc-g++-real
Now create to bash scripts avr-gcc and avr-g++ under /usr/bin/
script avr-gcc contains this line:
avr-gcc-real -fpermissive $#
script avr-g++ contains this line:
avr-g++-real -fpermissive $#
As you may know $# denotes the whole parameters passed to the script. Thus all the parameters transmitted by the IDE to compilers are transimitted to your bash scripts replacing them (which call the real compilers with your flags and the IDE one)
Don't forget to add executable property to your scripts:
chmod a+x avr-gcc
chmod a+x avr-g++
Under Windows I don't know if such a solution can be done.
I've added code to an existing large application and need to make GLib a requirement, as my code relies on it. For development, I just manually edited the Makefile to add
-lglib-2.0
To the LIBS= variable and
-I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include $<
to the line starting with ${CC}.
However, I am at a loss for how to make this permanent/portable in the app -- i.e. when someone executes ./configure in the future, the resulting Makefile should also include the above (as appropriate, since these depend on pkg-config output, I've learned). The codebase I updated includes the following files from the gnu tool chain:
Makefile.in
Makefile.manual
config.h.in
configure
configure.in
I only have a handful of CS degrees and a few years of development experience, so the GNU toolchain remains utterly impenetrable to me. :-/ From googling around, I'm under the impression there should also be a configure.ac file or something where I should add a macro for requiring glib, but no such file is included in the package and I'm at the point of learned helplessness with the whole automake/autoconf/configure/makefile business. Thanks in advance for any advice or pointers!
You should not edit any generated files manually. This includes the final Makefile used to build the application.
In configure.ac, every dependency is listed, thus checking for GLib should go in there. From this file, your final configure shell script is generated.
GLib provides a pkgconfig description so you almost always want to use this to get the correct compile and link flags.
Combining pkgconfig and Autotools is just a matter of calling the PKG_CHECK_MODULES macro. The Autotools Mythbuster is an excellent source that describes how to do it.
In the end it boils down to adding these lines to your configure.ac:
PKG_PROG_PKG_CONFIG
PKG_CHECK_MODULES([GLIB], [glib-2.0])
and these lines to your Makefile.am:
foo_CXXFLAGS = $(GLIB_CFLAGS)
foo_LIBS = $(GLIB_LIBS)
My main aim is to get the GSL Shell working on my OSX 10.7 system. So far I have the correct version of lua with the correct patches running. I also have a working version of GSL which compiles and runs example programs. I can build agg perfectly and also run their example programs by running make in the macosx_sdl folder.
My first question is how on earth do I create my own project with agg? I know that you are supposed to simply add the files to your project file and go, but this does not seem to want to compile for me. Is it simply a case of adding the include directory and the libagg.a?
Finally, how do I build gsl shell? Currently it complains about the agg-plot folder a lot, so where do I put the agg files to make this build, then when i've done it where do I place the agg files so that the lua scripts can get to them?!
Hope someone can help!
In general to use the AGG library you need to make sure that the compiler is able to find the headers files and, during the linking, the libraries, either in form of a static or dynamic libraries.
To make the headers files and the libraries available you need to take into account the system that is used to build the software. If a traditional makefile sistem is used you need to add some flags to make sure that the headers file can be found. This can be achieved by adding into the makefile something like:
CFLAGS += -I/path/to/agg/headers
and for the linker:
LIBS += -L/path/to/agg/library -lagg -lm
In the specific case of GSL Shell 1.1 the file "make-packages" is used in the Makefile to configure the required packages. You can add here the flags required to locate the AGG library:
AGG_INCLUDES = -I/usr/include/agg2
AGG_LIBS = -lagg -lX11 -lpthread -lsupc++
you should just modify the path provided with the "-I" option and, in AGG_LIBS, add an option "-L/path/to/agg/library" to specify the path where the AGG libraries are located.
Please note also that the agg libraries depends on other libraries. For example on linux it needs at least the X11 library. The libraries supc++ may be needed if the linking is made by invoking gcc instead of g++ because if gcc is used the C++ runtime libraries are not included.