I have 2 C libraries:
A dependency library, compiled as a shared-object
My library - which I also want to compile as a shared-object
I want to compile my library into a shared object, but I do not want the dependency library to be loaded on runtime.
I.e. I want the dependency library to become a part of the shared-object I create from my library, so one won't have to add the dependency to LD_LIBRARY_PATH when using my library.
How can I do it with GCC?
This seems to be a XY problem: You want to add a SO statically to avoid using LD_LIBRARY_PATH. Allow me to address the latter and ignore the former. ;-)
LD_LIBRARY_PATH is a means to temporarily overwrite the library search path. Note "overwrite": The paths given in LD_LIBRARY_PATH are searched first, and if a library is found in them, the standard search paths are not considered for that library. It "knocks out" other versions that might be installed in the standard paths. It's easy to see how this could have surprising and unwanted consequences. It is a debug feature, mostly, so you are right in not wanting to use it.
There are three "correct" ways to address this:
1) Install the dependency in one of the standard search paths (/usr/lib/* / /lib/*, check /etc/ld.so.conf for a list). This will require superuser priviledges.
2) Add the path of the dependency to the standard search paths (in /etc/ld.so.conf). This will require superuser priviledges.
3) Configure the path of the dependency into your library at compilation time (-Wl,-rpath=/path/to/lib/). This will add the given path to the paths searched by this library only. It does not require any special priviledges, and has none of the disadvantages that LD_LIBRARY_PATH has.
As an aside, you can check where your binary does "find" its libraries via ldd <filename>.
Related
We have an HPC environment with multiple versions of most packages, causing us to have designed a home-rolled way to install packages in unique locations and use environment modules for programmers/researchers to be able to identify which library versions they are using when they build a program, run a program, or both. Is there a relatively painless way to be able to perform builds in this environment. In my case, we're using OpenBLAS, ARPACK, LAPACK and SuperLU when building armadillo. In my case, I'm shooting for armadillo-0.3.7. It would be real nice if the use of switches as was done in the ./configure and make days would work. But all I've found so far is CMake builds, and it appears to be pretty much non-trivial to do a build.
Oh yeah. And, by the way, there's a need for the output Armadillo library to be static.
Thanks in advance for your help. The initial question may be a little vague, but I can get as specific as you like. I just didn't want to write a novel for the initial question on this issue.
Tools exist nowadays to handle the complexity of the build of these scientific software. I would suggest you look at either Spack or EasyBuild. Such tool will help you to save a lot of time by automatically building all the required dependencies and also generating the modulefiles for your users to use the software built.
The CMakeList.txt file and other CMake-related files can be modified to meet your needs. The flags are defined on line 48+, for instance set(ARMA_USE_LAPACK false)
The variables related to the LAPACK library are then defined in the include(ARMA_FindLAPACK) on line 250. The flag in toggled on on line 347 if lapack is found.
The customized path where LAPACK is located can be specified in the file cmake_aux/ARMA_FindLAPACK.cmake. If your customized path is stored as an environment variable as export PATHLAPACKLIB=/usr/lib/openblas-base, you can use it in the ARMA_FindLAPACK.cmake file by modifying line 11 ( see How to retrieve a user environment variable in CMake (Windows) and FIND_LIBRARY()):
message("Searching Lapack in $ENV{PATHLAPACKLIB}")
FIND_LIBRARY(LAPACK_LIBRARY
NAMES ${LAPACK_NAMES}
HINTS $ENV{PATHLAPACKLIB}
NO_DEFAULT_PATH
)
It is not a beautiful modification of the CMakefile, because it makes it not portable as its outcome depends on an environment variable. But, it you intend to to build and install Armadillo once for all, it works. Remember to delete the CMakeCache.txt file every time you modify a CMakeFile.txt, otherwise it keeps some trace of previous runs of cmake . and it looks as though the modification is consequenceless.
To make the library static, add the keyword static to the command add_library() on line 514 of CMakeFile.txt, as shown in CMake - Creating a static library :
add_library( armadillo STATIC ...)
Running cmake . and then make creates a small armadillo.a file, since most of the source consists in cpp headers.
Finally, the exemple1 is compiled as :
g++ -O2 -std=c++11 example1.cpp -o example1 -larmadillo -L/home/...../softs/armadillo-9.800.3/armadillo-9.800.3 -I/home/...../softs/armadillo-9.800.3/armadillo-9.800.3/include -lopenblas
According to the manual page for ld (and gcc used for linking by extension), if a -L option appears on the command line, it applies to all libraries specified by -l and takes precedence over the default search locations. However, that is not working in my link step. I have this on the command line:
-L /users/me/mylib -lpcre -lz
and /users/me/mylib contains (copies) of libpcre.so and libz.so
These libraries exist in other locations on the system (although not necessarily the same versions) and what I see (with ldd on Linux and otool on Mac) is a path that references the libraries in those locations. Some of those locations are on the LD_LIBRARY_PATH (which I cannot control in the build environment I am running in) and it appears that somehow those locations are being picked up in preference to my explicit setting with -L.
Just to be clear, this a link step problem and not a runtime problem. There is a lot of info on the web on how to affect/override library locations when executing and I am familiar with all that. In some sense what I am trying to do with the -L is create a completely specified setup. I know I can fix things up with install_name_tool on MacOS but I'd really like to understand why -L isn't doing what it claims to.
One thing I learned using gcc -Wl,-v is that gcc appears to forward all the LD_LIBRARY_PATH directories to ld. However, it places them after the ones explicitly listed by me and man ld says they are searched in order they appear on the line.
Just to be clear, this a link step problem and not a runtime problem.
From what you describe as the problem, I don't think you are right about this - it sounds like a runtime problem for which you are (justifiably) looking for a solution that you can employ while linking that will solve the problem you have at runtime.
The reason I say it does not appear to be a problem with linking is that it sounds like your linking is working as it is intended. LD (or GCC) are not complaining about the linking, and your linked executables are being produced just fine. The issue you are having is that when you subsequently go to run those executables, the loader is finding libraries other than the ones you intend. The purpose of the -L flag during linking is to let the linker know where it can find suitable libraries to use in preparing the linked binary. That is completely separate from where the loader will search for the required libraries at runtime.
As you say, you are already aware that there are ways you could employ at runtime (such as changing LD_LIBRARY_PATH) that would avoid the issue by changing the set of paths that the loader searches for libraries, but you'd rather not have to do that because for whatever reason you won't necessarily have control over the runtime environment, which is fair enough.
Luckily, there is a facility that I believe will get you what you want. Take a look at the ld option called -rpath (see the GNU ld man page for full documentation). Basically, if you add paths during linking using the -rpath option, those paths gets stored in the linked executable as preferred locations to find the libraries at runtime, in much the same way they would be searched if included in LD_LIBRARY_PATH. This should work on Linux or Mac OS X (at least since 10.5).
Passing the -rpath option to ld via gcc requires using the -Wl option to pass the flag through. To obtain an ld command line that contains ld -rpath /custom/path/to/libs requires a gcc invocation something like: gcc -Wl,-rpath,/custom/path/to/libs
In short, try replacing what you currently have: -L/users/me/mylib -lpcre -lz
With: -L/users/me/mylib -Wl,-rpath,/users/me/mylib -lpcre -lz
The resulting executable (or library) will then have /users/me/mylib stored as the place to go to find libraries, and it should find libpcre.so and libz.so there without needing to control LD_LIBRARY_PATH.
I am working on a library in C, let us call it ninja.
Ninja depends upon some under laying libraries (which we also provide) (e.g jutsu, goku, bla).
These are all placed in a shared library folder, let us say /usr/lib/secret/.
The clients whom are using this project wants to be able to havde ninja version 1 and 2 laying side by side, this it not so hard. The problem comes when ninja 1 dependes up on for instance jutsu 1 and ninja 2 depends upon jutsu 3. How the h... do we/I do so so that when installing ninja from our package repository. It knows the correct version of jutsu. Of course the rpm/deb package should depend upon the correct version of the jutsu package.
so what we want is when, we execute for instance zypper in ninja. and it installs and compiles on the system, it knows which jutsu library to take with out been given a version number.
So we in the make file don't have to do this:
gcc ninja.c -o ninja -L /usr/local/lib/secret/ -l jutsu_2
But just
gcc ninja.c -o ninja -L /usr/local/lib/secret/ -l jutsu
NOTE: I know it is random to use ninja and so on, but I am not allowed to publish the real library names
You want to use an SONAME. Describing all the steps necessary is probably too large a scope for a good StackOverflow answer, but I can give an overview and point to some documentation.
An SONAME is a special data field inside a shared library. It is typically used to indicate compatibility with other versions of the same library; if two different versions of a shared library have the same SONAME, the linkers will know that either one can fill the dependency on that library. If they have a different SONAME, they can't.
Example: I have libdns88 and libbind-dev version 1:9.8.4.dfsg.P1-6+nmu2+deb7u1 installed on a Debian wheezy system. I build a binary called samurai with -ldns. The GNU linker finds "libdns.so" in my library search path and dynamically links samurai with it. It reads the SONAME field from libdns.so (which is a symlink to libdns.so.88.1.1). The SONAME there is "libdns.so.88".
$ objdump -p /usr/lib/libdns.so | grep SONAME
SONAME libdns.so.88
The libdns developers (or maybe packagers) chose that SONAME to indicate that any version 88.* of libdns is expected to be binary compatible with any other version 88.*. They use that same SONAME for all versions with a compatible ABI. When the ABI had a change, they changed the SONAME to libdns.so.89, and so on. (Most well-managed libraries don't change their ABI that often.)
So the library dependency written into the samurai binary is just libdns.so.88. When I run samurai later, the dynamic linker/loader looks for a file called "libdns.so.88" instead of just "libdns.so".
Also by convention, the name of an rpm or deb package should change when the SONAME of the library contained changes. That's why there is a libdns88 package separate from the libdns100 package, and they can be installed side by side without interfering with each other. My samurai package will have a dependency on "libdns88" and I can expect that any package called libdns88 will have a compatible ABI to the one I built it against. Tools like dpkg-shlibdeps make it simple to create the right shared library package dependencies when SONAMEs and versioned symbols are used.
http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html
I want to pull all symbols from a static library (libyaml-cpp) into a shared one that I'm building (libLHAPDF), to avoid an explicit dependency for users of the latter. (It's not my favourite way to do things, but the difficulty of installing the prerequisite was a regular negative feedback about the package.) So I try to build the relevant bit of my shared library with an automake rule like this:
libLHAPDFInfo_la_LDFLAGS = $(AM_LDFLAGS) \
$(srcdir)/yamlcpp/yaml-cpp/libyaml-cpp.a
I specifically want something like this rather than
libLHAPDFInfo_la_LDFLAGS = -L$(srcdir)/yamlcpp/yaml-cpp -lyaml-cpp \
$(AM_LDFLAGS)
because there could be a shared lib version of libyaml-cpp installed elsewhere on the system, and I don't want to use it. (The odd flag ordering in that latter case is an attempt to make sure that -lyaml-cpp finds the one built in the srcdir... and yes it really is srcdir and not builddir in this case!)
However, the first version gives a warning that I'd rather not have:
*** Warning: Linking the shared library libLHAPDFInfo.la against the
*** static library ./yamlcpp/yaml-cpp/libyaml-cpp.a is not portable!
and more importantly it doesn't actually work: if I run nm on the generated library I see undefined symbols:
$ nm src/.libs/libLHAPDFInfo.a | grep YAML
U _ZN11LHAPDF_YAML4NodeC1Ev
U _ZN11LHAPDF_YAML4NodeD1Ev
...
(Details: libLHAPDFInfo.a is a noinst_LTLIBRARIES entry used as an intermediate to building the final shared lib. So this doesn't even work when linking one static lib against another. And to avoid symbol clashes, the bundled version is slightly hacked to rename the YAML namespace to LHAPDF_YAML: that doesn't change anything, but I thought I'd mention it just in case those symbol names seem strange to you.)
If I use the second form of the link flags (the -lyaml-cpp way), I get all the LHAPDF_YAML symbols in libLHAPDFInfo.a and thereafter into the shared library. And there is no longer any warning about non-portability of the static library linking (which was built with -fPIC, so it is valid to do this). But if a shared libyaml-cpp is also found on the system, it gets used regardless of the -L/-l flag ordering -- which I don't want.
Any suggestions of how I can make sure that the version of the library at a particular path will be used, while getting the symbols correctly copied across and avoiding libtool warnings?
EDIT: Having said that I could get symbols copied from libyaml-cpp.a into libLHAPDFInfo.a via the -lyaml-cpp flag, after more iterations I could no longer get this approach to show the expected symbols via nm. Looking at the ar/ranlib commands executed by libtool when making libLHAPDFInfo.a, the -l arguments seem to get completely dropped. So I don't know how it ever worked... as far as I can tell it's just not a mode that libtool supports, not that that is documented. In the end I renamed libyaml-cpp to liblhapdf-yaml-cpp.a as part of the build (since no lib with that name should be found accidentally), and linked that into the final shared libLHAPDF.so rather than the static convenience lib. Not as neat as I'd have liked -- I was hoping to isolate all the yaml-cpp dependency handling into the build of one convenience lib, and relying on file copies to disambiguate library lookup is unsatisfying -- but it works.
I maintain a library with FFI bindings on Hackage. So my Haskell library depends on the corresponding C library and its header files. Now I specify the external dependency in the .cabal file like this:
PkgConfig-Depends:
libfoo >= 1.2
And it works well for me in Linux. However, I have a user of the library who reports, that installing pkg-config on Windows is rather cumbersome, and instead he prefers
Includes:
foo.h
Extra-libraries:
foo
I'd like my library to be as easy to build as possible, and don't want to force build dependencies which are not strictly required. However, I see that Cabal manual suggests to use PkgConfig-Depends.
My questions:
Which way I should prefer for cross-platform packages?
Is it possible to write a .cabal file in such a way, that it can work with pkg-config and without?
And, by the way, is pkg-config included in the Haskell platform (I don't have a Windows machine to check right now)?
The pkg-config method is preferable because pkg-config knows where to find include and library files, which may be in nonstandard locations on some systems.
You can write the .cabal file to use both methods. Using a flag, as shown here, has the advantage that Cabal will automatically try the other flag value if the default fails. (Below example is not tested)
Flag UsePkgConfig
Description: Use pkg-config to check for library dependences
Default: True
Executable hax
if flag(UsePkgConfig)
PkgConfig-Depends: libfoo >= 1.2
else
Includes: foo.h
Extra-libraries: foo
pkg-config is not included in the Haskell Platform, nor could I imagine that it ever would be.
Usually I will use includes/Extra-libraries if they're relatively simple. But for complex packages that may have a lot of included libraries, such as gtk, it's much nicer to use pkg-config when available.
It is possible to write a .cabal file that will work with and without specific fields. Try this:
if os(windows)
Includes:
foo.h
Extra-libraries:
foo
else
PkgConfig-Depends:
libfoo >= 1.2
Also note that .cabal can run a configure script, which can help in some situations but isn't very windows-friendly.