Order in which library directories are searched and linked - gcc

I am having difficulty understanding the order in which directories are searched for linking to libraries. I have a CentOS6 system and 3 versions of gcc, 4.4.7, 4.7.2, 4.9.2. The system version is 4.4.7 and version 4.7.2 and 4.9.2 are modules. In /etc/ld.so.conf.d/ there two files, gcc-4.7.2.conf and gcc-4.9.2.conf which contain the paths to the 4.7.2 and 4.9.2 libraries.
I created a simple C++ program, main.cpp
#include <cstdio>
#include <iostream>
using namespace std;
int main(void)
{
cout << "Hello You!" << endl;
printf("Back at you!\n");
return 0;
}
compiling it with g++-4.4.7 and running ldd a.out, I see
linux-vdso.so.1 => (0x00007fff5535b000)
libstdc++.so.6 => /nonstandardpath/gcc-4.7.2/lib64/libstdc++.so.6 (0x00002ac12de73000)
libm.so.6 => /lib64/libm.so.6 (0x00002ac12e17a000)
libgcc_s.so.1 => /nonstandardpath/gcc-4.7.2/lib64/libgcc_s.so.1 (0x00002ac12e3ff000)
libc.so.6 => /lib64/libc.so.6 (0x00002ac12e614000)
/lib64/ld-linux-x86-64.so.2 (0x00002ac12dc51000)
Looking at man ld, it states (under -rpath-link=dir) :
The linker uses the following search paths to locate required shared
libraries:
Any directories specified by -rpath-link options.
Any directories specified by -rpath options. The difference between -rpath and
-rpath-link is that directories specified by -rpath options are included in the
executable and used at runtime, whereas the -rpath-link option is only effective at
link time. Searching -rpath in this way is only supported by native linkers and
cross linkers which have been configured with the --with-sysroot option.
On an ELF system, for native linkers, if the -rpath and -rpath-link options were not
used, search the contents of the environment variable "LD_RUN_PATH".
On SunOS, if the -rpath option was not used, search any directories specified using
-L options.
For a native linker, the search the contents of the environment variable
"LD_LIBRARY_PATH".
For a native ELF linker, the directories in "DT_RUNPATH" or "DT_RPATH" of a shared
library are searched for shared libraries needed by it. The "DT_RPATH" entries are
ignored if "DT_RUNPATH" entries exist.
The default directories, normally /lib and /usr/lib.
For a native linker on an ELF system, if the file /etc/ld.so.conf exists, the list
of directories found in that file.
If the required shared library is not found, the linker will issue a warning and
continue with the link.
It does not state the order in which the directories are searched. From my example above, it appears that /etc/ld.so.conf.d is searched before /usr/lib or /lib
QUESTION : what is the order in which linker searches for libraries (e.g. LD_LIBRARY_PATH, ld.so.conf.d, -rpath, -L)?

The online man page states the order for ld:
When resolving shared object dependencies, the dynamic linker first
inspects each dependency string to see if it contains a slash (this
can occur if a shared object pathname containing slashes was
specified at link time).
If a slash is found, then the dependency
string is interpreted as a (relative or absolute) pathname, and the
shared object is loaded using that pathname.
If a shared object dependency does not contain a slash, then it is searched for in the following order:
Using the directories specified in the DT_RPATH dynamic section attribute of the binary if present and DT_RUNPATH attribute does not exist. Use of DT_RPATH is deprecated.
Using the environment variable LD_LIBRARY_PATH (unless the executable is being run in secure-execution mode; see below). in which case it is ignored.
Using the directories specified in the DT_RUNPATH dynamic section attribute of the binary if present. Such directories are searched only to find those objects required by DT_NEEDED (direct dependencies) entries and do not apply to those objects' children, which must themselves have their own DT_RUNPATH entries. This is unlike DT_RPATH, which is applied to searches for all children in the dependency tree.
From the cache file /etc/ld.so.cache, which contains a compiled list of candidate shared objects previously found in the augmented library path. If, however, the binary was linked with the -z nodeflib linker option, shared objects in the default paths are skipped. Shared objects installed in hardware capability directories (see below) are preferred to other shared objects.
In the default path /lib, and then /usr/lib. (On some 64-bit architectures, the default paths for 64-bit shared objects are /lib64, and then /usr/lib64.) If the binary was linked with the -z nodeflib linker option, this step is skipped.
Taken from the 2017-09-15 version. I assume older versions of ld are similar.

Related

setting LD_LIBRARY_PATH doesn't have any effect in bash, why? [duplicate]

I'm building a simple C++ program and I want to temporarily substitute a system supplied shared library with a more recent version of it, for development and testing.
I tried setting the LD_LIBRARY_PATH variable but the linker (ld) failed with:
/usr/bin/ld: cannot find -lyaml-cpp
I expected that to work because according to the ld man page:
The linker uses the following search
paths to locate required shared
libraries: ... For a native linker,
the contents of the environment variable
"LD_LIBRARY_PATH"...
I then tried setting the LIBRARY_PATH, and that worked.
According to the GCC manual:
The value of
LIBRARY_PATH is a colon-separated list
of directories, much like PATH. When
configured as a native compiler, GCC
tries the directories thus specified
when searching for special linker
files, if it can't find them using
GCC_EXEC_PREFIX. Linking using GCC
also uses these directories when
searching for ordinary libraries for
the -l option (but directories
specified with -L come first).
As the (GCC) manual suggests, LIBRARY_PATH works because I link with GCC.
But..
Since I link with gcc why ld is
being called, as the error message
suggests?
What's the point of
having two variables serving the same
purpose? Are there any other
differences?
LIBRARY_PATH is used by gcc before compilation to search directories containing static and shared libraries that need to be linked to your program.
LD_LIBRARY_PATH is used by your program to search directories containing shared libraries after it has been successfully compiled and linked.
EDIT:
As pointed below, your libraries can be static or shared. If it is static then the code is copied over into your program and you don't need to search for the library after your program is compiled and linked. If your library is shared then it needs to be dynamically linked to your program and that's when LD_LIBRARY_PATH comes into play.
LD_LIBRARY_PATH is searched when the program starts, LIBRARY_PATH is searched at link time.
caveat from comments:
When linking libraries with ld (instead of gcc or g++), the LIBRARY_PATH or LD_LIBRARY_PATH environment variables are not read.
When linking libraries with gcc or g++, the LIBRARY_PATH environment variable is read (see documentation "gcc uses these directories when searching for ordinary libraries").
Since I link with gcc why ld is being called, as the error message suggests?
gcc calls ld internally when it is in linking mode.
LIBRARY_PATH is used by linker (ld)
LD_LIBRARY_PATH is used by loader (ld.so)

Linking .a files in Mac OSx

I have added libxxx.a in /usr/lib but when I perform otool -L myproject.so, libxxx.a was not included in the list of libraries. I have also included libxxx.a in my build file so I was thinking that I have successfully added it.
How can I like .a file?
I didn't have a problem with .dylib files though.
otool won't show the static library as they are included within the executable binary (a .dylib in this case). This is because static libraries are a collection of object (.o) files and it's pretty much the same as adding file1.o ... fileN.o to the linker command line and you can't see object files from otool either.
One way to check that your static library is part of the executable (other than it successfully linking) is to use the nm command which lists symbols. Providing the executable binary is not stripped, you would do something like:
$ nm /path/to/libLibrary.dylib | grep aClassOrFunctionInStaticLibrary
and the symbol being searched should have the letter t next to it, to indicate that it's part of the executable text section.
Also as mentioned by #PaulR, /usr/lib is part of the operating system and you should not add files there; use /usr/local/lib instead as /usr/local is designed for site-specific additions to the system and files there will survive an operating system update.

What does -lfoo link with if both libfoo.a and libfoo.so are on the library path?

Using gcc and gnu ld, I am compiling a binary (a shared object in this case) that depends on a library, let's say libfoo. libfoo comes as both .a and .so files, in the same directory.
Can I know which is being used, the static or the dynamic?
GNU ld on my Linux (Ubuntu 12.10 box) will load the .so file first.
From man ld:
-l namespec
--library=namespec
Add the archive or object file specified by namespec to the list of
files to link. This option may be used any number of times. If
namespec is of the form :filename, ld will search the library path
for a file called filename, otherwise it will search the library
path for a file called libnamespec.a.
On systems which support shared libraries, ld may also search for
files other than libnamespec.a. Specifically, on ELF and SunOS
systems, ld will search a directory for a library called
libnamespec.so before searching for one called libnamespec.a. (By
convention, a ".so" extension indicates a shared library.) Note
that this behavior does not apply to :filename, which always
specifies a file called filename.
gcc (and ld, which it's really running behind the scenes) default to dynamic linking if it's available. You can check the output binary to see which it did, if you want.
If you want to force it to use the static library, you can use the -static link option.

LD_LIBRARY_PATH vs LIBRARY_PATH

I'm building a simple C++ program and I want to temporarily substitute a system supplied shared library with a more recent version of it, for development and testing.
I tried setting the LD_LIBRARY_PATH variable but the linker (ld) failed with:
/usr/bin/ld: cannot find -lyaml-cpp
I expected that to work because according to the ld man page:
The linker uses the following search
paths to locate required shared
libraries: ... For a native linker,
the contents of the environment variable
"LD_LIBRARY_PATH"...
I then tried setting the LIBRARY_PATH, and that worked.
According to the GCC manual:
The value of
LIBRARY_PATH is a colon-separated list
of directories, much like PATH. When
configured as a native compiler, GCC
tries the directories thus specified
when searching for special linker
files, if it can't find them using
GCC_EXEC_PREFIX. Linking using GCC
also uses these directories when
searching for ordinary libraries for
the -l option (but directories
specified with -L come first).
As the (GCC) manual suggests, LIBRARY_PATH works because I link with GCC.
But..
Since I link with gcc why ld is
being called, as the error message
suggests?
What's the point of
having two variables serving the same
purpose? Are there any other
differences?
LIBRARY_PATH is used by gcc before compilation to search directories containing static and shared libraries that need to be linked to your program.
LD_LIBRARY_PATH is used by your program to search directories containing shared libraries after it has been successfully compiled and linked.
EDIT:
As pointed below, your libraries can be static or shared. If it is static then the code is copied over into your program and you don't need to search for the library after your program is compiled and linked. If your library is shared then it needs to be dynamically linked to your program and that's when LD_LIBRARY_PATH comes into play.
LD_LIBRARY_PATH is searched when the program starts, LIBRARY_PATH is searched at link time.
caveat from comments:
When linking libraries with ld (instead of gcc or g++), the LIBRARY_PATH or LD_LIBRARY_PATH environment variables are not read.
When linking libraries with gcc or g++, the LIBRARY_PATH environment variable is read (see documentation "gcc uses these directories when searching for ordinary libraries").
Since I link with gcc why ld is being called, as the error message suggests?
gcc calls ld internally when it is in linking mode.
LIBRARY_PATH is used by linker (ld)
LD_LIBRARY_PATH is used by loader (ld.so)

Questions on GCC linker

Apologize because for the moment I don't have the environment to experiment and sort out the following questions myself:
1) Let's say I have four library files: libmylib_super.a and libmylib_super.so, mylib_dumb.a and mylib_dumb.so. While specifying libraries to link to, what are the differences between the following approaches:
A) -l:libmylib_super.a
B) -llibmylib_super
C) -lmylib_super
D) -lmylib_dumb
2) Definition of -static from man page:
On systems that support dynamic
linking, this prevents linking with
the shared libraries. On other
systems, this option has no effect.
Does this linker option have anything to do with question #1? Or... by any chance will they interfere with each other?
Thanks.
--- edited 2009-12-28 ---
I just got my environment up and experimenting a bit, by linking to Boost date_time library. Say I have three library files: libboost_date_time-mt-d.a, libboost_date_time-mt-d.so.1.41.0, libboost_date_time-mt-d.so -> libboost_date_time-mt-d.so.1.41.0 (sym link).
A.1) -l:libboost_date_time-mt-d.a ==> linking OK, binary works even without the library file.
A.2) -l:libboost_date_time-mt-d.a with -static ==> linking error /usr/bin/ld: cannot find -lm
C.1) -lboost_date_time-mt-d ==> linking OK, binary works but requires the shared library file.
C.2) -lboost_date_time-mt-d with -static ==> linking error /usr/bin/ld: cannot find -lm
Any idea about the error in A.2 and C.2?
Additionally, while running the program in C.1, it seems searching the shared library file with name libboost_date_time-mt-d.so.1.41.0 but not the libboost_date_time-mt-d.so. Wouldn't that be inconvenient if the program is running on a system without the exact version of library? What could be the practical way to handle the version while using shared library?
According to the manual,
A) searches the library path for a file exactly named libmylib_super.a (the search first for a shared library behavior doesn't apply)
B) searches the library path for a file named liblibmylib_super.so first then liblibmylib_super.a or only searches for a file named liblibmylib_super.a if -static is used -- note it's the linker that adds the lib prefix and the file extension
C) searches the library path for a file named libmylib_super.so first then libmylib_super.a or only searches for a file named liblibmylib_super.so if -static is used
D) see C)
Note that B) won't work because it's the linker that should add the lib prefix to the library name.
Note that D) won't work because your mylib_dumb doesn't follow the naming convention.
See the GNU Linker Manual:
-l namespec
--library=namespec
Add the archive or object file specified by namespec to the list of files to link. This option may be used any number of times. If namespec is of the form :filename, ld will search the library path for a file called filename, otherwise it will search the library path for a file called libnamespec.a.
On systems which support shared libraries, ld may also search for files other than libnamespec.a. Specifically, on ELF and SunOS systems, ld will search a directory for a library called libnamespec.so before searching for one called libnamespec.a. (By convention, a .so extension indicates a shared library.) Note that this behavior does not apply to :filename, which always specifies a file called filename.
The linker will search an archive only once, at the location where it is specified on the command line. If the archive defines a symbol which was undefined in some object which appeared before the archive on the command line, the linker will include the appropriate file(s) from the archive. However, an undefined symbol in an object appearing later on the command line will not cause the linker to search the archive again.
See the -( option for a way to force the linker to search archives multiple times.
You may list the same archive multiple times on the command line.
This type of archive searching is standard for Unix linkers. However, if you are using ld on AIX, note that it is different from the behaviour of the AIX linker.

Resources