I'm building a simple C++ program and I want to temporarily substitute a system supplied shared library with a more recent version of it, for development and testing.
I tried setting the LD_LIBRARY_PATH variable but the linker (ld) failed with:
/usr/bin/ld: cannot find -lyaml-cpp
I expected that to work because according to the ld man page:
The linker uses the following search
paths to locate required shared
libraries: ... For a native linker,
the contents of the environment variable
"LD_LIBRARY_PATH"...
I then tried setting the LIBRARY_PATH, and that worked.
According to the GCC manual:
The value of
LIBRARY_PATH is a colon-separated list
of directories, much like PATH. When
configured as a native compiler, GCC
tries the directories thus specified
when searching for special linker
files, if it can't find them using
GCC_EXEC_PREFIX. Linking using GCC
also uses these directories when
searching for ordinary libraries for
the -l option (but directories
specified with -L come first).
As the (GCC) manual suggests, LIBRARY_PATH works because I link with GCC.
But..
Since I link with gcc why ld is
being called, as the error message
suggests?
What's the point of
having two variables serving the same
purpose? Are there any other
differences?
LIBRARY_PATH is used by gcc before compilation to search directories containing static and shared libraries that need to be linked to your program.
LD_LIBRARY_PATH is used by your program to search directories containing shared libraries after it has been successfully compiled and linked.
EDIT:
As pointed below, your libraries can be static or shared. If it is static then the code is copied over into your program and you don't need to search for the library after your program is compiled and linked. If your library is shared then it needs to be dynamically linked to your program and that's when LD_LIBRARY_PATH comes into play.
LD_LIBRARY_PATH is searched when the program starts, LIBRARY_PATH is searched at link time.
caveat from comments:
When linking libraries with ld (instead of gcc or g++), the LIBRARY_PATH or LD_LIBRARY_PATH environment variables are not read.
When linking libraries with gcc or g++, the LIBRARY_PATH environment variable is read (see documentation "gcc uses these directories when searching for ordinary libraries").
Since I link with gcc why ld is being called, as the error message suggests?
gcc calls ld internally when it is in linking mode.
LIBRARY_PATH is used by linker (ld)
LD_LIBRARY_PATH is used by loader (ld.so)
Related
I'm building a simple C++ program and I want to temporarily substitute a system supplied shared library with a more recent version of it, for development and testing.
I tried setting the LD_LIBRARY_PATH variable but the linker (ld) failed with:
/usr/bin/ld: cannot find -lyaml-cpp
I expected that to work because according to the ld man page:
The linker uses the following search
paths to locate required shared
libraries: ... For a native linker,
the contents of the environment variable
"LD_LIBRARY_PATH"...
I then tried setting the LIBRARY_PATH, and that worked.
According to the GCC manual:
The value of
LIBRARY_PATH is a colon-separated list
of directories, much like PATH. When
configured as a native compiler, GCC
tries the directories thus specified
when searching for special linker
files, if it can't find them using
GCC_EXEC_PREFIX. Linking using GCC
also uses these directories when
searching for ordinary libraries for
the -l option (but directories
specified with -L come first).
As the (GCC) manual suggests, LIBRARY_PATH works because I link with GCC.
But..
Since I link with gcc why ld is
being called, as the error message
suggests?
What's the point of
having two variables serving the same
purpose? Are there any other
differences?
LIBRARY_PATH is used by gcc before compilation to search directories containing static and shared libraries that need to be linked to your program.
LD_LIBRARY_PATH is used by your program to search directories containing shared libraries after it has been successfully compiled and linked.
EDIT:
As pointed below, your libraries can be static or shared. If it is static then the code is copied over into your program and you don't need to search for the library after your program is compiled and linked. If your library is shared then it needs to be dynamically linked to your program and that's when LD_LIBRARY_PATH comes into play.
LD_LIBRARY_PATH is searched when the program starts, LIBRARY_PATH is searched at link time.
caveat from comments:
When linking libraries with ld (instead of gcc or g++), the LIBRARY_PATH or LD_LIBRARY_PATH environment variables are not read.
When linking libraries with gcc or g++, the LIBRARY_PATH environment variable is read (see documentation "gcc uses these directories when searching for ordinary libraries").
Since I link with gcc why ld is being called, as the error message suggests?
gcc calls ld internally when it is in linking mode.
LIBRARY_PATH is used by linker (ld)
LD_LIBRARY_PATH is used by loader (ld.so)
I have ATLAS library "libatlas.so" in an unstandard directory "/opt/apps/gcc4_7/atlas/3.10.2/lib/"
I put the path along with other search paths in environment variable
LD_LIBRARY_PATH=/opt/apps/gcc4_7/atlas/3.10.2/lib:and_others
Then link my code with it as follow:
g++ mycode.o -L$LD_LIBRARY_PATH -latlas
It then say "ld: cannot find -latlas"
But if I add the path again as follow:
g++ mycode.o -L$LD_LIBRARY_PATH -L/opt/apps/gcc4_7/atlas/3.10.2/lib/ -latlas
I have no issues.
I cannot think of any reasons this could happen. Is it because the $LD_LIBRARY_PATH has too many paths in it?
According to the manual page for ld (and gcc used for linking by extension), if a -L option appears on the command line, it applies to all libraries specified by -l and takes precedence over the default search locations. However, that is not working in my link step. I have this on the command line:
-L /users/me/mylib -lpcre -lz
and /users/me/mylib contains (copies) of libpcre.so and libz.so
These libraries exist in other locations on the system (although not necessarily the same versions) and what I see (with ldd on Linux and otool on Mac) is a path that references the libraries in those locations. Some of those locations are on the LD_LIBRARY_PATH (which I cannot control in the build environment I am running in) and it appears that somehow those locations are being picked up in preference to my explicit setting with -L.
Just to be clear, this a link step problem and not a runtime problem. There is a lot of info on the web on how to affect/override library locations when executing and I am familiar with all that. In some sense what I am trying to do with the -L is create a completely specified setup. I know I can fix things up with install_name_tool on MacOS but I'd really like to understand why -L isn't doing what it claims to.
One thing I learned using gcc -Wl,-v is that gcc appears to forward all the LD_LIBRARY_PATH directories to ld. However, it places them after the ones explicitly listed by me and man ld says they are searched in order they appear on the line.
Just to be clear, this a link step problem and not a runtime problem.
From what you describe as the problem, I don't think you are right about this - it sounds like a runtime problem for which you are (justifiably) looking for a solution that you can employ while linking that will solve the problem you have at runtime.
The reason I say it does not appear to be a problem with linking is that it sounds like your linking is working as it is intended. LD (or GCC) are not complaining about the linking, and your linked executables are being produced just fine. The issue you are having is that when you subsequently go to run those executables, the loader is finding libraries other than the ones you intend. The purpose of the -L flag during linking is to let the linker know where it can find suitable libraries to use in preparing the linked binary. That is completely separate from where the loader will search for the required libraries at runtime.
As you say, you are already aware that there are ways you could employ at runtime (such as changing LD_LIBRARY_PATH) that would avoid the issue by changing the set of paths that the loader searches for libraries, but you'd rather not have to do that because for whatever reason you won't necessarily have control over the runtime environment, which is fair enough.
Luckily, there is a facility that I believe will get you what you want. Take a look at the ld option called -rpath (see the GNU ld man page for full documentation). Basically, if you add paths during linking using the -rpath option, those paths gets stored in the linked executable as preferred locations to find the libraries at runtime, in much the same way they would be searched if included in LD_LIBRARY_PATH. This should work on Linux or Mac OS X (at least since 10.5).
Passing the -rpath option to ld via gcc requires using the -Wl option to pass the flag through. To obtain an ld command line that contains ld -rpath /custom/path/to/libs requires a gcc invocation something like: gcc -Wl,-rpath,/custom/path/to/libs
In short, try replacing what you currently have: -L/users/me/mylib -lpcre -lz
With: -L/users/me/mylib -Wl,-rpath,/users/me/mylib -lpcre -lz
The resulting executable (or library) will then have /users/me/mylib stored as the place to go to find libraries, and it should find libpcre.so and libz.so there without needing to control LD_LIBRARY_PATH.
I have some fortran programs that would not compile in old versions of gfortran. I have to run multiple instances of this program and am using another system (a cluster system) which has centos5_x64 with gcc-4.1 !!
Therefore I had to build new version of gcc; I built both gcc-4.8.3 and gcc-4.9.2 in my home folder. These programs use hdf5 and so the latter also has to be compiled using the same compiler. I tested the fortran programs after removing the hdf5 dependency on both gfortran-4.8.3 and 4.8.9 and they get built and execute properly. I also tested simple C/C++ programs (with basic i/o and arithmetic) with the new gcc/g++(s); they work fine. Before compiling hdf5 libraries I set these environment variables:
PATH=<GCCPATH>:$PATH
LD_LIBRARY_PATH=<GCCLIB>
LD_RUN_PATH=<GCCLIB>
export PATH
export LD_LIBRARY_PATH
export LD_RUN_PATH
HDF5 specific instructions
4.3.7. Specifying other libraries and headers
Configure searches the standard places (those places known by the
systems compiler) for include files and header files. However,
additional directories can be specified by using the CPPFLAGS
and/or LDFLAGS variables:
$ CPPFLAGS=-I/home/robb/include \
LDFLAGS=-L/home/robb/lib \
LDFLAGS=-L<GCCLIB>
CPPFLAGS=-I<GCCINCLUDE>
then configured as:
./configure --prefix=$HOME/HDF5 --enable-fortran
During make I get this error:
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
This happens with both versions of gcc but not with the versions installed in the standard location (This happened in another system too which had gcc 4.4 installed). With general searching I got to know that this error is associated with the absence of a main() and in such cases -c flag has to be passed. However all examples of this error were from people's personal scripts and were not for libraries. Please let me know if I am missing something.
Upgrading the system OS is not a choice as of now.
As indicated in the comment by doqtor, I modified the Makefile post configure. It seems that CPPFLAGS and LDFLAGS were not set.
After setting the right CPPFLAGS and LDFLAGS in the Makefile, the compilation happened successfully.
I am currently working on a simple data synchonizer in a mixture of Fortran and C/C++ by using OpenMPI libraries. The synchonizer compiles and links correctly, as far as I can see:
f95 -o fortran_mpi_test *.o -L/usr/lib/gcc/x86_64-redhat-linux/4.1.1/
-L/usr/lib64/openmpi/1.4-gcc/lib/ -lmpi -lmpi_cxx -lstdc++
But when I execute the resulting executable on the same machined I get an error stating that one of the shared libraries is not found. That is confirmed by ldd.
Nevertheless the missing library libmpi_cxx.so.0 is located in one of the specified folders.
Could anyone give me a hint what I could have done wrong?
Check your environment variables. If your LIBRARY_PATH, LD_LIBRARY_PATH or similar vars have gotten out of sync or set to silly values you might not be searching the same directories for static libraries as you do for dynamics.
Also check out the ld.so manpage