Why is --rpath being superseded when building shared library? - gcc

Question

I am working on a CentOS machine. I have a file (test.c) that I'm compiling into a shared library.
test.c:
#include <stdio.h>
#include <stdlib.h>
#include <quadmath.h>
int my_func(void){
__float128 r;
r = strtoflt128 ("1.2345678", NULL);
int * b = malloc(sizeof(int) * 5);
double a = 10*M_PI_2q;
printf("Hello world %f\n",a);
return 0;
}
To compile it :
$ gcc -c -fPIC test.c -o test.o
$ gcc -shared -Wl,--verbose -Wl,-soname,poo.so -Wl,-rpath,/opt/gcc/5.5.0/lib/ -o poo.so test.o -lquadmath
$ echo $LD_LIBRARY_PATH ### This shouldn't matter?
/opt/gcc/5.5.0/lib
From the verbose output from ld, it looks like it /opt/gcc/5.5.0/lib64/libquadmath.so.0 is found (which is what I want). I.e.
.
.
.
attempt to open /opt/gcc/5.5.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/libquadmath.so failed
attempt to open /opt/gcc/5.5.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/libquadmath.a failed
attempt to open /opt/gcc/5.5.0/bin/../lib/gcc/libquadmath.so failed
attempt to open /opt/gcc/5.5.0/bin/../lib/gcc/libquadmath.a failed
attempt to open /opt/gcc/5.5.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../lib64/libquadmath.so succeeded
-lquadmath (/opt/gcc/5.5.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/5.5.0/../../../../lib64/libquadmath.so)
.
.
.
However, when I look with ldd poo.so, it does not have the correct library. I.e.
$ ldd poo.so
linux-vdso.so.1 => (0x00007fffc0e8e000)
libquadmath.so.0 => /act/gcc-4.7.2/lib64/libquadmath.so.0 (0x00002aea8fd39000)
libc.so.6 => /lib64/libc.so.6 (0x00002aea8ff6f000)
libm.so.6 => /lib64/libm.so.6 (0x00002aea90303000)
/lib64/ld-linux-x86-64.so.2 (0x00002aea8f8f2000)
I am using gcc version 5.5.0 and would like poo.so to link to /opt/gcc/5.5.0/lib/libquadmath.so.0 instead of the 4.7.2 version.
I think this is related to the existence of files in /etc/ld.so.conf specifying the location of the older libraries. I.e.
$ cat /etc/ld.so.conf.d/gcc-4.7.2.conf
/act/gcc-4.7.2/lib64
/act/gcc-4.7.2/lib
The man page of ld (as I inquired here) is less than useful.
QUESTION : I specify -rpath when linking and building the shared library, yet I don't understand why it is finding the older gcc-4.7.2 version of the quadmath library. Why does the linker seem to prioritize the paths in /etc/ld.so.conf and ignore the -rpath option? How can I specify the library location at compile time and how do I fix this?

Answer 1

To understand the solution, first we must understand the incorrect thinking.
I did not realize that there two utilities doing linking. There is ld (which is used at compile time) and ld.so (which is used at run time). The man page of ld.so describes how libraries are found at run time. To summarize it searches :
a) Directories specified in DT_RPATH
b) Directories specified in LD_LIBRARY_PATH
c) Directories in DT_RUNPATH dynamic section of binary (if present)
d) Cache file /etc/ld.so.cache
e) The default path /lib and then /usr/lib
The man page of ld primarily confused me. It says under the -rpath option:
Add a directory to the runtime library search path. This is used when linking an ELF executable with shared objects
I thought that passing -rpath when building a shared library would
guarantee that it would find the library without needing to set
LD_LIBRARY_PATH. Wrong. This feature only seems to be useful
when building an executable, not a shared library. There is an -rpath-link option, but I couldn't figure out how to make it work.
You can see this by doing the following:
$ cat test2.c
#include <stdio.h>
#include <stdlib.h>
#include <quadmath.h>
int main(void){
__float128 r;
r = strtoflt128 ("1.2345678", NULL);
int * b = malloc(sizeof(int) * 5);
double a = 10*M_PI_2q;
printf("Hello world %f\n",a);
return 0;
}
$ export LD_LIBRARY_PATH=
$ which gcc
/opt/gcc/5.5.0/bin/gcc
$ gcc -Wl,-rpath,/opt/gcc/5.5.0/lib64 test2.c -lquadmath
$ ldd a.out
linux-vdso.so.1 => (0x00007fffff34f000)
libquadmath.so.0 => /opt/gcc/5.5.0/lib64/libquadmath.so.0 (0x00002b9c35e07000)
libc.so.6 => /lib64/libc.so.6 (0x00002b9c36069000)
libm.so.6 => /lib64/libm.so.6 (0x00002b9c363fd000)
/lib64/ld-linux-x86-64.so.2 (0x00002b9c35be5000)
$ readelf -a a.out | grep -i rpath
0x000000000000000f (RPATH) Library rpath: [/opt/gcc/5.5.0/lib64]
The way that the linker found the correct library was by utilizing the path to /opt/gcc/5.5.0/bin/gcc and looking at relative paths with respect to it. I checked this by setting export LD_LIBRARY_PATH= and excluding the rpath argument, i.e. gcc -shared -Wl,--verbose -Wl,-soname,poo.so -o poo.so test.o -lquadmath. Even then, it was able to link to the correct libquadmath.so.0 library.
The reason that ldd poo.so could not find the correct library was because it wanted the 64-bit version of the libquadmath.so.0 library IS NOT /opt/gcc/5.5.0/lib/libquadmath.so.0. That is the 32-bit version of the library. This is obvious once you inspect both versions of the library and see that the they are ELF64 and ELF32 respectively (e.g. readelf -a /opt/gcc/5.5.0/lib64/libquadmath.so.0 | grep Class).
Since I never specify the path to the correct library that it is looking for, ld.so by default looked at /etc/ld.conf.cache (which builds on `/etc/ld.conf.d/') to decide which directories to look in. This is why it found the 4.7.2 version of quadmath.
To get the correct library at runtime, I need to set export
LD_LIBRARY_PATH=/opt/gcc/5.5.0/lib64.
Summary, it was finding the correct library at compile time using ld (b/c it looked at the relative paths of gcc) but could not find the correct library at run time (b/c LD_LIBRARY_PATH was incorrectly set). The solution is :
export LD_LIBRARY_PATH=/opt/gcc/5.5.0/lib64. -rpath is not a valid option when building a shared library.

Answer 2

The path specified by --rpath option will be hard coded into binary you created. At running time, ld will search dependent libraries from those paths preferentially. This is determined by ld.so.
But when you compile program, The linker will search dependent libraries from LD_LIBRARY_PATH and those folder you specified by -L option. At compiling time, the path specified by --rpath do not take effect. This is determined by ld, the GNU linker.