Is it possible to create a shared library which is itself statically linked, i.e. it does not depend on other shared libraries?
Let me be a little bit more concrete..
I want to create a shared library, say mylib.so, which makes use of some other special libraries (in my case its intel mkl and openMP). Since I have installed these libraries I can build mylib.so and include it in other programs without any problem.
However, if I want to use the library (or the executables including it) on another machine I first have to install all the intel stuff. Is there a way to avoid this? My first try was to add the option -static when building mylib.so but this doesn't seem to do anything..
I'm using icc..
Is it possible to create a shared library which is itself statically linked, i.e. it does not depend on other shared libraries?
Not on Linux, not when using GLIBC (your shared library will always depend on at least ld-linux*.so*).
I want to create a shared library, say mylib.so, which makes use of some other special libraries (in my case its intel mkl and openMP).
There is no problem1 statically linking Intel MKL and OpenMP libraries into mylib.so -- you just don't want to depend on these libraries dynamically (in other words, you are asking for an impossible thing which you don't actually need).
To do so, you need two things:
Link mylib.so with archive versions of the libraries you don't want to depend on dynamically, e.g. gcc -o mylib.so -shared mylib.c .../libmkl.a ...
The libraries which you want to statically link into mylib.so must have been built with position-independent code (i.e. with -fPIC flag).
Update:
What if the archived version isn't available?
Then you can't link it into your library.
Eg I'm using intel/oneapi/intelpython/latest/lib/libstdc++.so and there is no corresponding .a file..
This is a special case: you wouldn't want to link that version into your library even if it were available.
Instead, your program should use the version installed on the target system.
Having two separate versions of libstdc++ (e.g. one statically linked, and the other dynamically linked) into a single process will end very badly -- either with a crash, or with silent stack or heap corruption.
1 Note that linking in somebody else's library and distributing it may have licensing implications.
Related
I am trying to cross compile some dependency libs for RaspberryPi target system, and host system is Linux with GCC compiler. For example, let's say that one of those libs has dependency on linkage stage and being linked with one of the system's static or dynamic libraries.
How this case is resolved by linker? (Because those .a or .so files can be different on target system, so probably program on RaspberryPi will crash in this case). How to make it work in a right way?
The build environment that the cross-compiler provides is more accurately described as a cross-toolchain. It needs to provide everything you need: Not just the compiler, but also the assembler, linker, and all run-time support libraries. That includes a C library (maybe glibc, maybe something else), the GCC run-time library (libgcc and libgcc_s), and the C++ run-time library (libstdc++). But the build environment also needs copies of all the libraries your software needs to build, typically both header files and static libraries or dynamic shared objects for the target. In particular, you cannot use the installed header files on the host because they might have the wrong definitions and declarations for the target.
Some programmers simply copy their dependencies (which are not system libraries) into their source tree, so that the cross-build environment can stay minimal. But then these libraries have to be tracked and updated as part of the project, which can be cumbersome.
I am working on a product that is composed of multiple C++ executables and libraries that have various dependencies on one another. I am building them all with GCC and -fsanitize-address.
From what I understand, if I want to use address sanitizer with a library I have to build it as a shared object (which is the default option for GCC). Because of this, I thought the best option would be to build address sanitizer statically with -static-libasan for the executables and build it dinamically for the libraries. However, when I do that I get a link error when building one of the C++ executables:
==10823==Your application is linked against incompatible ASan runtimes
This makes me think that static and dynamic version of address sanitizer cannot be mixed with GCC, am I right? I was not able to find any information about this on the sanitizers GitHub page.
TLDR:
If you use GCC/Clang and both main executable and shlibs are sanitized, you don't need to do anything special - just stick with default -fsanitize=address.
If you use GCC and only shlibs are sanitized, again keep using -fsanitize=address and additionally export LD_PRELOAD=$(gcc -print-file-name=libasan.so) when running your application.
If you use Clang and only shlibs are sanitized, compile/link with -fsanitize-address -shared-libasan and additionally export LD_PRELOAD=$(clang -print-file-name=libclang_rt.asan-x86_64.so) when running the app.
Now some explanations. Originally Asan existed only in Clang which by default used (and still uses) -static-libasan. When it was ported to GCC, GCC developers decided that shared runtime is preferred (e.g. because it allows one to sanitize just one shared library and keep main executable unsanitized e.g. sanitize Python module without recompiling python.exe, see wiki for other examples). Both approaches are binary incompatible so you can't link part of your applications with static runtime and part with dynamic runtime.
Roughly
GCCs -fsanitize=address is equivalent to Clangs -fsanitize=address -shared-libasan (and -shared-libasan is second-class citizen in Clang so not as well supported)
Clangs -fsanitize=address is equivalent to GCCs -fsanitize=address -static-libasan (and again, -static-libasan is second-class citizen in GCC so has some issues)
As a side note, for other GCC/Clang Asan differences see this helpful wiki.
Background:
I am trying to provide a library wrapped within an interface, to be used in a third party application (MATLAB).
My problem is that both my library and the third party application rely on Boost libraries.
To make things even worse, not only they rely on two different versions of Boost, but my library is built against a patched version of Boost, and they are statically linked. Third party libraries are instead dynamically linked against Boost.
This is how I link (simplified output of make -n)
g++ -fPIC ${CPPFLAGS} -shared -lc -ldl ... -Wl,-h,mylib.so -o mylib.so \
${MYSTATICLIBS} \
-L${MATLABLIBSPATH} -lmx -lmex \
-L${MYBOOSTVERSIONPATH} -lboost_thread-gcc49-mt-1_52 ...
When I run the library fron within MATLAB then, the wrong boost functions are called from my internal functions (and everything crashes).
My naive intuition was that since i statically linked my Boost, all calls to them from within my library should point to my Boost, while calls from within libmx.so or libmex.so would point to their dynamically loaded dependencies. However this proves my naive intuition was very wrong.
I found different hints here on SO and around the web describing how linking order may break similar situations, however everything describes either all static or all dynamic linking of conflicting symbols.
Suggested solutions are
using -Bsymbolic -Bsymbolic-functions
upgrade my library to the same boost version
changing linking order ( first boost, then -lmx -lmex )
sell all my earthly possessions, buy a bar on the beach somewhere hot and try making a living out of it
other??
Can someone explain if and why these solution can work?
( my other problem is, I am not able to test with MATLAB myself, as I have no access to the machines on which it is installed. this is why I'm also asking "if" the proposed solutions would work)
First of all, welcome to the wonderful world of Linux symbol interposition ;)
I think what happens is your library exports all symbols from Boost library it has been linked with (can be verified with readelf --dyn-syms -W). At runtime, dynamic linker will note that and instead rebind symbols to a different version of Boost (because it happens to be loaded earlier and so overrides symbols used by your library).
What you want to do is to tell gcc that you do not want to export Boost symbols or you do not want them to be interposable at runtime. There are several solutions for this, the best being to compile and link with -fvisibility=hidden (note that Boost will also need to be linked with this flag). This will prevent any functions from being exported from your library unless you explicitly mark them with __attribute__((visibility("hidden"))) (you'll need to annotate library interface functions of course).
Another option is -Bsymbolic - this would not prevent spurious Boost exports from your library but would at least prevent dynamic linker from overriding them with random versions of Boost functions which happen to be loaded by other libraries.
Now to remaining questions.
upgrade my library to the same boost version
You could do that but this may break if 3-rd party upgrades their Boost at some point.
changing linking order ( first boost, then -lmx -lmex )
Does not make much sense... Have you tried?
I have a question about GCC, ThreadSanitizer and the use of archives, shared libraries and PIE and PIC.
I've been reading about as best I can all morning, but I just can't find useful, clear information on-line.
I understand what PIC does. I think I understand that PIE is if you like an optimized version of PIC, which is only for executables.
Now come the questions...
Can I compile an executable with PIC, rather than PIE?
If I compile a shared library (.so) with PIC, must I then use PIC with any executable which uses that library, rather than PIE?
If I compile an archive (.a), can I use PIE? (I have read -static and -pie should not be used together, which implies not).
I'm using ThreadSanitizer. This requires PIC (and perhaps PIE is okay too - but as you can see, I'm not clear about this). I have a library, which can be compiled as an archive (.a) or a shared library (.so). The library needs to use ThreadSanitizer. However, the binary which uses it also needs to use ThreadSanitizer (as it has some code of its own which needs checking).
The library when built as a shared library in fact fails to link when used wih ThreadSanitizer - I think the link is failing to link to libtsan (but this is I suspect not a real library, but a bunch of compiler instrincs built into GCC). This is almost certainly me getting something wrong somewhere.
What I really want to do is use an archive (.a) since the binary is a test programme and should be able to compile without the library being installed (so users can conveniently check/test the library - the makefile for the test binary has a hard coded path to the archive binary).
If I can use PIE with archives (.a), then I'd PIE the library and the test binary. If PIE cannot be used with archives, then I think I need to use PIC with both the library and test binary. I don't want to use a shared library at all, since ThreadSanitizer uses TLS (thread local store) heavily and shared libraries with PIC have absolutely terrible TLS performance.
Ultimate functionality of pic and pie are same but in gcc -fpic is used to create shared libraries whereas -fpie is used to for exes.
No you cannot use pic for an executable
Shared libraries don't care where pie (PIE just make the exec position independent) or normal execs are using it. It is dynamic linker's (ld.so) job to link the shared library.
No you can't make a exec position independent while using a static library. When you link a exec with static library it creates a dependency on exec and the symbols have to be resolved at compile-time.So in short you can't.
I've to go answer rest after office
I've used zlib for ages and never thought about the fact that it is named slightly unconventionally. While most libraries on Linux follow the naming convention of lib<name>.so for shared objects and lib<name>.a for archives, zlib is named zlib.so/zlib.a. My question is: how does gcc/ld know to look for zlib.so when I use -lz as a link flag?
I understand that for linking, gcc invokes ld, which searches for libraries in certain default paths and any path specified with -L, and it appends the lib and .so or .a. parts as necessary. Oddly, gcc's manual page for linking options only mentions that the linker can find archives; there is no mention of the .so extension. The man page for ld at least mentions both extensions, but still only mentions searching by prepending lib to the specified library name. How does ld know to add the lib after the z for zlib? I've never seen this happen to another library.
gcc has several different methods for linking libraries, shared or static. If you specify -lz, gcc is going to look for libz.so (possibly with some version bits between the libz and the .so, but the important part is the file name will start with libz and end with .so), or for libz.a (again, possibly with version info) if you are compiling statically, or as a fallback if the shared library does not exist. If you specify -lzlib it will look for libzlib.so (which is not the standard name - the package is often named zlib, but the library itself is libz). Another way of linking would be to not use the -l<lib> option, and just specify /path/to/zlib.so or -L /path/to zlib.so (or zlib.a if you want). In this case, the library doesn't have to have the lib prefix, but you would have to explicitly provide any version info, unless provisions are made for a symbolic link or something similar to provide the literal name zlib.so.
Applications can also load shared libraries at runtime via dlopen() and it's other associated functions, in which case the library can also be named whatever you want it to be (this doesn't work for static libraries, of course).
So, if the library you are looking at is actually called zlib.so, then it is not being found by gcc ... -lz, unless it just happens to be a symbolic link to libz.so (or vice versa, in which case gcc is really just using libz.so, which happens to have the same content as your zlib.so). However gcc might be using it if the build process explicitly names the library in the link stage (not using -l<lib>) or if your application loads it via dlopen() (but in that case, it's not really linked to your program - it's just loaded at run time).