Compiling an executable with gccgo creates a dynamically linked executable, linked against libgo.so. I am using gccgo-11 on ubuntu 20.04, pulled from apt.
The resulting executable is small, as expected. About 50Kb for a basic hello world.
As soon as I start using external modules (in this case viper and gorilla/websocket) the executable size starts creeping back to the original, statically linked size (~9MB).
If I compile with go alone (not using gccgo) the exeutable is about 9MB also.
This look to me, like the modules are compiled separately, and statically, and then linked, statically into the final executable.
Am I overlooking something here? Is there some option that must be enabled?
I would expect the module dependencies to also be compiled by gccgo, and linked dynamically against the shared libgo.so.
Related
Is it possible to create a shared library which is itself statically linked, i.e. it does not depend on other shared libraries?
Let me be a little bit more concrete..
I want to create a shared library, say mylib.so, which makes use of some other special libraries (in my case its intel mkl and openMP). Since I have installed these libraries I can build mylib.so and include it in other programs without any problem.
However, if I want to use the library (or the executables including it) on another machine I first have to install all the intel stuff. Is there a way to avoid this? My first try was to add the option -static when building mylib.so but this doesn't seem to do anything..
I'm using icc..
Is it possible to create a shared library which is itself statically linked, i.e. it does not depend on other shared libraries?
Not on Linux, not when using GLIBC (your shared library will always depend on at least ld-linux*.so*).
I want to create a shared library, say mylib.so, which makes use of some other special libraries (in my case its intel mkl and openMP).
There is no problem1 statically linking Intel MKL and OpenMP libraries into mylib.so -- you just don't want to depend on these libraries dynamically (in other words, you are asking for an impossible thing which you don't actually need).
To do so, you need two things:
Link mylib.so with archive versions of the libraries you don't want to depend on dynamically, e.g. gcc -o mylib.so -shared mylib.c .../libmkl.a ...
The libraries which you want to statically link into mylib.so must have been built with position-independent code (i.e. with -fPIC flag).
Update:
What if the archived version isn't available?
Then you can't link it into your library.
Eg I'm using intel/oneapi/intelpython/latest/lib/libstdc++.so and there is no corresponding .a file..
This is a special case: you wouldn't want to link that version into your library even if it were available.
Instead, your program should use the version installed on the target system.
Having two separate versions of libstdc++ (e.g. one statically linked, and the other dynamically linked) into a single process will end very badly -- either with a crash, or with silent stack or heap corruption.
1 Note that linking in somebody else's library and distributing it may have licensing implications.
I am trying to understand how linking and loading work. My understanding is that the Unix program "ld" contains both linking and loading functionality. When gcc is invoked, after preprocessing, compiling, and assembling, the linker is called which links all object files and .a files into an executable, along with minimal instructions for how shared libraries should be "connected" (what is the correct terminology here?) at runtime. This linker is ld.
At runtime, my understanding is that the executable is loaded into memory, although I'm not sure how. My specific questions are as follows:
1) Are shared object files being "linked" at compile time, or is there another word for what is happening?
2) At runtime, is ld being called for a second time? How can I see proof of this for my executable (on Linux and on MacOS)?
3) Are shared object files being "linked" at runtime, or is there another word for the process when shared objects are read from the location in LD_LIBRARY_PATH at runtime?
Thanks!
Is ld called at both compile time and runtime?
No: ld is not called at either compile or runtime.
When gcc is invoked, after preprocessing, compiling, and assembling, the linker is called which links all object files and .a files into an executable
Most moderately complicated programs use separate compilation and linking steps.
At compilation, a set of relocatable object files is produced (preprocessing, compilation and assembling are invoked at that step). Optionally the .o files are archived into an archive library (libsomething.a).
Then a link step is performed (often this is called "static linking", to differentiate this step from "dynamic loading" that will happen at runtime), producing an executable, or a shared library. Only at this step is /usr/bin/ld is invoked. On Linux, ld is part of the binutils package.
along with minimal instructions for how shared libraries should be "connected"
The linker records which shared libraries are required at runtime, and possibly which versions of libraries or symbols are required.
It also records which runtime loader should be used to load the required shared libraries.
At runtime, my understanding is that the executable is loaded into memory, although I'm not sure how.
The kernel loads executable into memory, and checks whether runtime loader was requested at static link time. If it was, the dynamic loader is also loaded into memory, and execution control is passed to it (instead of the main executable).
It is then the job of the dynamic loader to examine the executable for instructions on which other libraries are required, check whether correct versions can be found, loading them into memory, and arranging things such that symbol resolution will work between the main executable and the shared libraries. This is the runtime loading step, often also called dynamic linking.
The dynamic loader can be part of the OS, but on Linux it's part of libc (GLIBC, uClibc and musl each have their own loader).
No. ld is linking as in creating a library or exe, ld*.so is the loading part. Also ld*.so is part of the OS, not the gcc suite afaik. ld is generally part of (GNU) binutils on a gcc based system (but e.g. usually LLVM lld in a LLVM based system)
ld*.so is ld-linux-{arch}.so.2 on Linux and /libexec/ld-elf.so on e.g. FreeBSD.
I am building Go code that uses CGo heavily and this code must be compiled into a shared or static library (static is highly preferred). (code for reference)
It all works just fine on Linux and Mac, but on Windows it fails on linker stage either saying that all 4 modes (c-shared, shared, c-archive, archive) are not available or if invoke go tool link -shared manually complains about missing windows specific instructions.
My understanding is that all I need to build usable lib.a is to compile everything I will use into object files (*.o) and then put it through ar to produce usable static library.
Now the question is whether I can completely skip Go's linker and based on prepared .o files create .a manually?
How would I go about doing that if that is even possible?
Looks like gcc on windows is unable to automatically discover necessary shared libraries. The problem was caused by GCC and not by Go.
Although for compiling Go I had to use self-compiled master tip as current release (1.6.2) does not support shared/static libraries on windows/amd64.
Manually feeding gcc with each shared library (ntdll, winmm etc) in default location (C:\Windows\SysWOW64) has fixed the problem.
In one of the Go seminars, the lecturer said that when he compiled the Go application, statically linked, the size of the resulting binary was about 600 MB big, but when he compiled the same app with dynamic linking, the resulting binary turned into 10 MB.
I'm not sure what he is talking about, can dynamic vs. static linking during compilation make that difference in space of binary, and do I have control over that?
By default Go uses static linking so everything (your code and packages sources) compiles in one big binary.
Since Go 1.5 released you can compile Go shared library using -buildmode=shared option for go build or go install. Then you can compile your app binary with -linkshared flag. Details can be found here.
Ofcourse, if you link packages dynamically your binary size will be less than with static linking, but total application size will not be reduced because you just "put your code in other place".
So dynamic linking makes sense only if you need to share the same packages between different apps.
I have a question about GCC, ThreadSanitizer and the use of archives, shared libraries and PIE and PIC.
I've been reading about as best I can all morning, but I just can't find useful, clear information on-line.
I understand what PIC does. I think I understand that PIE is if you like an optimized version of PIC, which is only for executables.
Now come the questions...
Can I compile an executable with PIC, rather than PIE?
If I compile a shared library (.so) with PIC, must I then use PIC with any executable which uses that library, rather than PIE?
If I compile an archive (.a), can I use PIE? (I have read -static and -pie should not be used together, which implies not).
I'm using ThreadSanitizer. This requires PIC (and perhaps PIE is okay too - but as you can see, I'm not clear about this). I have a library, which can be compiled as an archive (.a) or a shared library (.so). The library needs to use ThreadSanitizer. However, the binary which uses it also needs to use ThreadSanitizer (as it has some code of its own which needs checking).
The library when built as a shared library in fact fails to link when used wih ThreadSanitizer - I think the link is failing to link to libtsan (but this is I suspect not a real library, but a bunch of compiler instrincs built into GCC). This is almost certainly me getting something wrong somewhere.
What I really want to do is use an archive (.a) since the binary is a test programme and should be able to compile without the library being installed (so users can conveniently check/test the library - the makefile for the test binary has a hard coded path to the archive binary).
If I can use PIE with archives (.a), then I'd PIE the library and the test binary. If PIE cannot be used with archives, then I think I need to use PIC with both the library and test binary. I don't want to use a shared library at all, since ThreadSanitizer uses TLS (thread local store) heavily and shared libraries with PIC have absolutely terrible TLS performance.
Ultimate functionality of pic and pie are same but in gcc -fpic is used to create shared libraries whereas -fpie is used to for exes.
No you cannot use pic for an executable
Shared libraries don't care where pie (PIE just make the exec position independent) or normal execs are using it. It is dynamic linker's (ld.so) job to link the shared library.
No you can't make a exec position independent while using a static library. When you link a exec with static library it creates a dependency on exec and the symbols have to be resolved at compile-time.So in short you can't.
I've to go answer rest after office