I am building a fork of caffe (3D-caffe), on my university cluster which has a different builds of different versions of different software in /software/software/<package>/<version-compiler>/
When cmake executes this compilation step:
g++ -fPIC -Wall -Wno-sign-compare -Wno-uninitialized -O3 -DNDEBUG \
CMakeFiles/caffe.bin.dir/caffe.cpp.o -o caffe -rdynamic ../lib/libcaffe.so \
../lib/libproto.a /software/software/Boost/1.61.0-foss-2016a/lib/libboost_system.so \
/software/software/Boost/1.61.0-foss-2016a/lib/libboost_thread.so \
-lpthread -lglog /software/software/gflags/2.1.2-foss-2016a/lib/libgflags.so \
-lprotobuf -lpthread -lglog \
/software/software/gflags/2.1.2-foss-2016a/lib/libgflags.so \
-lprotobuf -lhdf5_hl -lhdf5 -lrt -lsz -lz -ldl -lm -lpthread -llmdb -lleveldb -lsnappy \
-lcudart -lcurand -lcublas -lcudnn \
/software/software/OpenCV/3.1.0-foss-2016a/lib/libopencv_highgui.so.3.1.0 \
/software/software/OpenCV/3.1.0-foss-2016a/lib/libopencv_imgcodecs.so.3.1.0 \
/software/software/OpenCV/3.1.0-foss-2016a/lib/libopencv_imgproc.so.3.1.0 \
/software/software/OpenCV/3.1.0-foss-2016a/lib/libopencv_core.so.3.1.0 \
-lopenblas -Wl,-rpath,/home/p253591/3D-Caffe/build/lib:/software/software/Boost/1.61.0-foss-2016a/lib:/software/software/gflags/2.1.2-foss-2016a/lib
TL;DR: The essentials form is (note the -lz for linking to libz):
g++ ... .../caffe.cpp.o -o caffe .../libcaffe.so .../libproto.a .../libboost_system.so ...more .so's and -lxyz's... -lz
It errors:
/software/software/libpng/1.6.23-foss-2016a/lib/libpng16.so.16: undefined reference to `inflateReset2#ZLIB_1.2.3.4'
collect2: error: ld returned 1 exit status
And this seems to be correct, because when I specify the -print-file-name=libz.so.1 option to g++, it prints /lib/../lib64/libz.so.1, which is slightly too old regarding version.
I have a newer version in /software/software/zlib/1.2.8-foss-2016a/lib/, but when I:
prepend this path to LD_LIBARY_PATH,
append it to the -rpath option, or
add it using the -L option,
g++ keeps using /lib/../lib64/libz.so.1 as it reports using -print-file-name
What does work, is removing the -lz option and adding /software/software/zlib/1.2.8-foss-2016a/lib/libz.so.1 explicitly as a file to include during compilation/linking. But that seems like a hack.
How can I tell g++ to use /software/software/zlib/1.2.8-foss-2016a/lib/libz.so.1 instead of /lib/../lib64/libz.so.1 when fulfilling the -lz option? Preferably indirectly via cmake (when building caffe).
Thank you in advance!
EDIT
ldd states the right .so is used:
# ldd /software/software/libpng/1.6.23-foss-2016a/lib/libpng16.so.16
linux-vdso.so.1 => (0x00007fff55599000)
libz.so.1 => /software/software/zlib/1.2.8/lib/libz.so.1 (0x00007f7875b29000)
libm.so.6 => /lib64/libm.so.6 (0x00007f7875888000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f787566b000)
libc.so.6 => /lib64/libc.so.6 (0x00007f78752d7000)
/lib64/ld-linux-x86-64.so.2 (0x0000003328800000)
Moreover, the symbol should be there in this libz.so.1:
# nm /software/software/zlib/1.2.8/lib/libz.so.1
...
0000000000008270 T inflateReset2
...
0000000000000000 A ZLIB_1.2.3.4
....
#
I'm assuming the #-notation in de undefined symbol-error somehow needs the A-type symbol to be there as a version check (but I could not find any documentation on this in ld.so man pages)
My guess is that you add -L after -lz which won't work as -L only affects -l which follow it. Also note that -rpath and LD_LIBRARY_PATH are influencing runtime library search (and also ldd output) so they won't have much influence on compilation.
Related
I am working on a bare-metal free standing software on a STM32H753. I'm not using neither the libc nor the crt.
Here is the link command line:
arm-none-eabi-gcc -T"xxx.ld" -mfpu=fpv5-d16 -mfloat-abi=hard -mthumb -ffreestanding -nostdlib -nostartfiles --specs=nosys.specs -Wl,--start-group -lc -lm -Wl,--end-group -Wl,-Map=xxx.map -o xxx.elf <list of .o>
Now I need to include math library since I am using sqrt function. i thought the link command line would be sufficient but I get a "sqrt undefined" error.
I tried to add the path to the libm.a: (also tried without -Wl)
arm-none-eabi-gcc -T"xxx.ld" -Wl,-L/opt/gcc-arm-none-eabi-10-2020-q4-major/arm-none-eabi/lib/ -mfpu=fpv5-d16 -mfloat-abi=hard -mthumb -ffreestanding -nostdlib -nostartfiles --specs=nosys.specs -Wl,--start-group -lc -lm -Wl,--end-group -Wl,-Map=build_uP1/base_gen_uP1.map -o build_uP1/base_gen_uP1.elf <list of .o>
But I still get the same error.
I don't understand what options to choose to link with the correct library
Sorry for this self answer but I think I 've found the solution.
my first mistake is that the library must be put at the end of the command line. the order of arguments does matter.
then there are many versions of libm.a in the gcc install so I had to pick the right one
The following line is working:
arm-none-eabi-gcc -T"xxx.ld" -L/opt/gcc-arm-none-eabi-10-2020-q4-major/arm-none-eabi/lib/thumb/v7+fp/hard/ -mfpu=fpv5-d16 -mfloat-abi=hard -mthumb -ffreestanding -nostdlib -nostartfiles --specs=nosys.specs -Wl,-Map=xxx.map -o xxx.elf <list of .o> -lm -lc
I've noticed that one symbol and some data from libc.a are needed: __errno, and impure_data
I have converted c program into assembly code using following commands in LLVM :
clang -emit-llvm matrix.c -c -o matrix.bc
llc -march=alpha matrix.bc -o matrix.s
Now how to convert matrix.s assembly file into executable file of alpha.
How to do that?
clang can also be used
clang matrix.s -L [additional library locations] -mllvm -Wall -g -L. -Wl,-pie -I. -I[additional include locations] -o [executable output]
Adjust the flags as your needs dictate.
EDIT
Without the need for other includes or libraries just call:
clang matrix.s -mllvm -Wall -g -Wl,-pie -o matrix.out
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/vagrant/python/include/python2.7 -c external/KentLib/wWigIO/wWigIO.c -o build/temp.linux-i686-2.7/external/KentLib/wWigIO/wWigIO.o -w -shared -fPIC -p -Iexternal/KentLib/inc
Then:
gcc -pthread -shared build/temp.linux-i686-2.7/external/KentLib/wWigIO/wWigIO.o -o build/lib.linux-i686-2.7/wWigIO.so -DMACHTYPE_x86_64 -lz -lm external/KentLib/lib/jkweb.a
(sorry for the messy-ness of these commands I wanted to copy them verbatim to avoid leaving out important details)
Then, I look at the symbols, and notice that compress is not defined:
$ nm build/lib.linux-i686-2.7/wWigIO.so | grep compress
U compress
0002486d t getDecompressor
00024b28 T lineFileDecompress
00024c0f T lineFileDecompressFd
00024c8b T lineFileDecompressMem
U uncompress
00037cd2 T zUncompress
It doesn't seem to be linking to the either libm or libz:
$ ldd build/lib.linux-i686-2.7/wWigIO.so
linux-gate.so.1 => (0xb76e2000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb7668000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb74be000)
/lib/ld-linux.so.2 (0xb76e3000)
I know that libz is installed and it's in the search path:
$ sudo cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf
$ sudo cat /etc/ld.so.conf.d/*.conf
# Multiarch support
/lib/i386-linux-gnu
/usr/lib/i386-linux-gnu
/lib/i686-linux-gnu
/usr/lib/i686-linux-gnu
# libc default configuration
/usr/local/lib
libz is in those locations:
$ locate libz
/lib/i386-linux-gnu/libz.so.1
/lib/i386-linux-gnu/libz.so.1.2.3.4
/usr/lib/i386-linux-gnu/libz.a
/usr/lib/i386-linux-gnu/libz.so
I can see the symbol defined in libz.so
$ nm -D /usr/lib/i386-linux-gnu/libz.so | grep compress
00001d60 T compress
00001c70 T compress2
00001da0 T compressBound
00003d20 T uncompress
The only way I can get this to work is to change the gcc command to this (bold part is added):
gcc -pthread -shared build/temp.linux-i686-2.7/external/KentLib/wWigIO/wWigIO.o -o build/lib.linux-i686-2.7/wWigIO.so -DMACHTYPE_x86_64 -lz -lm external/KentLib/lib/jkweb.a /usr/lib/i386-linux-gnu/libz.a
This makes no sense to me. Why wouldn't libz be linking?
I sort of figured out a solution.
If I set LDFLAGS='-Wl,--no-as-needed -lz' then it does indeed link to libz and the compiled shared library now does not get the undefined symbol error.
However, I'm still confused as to why it doesn't link to libz by just providing the -lz flag.
I'm trying to understand the process of static linking, loading of GCC:
I have the following toy program
#include "stdio.h"
int main() {
fprintf(stdout, "Hello World \n");
return 0 ;
}
I can compile it and run file as follows:
gcc -static -std=gnu99 -Wall -Wno-unused -g test.c -o test;
But as soon as I try to separate out the compile and linking process as follows:
gcc -static -std=gnu99 -Wall -Wno-unused -g test.c -c;
ld -o test -T link.lds test.o
where the link.lds is
SECTIONS
{
. = 0x10000;
.text : { *(.text) }
. = 0x8000000;
.data : { *(.data) }
.bss : { *(.bss) }
}
I get the error "undefined reference to stdouttest.o: In function `main':
test.c:(.text+0x7): undefined reference to `stdout'
test.c:(.text+0x1e): undefined reference to `fwrite'
If I try adding the flag -lc to ld, it tells me that it is not found. I've tried running gcc with -lc
and/or -static-libgcc but I have the same problem.
What am I doing wrong?
Do
gcc -v -static -std=gnu99 -Wall -Wno-unused -g test.c
and look for the collect2 tag.
In my case it is
collect2 --sysroot=/ --build-id -m elf_x86_64 --hash-style=gnu --as-needed -static -z relro -o test /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginT.o -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. /tmp/ccoR98Xr.o --start-group -lgcc -lgcc_eh -lc --end-group /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o
You have to replace the temporary object file. In my case I replaced /tmp/ccoR98Xr.o with test.o. Then do
gcc -c -std=gnu99 -Wall -Wno-unused -g test.c
ld --sysroot=/ --build-id -m elf_x86_64 --hash-style=gnu --as-needed -static -z relro -o test /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginT.o -L/usr/lib/gcc/x86_64-linux-gnu/4.8 -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.8/../../.. test.o --start-group -lgcc -lgcc_eh -lc --end-group /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o
It links to the object files : crt1.o, crti.o, crtbeginT.o, crtend.o, and crtn.o.
It links to the libraires: libgcc.a, libgcc_eh.a, and libc.a.
You can replace --start-group -lgcc -lgcc_eh -lc --end-group with -lgcc -lc -lgcc_eh -lc if you like.
Knowing this we can simply this to
ln -s `gcc -print-file-name=crt1.o`
ln -s `gcc -print-file-name=crti.o`
ln -s `gcc -print-file-name=crtn.o`
ln -s `gcc -print-file-name=libgcc_eh.a`
ln -s `gcc -print-file-name=libc.a`
gcc -c -std=gnu99 -Wall -Wno-unused -g test.c
ld -m elf_x86_64 -o test crt1.o crti.o test.o libc.a libgcc_eh.a libc.a crtn.o
I did not use crtbeginT.o, crtend.o, and libgcc.a because it worked without them.
Take a look at this, strace does the job (show you the secrets) but you will soon realize there are tons of options in it... You need to link a few stuff together (from GNU C lib) to get your executable, not only your object... You can add grep 'exec' in the end to make it cleaner.
Uhhh, also you need to do this:
as obj.s -o obj.o
Use GNU assembler to convert your .s to .o then link with ld.
Recently I was creating a loadable module and found that both
gcc -fPIC --shared -o foo.so.1 foo.c
and
gcc -fPIC --shared -c foo.c
ld --shared -o foo.so.2 foo.o
can achieve the same effect.
I also discovered that foo.so.1 is larger than foo.so.2 by about 3KB, and
gcc -### -fPIC --shared -o foo.so.1 foo.c
revealed that GCC added stuffs other than foo.c into foo.so.1 (e.g, crtendS.o and crtn.o):
/usr/lib/gcc/x86_64-linux-gnu/4.7/collect2 "--sysroot=/" --build-id --no-add-needed --eh-frame-hdr -m elf_x86_64 "--hash-style=both" -shared -o foo.so.1 /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.7/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/4.7 -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/4.7/../../.. /tmp/cc3JBdCJ.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/4.7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/crtn.o
Since both foo.so.1 and foo.so.2 can be loaded via dlopen, I was wondering:
What's the difference between these 2 linking methods?
Do crtendS.o and crtn.o make any difference to functions in created libraries?
There's no difference in principle. When you "link by gcc" it actually calls ld. If you get a message at the linking stage when "linking by gcc" you'll immediately see that it is actually from ld. If you want to pass some ld-specific command-line options to ld, gcc's command-line interface has features intended specifically for that purpose (-Xlinker and -Wl options).
As for the additional objects files... they probably contain global load-time library initialization/de-initialization code implicitly added by the compiler. (Requested by the standard library?) You can find some information about it here: https://gcc.gnu.org/onlinedocs/gccint/Initialization.html