I'm having some trouble building shared libraries from Ada packages without using GPR's.
I have a package, Numerics, in files "numerics.ads" and "numerics.adb". They have no dependencies. There is a small build script which does:
gnatmake -Os numerics.ad[bs] -cargs -fPIC
gcc -shared numerics.o -o libnumerics.so -Wl,-soname,libnumerics.so
The .so and .ali files are installed at /usr/lib, and the .ads file is installed at /usr/include.
gnatls -v outputs the following relevant parts:
Source Search Path:
<Current_Directory>
/usr/include
/usr/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/adainclude
Object Search Path:
<Current_Directory>
/usr/lib
/usr/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/adalib
So GNAT should have no problem finding the files.
Then, trying to compile a package that depends on Numerics:
gnatmake -O2 mathematics.ad[bs] -cargs -fPIC
outputs:
gcc -c -fPIC mathematics.adb
gcc -c -I./ -fPIC -I- /usr/include/numerics.ads
cannot generate code for file numerics.ads (package spec)
gnatmake: "/usr/include/numerics.ads" compilation error
This error has me thinking GNAT doesn't recognize the shared library, and is trying to rebuild Numerics.
I'd like to be building shared libraries, and only supply the spec for reference/documentation purposes.
edit:
So, it looks like gprbuild does two things I'm not doing. The first, is also passing -lnumerics to the compiler. The second, which shouldn't matter since libnumerics.so is in a standard directory anyways, is -L«ProjectDirectory». GPRbuild is obviously not doing desired behavior either, even though it's building the dependent project. It should be using the installed library /usr/lib/libnumerics.so, but instead is using «path»/Numerics/build/libnumerics.so. Furthermore, after building Numerics with GPRbuild, and then renaming the body to make it as if the body didn't exist (like with the installed files), when building Mathematics with GPRbuild, it complains about the exact same problem. It's as if the libraries aren't even shared, and GPRBuild is just making them look that way (except readelf reports the correct dependencies inside the libraries).
Adding -lnumerics to the build script accomplishes nothing; the build error is exactly the same. I'm completely lost at this point.
edit:
Following the link from Simon, the buildscript has changed to:
gnatmake -O2 mathematics.ad[bs] \
-aI/usr/include \
-aO/usr/lib \
-cargs -fPIC \
-largs -lnumerics
The error is essentially the same:
gcc -c -O2 -I/usr/include/ -fPIC mathematics.adb
gcc -c -I./ -O2 -I/usr/include/ -fPIC -I- /usr/include/numerics.ads
cannot generate code for file numerics.ads (package spec)
gnatmake: "/usr/include/numerics.ads" compilation error
I thought to check libnumerics.so is actually a correct shared library. ldd reports:
linux-vdso.so.1 (0x00007ffd944c1000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f50d3927000)
/usr/lib64/ld-linux-x86-64.so.2 (0x00007f50d3ed4000)
So I'm thinking yes, the library is fine, and gnatmake still isn't recognizing it.
In general, you need to install the body of the packages as well (numerics.adb in your case). Also, I suspect you want to set the ALI files
(numerics.ali) as read-only, so that gnatmake does not try to recompile them.
Related
Background
I'm compiling, on a machine which runs macOS 10.15, a project which ships an old version of libpng (1.6.17) as a submodule. The corresponding code is available at https://github.com/glennrp/libpng. I also have libpng 1.6.37 installed by Homebrew.
Until not so long ago, I was able to compile libpng 1.6.17 using CMake without trouble. Since very recently (but the exact date is unknown to me), build fails with errors like:
FAILED: CMakeFiles/png16_static.dir/pngwutil.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -I/usr/local/include -I. -I../ -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -MD -MT CMakeFiles/png16_static.dir/pngwutil.o -MF CMakeFiles/png16_static.dir/pngwutil.o.d -o CMakeFiles/png16_static.dir/pngwutil.o -c ../pngwutil.c
../pngwutil.c:2413:20: error: use of undeclared identifier 'PNG_WEIGHT_SHIFT'
PNG_WEIGHT_SHIFT;
^
I ran a few checks against a copy of my project I had which still compiled correctly because CMake wasn't re-running itself on it. The only difference between the two cases is a -I/usr/local/include flag added to compiler calls (I added some markup to help see the difference):
suceeds:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -Dpng16_EXPORTS -Iext_build/libpng -I../../ext/libpng -O3 -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -fPIC -fno-stack-protector -fomit-frame-pointer -fno-math-errno -ffp-contract=fast -march=native -MD -MT ext_build/libpng/CMakeFiles/png16.dir/pngrio.o -MF ext_build/libpng/CMakeFiles/png16.dir/pngrio.o.d -o ext_build/libpng/CMakeFiles/png16.dir/pngrio.o -c ../../ext/libpng/pngrio.c
<---------------------------------->
fails:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -Dpng16_EXPORTS -I/usr/local/include -Iext_build/libpng -I../../ext/libpng -O3 -DNDEBUG -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -fPIC -fno-stack-protector -fomit-frame-pointer -fno-math-errno -ffp-contract=fast -march=native -MD -MT ext_build/libpng/CMakeFiles/png16.dir/pngrio.o -MF ext_build/libpng/CMakeFiles/png16.dir/pngrio.o.d -o ext_build/libpng/CMakeFiles/png16.dir/pngrio.o -c ../../ext/libpng/pngrio.c
<------------------------------------------------------->
I re-ran CMake on the copy of the project which was working and I got the same error, which pointed me to a system-related problem. I then checked out directly the libpng sources and got the same error.
Steps to reproduce
Clone the libpng repo and check out v1.6.17
git clone https://github.com/glennrp/libpng.git
cd libpng
git checkout v1.6.17
Build libpng
cmake . -B build && cmake --build build
Question
What did add this -I/usr/local/include flag to my compiler calls?
Bonus question (maybe more interesting)
Now, it gets funny. If you checkout a more recent libpng (I tried with 1.6.21, 1.6.25, 1.6.28, 1.6.33 and 1.6.37), the problem goes away, although the flag is still here:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -I/usr/local/include -I. -I../ -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -MD -MT CMakeFiles/png16_static.dir/pngrio.o -MF CMakeFiles/png16_static.dir/pngrio.o.d -o CMakeFiles/png16_static.dir/pngrio.o -c ../pngrio.c
This means that I could update my submodule use one of these releases and my problem would go away.
However, if I'm not not mistaken, -I flags are resolved from left to right: I therefore suspect that my Homebrew headers are used instead of the source ones. If I'm right, then this doesn't guarantee that a Homebrew upgrade of libpng won't break the build again: it just shows that libpng's API has been stable since v1.6.21 and that I can use my Homebrew headers with the source I'm trying to compile. Am I right, or am I missing something?
answer to question: the system include path is added by compiler/preprocessor (this page will explain more datails)
the order of included in CMake project may be changed (not sure how much), CMake allows prepend includes in the list; try compare CMake build scripts between releases. I believe there will be mentioned change.
Yes sometimes cmake will pick up a variety of libs and include dirs and freely mix and match. If you need to keep multiple versions up, you should use the cmake command to pass the correct packages.
I finally got to the bottom of it. It turned out that libpng's zlib dependency was the one giving me trouble (indirectly).
What happened
MacOS ships zlib and it is usually known to developers who therefore don't feel the need to install a 3rd party zlib. However, CMake's find_package doesn't know about this preference and will pick up a zlib implementation found in /usr/local, for instance if it is installed by Homewbrew. For some reason, a zlib was installed by 3rd party software at this location on my system—not a Homebrew package, which made the detection more difficult, and it was detected by find_package.
The corresponding include directory is /usr/local/include. libpng's CMake code is such that this directory is then added to the list of include directories, which leads to the header conflicts mentioned in the question. I understood what was happening by going through CMakeCache.txt (searching for /usr/local/include), so the main lesson is: don't forget to check your CMake cache in such situations.
How to solve the problem
The lazy way. Remove the undesired lib files. I ran brew doctor and removed the files which were not supposed to be here. This, however, might have undesired consequences if the specific zlib version sitting in /usr/local is actually required by some piece of software.
The dirty CMake way. Modify the top-level CMake code to hint find_package about where it should pick zlib. Either hard-code the hint using the PATHS argument or set it using the ZLIB_ROOT argument (you might have to define policies for that).
I'm sure there is a better way to handle this by "doing CMake right" in libpng and forcing library search in system paths, but my CMake skills are not good enough to say what should be done exactly. And anyway, it's a bit off-topic with respect to the question.
I have inherited a Makefile which builds a .so file. It is linking with -lcrypto from OpenSSL on Ubuntu with gcc 4.7.4. Critically, it is NOT linking with -lssl nor -ldl, and when I run nm -g thelib.so, it only has the ~15 symbols from openssl crypto. However, they are all U (undefined).
I'm refactoring the Makefile on another Ubuntu machine. When I link with -lcrypto, it fails due to undefined symbols needed from dl. When I add linking to -ldl, those errors go away and linking succeeds. However, my .so file is 1.5 MB bigger than the original, and there are at least a hundred symbols related to SSL, which are all T (defined), which seem to indicate that -lssl is happening implicitly somehow.
While it would seem prudent and good that they are all defined in my case, I need to figure out how to produce the same result just as it is.
So, my question is, how does one get GCC to allow the linking of a .so file and accept undefined references? I've compared our commands, and there are little differences which I've tried to eliminate, but nothing seems to work. I read that it might be related to -Wl,--no-as-needed, but i'm using that. Here's my linker flags.
g++ -shared -o mylib.so myobjs.o -fPIC -lstdc++ -lm -z defs -Wl,-soname,mylib -Wl,--no-as-needed -lpthread -lcrypto -lz
On the other system (the one with the larger result), OpenSSL has apparently not been built as a shared object, only as a static library (but maybe as PIC, so that you can link the result into a shared object). You will have to install the packages that provide the shared object and the corresponding .so symbolic link.
We are catching link errors on Solaris with makefiles generated by CMake 3.6.2. In the testing below, we are using GCC and not SunCC. From the looks of it, CMake is applying our options inconsistently:
Typical compile command
[ 2%] Building CXX object CMakeFiles/cryptopp-object.dir/cpu.cpp.o
/bin/c++ -fPIC -march=native -m64 -Wa,--divide -o CMakeFiles/cryptopp-object.dir/cryptlib.cpp.o
-c /export/home/jwalton/cryptopp/cpu.cpp
Abbreviated link command
/bin/c++ CMakeFiles/cryptest.dir/bench1.cpp.o CMakeFiles/cryptest.dir/bench2.cpp.o
...
CMakeFiles/cryptest.dir/fipstest.cpp.o -o cryptest.exe libcryptopp.a -lnsl -lsocket
Typical link error
ld: fatal: file CMakeFiles/cryptopp-object.dir/cryptlib.cpp.o: wrong ELF class: ELFCLASS64
Notice the file was compiled with -march=native -m64 (its a 64-bit capable machine and kernel), but the link invocation is missing it (the default is 32-bit on Solaris).
Attempting to search for "cmake use CXXFLAGS link" is producing too much irrelevant noise, and I'm not having much luck finding the CMakeList.txt option. I also want to avoid duplicating the work into LDFLAGS, or performing the work of reformatting the options (CXXFLAGS option -Wl,-x becomes LDFLAGS option -x).
How do I instruct CMake to use both CXX and CXXFLAGS when driving link?
I found Running a different program for the linker on the CMake users mailing list, but it does not feel right to me (also, the problem and context are slightly different). It also does not work.
Here is a small example:
PROJECT(foo)
SET(CMAKE_CXX_LINK_EXECUTABLE
"purify <CMAKE_CXX_COMPILER> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
ADD_EXECUTABLE(foo foo.cxx)
I also found Setting global link flags on the mailing list. It does not work, either.
SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_CXX_FLAGS}")
SET(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_CXX_FLAGS}")
SET(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_CXX_FLAGS}")
One of my users is getting an error message when trying to compile a C part of our mixed C/C++ codebase on ubuntu 12.04 with gcc 4.8.1
We have a library in C++ with some C-linkage functions in, and want to compile a C program linking to it. The library is compiled with g++ and builds fine. The c program fails like this:
> gcc -O3 -g -fPIC -I/media/Repo/lcdm/code/cosmosis/ -Wall -Wextra -pedantic -Werror -std=c99 -o c_datablock_t c_datablock_test.c -L . -lcosmosis
cc1plus: error: command line option ‘-std=c99’ is valid for C/ObjC but not for C++ [-Werror]
The program has a lower case .c file suffix, so why does gcc try to compile it as c++ ? We have not seen this on other OSes.
(I know we could kick the problem down the road by removing -Werror or handle this particular file with -x c but I'd like to solve the real problem.)
why does gcc try to compile it as c++
I can think of only two plausible explanations, and they both are end-user's fault.
It could be that the user transferred sources via Windows, and the file is really called C_DATABLOCK_TEST.C, and the user is misleading you.
It could also be that the user overwrote his gcc with g++ (surprisingly many people believe that gcc and g++ are the same thing, but they are not).
To disprove the first possibility, ask the user to execute his build commands under script, and send you resulting typescript.
To disprove the second, ask the user to add -v to the compile command.
This look like GCC Bug 54641, which has been fixed in a later release of GCC. It is only a warning, but your compile flags are causing GCC to treat all warnings as errors.
I am trying to write a matlab mex function which uses libhdf5; My Linux install provides libhdf5-1.8 shared libraries and headers. However, my version of Matlab, r2007b, provides a libhdf5.so from the 1.6 release. (Matlab .mat files bootstrap hdf5, evidently). When I compile the mex, it segfaults in Matlab. If I downgrade my version of libhdf5 to 1.6 (not a long-term option), the code compiles and runs fine.
question: how do I solve this problem? how do I tell the mex compilation process to link against /usr/lib64/libhdf5.so.6 instead of /opt/matlab/bin/glnxa64/libhdf5.so.0 ? When I try to do this using -Wl,-rpath-link,/usr/lib64 in my compilation, I get errors like:
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../x86_64-pc-linux-gnu/bin/ld: warning: libhdf5.so.0, needed by /opt/matlab/matlab75/bin/glnxa64/libmat.so, may conflict with libhdf5.so.6
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: ld returned 1 exit status
mex: link of 'hdf5_read_strings.mexa64' failed.
make: *** [hdf5_read_strings.mexa64] Error 1
ack. the last resort would be to download a local copy of the hdf5-1.6.5 headers and be done with it, but this is not future proof (a Matlab version upgrade is in my future.). any ideas?
EDIT: per Ramashalanka's excellent suggestions, I
A) called mex -v to get the 3 gcc commands; the last is the linker command;
B) called that linker command with a -v to get the collect command;
C) called that collect2 -v -t and the rest of the flags.
The relevant parts of my output:
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/crti.o
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/crtbeginS.o
hdf5_read_strings.o
mexversion.o
-lmx (/opt/matlab/matlab75/bin/glnxa64/libmx.so)
-lmex (/opt/matlab/matlab75/bin/glnxa64/libmex.so)
-lhdf5 (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/libhdf5.so)
/lib64/libz.so
-lm (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/libm.so)
-lstdc++ (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/libstdc++.so)
-lgcc_s (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/libgcc_s.so)
/lib64/libpthread.so.0
/lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2
-lgcc_s (/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/libgcc_s.so)
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/crtendS.o
/usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/crtn.o
So, in fact the libhdf5.so from /usr/lib64 is being referenced. However, this is being overriden, I believe, by the environment variable LD_LIBRARY_PATH, which my version of Matlab automagically sets at run-time so it can locate its own versions of e.g. libmex.so, etc.
I am thinking that the crt_file.c example works either b/c it does not use the functions I am using (H5DOpen, which had a signature change in the move from 1.6 to 1.8 (yes, I am using -DH5_USE_16_API)), or, less likely, b/c it does not hit the parts of Matlab internals that need hdf5. ack.
The following worked on my system:
Install hdf5 version 1.8.4 (you've already done this: I installed the source and compiled to ensure it is compatible with my system, that I get gcc versions and that I get the static libraries - e.g. the binaries offered for my system are icc specific).
Make a target file. You already have your own file. I used the simple h5_crtfile.c from here (a good idea to start with this simple file first a look for warnings). I changed main to mexFunction with the usual args and included mex.h.
Specify the static 1.8.4 library you want to load explicitly (the full path with no -L for it necessary) and don't include -lhdf5 in the LDFLAGS. Include a -t option so you can ensure that there is no dynamic hdf5 library being loaded. You also need -lz, with zlib installed. For darwin we also need a -bundle in LDFLAGS:
mex CFLAGS='-I/usr/local/hdf5/include' LDFLAGS='-t /usr/local/hdf5/lib/libhdf5.a -lz -bundle' h5_crtfile.c -v
For linux, you need an equivalent position-independent call, e.g. fPIC and maybe -shared, but I don't have a linux system with a matlab license, so I can't check:
mex CFLAGS='-fPIC -I/usr/local/hdf5/include' LDFLAGS='-t /usr/local/hdf5/lib/libhdf5.a -lz -shared' h5_crtfile.c -v
Run the h5_crtfile mex file. This runs without problems on my machine. It just does a H5Fcreate and H5Fclose to create "file.h5" in the current directory, and when I call file file.h5 I get file.h5: Hierarchical Data Format (version 5) data.
Note that if I include a -lhdf5 above in step 3, then matlab aborts when I try to run the executable (because it then uses matlab's dynamic libraries which for me are version 1.6.5), so this is definitely solving the problem on my system.
Thanks for the question. My solution above is definitely much easier for me than what I was doing before. Hopefully the above works for you.
I am accepting Ramashalanka's answer because it led me to the exact solution which I will post here for completeness only:
download the hdf5-1.6.5 library from the hdf5 website, and install the header files in a local directory;
tell mex to look for "hdf5.h" in this local directory, rather than in the standard location (e.g. /usr/include.)
tell mex to compile my code and the shared object library provided by matlab, and do not use the -ldfh5 flag in LDFLAGS.
the command I used is, essentially:
/opt/matlab/matlab_default/bin/mex -v CC#gcc CXX#g++ CFLAGS#"-Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include" CXXFLAGS#"-Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include " -O -lmwblas -largeArrayDims -L/usr/lib64 hdf5_read_strings.c /opt/matlab/matlab_default/bin/glnxa64/libhdf5.so.0
this gets translated by mex into the commands:
gcc -c -I/opt/matlab/matlab75/extern/include -DMATLAB_MEX_FILE -Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include -O -DNDEBUG hdf5_read_strings.c
gcc -c -I/opt/matlab/matlab75/extern/include -DMATLAB_MEX_FILE -Wall -O3 -fPIC -I./hdf5_1.6.5/src -I/usr/include -I/opt/matlab/matlab_default/extern/include -O -DNDEBUG /opt/matlab/matlab75/extern/src/mexversion.c
gcc -O -pthread -shared -Wl,--version-script,/opt/matlab/matlab75/extern/lib/glnxa64/mexFunction.map -Wl,--no-undefined -o hdf5_read_strings.mexa64 hdf5_read_strings.o mexversion.o -lmwblas -L/usr/lib64 /opt/matlab/matlab_default/bin/glnxa64/libhdf5.so.0 -Wl,-rpath-link,/opt/matlab/matlab_default/bin/glnxa64 -L/opt/matlab/matlab_default/bin/glnxa64 -lmx -lmex -lmat -lm -lstdc++
this solution should work on all my various target machines and at least until I upgrade to matlab r2009a, which I believe uses hdf5-1.8. thanks for all the help, sorry for being so dense with this--I think I was overly-committed to using the packaged version of hdf5, rather than a local set of header files.
Note this would all have been trivial if Mathworks had provided a set of the header files with the Matlab distribution...