I'm trying to install GalSim on OSX Mavericks, using the anaconda python distribution, but keep running into the error:
Unable to build a python loadable module using the python executable:
/usr/bin/env python,
the library name libpython2.7.a,
and the libdir /Users/harrison/anaconda/lib/python2.7/config.
If these are not the correct library names, you can tell scons the
correct names to use with the flags EXTRA_LIB_PATH and/or EXTRA_LIBS.
the /usr/bin/env python is the anaconda one and there is indeed a libpython2.7.a in that libdir.
scons is installed via the conda package management system.
boost is installed manually as recommended via
./bootstrap.sh --with-python=/Users/harrison/anaconda/bin/python --with-python-root=/Users/harrison/anaconda/
./b2 -a link=shared
sudo ./b2 -a --prefix=/usr/local link=shared install
checking the version of python boost links to gives
$ otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python.dylib:
libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
Is this correct (it doesn't seem so to me -- boost seems to be linking to the libpython2.7 in /usr/local/) and is this what is causing the problem installing GalSim?
Thanks
Ian
Full output:
$ scons
scons: Reading SConscript files ...
SCons is version 2.3.0 using python version 2.7.6
Python is from /Users/harrison/anaconda/include/python2.7
Using the following (non-default) scons options:
CXX = g++
These can be edited directly in the file gs_scons.conf.
Type scons -h for a full list of available options.
Using python = /usr/bin/env python
Using default PYPREFIX = /Users/harrison/anaconda/lib/python2.7/site-packages
Using compiler: /usr/bin/g++
compiler version: 4.2.1
Determined that a good number of jobs = 4
Checking for C++ header file fftw3.h... yes
Checking for correct FFTW linkage... yes
Checking for C++ header file boost/shared_ptr.hpp... yes
Checking for C++ header file TMV.h... yes
Using TMV_LINK file: /usr/local/share/tmv/tmv-link
-L/usr/local/Cellar/tmv-cpp/0.71/lib -ltmv -lblas
Mac version is 10.9
WARNING: The Apple BLAS library has been found not to be thread safe on
Mac OS version 10.7 (and possibly higher), even across multiple
processes (i.e. not just multiple threads in the same process).
The symptom is that `scons tests` will hang when running nosetests
using multiple processes.
If this occurs, the solution is to compile TMV either with a
different BLAS library (e.g. ATLAS) or with no BLAS library at
all (using WITH_BLAS=false).
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage... yes
Checking if we can build against Python...
Unable to build a python loadable module using the python executable:
/usr/bin/env python,
the library name libpython2.7.a,
and the libdir /Users/harrison/anaconda/lib/python2.7/config.
If these are not the correct library names, you can tell scons the
correct names to use with the flags EXTRA_LIB_PATH and/or EXTRA_LIBS.
Please fix the above error(s) and rerun scons.
Note: you may want to look through the file INSTALL.md for advice.
Also, if you are having trouble, please check the INSTALL FAQ at
https://github.com/GalSim-developers/GalSim/wiki/Installation%20FAQ
Ian, thanks for posting that. Based on the config.log and my own experience, I have an answer that should solve at least this problem (no guarantees that you won't hit something else later in the build).
The key problem here is that if you tell it the compiler is gcc (as is the default), it will try to use openMP. But actually invoking gcc on Mavericks gives you clang++, for which openMP doesn't work. So, you have to explicitly tell galsim that the compiler is clang++ so it won't try to use openMP. (The clue to this is in config.log, there are messages in various places about undefined symbols for x86_64 architecture... I had this problem on my own system when I started using Mavericks. See also https://github.com/GalSim-developers/GalSim/issues/493 )
So, please do scons CXX=clang++ and this problem should go away. I cannot guarantee that you won't hit a boost problem later on, however.
Please also see this question which has two suggestions which may help solve the problem for you as well.
Related
My ultimate goal is to get python package graph_tool working on my system and also on ipynb if possible. I have already brew install graph-tool, as indicated here, but that's still insufficient.
So I follow conda instructions here, and I make decent progress, until I encounter a compiler issue with the configuration.
$ conda create -n py36env python=3.6.3 anaconda
(py36env) $ conda install -c conda-forge cgal
... and so forth with the other required libraries
(py36env) Tams-MacBook-Pro:graph-tool-2.25 tamtam$ ./configure --prefix=/Users/tamtam/anaconda3/envs/py36env/ --with-python-module-path=/Users/tamtam/anaconda3/envs/py36env/lib/python3.6/site-packages
.
.
.
checking whether g++ supports C++14 features by default... no
checking whether g++ supports C++14 features with -std=gnu++14... no
checking whether g++ supports C++14 features with -std=gnu++1y... no
configure: error: *** A compiler with support for C++14 language features is required.
I'm not very familiar with compiler things, but I check my system as follows:
(py36env) Tams-MacBook-Pro:graph-tool-2.25 tamtam$ conda list | grep gcc
gcc 4.8.5 8
And outside of the conda env:
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/Users/tamtam/anaconda3/envs/py36env/bin/../libexec/gcc/x86_64-apple-darwin11.4.2/4.8.5/lto-wrapper
Target: x86_64-apple-darwin11.4.2
Configured with: ./configure --prefix=/Users/ray/mc-x64-3.5/conda-bld/gcc-4.8_1477649012852/
Thread model: posix
gcc version 4.8.5 (GCC)
And with Homebrew
$ brew list --versions | grep gcc
gcc 7.1.0 7.2.0
$ /usr/local/bin/gcc-7 --version
gcc-7 (Homebrew GCC 7.2.0) 7.2.0
$ echo $PATH
/Users/tamtam/anaconda3/bin:/Users/tamtam/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
So my questions are: EDITS INCLUDED
1) What's the difference between the conda gcc 4.8.5 and Homebrew gcc 7.2 on my system? I'm confused because they're each up-to-date packages.
EDIT: Nowhere online that I perused explicitly states this, but I am understanding (aka educated guess bc it's 3am and I need sleep) conda gcc 4.8.5 to just be equivalent to brew gcc 4.8.5, and that Anaconda is just way behind on developing an 'official' gcc supporting C++14. In the meantime I conda install -c salford_systems gcc-5 and ./configure seems to find itself a C++14 compiler. YAY (but this also led me to a new problem T.B.A.)
NOTE: I had also tried running the ./configure file with both conda gcc 4.8.5 and brew gcc 7.2.0 (switching of default gcc compiler guided here), but still same error of ./configure not finding C++14 compile (so ignore this)
2) I also have Xcode 8.3.3 that also comes with clang, which is what gcc is symlinked to... ?? No idea about the setup/relation
3) How can I adjust my system (for example, paths) so that the configuration process will detect a C++14 compiler? EDIT: resolved per 1)
(This solution is also included within the edits at the bottom of my question.)
Nowhere online that I perused explicitly states this, but I am understanding (aka educated guess bc it's 3am and I need sleep) conda gcc 4.8.5 to just be equivalent to brew gcc 4.8.5, and that Anaconda is just way behind on developing an 'official' gcc supporting C++14. In the meantime I conda install -c salford_systems gcc-5 and ./configure seems to find itself a C++14 compiler. YAY (but this also led me to a new problem)
NOTE: I had also tried running the ./configure file with both conda gcc 4.8.5 and brew gcc 7.2.0 (switching of default gcc compiler guided here), but still same error of ./configure not finding C++14 compile arises (so ignore this)
As a rule of thumb, if a product provides its own build tools (compiler, C headers, static libraries, include/library paths etc) instead of using the system's ones, you should use those when building things to use within its environment.
That's because, depending on include files, compiler, compiler version, compiler flags, predefined macros etc the same sources and libraries built with different toolchains can be binary incompatible. This goes double for C++ where there are no ABI standards at all. (In C, there are de facto ones for most platforms where there are competing compilers due to the need to do system calls.)
Now, what you have on your system:
The system's stock gcc (if any) is at /usr/bin (so includes are at /usr/include, static libraries at /usr/lib etc)
Homebrew's gcc 7.2 at /usr/local/bin (includes are probably at /usr/local/include)
Anaconda also has a gcc package that, if installed, would reside somewhere like /Users/<login>/anaconda3/envs/<env name>/bin
If an environment provides an own toolchain, it provides some way to make build scripts select that toolchain when building things. A common way is to set relevant environment variables (PATH, INCLUDE, LIB) - either directly or through some tool that the environment provides. Homebrew executables reside in /usr/local/bin which is always present in UNIX PATH as per the FHS; Anaconda adds itself to PATH whenever its environment is activated.
So, as you yourself guessed, since Anaconda provides its own gcc packages (thus enforces the compatibility of their output with its binary packages via package metadata - if there are known incompatibilities, the installation will fail due to requirement conflicts), you need to install one that meets graph-tool requirements -- which is gcc 5, available via gcc-5 package from salford_systems channel. Then specify --prefix as you already have.
You may need to install other dependencies, too, if you didn't already or you need optional features that they're required for.
I'm starting to learn Haskell and I found out that ghc can compile using LLVM with the -fllvm flag. Whenever I use the flag, I get the error message:
<no location info>: error:
Warning: Couldn't figure out LLVM version!
Make sure you have installed LLVM 3.7
ghc: could not execute: opt
However, I have opt in my /usr/local/Cellar/llvm/3.9.0/ folder. I'm on a Mac OS X and I've installed the full LLVM with brew install llvm but error persists. Is this a genuine version problem where I have to unistall LLVM and reinstall its 3.7 version? Or is ghc having trouble finding opt and there is some kind of search path I can modify to fix the problem? Thanks for the help and have a great day.
The GHC documentation says that it's compatible with llvm-2.8+, but as you've discovered, it actually requires llvm-3.7.
The simplest way to get it is:
brew install llvm#3.7
This installs llvm binaries in your path with a -3.7 suffix, like clang-3.7. GHC will need the unadorned names, which are in a subdirectory:
export PATH=/usr/local/opt/llvm#3.7/lib/llvm-3.7/bin:$PATH
I have installed fftw and boost with conda;
conda install -c https://conda.anaconda.org/anaconda boost
conda install -c https://conda.anaconda.org/nlesc fftw
tmv with scons
tmv0.72){58}> scons install PREFIX=/home/rgm/local
the scons with the various variables specified earlier as shown below:
scons
scons: Reading SConscript files ...
SCons is version 2.3.5 using python version 2.7.10
Python is from /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
Using the following (non-default) scons options:
PREFIX = /home/rgm/local/
TMV_DIR = /Users/rgm/local
FFTW_DIR = /Users/rgm/anaconda/
BOOST_DIR = /Users/rgm/anaconda/
These can be edited directly in the file gs_scons.conf.
Type scons -h for a full list of available options.
Using python = /home/rgm/anaconda/bin/python
Using default PYPREFIX = /home/rgm/anaconda/lib/python2.7/site-packages
Using compiler: /home/rgm/anaconda/bin/g++
compiler version: 4.8.5
Determined that a good number of jobs = 4
Checking for C++ header file fftw3.h... yes
Checking for correct FFTW linkage... yes
Checking for boost header files... yes
Boost version is 1.57.0
Checking for C++ header file TMV.h... yes
TMV version is 0.72
Using TMV_LINK file: /Users/rgm/local/share/tmv/tmv-link
-L/home/rgm/local/lib -ltmv -lblas -lpthread -fopenmp
Mac version is 10.10.5
XCode version is 7.2
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage...
Error: TMV file failed to link correctly
Check that the correct location is specified for TMV_DIR
looking in config.log there are lines like:
Undefined symbols for architecture x86_64:
"_fftw_destroy_plan", referenced from:
_main in conftest_2.o
The proximate problem here is that it cannot find the OpenMP library (libgomp.1.dylib) at runtime. Here are the relevant lines from the config.log:
g++ -o .sconf_temp/conftest_16 -fopenmp .sconf_temp/conftest_16.o -Llib -L/Users/rgm/anaconda/lib -L/Users/rgm/local/lib -L/home/rgm/local/lib -ltmv_symband -lfftw3 -lpthread -ltmv -lblas -lpthread
.sconf_temp/conftest_16 > .sconf_temp/conftest_16.out
dyld: Library not loaded: #rpath/./libgomp.1.dylib
Referenced from: /data/vault/rgm/soft/MACOS/GalSim/.sconf_temp/conftest_16
Reason: image not found
sh: line 1: 20592 Trace/BPT trap: 5 .sconf_temp/conftest_16 > ".sconf_temp/conftest_16.out"
I've had similar troubles with Anaconda before on Macs. They seem to try to force the Mac to use Unix-style run-time library resolution, rather than follow the normal way Macs do it, which is fundamentally different. In particular, this #rpath thing works really nicely on Linux/Unix systems, but not so much on Macs. I guess with their own conda-installed libraries, they have a way to make it all work correctly , but it doesn't mesh very well with outside libraries that are installed in the normal Mac way.
Anyway, since it's just the OpenMP stuff in TMV, and the matrix operations in GalSim are never time critical, your easiest option is probably to install TMV without OpenMP support using scons install WITH_OPENMP=false.
Then try installing GalSim again.
I am trying to build GraphChi on OS X Yosemite but get the following error:
fatal error: 'omp.h' file not found
From this question - How to include omp.h in OS X? - I learned that Yosemite uses Clang instead of gcc, which does not include omp.h.
$ which gcc
/usr/bin/gcc
$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.0 (clang-600.0.56) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.1.0
Thread model: posix
Next, I installed gcc via Homebrew
$ brew info gcc
gcc: stable 4.9.2 (bottled)
http://gcc.gnu.org
/usr/local/Cellar/gcc/4.9.2_1 (1092 files, 177M)
Built from source with: --without-multilib
and updated $PATH to include the path to the new gcc version
$ echo $PATH
/usr/local/Cellar/gcc/4.9.2_1:usr/local/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin
however, gcc -v and which gcc still point to the old version, and building GraphChi still doesn't work due to the missing omp.h file
Does anyone know what else I need to do?
Update
locate omp.h returned:
/usr/local/Cellar/apple-gcc42/4.2.1-5666.3/lib/gcc/i686-apple-darwin11/4.2.1/include/omp.h
/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin14.1.0/4.9.2/include/omp.h
/usr/local/Cellar/gfortran/4.8.2/gfortran/lib/gcc/x86_64-apple-darwin13.0.0/4.8.2/include/omp.h
my ~/.profile:
export PATH=/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin14.1.0/4.9.2/include:/usr/local/Cellar/gcc/4.9.2_1/bin:usr/local/bin:/opt/local/bin:/opt/local/sbin:$PATH
I solved this with installing gcc with homebrew:
brew install gcc --without-multilib
and then building the source code with
CC=gcc-5 CXX=g++-5 cmake ..
CC=gcc-5 CXX=g++-5 make -j7
Once you have installed gcc-4.9 with homebrew, it will automatically be in your path. To use OpenMP, you just need to make sure you are using the newly installed gcc-4.9, and it will be able to find omp.h.
In the case of GraphChi, you will have to go change line 3 of the Makefile to be gcc-4.9. From there, running make should just work. They describe this in their README, but at least the version they describe is out of date https://github.com/GraphChi/graphchi-cpp#problems-compiling-on-mac.
clang does not support OpenMP yet. Also gcc by default links to Apple's LLVM clang compiler (not the GCC installed from brew).
Instead gcc-4.9 would link to GCC. I think if -fopenmp is specified omp.h is included automatically.
It is possible to manually build a version of clang with OpenMP support, see http://clang-omp.github.io
You shouldn't add the include path to PATH; instead, specify it as CFLAGS, including the -I option. You can export the CFLAGS variable, or set it on the fly.
Depending on how you compile things, you could do
CFLAGS=-I/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin14.1.0/4.9.2/include/omp.h gcc <whatever>
Of course, in this case you can specify it directly on the gcc command (as -I/usr/local/....), but the CFLAGS variable also works with configure (as configure often won't have an option to specify where it should look for specific include files); probably with make, or even for those installing a Python package: CFLAGS=-I... pip install <some-package>.
Other flags to consider are
CXXFLAGS: C++ specific pre-processor flags
LDFLAGS: linker specific flags (e.g. LDFLAGS=-L/some/path/... for linking with dynamic libraries).
CC: specify the C compiler to use. This is an easy way to avoid the built-in gcc alias for clang on OS X. Just use CC=/usr/local/bin/gcc-4 make or similar.
CXX: specify the C++ compiler to use.
I have a C++ shared library being called from my Python program. The C++ lib is compiled with HDF5 which I installed using homebrew on OS X, so it resides in /usr/local/lib. Now the problem is that I also have PyTables installed, which includes an older version of libhdf5, installed somewhere in /Library/Frameworks/EPD64.framework/Versions/Current/...
I can compile and link my library just fine, and otool -L indeed points to /usr/local/lib/libhdf5.dylib. However, when I try to run it from Python, there is a version mismatch error. Somehow the runtime linker is loading the older hdf5 library instead.
One workaround is to build hdf5 as a static library instead, but I'd like to find a neater solution - how can I persuade the runtime linker to use the newer library? I already tried setting DYLD_LIBRARY_PATH but that just broke everything else (Python and MacVim wouldn't start).
Can you provide more info? I have a few alternatives (and questions).
If I understood correctly, you have a library that uses lib_ver_a.dyld and pyTables that uses lib_ver_b.dyld. I.e. the same library but different version.
You could either:
Compile your library to use the same version as pyTables (DYLD_LIB... while you compile) and run your program with pyTables library: DYLD_LIBRARY_PATH=/correct/path/lib python myprog.py
Upgrade your xcode (which version you have? also which osx version?). This upgrades your libraries and pyTables (be careful, newest is not necessary the most stable)
You should also check which python you are calling (a custom one or apple one).
There is also a possibility to define a specific dyld while linking (Do you use gcc or ld for linking? I might miss some options here):
gcc /path/lib1.dyld myLib.c -o myLib.o
This way your library is linked to certain version of the library (caution! I don't remember the exact details, but I can dig them out).
I hope this helps.
br,
Juha