How to change NumPy and conda MKL version - performance

How can I change the MKL (Math Kernel Library) version used by NumPy and Miniconda?
Intel's MKL doesn't perform well because on AMD processors, as MKL deliberately chooses the slowest path on non-intel CPUs, "crippling" numerical processes on AMD processors. This is very problematic for scientific work on AMD CPUs where numerical calculations using NumPy are greatly affected by this "cripple AMD" functionality.
The Python I am using (Python 3.9.5) is distributed by Miniconda, and NumPy was installed using conda install numpy. This installed NumPy with MKL version 2021.0.3.
Previously a workaround for the "cripple AMD" function was to set an environment variable MKL_DEBUG_CPY_TYPE=5. But starting MKL 2020, this variable was removed (intel, why??), so it was no longer able to get this workaround working.
So the solution now is to downgrade NumPy's (and conda's) MKL version to 2019. How can this be achieved? How can I change the MKL version used by NumPy and Conda, from 2021.0.3 to 2019?
System info:
Python: 3.9.5
conda: 4.10.3
NumPy: 1.20.3
MKL (mkl.get_version_string()): 2021.3
NumPy MKL (np.__mkl_version__): 2021.0.3
Please let me know because this is a critical issue for scientific computation on AMD CPUs.
Thank you in advance!!!
PS: Before you say "MKL is written by Intel for Intel processors, so its ok to cripple other processors!", please keep in mind that there should be a competitive spirit during competition, such as amazing innovations, not anti-competitive actions like deliberately slowing down performance on competitor's CPUs . If you want to win a race, practice and improve your running technique, don't break your competitor's legs.
Please avoid a debate, and try to answer my question if you can. If you can't, just ignore and leave.

I would make a new environment, and probably source from the Anaconda channel. The following works for me:
Bash
## create environment
conda create -n foo -c anaconda python numpy mkl=2019.* blas=*=*mkl
## activate and launch python
conda activate foo
python
Python
import mkl
import numpy as np
mkl.get_version()
## 'Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications'
np.__mkl_version__
## '2019.4'
np.show_config()
## blas_mkl_info:
## libraries = ['mkl_rt', 'pthread']
## library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
## define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
## include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']
## blas_opt_info:
## libraries = ['mkl_rt', 'pthread']
## library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
## define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
## include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']
## lapack_mkl_info:
## libraries = ['mkl_rt', 'pthread']
## library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
## define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
## include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']
## lapack_opt_info:
## libraries = ['mkl_rt', 'pthread']
## library_dirs = ['/Users/mfansler/miniconda3/envs/foo/lib']
## define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
## include_dirs = ['/Users/mfansler/miniconda3/envs/foo/include']
Notes
I'm leaving this here as an "as-is" answer, since there seems to be some complications that are beyond me. Namely, Anaconda and Conda Forge appear to have different integration strategies when it comes to NumPy + MKL. Anaconda builds NumPy with direct integration (including the np.__mkl_version__ extension); Conda Forge appears to generically build NumPy with BLAS/LAPACK, and in turn builds libblas, liblapack variants based on MKL implementations. Not sure what differences these strategies might make.
Anaconda channel only has Python 3.8 at this point - but that fits with the intended MKL 2019.* anyway. Python 3.9 was a late 2020 release.
The blas=*=*mkl is crucial: that is what constrains to using an MKL build for NumPy.
This was on osx-64 platform - hopefully, the differences are not substantial.

According to conda-forge documentation:
You can switch your BLAS implementation by doing,
conda install "libblas=*=*mkl"
conda install "libblas=*=*openblas"
conda install "libblas=*=*blis"
conda install "libblas=*=*accelerate"
conda install "libblas=*=*netlib"
This would change the BLAS implementation without changing the conda packages depending on BLAS.
The following legacy commands are also supported as well.
conda install "blas=*=mkl"
conda install "blas=*=openblas"
conda install "blas=*=blis"
conda install "blas=*=accelerate"
conda install "blas=*=netlib"

Related

Why does pip3 install suddenly not work with a virtual environment

I am working on a python package using a virtual environment on Mac OS version 11.1 . I use the following command to create the virtual environment.
python3 -m venv .venv
Then I use this command to activate the virtual environment;
source .venv/bin/activate
I have used virtual environments in packages many times and have never had a problem. Suddenly with I try a pip install, lets say pip3 install NumPy the install fails with a massive error description. The first few lines state the following;
ERROR: Command errored out with exit status 1:
command: /Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/bin/python3 /Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/rb/11zk766j0jngjmv3qmvcfg7c0000gn/T/tmpc_rc8z1m
cwd: /private/var/folders/rb/11zk766j0jngjmv3qmvcfg7c0000gn/T/pip-install-bx_8yuq0/numpy
Complete output (2154 lines):
Running from numpy source directory.
numpy/random/_bounded_integers.pxd.in has not changed
numpy/random/_philox.pyx has not changed
numpy/random/_bounded_integers.pyx.in has not changed
numpy/random/_sfc64.pyx has not changed
numpy/random/_mt19937.pyx has not changed
numpy/random/bit_generator.pyx has not changed
Processing numpy/random/_bounded_integers.pyx
numpy/random/mtrand.pyx has not changed
numpy/random/_generator.pyx has not changed
numpy/random/_pcg64.pyx has not changed
numpy/random/_common.pyx has not changed
Cythonizing sources
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
libraries mkl_rt not found in ['/Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
blis_info:
libraries blis not found in ['/Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
openblas_info:
libraries openblas not found in ['/Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
libraries tatlas not found in ['/Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_3_10_blas_info:
libraries satlas not found in ['/Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_blas_threads_info:
Setting PTATLAS=ATLAS
libraries ptf77blas,ptcblas,atlas not found in ['/Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
atlas_blas_info:
libraries f77blas,cblas,atlas not found in ['/Users/jonwebb/Desktop/Code_Dev/Python/core_utilities/.venv/lib', '/usr/local/lib', '/usr/lib']
NOT AVAILABLE
/private/var/folders/rb/11zk766j0jngjmv3qmvcfg7c0000gn/T/pip-install-bx_8yuq0/numpy/numpy/distutils/system_info.py:1989: UserWarning:
The last few lines look like;
########### CLIB COMPILER OPTIMIZATION ###########
Platform :
Architecture: x64
Compiler : clang
CPU baseline :
Requested : 'min'
Enabled : SSE SSE2 SSE3
Flags : -msse -msse2 -msse3
Extra checks: none
CPU dispatch :
Requested : 'max -xop -fma4'
Enabled : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
Generated : none
CCompilerOpt._cache_write[796] : write cache to path -> /private/var/folders/rb/11zk766j0jngjmv3qmvcfg7c0000gn/T/pip-install-bx_8yuq0/numpy/build/temp.macosx-11-x86_64-3.9/ccompiler_opt_cache_clib.py
----------------------------------------
ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly
Until recently this was not a problem and I can still conduct pip3 installs outside of a virtual environment. Does anyone know what the cause of this issue is?

osx - C++14 compiler not detected, multiple versions of gcc when compiling graph-tool in anaconda

My ultimate goal is to get python package graph_tool working on my system and also on ipynb if possible. I have already brew install graph-tool, as indicated here, but that's still insufficient.
So I follow conda instructions here, and I make decent progress, until I encounter a compiler issue with the configuration.
$ conda create -n py36env python=3.6.3 anaconda
(py36env) $ conda install -c conda-forge cgal
... and so forth with the other required libraries
(py36env) Tams-MacBook-Pro:graph-tool-2.25 tamtam$ ./configure --prefix=/Users/tamtam/anaconda3/envs/py36env/ --with-python-module-path=/Users/tamtam/anaconda3/envs/py36env/lib/python3.6/site-packages
.
.
.
checking whether g++ supports C++14 features by default... no
checking whether g++ supports C++14 features with -std=gnu++14... no
checking whether g++ supports C++14 features with -std=gnu++1y... no
configure: error: *** A compiler with support for C++14 language features is required.
I'm not very familiar with compiler things, but I check my system as follows:
(py36env) Tams-MacBook-Pro:graph-tool-2.25 tamtam$ conda list | grep gcc
gcc 4.8.5 8
And outside of the conda env:
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/Users/tamtam/anaconda3/envs/py36env/bin/../libexec/gcc/x86_64-apple-darwin11.4.2/4.8.5/lto-wrapper
Target: x86_64-apple-darwin11.4.2
Configured with: ./configure --prefix=/Users/ray/mc-x64-3.5/conda-bld/gcc-4.8_1477649012852/
Thread model: posix
gcc version 4.8.5 (GCC)
And with Homebrew
$ brew list --versions | grep gcc
gcc 7.1.0 7.2.0
$ /usr/local/bin/gcc-7 --version
gcc-7 (Homebrew GCC 7.2.0) 7.2.0
$ echo $PATH
/Users/tamtam/anaconda3/bin:/Users/tamtam/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
So my questions are: EDITS INCLUDED
1) What's the difference between the conda gcc 4.8.5 and Homebrew gcc 7.2 on my system? I'm confused because they're each up-to-date packages.
EDIT: Nowhere online that I perused explicitly states this, but I am understanding (aka educated guess bc it's 3am and I need sleep) conda gcc 4.8.5 to just be equivalent to brew gcc 4.8.5, and that Anaconda is just way behind on developing an 'official' gcc supporting C++14. In the meantime I conda install -c salford_systems gcc-5 and ./configure seems to find itself a C++14 compiler. YAY (but this also led me to a new problem T.B.A.)
NOTE: I had also tried running the ./configure file with both conda gcc 4.8.5 and brew gcc 7.2.0 (switching of default gcc compiler guided here), but still same error of ./configure not finding C++14 compile (so ignore this)
2) I also have Xcode 8.3.3 that also comes with clang, which is what gcc is symlinked to... ?? No idea about the setup/relation
3) How can I adjust my system (for example, paths) so that the configuration process will detect a C++14 compiler? EDIT: resolved per 1)
(This solution is also included within the edits at the bottom of my question.)
Nowhere online that I perused explicitly states this, but I am understanding (aka educated guess bc it's 3am and I need sleep) conda gcc 4.8.5 to just be equivalent to brew gcc 4.8.5, and that Anaconda is just way behind on developing an 'official' gcc supporting C++14. In the meantime I conda install -c salford_systems gcc-5 and ./configure seems to find itself a C++14 compiler. YAY (but this also led me to a new problem)
NOTE: I had also tried running the ./configure file with both conda gcc 4.8.5 and brew gcc 7.2.0 (switching of default gcc compiler guided here), but still same error of ./configure not finding C++14 compile arises (so ignore this)
As a rule of thumb, if a product provides its own build tools (compiler, C headers, static libraries, include/library paths etc) instead of using the system's ones, you should use those when building things to use within its environment.
That's because, depending on include files, compiler, compiler version, compiler flags, predefined macros etc the same sources and libraries built with different toolchains can be binary incompatible. This goes double for C++ where there are no ABI standards at all. (In C, there are de facto ones for most platforms where there are competing compilers due to the need to do system calls.)
Now, what you have on your system:
The system's stock gcc (if any) is at /usr/bin (so includes are at /usr/include, static libraries at /usr/lib etc)
Homebrew's gcc 7.2 at /usr/local/bin (includes are probably at /usr/local/include)
Anaconda also has a gcc package that, if installed, would reside somewhere like /Users/<login>/anaconda3/envs/<env name>/bin
If an environment provides an own toolchain, it provides some way to make build scripts select that toolchain when building things. A common way is to set relevant environment variables (PATH, INCLUDE, LIB) - either directly or through some tool that the environment provides. Homebrew executables reside in /usr/local/bin which is always present in UNIX PATH as per the FHS; Anaconda adds itself to PATH whenever its environment is activated.
So, as you yourself guessed, since Anaconda provides its own gcc packages (thus enforces the compatibility of their output with its binary packages via package metadata - if there are known incompatibilities, the installation will fail due to requirement conflicts), you need to install one that meets graph-tool requirements -- which is gcc 5, available via gcc-5 package from salford_systems channel. Then specify --prefix as you already have.
You may need to install other dependencies, too, if you didn't already or you need optional features that they're required for.

Anaconda cython-built module is looking for wrong GLIBCXX

Let me explain the situation.
I compiled a c++ library using system g++ (under linux).
Then built a cython module which calls a function in the pre-compiled library. Building the cython module was done under an anaconada venv (but no g++ installed inside the venv. Hence cython must had used system gcc/g++).
Now under the same anaconda venv, importing the cython module results in an error as:
anaconda3/envs/hspy/lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./libc_rd_halo.so).
It seems like the c library (libc_rd_halo.so) which was compiled by system g++ is now looking for an anaconda g++ library. And I don't have g++ installed in anaconda venv.
I can fix the problem by using anaconda g++ to compile the c library from the beginning. Build the cython module again, and then I can import the module.
But I'd like to let anaconda cython use system compiler and look for the system compiler library when the module is being imported.
My questions are:
why does anaconda cython look for a local anaconda g++ by default, but not the system compiler? any benefit doing so?
How can I make anaconda cython look for system compiler library?
why does anaconda cython look for a local anaconda g++ by default, but not the system compiler? any benefit doing so?
How can I make anaconda cython look for system compiler library?
Because libraries compiled with one version of GCC can only be linked with other libraries that were compiled with a compatible version of GCC, as you found out.
You don't. Per above, binaries build with the system compiler may or may not work with other anaconda binaries built using the anaconda compiler.
If you want your binary to use system libraries then you need to make a system package, not an anaconda package. The system package will then only work on that system (and version of).

Questions about Octave 3.8.2 compilation on CentOS 6.4

I try to compile Octave 3.8.2 on CentOS 6.4 (serve), but there's an error:
A BLAS library was detected but found incompatible with your Fortran 77 compiler settings.
I'm so confused. I compiled a BLAS and a LAPACK myself, and I added their path to bashrc. But this error appeared again. I try to change fflags to ff2c as other resources suggested, but this error also occurred.
I found the following guide and discussion of my error.
I am going through this, and I had the same issues that you are mentioning. I solved it by using the auto config options to point it to the correct library, e.g.
./configure --with-lapack=/path/to/liblapack.so.3 --with-blas=/path/to/libf77blas.so.3
It compiles and produces an executable.
I would suggest using OpenBLAS instead of the common BLAS and LAPACK packages. It is written in the Octave documentation:
"Accelerated BLAS libraries such as OpenBLAS (https://www.openblas.net/) or ATLAS (http://math-atlas.sourceforge.net) are recommended for best performance. The reference implementation (http://www.netlib.org/blas) is slow, unmaintained, and suffers from certain bugs in corner case inputs."
You can download it from https://www.openblas.net/ and install it with the classical ./configure && make %% sudo make install. After this move the libraries generated by OpenBLAS to /usr/lib64 and run:
./configure -with-blas="-lopenblas"
Add other flags and arguments to configure if needed.

Installing GalSim, on OSX with anaconda

I'm trying to install GalSim on OSX Mavericks, using the anaconda python distribution, but keep running into the error:
Unable to build a python loadable module using the python executable:
/usr/bin/env python,
the library name libpython2.7.a,
and the libdir /Users/harrison/anaconda/lib/python2.7/config.
If these are not the correct library names, you can tell scons the
correct names to use with the flags EXTRA_LIB_PATH and/or EXTRA_LIBS.
the /usr/bin/env python is the anaconda one and there is indeed a libpython2.7.a in that libdir.
scons is installed via the conda package management system.
boost is installed manually as recommended via
./bootstrap.sh --with-python=/Users/harrison/anaconda/bin/python --with-python-root=/Users/harrison/anaconda/
./b2 -a link=shared
sudo ./b2 -a --prefix=/usr/local link=shared install
checking the version of python boost links to gives
$ otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python.dylib:
libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
Is this correct (it doesn't seem so to me -- boost seems to be linking to the libpython2.7 in /usr/local/) and is this what is causing the problem installing GalSim?
Thanks
Ian
Full output:
$ scons
scons: Reading SConscript files ...
SCons is version 2.3.0 using python version 2.7.6
Python is from /Users/harrison/anaconda/include/python2.7
Using the following (non-default) scons options:
CXX = g++
These can be edited directly in the file gs_scons.conf.
Type scons -h for a full list of available options.
Using python = /usr/bin/env python
Using default PYPREFIX = /Users/harrison/anaconda/lib/python2.7/site-packages
Using compiler: /usr/bin/g++
compiler version: 4.2.1
Determined that a good number of jobs = 4
Checking for C++ header file fftw3.h... yes
Checking for correct FFTW linkage... yes
Checking for C++ header file boost/shared_ptr.hpp... yes
Checking for C++ header file TMV.h... yes
Using TMV_LINK file: /usr/local/share/tmv/tmv-link
-L/usr/local/Cellar/tmv-cpp/0.71/lib -ltmv -lblas
Mac version is 10.9
WARNING: The Apple BLAS library has been found not to be thread safe on
Mac OS version 10.7 (and possibly higher), even across multiple
processes (i.e. not just multiple threads in the same process).
The symptom is that `scons tests` will hang when running nosetests
using multiple processes.
If this occurs, the solution is to compile TMV either with a
different BLAS library (e.g. ATLAS) or with no BLAS library at
all (using WITH_BLAS=false).
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage... yes
Checking if we can build against Python...
Unable to build a python loadable module using the python executable:
/usr/bin/env python,
the library name libpython2.7.a,
and the libdir /Users/harrison/anaconda/lib/python2.7/config.
If these are not the correct library names, you can tell scons the
correct names to use with the flags EXTRA_LIB_PATH and/or EXTRA_LIBS.
Please fix the above error(s) and rerun scons.
Note: you may want to look through the file INSTALL.md for advice.
Also, if you are having trouble, please check the INSTALL FAQ at
https://github.com/GalSim-developers/GalSim/wiki/Installation%20FAQ
Ian, thanks for posting that. Based on the config.log and my own experience, I have an answer that should solve at least this problem (no guarantees that you won't hit something else later in the build).
The key problem here is that if you tell it the compiler is gcc (as is the default), it will try to use openMP. But actually invoking gcc on Mavericks gives you clang++, for which openMP doesn't work. So, you have to explicitly tell galsim that the compiler is clang++ so it won't try to use openMP. (The clue to this is in config.log, there are messages in various places about undefined symbols for x86_64 architecture... I had this problem on my own system when I started using Mavericks. See also https://github.com/GalSim-developers/GalSim/issues/493 )
So, please do scons CXX=clang++ and this problem should go away. I cannot guarantee that you won't hit a boost problem later on, however.
Please also see this question which has two suggestions which may help solve the problem for you as well.

Resources