After upgrading python 3.8.6 to 3.9.10 using homebrew, my Cython extensions no longer without explicitly adding /usr/local/include (for my Intel MacBook) or /opt/homebrew/include to the include_dirs of my extension.
My setup.py.in:
import os, sys
from numpy.distutils.core import setup, Extension
from Cython.Build import cythonize
link_arguments = []
extra_include_dirs = []
if (sys.platform == 'darwin'):
link_arguments.append("-Wl,-rpath")
link_arguments.append("-Wl,#loader_path/")
if os.path.exists('/opt/homebrew/'):
extra_include_dirs.append("/opt/homebrew/include/")
else:
extra_include_dirs.append("/usr/local/include/")
else:
link_arguments.append("-Wl,-rpath=${CMAKE_SOURCE_DIR}/lib/")
pynwp_extension = Extension(
name="pynwp",
sources=["${CMAKE_CURRENT_SOURCE_DIR}/lambert.f90", "${CMAKE_CURRENT_SOURCE_DIR}/pynwp.f90", "${CMAKE_CURRENT_SOURCE_DIR}/readAtmosphereGen.f90", "${CMAKE_CURRENT_SOURCE_DIR}/ptogrot.f", "${CMAKE_CURRENT_SOURCE_DIR}/bilin1.f", "${CMAKE_CURRENT_SOURCE_DIR}/fl2pres_f.f","${CMAKE_CURRENT_SOURCE_DIR}/message.c","${CMAKE_CURRENT_SOURCE_DIR}/gridWindDirCorrection.F"],
libraries=["HirlamUtils_fPIC", "eccodes_f90", "jasper"],
library_dirs=["${PROJECT_BINARY_DIR}", "${CMAKE_SOURCE_DIR}/build${CMAKE_BUILD_TYPE}/src/libHirlamUtils/", "/opt/homebrew/lib/"],
extra_link_args = link_arguments,
include_dirs=["${CMAKE_SOURCE_DIR}/include", "/usr/lib64/gfortran/modules/",
"${CMAKE_SOURCE_DIR}/build${CMAKE_BUILD_TYPE}"] + extra_include_dirs,
extra_f90_compile_args=["-DLINUX", "-DIS_LITTLE_ENDIAN", "-DUSEWALLTIME", "-DHAS_BLAS", "-DHAS_LAPACK", "-DGRIB32", "-DTIMING", "-DPREC32", "-fno-whole-file", "-g", "-fbounds-check"]
#compiler_directives={'language_level' : "3"}
)
setup(name="pynwp",
author="me",
author_email="me!me.com",
version="1.0.1",
description="Python wrapper for pynwp",
package_dir={"": "${CMAKE_CURRENT_SOURCE_DIR}"},
url="http://emaddc.eu",
license="MIT License",
ext_modules=[pynwp_extension]
)
In the file above, I have hardcoded the location (temporarily) of the homebrew library dir and added some functionality for the include dir based on /opt/homebrew being found. If I remove this from the file, compilation fails as eccodes.mod cannot be found, see the output when I run the command generate by python/CMake manually:
buildDebug git:(master) ✗ /opt/homebrew/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops -I<project_dir>//include -I/usr/lib64/gfortran/modules/ -I<project_dir>//buildDebug -Ibuild/src.macosx-12-arm64-3.9/build/src.macosx-12-arm64-3.9 -I/opt/homebrew/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.macosx-12-arm64-3.9/numpy/distutils/include -I/opt/homebrew/opt/python#3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c -c <project_dir>//src/pynwp/readAtmosphereGen.f90 -o build/temp.macosx-12-arm64-3.9<project_dir>//src/pynwp/readAtmosphereGen.o -DLINUX -DIS_LITTLE_ENDIAN -DUSEWALLTIME -DHAS_BLAS -DHAS_LAPACK -DGRIB32 -DTIMING -DPREC32 -fno-whole-file -g -fbounds-check
f951: Warning: Nonexistent include directory '/usr/lib64/gfortran/modules/' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory 'build/src.macosx-12-arm64-3.9/build/src.macosx-12-arm64-3.9' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory 'build/src.macosx-12-arm64-3.9/numpy/distutils/include' [-Wmissing-include-dirs]
<project_dir>//src/pynwp/readAtmosphereGen.f90:3:7:
3 | use eccodes
| 1
Fatal Error: Cannot open module file 'eccodes.mod' for reading at (1): No such file or directory
compilation terminated.
This extension is part of a larger project that is build using CMake. When running the gfortran command manually, the same errors indicating that eccodes.mod cannot be found. The file is however located on a default location on the gfortran/gcc search path:
locate eccodes.mod
/opt/homebrew/Cellar/eccodes/2.24.2/include/eccodes.mod
/opt/homebrew/include/eccodes.mod
And the search path for gfortran:
gfortran -E -Wp,-v -
#include <...> search starts here:
/opt/homebrew/include
/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../lib/gcc/11/gcc/aarch64-apple-darwin21/11/include
/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../lib/gcc/11/gcc/aarch64-apple-darwin21/11/include-fixed
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks
End of search list.
Another project uses gcc/ld and has a similar problem. I need to explicitly add LINK_DIRECTORIES(/opt/homebrew/lib) to CMakeList.txt in order for gcc to find the eccodes library. Without that, I get:
gcc-11: warning: this compiler does not support X86 (arch flags ignored)
ld: library not found for -leccodes
collect2: error: ld returned 1 exit status
make[2]: *** [src/smoothModeS-v51/smoothModeS-v51.x] Error 1
make[1]: *** [src/smoothModeS-v51/CMakeFiles/smoothModeS-v51.x.dir/all] Error 2
make: *** [all] Error 2
Adding the paths to CPATH and LIBRARY_PATH had no effect.
This method works but seems hard to maintain. What am I missing and what has changed that gfortran/gcc no longer search in the default paths for but libraries and headers/modules?
EDIT
Just found that the standalone executable that uses similar code as the python extension and also uses eccodes has a similar issue. If I do not include INCLUDE_DIRECTORIES(/opt/homebrew/include) in CMakeLists.txt , I get:
cd <project_dir>/buildDebug/src/collocEHS && /opt/homebrew/bin/gfortran -I<project_dir>/buildDebug -I<project_dir>/include -I<project_dir>/src/readASTERIX2 -I<project_dir>/src/geomag70_linux -I<project_dir>/src/libDTG -fallow-argument-mismatch -ffpe-trap=invalid,zero,overflow -DPREC32 -DLINUX -DIS_LITTLE_ENDIAN -DUSEWALLTIME -DHAS_BLAS -DHAS_LAPACK -DGRIB32 -DTIMING -ffixed-line-length-none -g -fcheck=all -Wall -fcheck=bounds -O0 -g -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk -c <project_dir>/src/pynwp/readAtmosphereGen.f90 -o CMakeFiles/collocEHSv2.dir/__/pynwp/readAtmosphereGen.f90.o
<project_dir>/src/pynwp/readAtmosphereGen.f90:3:7:
3 | use eccodes
| 1
Fatal Error: Cannot open module file 'eccodes.mod' for reading at (1): No such file or directory
compilation terminated.
make[2]: *** [src/collocEHS/CMakeFiles/collocEHSv2.dir/__/pynwp/readAtmosphereGen.f90.o] Error 1
make[1]: *** [src/collocEHS/CMakeFiles/collocEHSv2.dir/all] Error 2
make: *** [all] Error 2
So I guess the problem is unrelated to python but more with gcc/gfortran (gcc version 11.2.0 (Homebrew GCC 11.2.0_3).
EDIT 2
A reboot of the laptop fixed the issue with the python extension build and setup.py requiring additional directories. For normal builds using CMake, I still require the extra INCLUDE_DIRECTORIES and LINK_DIRECTORIES order for gcc/gfortran to find libraries installed by brew in /opt/homebrew (or /usr/local for intel MacBook).
As per homebrew devs, this is desired behaviour: /opt/homebrew and /usr/local are "special" directories to be manually added in e.g., CMake projects. This is explained in my bug report on home-brew's GitHub, see https://github.com/Homebrew/homebrew-core/issues/95561.
I haven't been able to confirm this with documentation.
I'm trying make a program which requires Armadillo as a dependency. I've installed Armadillo in a local directory (I'm working on a cluster without root privileges), but am unable to install the program because it can't find armadillo. How would I specify that? Thanks in advance for the help!
What I'm doing & seeing:
$ make
g++ -O3 -o ancestry_hmm ancestry_hmm.cpp -larmadillo
ancestry_hmm.cpp:22:21: fatal error: armadillo: No such file or directory
#include <armadillo>
^
compilation terminated.
make: *** [all] Error 1
The current Make file:
TCFLAGS = -ltcmalloc
ARMAFLAGS = -larmadillo
all:
$(LINK.cc) -O3 -o ancestry_hmm ancestry_hmm.cpp $(ARMAFLAGS)
Path to armadillo:
/{user}/Programs/Anaconda/*.so
/{user}/Programs/Anaconda/
Answer was provided by Claes Rolen in the comments. I had to also add -lgfortran, but it works, thanks!
I guess I don't have enough experience installing software with g++. I am trying to install FIt-SNE. I am following the installation instructions. I have already installed FFTW, and I just need to execute the following line:
g++ -std=c++11 -O3 src/sptree.cpp src/tsne.cpp src/nbodyfft.cpp -o bin/fast_tsne -pthread -lfftw3 -lm -Wno-address-of-packed-member
And I get this error:
In file included from src/tsne.cpp:40:0:
src/nbodyfft.h:7:10: fatal error: fftw3.h: No such file or directory
#include <fftw3.h>
^~~~~~~~~
compilation terminated.
In file included from src/nbodyfft.cpp:5:0:
src/nbodyfft.h:7:10: fatal error: fftw3.h: No such file or directory
#include <fftw3.h>
^~~~~~~~~
compilation terminated.
I understand that the installer is not finding the FFTW package, and I think I need to provide the path to the installation somehow to the installer, but I don't know how to do it. Can anyone help me with this?
I have been following the instructions to install CalculiX and its dependencies as best as I could from the wiki page https://github.com/precice/calculix-adapter/wiki/Installation-instructions-for-CalculiX
I have already installed all the dependencies.
In the installation when I say make I get the following error:
gfortran -fopenmp -Wall -O3 -o bin/ccx_preCICE bin/ccx_2.15.o bin/ccx_2.15.a /home/prasad/SPOOLES.2.2//spooles.a -L/usr/local/lib -lprecice -lstdc++ -L/home/prasad/yaml-cpp-yaml-cpp-0.6.2//build -lyaml-cpp /home/prasad/ARPACK//libarpack_INTEL.a -lpthread -lm -lc
/home/prasad/ARPACK//libarpack_INTEL.a(second.o): In function `second_':
second.f:(.text+0x14): undefined reference to `etime_'
collect2: error: ld returned 1 exit status
Makefile:101: recipe for target 'bin/ccx_preCICE' failed
make: *** [bin/ccx_preCICE] Error 1
I tried changing the line 24 in ARPACK/UTIL/second.f to * EXTERNAL ETIME, but I am still getting the same error.
Can anyone tell me how to fix it?
Thank you in advance for the help.
Edit line 24 in ARPACK/UTIL/second.f
Append * to get
* EXTERNAL ETIME
If you get this error when you are building the adapter for CalculiX, go to ARPACK
make lib
Go back to the calculiX-adapter-master directory or the directory from where you were building CalculiX and its adapter
make clean
make
For detailed guide please check preCICE wiki for CalculiX adapter
I am following http://openrisc.net/toolchain-build.html to build gcc toolchain for openrisc or32.
I'm doing 'building by hand' flow and had passed
binutils
stage 1 gcc
install linux headers
and was to do 'compile uClibc' which is composed of commands below.
$ git clone git://openrisc.net/jonas/uClibc
$ cd uClibc
$ make ARCH=or32 defconfig
$ make PREFIX=${SYSROOT}
$ make PREFIX=${SYSROOT} install <br>
when I run 'make ARCH=or32 defconfig', I get this error.
CC libpthread/linuxthreads.old/attr.o
In file included from libpthread/linuxthreads.old/internals.h:30:0,
from libpthread/linuxthreads.old/attr.c:26:
./libpthread/linuxthreads.old/sysdeps/or32/pt-machine.h: In function 'testandset':
./libpthread/linuxthreads.old/sysdeps/or32/pt-machine.h:41:8: error: '__NR_or1k_atomic' undeclared (first use in this function)
./libpthread/linuxthreads.old/sysdeps/or32/pt-machine.h:41:8: note: each undeclared identifier is reported only once for each function it appears in
In file included from libpthread/linuxthreads.old/../linuxthreads.old_db/proc_service.h:20:0,
from libpthread/linuxthreads.old/../linuxthreads.old_db/thread_dbP.h:9,
from libpthread/linuxthreads.old/internals.h:32,
from libpthread/linuxthreads.old/attr.c:26:
./include/sys/procfs.h: At top level:
./include/sys/procfs.h:32:21: fatal error: asm/elf.h: No such file or directory
compilation terminated.
make: *** [libpthread/linuxthreads.old/attr.o] Error 1
Has anybody had same problem? I use CentOS 6.4.
gcc searches for the header file from the system in the order
/usr/local/include
libdir/gcc/target/version/include (libdir was /usr/lib in my case)
/usr/target/include
/usr/include
my system had sys/syscall.h under /usr/include so that file was used when sys/syscall under uClib/include should have been used. So I added -nostdinc so that gcc doesn't search the standard include path. Now it became
make PREFIX=${SYSROOT} -nostdinc
and it works!
The following command was also modified
make PREFIX=${SYSROOT} -nostdinc install
Cheers!