After upgrading python 3.8.6 to 3.9.10 using homebrew, my Cython extensions no longer without explicitly adding /usr/local/include (for my Intel MacBook) or /opt/homebrew/include to the include_dirs of my extension.
My setup.py.in:
import os, sys
from numpy.distutils.core import setup, Extension
from Cython.Build import cythonize
link_arguments = []
extra_include_dirs = []
if (sys.platform == 'darwin'):
link_arguments.append("-Wl,-rpath")
link_arguments.append("-Wl,#loader_path/")
if os.path.exists('/opt/homebrew/'):
extra_include_dirs.append("/opt/homebrew/include/")
else:
extra_include_dirs.append("/usr/local/include/")
else:
link_arguments.append("-Wl,-rpath=${CMAKE_SOURCE_DIR}/lib/")
pynwp_extension = Extension(
name="pynwp",
sources=["${CMAKE_CURRENT_SOURCE_DIR}/lambert.f90", "${CMAKE_CURRENT_SOURCE_DIR}/pynwp.f90", "${CMAKE_CURRENT_SOURCE_DIR}/readAtmosphereGen.f90", "${CMAKE_CURRENT_SOURCE_DIR}/ptogrot.f", "${CMAKE_CURRENT_SOURCE_DIR}/bilin1.f", "${CMAKE_CURRENT_SOURCE_DIR}/fl2pres_f.f","${CMAKE_CURRENT_SOURCE_DIR}/message.c","${CMAKE_CURRENT_SOURCE_DIR}/gridWindDirCorrection.F"],
libraries=["HirlamUtils_fPIC", "eccodes_f90", "jasper"],
library_dirs=["${PROJECT_BINARY_DIR}", "${CMAKE_SOURCE_DIR}/build${CMAKE_BUILD_TYPE}/src/libHirlamUtils/", "/opt/homebrew/lib/"],
extra_link_args = link_arguments,
include_dirs=["${CMAKE_SOURCE_DIR}/include", "/usr/lib64/gfortran/modules/",
"${CMAKE_SOURCE_DIR}/build${CMAKE_BUILD_TYPE}"] + extra_include_dirs,
extra_f90_compile_args=["-DLINUX", "-DIS_LITTLE_ENDIAN", "-DUSEWALLTIME", "-DHAS_BLAS", "-DHAS_LAPACK", "-DGRIB32", "-DTIMING", "-DPREC32", "-fno-whole-file", "-g", "-fbounds-check"]
#compiler_directives={'language_level' : "3"}
)
setup(name="pynwp",
author="me",
author_email="me!me.com",
version="1.0.1",
description="Python wrapper for pynwp",
package_dir={"": "${CMAKE_CURRENT_SOURCE_DIR}"},
url="http://emaddc.eu",
license="MIT License",
ext_modules=[pynwp_extension]
)
In the file above, I have hardcoded the location (temporarily) of the homebrew library dir and added some functionality for the include dir based on /opt/homebrew being found. If I remove this from the file, compilation fails as eccodes.mod cannot be found, see the output when I run the command generate by python/CMake manually:
buildDebug git:(master) ✗ /opt/homebrew/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops -I<project_dir>//include -I/usr/lib64/gfortran/modules/ -I<project_dir>//buildDebug -Ibuild/src.macosx-12-arm64-3.9/build/src.macosx-12-arm64-3.9 -I/opt/homebrew/lib/python3.9/site-packages/numpy/core/include -Ibuild/src.macosx-12-arm64-3.9/numpy/distutils/include -I/opt/homebrew/opt/python#3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c -c <project_dir>//src/pynwp/readAtmosphereGen.f90 -o build/temp.macosx-12-arm64-3.9<project_dir>//src/pynwp/readAtmosphereGen.o -DLINUX -DIS_LITTLE_ENDIAN -DUSEWALLTIME -DHAS_BLAS -DHAS_LAPACK -DGRIB32 -DTIMING -DPREC32 -fno-whole-file -g -fbounds-check
f951: Warning: Nonexistent include directory '/usr/lib64/gfortran/modules/' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory 'build/src.macosx-12-arm64-3.9/build/src.macosx-12-arm64-3.9' [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory 'build/src.macosx-12-arm64-3.9/numpy/distutils/include' [-Wmissing-include-dirs]
<project_dir>//src/pynwp/readAtmosphereGen.f90:3:7:
3 | use eccodes
| 1
Fatal Error: Cannot open module file 'eccodes.mod' for reading at (1): No such file or directory
compilation terminated.
This extension is part of a larger project that is build using CMake. When running the gfortran command manually, the same errors indicating that eccodes.mod cannot be found. The file is however located on a default location on the gfortran/gcc search path:
locate eccodes.mod
/opt/homebrew/Cellar/eccodes/2.24.2/include/eccodes.mod
/opt/homebrew/include/eccodes.mod
And the search path for gfortran:
gfortran -E -Wp,-v -
#include <...> search starts here:
/opt/homebrew/include
/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../lib/gcc/11/gcc/aarch64-apple-darwin21/11/include
/opt/homebrew/Cellar/gcc/11.2.0_3/bin/../lib/gcc/11/gcc/aarch64-apple-darwin21/11/include-fixed
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks
End of search list.
Another project uses gcc/ld and has a similar problem. I need to explicitly add LINK_DIRECTORIES(/opt/homebrew/lib) to CMakeList.txt in order for gcc to find the eccodes library. Without that, I get:
gcc-11: warning: this compiler does not support X86 (arch flags ignored)
ld: library not found for -leccodes
collect2: error: ld returned 1 exit status
make[2]: *** [src/smoothModeS-v51/smoothModeS-v51.x] Error 1
make[1]: *** [src/smoothModeS-v51/CMakeFiles/smoothModeS-v51.x.dir/all] Error 2
make: *** [all] Error 2
Adding the paths to CPATH and LIBRARY_PATH had no effect.
This method works but seems hard to maintain. What am I missing and what has changed that gfortran/gcc no longer search in the default paths for but libraries and headers/modules?
EDIT
Just found that the standalone executable that uses similar code as the python extension and also uses eccodes has a similar issue. If I do not include INCLUDE_DIRECTORIES(/opt/homebrew/include) in CMakeLists.txt , I get:
cd <project_dir>/buildDebug/src/collocEHS && /opt/homebrew/bin/gfortran -I<project_dir>/buildDebug -I<project_dir>/include -I<project_dir>/src/readASTERIX2 -I<project_dir>/src/geomag70_linux -I<project_dir>/src/libDTG -fallow-argument-mismatch -ffpe-trap=invalid,zero,overflow -DPREC32 -DLINUX -DIS_LITTLE_ENDIAN -DUSEWALLTIME -DHAS_BLAS -DHAS_LAPACK -DGRIB32 -DTIMING -ffixed-line-length-none -g -fcheck=all -Wall -fcheck=bounds -O0 -g -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk -c <project_dir>/src/pynwp/readAtmosphereGen.f90 -o CMakeFiles/collocEHSv2.dir/__/pynwp/readAtmosphereGen.f90.o
<project_dir>/src/pynwp/readAtmosphereGen.f90:3:7:
3 | use eccodes
| 1
Fatal Error: Cannot open module file 'eccodes.mod' for reading at (1): No such file or directory
compilation terminated.
make[2]: *** [src/collocEHS/CMakeFiles/collocEHSv2.dir/__/pynwp/readAtmosphereGen.f90.o] Error 1
make[1]: *** [src/collocEHS/CMakeFiles/collocEHSv2.dir/all] Error 2
make: *** [all] Error 2
So I guess the problem is unrelated to python but more with gcc/gfortran (gcc version 11.2.0 (Homebrew GCC 11.2.0_3).
EDIT 2
A reboot of the laptop fixed the issue with the python extension build and setup.py requiring additional directories. For normal builds using CMake, I still require the extra INCLUDE_DIRECTORIES and LINK_DIRECTORIES order for gcc/gfortran to find libraries installed by brew in /opt/homebrew (or /usr/local for intel MacBook).
As per homebrew devs, this is desired behaviour: /opt/homebrew and /usr/local are "special" directories to be manually added in e.g., CMake projects. This is explained in my bug report on home-brew's GitHub, see https://github.com/Homebrew/homebrew-core/issues/95561.
I haven't been able to confirm this with documentation.
I'm triing to compile the superoptimizer on windows. (https://github.com/bonzini/superopt)
Unfortunatly my knowledge of make is very limited. I tried:
make CPU=-D386 superopt
Which gives me the error:
cc superopt.c -o superopt
process_begin: CreateProcess(NULL, cc superopt.c -o superopt, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [superopt] Fehler 2
After setting the compiler to gcc manually with:
make CPU=-D386 superopt CC=gcc
I get the following error:
gcc superopt.c -o superopt
In file included from superopt.c:27:0:
superopt.h:104:2: error: #error You have to choose target CPU type (--with-arch).
#error You have to choose target CPU type (--with-arch).
^
In file included from superopt.h:130:0,
from superopt.c:27:
longlong.h:1465:14: error: unknown type name 'UQItype'
extern const UQItype __clz_tab[];
^
superopt.c:32:21: fatal error: version.h: No such file or directory
compilation terminated.
make: *** [superopt] Fehler 1
It seems it does't properly selects i386.
Any hints would be greatly appreciated.
That is not the source code of
GNU superopt. It is the source code of someone's project
to patch GNU superopt, last updated 2008, and seemingly
abandoned a hard-hat area.
This is the source code of
GNU superopt. Extract the tarball and build as you have attempted
with:
make CC=gcc CPU=-DI386 superopt
Note: I386, not 386.
You will see warnings like:
warning: incompatible implicit declaration of built-in function 'foo'
because the 20 yearold C code does not diligently include the standard
headers that prototype the standard functions that it calls, but superopt will build successfully.
I am trying to install PythonMagick following these instructions. https://gist.github.com/tomekwojcik/2778301
When I get to $ make I get this error
Making all in pythonmagick_src
CXX libpymagick_la-_DrawableFillRule.lo
_DrawableFillRule.cpp:3:10: fatal error: 'boost/python.hpp' file not found
#include <boost/python.hpp>
^
1 error generated.
make[1]: *** [libpymagick_la-_DrawableFillRule.lo] Error 1
make: *** [all-recursive] Error 1
How do I get PythonMagick installed in my project? Any way that will work. I can't find useful instructions anywhere on the internet.
Make sure you have boost-python brew boost-python. Note the version number, as you'll need to replace 1.59.0 below with the correct version.
$ BOOST_ROOT=/usr/local/Cellar/boost/1.59.0
$ ./configure
Edit Makefile and pythonmagick_src/Makefile to include the boost library. You are looking for two lines: DEFAULT_INCLUDES and LDFLAGS. You'll add boost paths to the end of those lines, making them look something like this:
DEFAULT_INCLUDES = -I. -I$(top_builddir)/config -I/usr/local/Cellar/boost/1.59.0/include
LDFLAGS = -L/usr/local/Cellar/boost-python/1.59.0/lib
That should resolve the compile/link errors.
I am using a java program. It automatically creates log files in a directory, but I am doing that myself a different way with tee. I cannot find an easy way to disable the logs, so I am resorting to using nullfs.
I cloned it with
git clone https://github.com/xrgtn/nullfs.git
and I ran
make nul1fs
as instructed. It terminates within a second, with the following output:
cc "-lfuse" nul1fs.c -o nul1fs
nul1fs.c:13:18: fatal error: fuse.h: No such file or directory
compilation terminated.
make: *** [nul1fs] Error 1
I tried apt-get source fuse and copying fuse.h into the nullfs directory, but nothing changed.
I have FUSE installed. I'm running Debian wheezy x86_64.
You need the development package of FUSE, which contains the fuse.h you're missing. Do a apt-get install libfuse-dev and it should work.
Copying the header file in the source directory did not work, because in nul1fs.c you'll notice that fuse.h is included with angle brackets. This means, the header file will be searched in the system-wide include paths. That usually means /usr/include.
Note that you then may run into this error:
$ make nul1fs
cc "-lfuse" nul1fs.c -o nul1fs
/tmp/ccbt0X7c.o: In function `main':
nul1fs.c:(.text+0x3c3): undefined reference to `fuse_main_real'
collect2: error: ld returned 1 exit status
make: *** [nul1fs] Error 1
It's a documented bug with a workaround: put the linker flags after the file lists. I.e. compile nul1fs with:
cc nul1fs.c -o nul1fs -lfuse
and not with make nul1fs, which boils down to
cc -lfuse nul1fs.c -o nul1fs
I want to compile opentracker(https://erdgeist.org/arts/software/opentracker) on my machine osx 10.9. when run "make" command I get this error:
cc -c -o opentracker.o -I../libowfat -Wall -pipe -Wextra -O3 -DWANT_FULLSCRAPE
opentracker.c
opentracker.c:25:10: fatal error: 'io.h' file not found
#include "io.h"
^
1 error generated.
make: *** [opentracker.o] Error 1
what is the problem? Is there any way to install io.h through homebrew?
You may have sorted this out already but i'll put this here for anyone else who has the same problem.
The "-I../libowfat" tells the compiler to include that path in the search path for header files. io.h is part of libowfat which you need to compile opentracker.
You can get libowfat here http://www.fefe.de/libowfat/
Unpack it and place it one directory up from where you are compiling opentracker. Alternatively you can change the -I to point to where you have libowfat unpacked.